A sound dynamic regulation and optimization method and system based on a convolutional neural network

Abstract

This invention relates to the field of audio signal processing technology, and discloses a method and system for dynamic audio control optimization based on convolutional neural networks. The method includes: separating the mixed audio signals using a source separation convolutional neural network to obtain independent estimated spectra for each source category; extracting energy demand prediction vectors and transient feature descriptors for each source category using a spectral residual convolutional neural network; calculating the driving characteristic matching score between each source category and each speaker unit; performing propagation calculations based on the room transfer function and constructing an arrival power prediction matrix; calculating the masking threshold matrix and perceived loudness estimate for each listener position; solving for the optimal power allocation coefficient through a joint optimization objective function; extracting position-specific perception sensitivity coefficients using an auditory masking perception convolutional neural network; generating adaptive dynamic compression gain values ​​and finally outputting the driving signals for each speaker unit.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing technology, and more specifically, to a method and system for dynamic audio control optimization based on convolutional neural networks. Background Technology

[0002] In performance sound systems equipped with multi-unit speaker arrays, the mixed audio signal simultaneously contains multiple sound source components such as vocals, instruments, and percussion, while the audience is dispersed in spatial locations at varying distances and angles from the speakers. Existing dynamic sound control methods typically involve two independently operating stages: a speaker power allocation stage based on frequency band energy analysis and a dynamic compression stage based on psychoacoustic models.

[0003] The above method has two technical drawbacks. First, the speaker power allocation stage analyzes the overall frequency energy of the mixed signal, failing to distinguish the differentiated driving needs of different sound sources within the same frequency band. For example, the fundamental frequency of a human voice and the fundamental frequency of a guitar overlap in the mid-frequency range. Human voices require a rapid transient response to maintain clear articulation, while guitar chords require a sustained steady-state output to maintain full timbre. These two require drastically different physical response characteristics from the speaker units, and a uniform power allocation leads to distortion in sound source reproduction. Second, the speaker power allocation stage and the dynamic compression stage operate independently with disconnected objective functions. The speaker power allocation stage does not consider the differences in sound pressure distribution at different listener positions, and the dynamic compression stage does not consider the specific allocation scheme of each sound source component on each speaker unit. This results in listeners near specific speaker units experiencing excessive sound pressure levels, while listeners further away may lack dynamic detail. Overall, the dynamic control of the sound system cannot achieve optimal perception simultaneously at multiple listener positions. Summary of the Invention

[0004] This invention provides a method and system for dynamic sound control optimization based on convolutional neural networks, which solves the technical problems in related technologies where multi-speaker array sound systems cannot adaptively allocate power according to the different characteristics of different sound sources and the differences in the positions of listeners, resulting in uneven perceived loudness at each listener's position and mismatch between the speaker driving characteristics and the characteristics of the sound source.

[0005] This invention discloses a method for dynamic audio control optimization based on convolutional neural networks, comprising: performing a short-time Fourier transform on a mixed audio signal to obtain a mixed spectrum; inputting the mixed spectrum into a pre-trained sound source separation convolutional neural network to obtain a separation spectrum mask matrix for each sound source category; and obtaining an independent estimated spectrum for each sound source category based on the element-wise multiplication of the separation spectrum mask matrix with the mixed spectrum. Feature extraction is performed on the independently estimated spectra of each sound source category to obtain the energy demand prediction vector and transient feature descriptor for each sound source category; The driving characteristic matching score between each sound source category and each speaker unit is calculated based on the transient feature descriptor and the physical response characteristic parameters of each speaker unit. Based on the room transfer function, propagation calculations are performed on multiple candidate power allocation schemes to obtain the arrival power prediction matrix for each audience location; Based on the arrival power prediction matrix, the masking threshold matrix and perceived loudness estimate of each audience position under each candidate power allocation scheme are calculated. The joint optimal power allocation coefficients are solved by taking the weighted combination of minimizing the perceived loudness variance across listener locations and maximizing the driving characteristic matching score as the joint optimization objective function. The actual received spectrum at each listener's location is calculated based on the joint optimal power allocation coefficient, and the location-specific perception sensitivity coefficient is extracted. Generate adaptive dynamic compression gain values ​​based on location-specific sensing sensitivity coefficients; The drive signal for each speaker unit is generated based on the adaptive dynamic compression gain value and the joint optimal power allocation coefficient.

[0006] Furthermore, the source separation convolutional neural network consists of an encoder, a separation backbone network, and a decoder. The encoder receives the mixed spectrum and extracts multi-scale spectral feature representations through multi-layer convolution operations. The separation backbone network is composed of stacked multi-layer temporal convolutional modules, each performing one-dimensional convolution operations along the time axis and frequency axis to output separation features corresponding to each source category. The decoder receives the separation features of each source category, restores them to the same time-frequency resolution as the mixed spectrum through transpose convolution operations, and outputs a separation spectrum mask matrix with values ​​ranging from zero to one after passing through a sigmoid activation function. The source separation convolutional neural network is pre-trained using the mean square error between the independent estimated spectrum of each source category and the corresponding real independent source spectrum as the loss function.

[0007] Furthermore, feature extraction for the independently estimated spectra of each sound source category includes: calculating the Mel frequency cepstral coefficients and logarithmic power spectrum for each independently estimated spectrum of each sound source category; stacking the Mel frequency cepstral coefficients and logarithmic power spectra of each frame along the time axis in a sliding window manner to generate a spectral evolution map for each sound source category; inputting each spectral evolution map into a shared-weight spectral residual convolutional neural network to output the energy demand prediction vector and transient feature descriptor for each sound source category in each frequency band; the transient feature descriptor contains three components: attack time estimate, sustain time estimate, and decay slope estimate, which respectively characterize the rise time of the sound source from silence to peak, the duration of the sound source near the peak, and the decay rate of the sound source from the peak; before using the energy demand prediction vector and the transient feature descriptor to calculate the driving characteristic matching score, Z-score normalization is applied to each component of the energy demand prediction vector and the transient feature descriptor.

[0008] Furthermore, the spectral residual convolutional neural network consists of an input convolutional layer, multiple residual convolutional blocks, and an output branch layer. The input convolutional layer receives the spectral evolution map and extracts the initial feature map through two-dimensional convolution operations. Each residual convolutional block contains two layers of convolution operations and skip connections. After each convolutional layer, batch normalization and nonlinear activation are performed. The skip connections directly add the input to the convolutional output. The output branch layer contains two parallel fully connected branches. The energy demand branch performs global average pooling and fully connected operations on the deep feature map to output an energy demand prediction vector. The transient feature branch performs global average pooling and fully connected operations on the deep feature map to output a transient feature descriptor. The spectral residual convolutional neural network is pre-trained using a weighted sum of the energy demand prediction error and the transient feature prediction error as the loss function.

[0009] Furthermore, calculating the driving characteristic matching score between each sound source category and each speaker unit includes: obtaining the transient response time constant and frequency response gain of each speaker unit, and performing Z-score normalization respectively; for each sound source category and each speaker unit in each frequency band, the driving characteristic matching score is determined by the product of two exponential factors. The first exponential factor is negatively attenuated based on the absolute value of the deviation of the ratio of the estimated attack time to the transient response time constant from one, according to a preset attenuation coefficient, to measure the time scale matching degree between the sound source attack time and the speaker transient response time; the second exponential factor is negatively attenuated based on the absolute value of the deviation of the product of the estimated attenuation slope and the frequency response gain from a preset attenuation gain reference value, according to a preset attenuation coefficient, to measure the gain adaptation degree between the sound source attenuation characteristics and the speaker frequency response; the value range of the driving characteristic matching score is greater than zero and does not exceed one.

[0010] Furthermore, the propagation calculation of multiple candidate power allocation schemes based on the room transfer function includes: obtaining the room transfer function from each representative listener position to each loudspeaker unit; for each sound source category in each frequency band, within a simplex space satisfying normalization constraints, using the distribution after normalization of the driving characteristic matching score as the sampling center, and generating multiple candidate power allocation ratios in its neighborhood according to the Latin hypercube sampling strategy; for each candidate power allocation scheme, multiplying the power allocation ratio of the sound source category in each frequency band with the corresponding predicted energy demand value and the power gain of the room transfer function, and summing along the loudspeaker unit dimension to obtain the arrival power of each sound source category received at each listener position, and organizing it into an arrival power prediction matrix.

[0011] Furthermore, the calculation of the masking threshold matrix and perceived loudness estimate for each listener position under each candidate power allocation scheme includes: superimposing the arrival power of each sound source category at each listener position along the sound source dimension to obtain the total arrival power; performing frequency domain expansion calculation on the total arrival power according to the psychoacoustic expansion function to obtain the simultaneous masking amount, wherein the psychoacoustic expansion function performs weighted summation of the power of each frequency component according to the expansion weight determined by the equivalent rectangular bandwidth auditory filter model; calculating the temporal masking amount based on the power change relationship between adjacent frames according to the exponential decay model; taking the larger value between the simultaneous masking amount and the temporal masking amount as the comprehensive masking threshold, and organizing it into a masking threshold matrix; performing characteristic loudness conversion on the power of each frequency band according to the psychoacoustic loudness calculation standard based on the total arrival power and integrating along the frequency axis to obtain the perceived loudness estimate; the perceived loudness variance in the joint optimization objective function is calculated for the spatial variance of the perceived loudness estimate of each listener position at the corresponding time of the current frame, and the power allocation coefficient is dynamically updated in each frame according to the acoustic state at the current time.

[0012] Furthermore, solving for the joint optimal power allocation coefficients includes: the joint optimization objective function is to minimize the variance of perceived loudness across all listener positions multiplied by a first weighting coefficient minus the sum of the products of the power allocation coefficients and driving characteristic matching scores of each loudspeaker unit in each frequency band of each sound source category multiplied by a second weighting coefficient; the perceived loudness estimate is normalized to the mean based on the range before being substituted into the joint optimization objective function; the constraints include that the total allocated power of each loudspeaker unit in each frequency band does not exceed the available dynamic margin determined by the difference between the maximum power that the unit can withstand in each frequency band and the currently allocated power, the peak perceived loudness at any listener position does not exceed the preset comfort upper limit and the minimum perceived loudness is not lower than the preset audible lower limit, and the allocation ratio of each sound source category in each frequency band satisfies the normalization constraint; the alternating direction multiplier method is used to solve the problem, and the initial point of iteration is taken as the scheme with the optimal joint optimization objective function value among the candidate power allocation schemes.

[0013] Furthermore, the extraction of location-specific perception sensitivity coefficients includes: calculating the actual received spectrum for each listener location based on the joint optimal power allocation coefficient and the room transfer function; for each listener location, the masking threshold matrix and the actual arrival power matrix are respectively processed by Z-score normalization and then concatenated along the channel dimension to generate a dual-channel auditory perception feature map, which is then stacked along the spatial location dimension to form a multi-location auditory perception feature tensor; the multi-location auditory perception feature tensor is input into an auditory masking perception convolutional neural network to output the location-specific perception sensitivity coefficients for each location in each frequency band; the auditory masking perception convolutional neural network consists of a frequency band feature extraction layer, a cross-location spatial convolutional layer, and a sensitivity output layer. The frequency band feature extraction layer independently performs a one-dimensional convolution operation along the frequency band dimension for each location to extract local frequency band masking features, and the cross-location spatial convolutional layer along the spatial location dimension... The algorithm performs one-dimensional convolution operations and uses multi-scale convolution kernel combinations to extract spatial difference patterns of masking structures between different listener positions. The sensitivity output layer outputs position-specific perception sensitivity coefficients after position-by-position fully connected operations and sigmoid activation operations. The adaptive dynamic compression gain value is generated based on the position-specific perception sensitivity coefficients by: performing spatial weighted fusion on the position-specific perception sensitivity coefficients of each position based on the weight factor assigned by the spatial distribution density normalization to generate a comprehensive perception sensitivity coefficient; determining the compression ratio of each frequency band by performing linear modulation between the preset upper limit and lower limit of the compression ratio based on the comprehensive perception sensitivity coefficients; calculating the adaptive dynamic compression gain value according to the compression ratio for the portion of the current frame input power of each frequency band that exceeds the compression start power threshold; and performing moving average smoothing processing on the adaptive dynamic compression gain value along the frequency band dimension.

[0014] This invention provides a dynamic audio control and optimization system based on a convolutional neural network, comprising: The sound source separation module is used to perform a short-time Fourier transform on the mixed audio signal to obtain the mixed spectrum, input the mixed spectrum into a pre-trained sound source separation convolutional neural network to obtain the separation spectrum mask matrix of each sound source category, and obtain the independent estimated spectrum of each sound source category based on the element-wise multiplication of the separation spectrum mask matrix and the mixed spectrum. The feature extraction module is used to extract features from the independently estimated spectra of each sound source category, and obtain the energy demand prediction vector and transient feature descriptor for each sound source category; The matching score calculation module is used to calculate the driving characteristic matching score between each sound source category and each speaker unit based on the transient feature descriptor and the physical response characteristic parameters of each speaker unit; The propagation calculation module is used to perform propagation calculations on multiple candidate power allocation schemes based on the room transfer function, and obtain the arrival power prediction matrix for each audience location. The perception calculation module is used to calculate the masking threshold matrix and perceived loudness estimate of each listener position under each candidate power allocation scheme based on the arrival power prediction matrix; The joint optimization module is used to solve for the joint optimal power allocation coefficients by taking the weighted combination of minimizing the perceived loudness variance across listener locations and maximizing the driving characteristic matching score as the joint optimization objective function. The perception sensitivity extraction module is used to calculate the actual received spectrum of each listener's location based on the joint optimal power allocation coefficient, and extract the location-specific perception sensitivity coefficient. The dynamic compression module is used to generate an adaptive dynamic compression gain value based on the location-specific sensing sensitivity coefficient. The drive signal generation module is used to generate drive signals for each speaker unit based on the adaptive dynamic compression gain value and the joint optimal power allocation coefficient.

[0015] This invention decomposes mixed audio signals into independent estimated spectra for each sound source category using a source-separation convolutional neural network. Then, it utilizes a spectral residual convolutional neural network to output transient feature descriptors for each sound source category. This allows the driving characteristic matching score to quantify the degree of fit between the physical response characteristics of each speaker unit and the transient requirements of each sound source category. This solves the technical problem of sound source reproduction distortion caused by the confusion of driving requirements of different sound sources within the same frequency band, achieving the technical effect of refining power allocation decisions from the overall frequency band energy level to the sound source level. Furthermore, this invention simultaneously introduces room transfer function propagation calculation, cross-location perceived loudness variance term, and driving characteristic matching score term into the joint optimization objective function. Based on the joint optimal power allocation coefficient result, it drives adaptive modulation of the dynamic compression strategy, solving the technical problem of the disconnect between the objective functions of the speaker power allocation stage and the dynamic compression stage, which prevents the dynamic control of sound at multiple listener positions from simultaneously achieving perceptual optimality. This achieves the technical effect of enabling the two stages to operate in coordination based on the same acoustic prediction. Attached Figure Description

[0016] Figure 1 This is a flowchart of the audio dynamic control optimization method based on convolutional neural networks provided in this embodiment of the invention; Figure 2 This is a schematic diagram showing the distribution of separation mask values ​​for each sound source in different frequency bands, provided in an embodiment of the present invention. Figure 3 This is a schematic diagram comparing the transient characteristic parameters of various sound sources provided in the embodiments of the present invention; Figure 4 This is a schematic diagram of the matching fraction heatmap of the driving characteristics of each sound source and speaker unit in the mid-frequency band provided in the embodiments of the present invention; Figure 5This is a schematic diagram of the perceived loudness estimation values ​​at each listener location for candidate scheme Q017 provided in this embodiment of the invention; Figure 6 This is a schematic diagram comparing the perceived loudness at different listener positions before and after optimization, provided in an embodiment of the present invention. Figure 7 This is a schematic diagram of the joint optimal power allocation coefficient distribution (mid-frequency band 1kHz) provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the integrated sensing sensitivity coefficient and adaptive compression gain value for each frequency band provided in the embodiments of the present invention; Figure 9 This is a schematic diagram of the compression ratio and input power distribution of each frequency band provided in the embodiments of the present invention. Detailed Implementation

[0017] In performance sound systems equipped with multi-unit speaker arrays, the mixed audio signal simultaneously contains multiple sound source components such as vocals, instruments, and percussion, while the audience is dispersed in spatial locations at varying distances and angles from the speakers. Existing dynamic sound control methods typically involve two independently operating stages: a speaker power allocation stage based on frequency band energy analysis and a dynamic compression stage based on psychoacoustic models.

[0018] The above method has two technical drawbacks. First, the speaker power allocation stage analyzes the overall frequency energy of the mixed signal, failing to distinguish the differentiated driving needs of different sound sources within the same frequency band. For example, the fundamental frequency of a human voice and the fundamental frequency of a guitar overlap in the mid-frequency range. Human voices require a rapid transient response to maintain clear articulation, while guitar chords require a sustained steady-state output to maintain full timbre. These two require drastically different physical response characteristics from the speaker units, and a uniform power allocation leads to distortion in sound source reproduction. Second, the speaker power allocation stage and the dynamic compression stage operate independently with disconnected objective functions. The speaker power allocation stage does not consider the differences in sound pressure distribution at different listener positions, and the dynamic compression stage does not consider the specific allocation scheme of each sound source component on each speaker unit. This results in listeners near specific speaker units experiencing excessive sound pressure levels, while listeners further away may lack dynamic detail. Overall, the dynamic control of the sound system cannot achieve optimal perception simultaneously at multiple listener positions.

[0019] According to an embodiment of this invention, a method for dynamic audio control optimization based on a convolutional neural network is provided. It should be understood that the hardware environment for implementing this method includes: a system configured with... A speaker array with one speaker unit, corresponding The system includes a power amplification channel, a digital signal processing platform (for performing source separation inference, feature extraction, optimization, and compression operations), and pre-calibrated... The room transfer function data from representative listener positions to each speaker unit. This refers to the total number of speaker units. This represents the total number of representative audience positions. The physical response characteristics of each speaker unit (including the maximum power handling and transient response time constant for each frequency band) have been pre-measured and stored in the digital signal processing platform.

[0020] At least one embodiment of the present invention discloses a method for dynamic sound control optimization based on a convolutional neural network, such as... Figure 1 As shown, it includes the following steps: Step 1: Separate the sound sources from the mixed audio signal and obtain the independent estimated spectrum of each sound source category; Obtain a continuous frame sequence of the mixed audio signal to be played, perform a short-time Fourier transform on each frame, and obtain the mixed spectrum. ,in For frame index, For frequency indexing, the mixed spectrum is input into a pre-trained sound source separation convolutional neural network, which outputs a separation spectrum mask matrix for each sound source category. ,in This provides an index for the sound source categories. Each mask matrix is ​​element-wise multiplied by the mixed spectrum to obtain an independent estimated spectrum for each sound source category. .

[0021] It should be noted that the aforementioned sound source separation convolutional neural network consists of three components: an encoder, a separation backbone network, and a decoder. The encoder receives the mixed spectrum. The algorithm extracts multi-scale spectral feature representations through multi-layer convolutional operations and then passes these representations to a separation backbone network. The separation backbone network consists of stacked multi-layer temporal convolutional modules. Each module performs one-dimensional convolutional operations along both the time and frequency axes to learn the differences in time-frequency distribution patterns of different sound sources, outputting separation features corresponding to each sound source category. The decoder receives the separation features of each sound source category, restores them to the same time-frequency resolution as the mixed spectrum through transposed convolutional operations, and outputs the features after passing them through a sigmoid activation function, with values ​​ranging from [value range missing]. Mask matrix inside .

[0022] Furthermore, the input to the aforementioned sound source separation convolutional neural network is a mixed spectrum. The output is a mask matrix for each sound source category. The sound source separation convolutional neural network is pre-trained in a supervised manner on a training dataset containing a mixture of multiple sound sources and corresponding independent source labels, to independently estimate the spectrum for each sound source category. With the corresponding real independent source spectrum The mean squared error between them is used as the loss function, i.e. ,in The source separation loss function is used, and the Adam optimization algorithm is employed to update the network weights.

[0023] Step 2: Extract features from the independently estimated spectra of each sound source category to obtain the energy demand prediction vector and transient feature descriptor for each sound source category; Independently estimated spectrum for each sound source category Mel-frequency cepstral coefficients and logarithmic power spectra are calculated separately. The Mel-frequency cepstral coefficients and logarithmic power spectra of each frame are stacked along the time axis using a sliding window method to generate spectral evolution diagrams for each sound source category. The spectral evolution maps are input into a weighted spectral residual convolutional neural network, which outputs a predicted energy demand vector for each sound source category in each frequency band. and transient feature descriptors .

[0024] It should be noted that the above transient feature descriptors It contains three components: attack time estimate (Rise time of the sound source from silence to peak value), duration estimate (Characterizing the duration of the sound source near its peak) and estimated attenuation slope (Characterizing the rate at which a sound source decays from its peak value). These three components together describe the time-domain envelope dynamics of each sound source category across each frequency band.

[0025] It should be noted that the aforementioned spectral residual convolutional neural network consists of three components: an input convolutional layer, multiple residual convolutional blocks, and an output branch layer. The diagram shows the evolution of the received spectrum at the input convolutional layer. The network extracts initial feature maps through two-dimensional convolution operations and passes these maps to residual convolutional blocks. Each residual convolutional block contains two layers of convolution operations and skip connections. After each convolutional layer, batch normalization and non-linear activation are performed. Skip connections directly add the input to the convolutional output, allowing the network to learn residual features. The deep feature maps, processed layer by layer by multiple residual convolutional blocks, are then passed to the output branch layer. The output branch layer contains two parallel fully connected branches: the energy demand branch performs global average pooling and fully connected operations on the deep feature maps, outputting energy demand prediction vectors for each frequency band. The transient feature branch performs global average pooling and fully connected operations on the deep feature map, outputting transient feature descriptors for each frequency band. The spectral evolution maps of each sound source category share the same set of network weights, enabling the network to extract consistent features from different sound source categories within a unified feature space.

[0026] Furthermore, the input to the aforementioned spectral residual convolutional neural network is the spectral evolution graph. The output is an energy demand prediction vector. and transient feature descriptors The spectral residual convolutional neural network is pre-trained in a supervised manner on a training dataset containing spectral evolution maps of each sound source category, along with corresponding energy demand and transient parameter annotations. The loss function is a weighted sum of the energy demand prediction error and the transient feature prediction error, i.e. ,in The feature extraction loss function is... and These are the corresponding actual labeled values. To balance the preset weight coefficients of the two losses, the Adam optimization algorithm is used to update the network weights.

[0027] It should be noted that the energy demand prediction vector output by the spectral residual convolutional neural network and transient feature descriptors The dimensions and numerical ranges of the components in the energy demand prediction vector differ due to their different physical meanings: the units of each component are power units, and the attack time estimate... and maintenance time estimate The unit is time, and the estimated decay slope is... This is a dimensionless slope value. To eliminate the influence of dimensional differences on the calculation of the driving characteristic matching score in subsequent steps, the energy demand prediction vector is modified before using the above output in step 3. Z-score normalization is applied to the transient feature descriptors. Each component is standardized using Z-score to ensure that all quantities are consistent on the numerical scale.

[0028] Step 3: Based on the transient feature descriptor and the physical response characteristics of the loudspeaker unit, calculate the driving characteristic matching score between each sound source category and each loudspeaker unit; Obtain the physical response characteristic parameters of each speaker unit, including the transient response time constant of each frequency band. and frequency response gain ,in Index the speaker unit. The transient response time constant is... and frequency response gain The transient response time constant is used before calculating the driving characteristic matching score. Z-score normalization was applied to the frequency response gain. Z-score normalization was applied to ensure that the components of the transient feature descriptor were on a consistent numerical scale with those of the normalized descriptor. For each sound source category... In each frequency band With each speaker unit Based on the ratio between the attack time estimate and the transient response time constant, and the fit between the attenuation slope estimate and the frequency response gain, the driving characteristic matching score is calculated. : in, Represents an exponential function; and The preset attenuation coefficient controls the sensitivity of the drive characteristic matching score to various deviations; The preset attenuation gain reference value represents the ideal target value of the product of the sound source attenuation slope and the speaker frequency response gain. When the product equals The second exponential factor reaches its maximum value. deviation The larger the value, the lower the driving characteristic matching score. All inputs in the above formula have been standardized, ensuring consistent dimensions and valid calculation. Driving characteristic matching score The range of values ​​is The closer the value is to Characterizing the loudspeaker unit Physical response characteristics and sound source type In frequency band The higher the degree of adaptation between transient requirements.

[0029] Furthermore, the first exponential factor in the above-mentioned driving characteristic matching score formula is used to calculate the attack time estimate. With transient response time constant The ratio of deviation The absolute value of the ratio is used to measure the degree of time scale matching between the two: when the ratio equals At that time, the sound source attack time and the speaker transient response time perfectly match, and the first term reaches its maximum value. Ratio deviation The larger the value, the greater the time-scale difference between the transient demand of the sound source and the response capability of the loudspeaker; the first term decays exponentially accordingly. The second exponential factor is estimated by calculating the decay slope. With frequency response gain The product deviates from the reference value The absolute value of the product is used to measure the degree of gain fit between the sound source attenuation characteristics and the speaker frequency response: the closer the product is to the absolute value of the gain fit between the sound source attenuation characteristics and the speaker frequency response, the better. This indicates that the better the speaker's gain characteristics in the corresponding frequency band can support the attenuation slope requirements of the sound source, the closer the second term is to... ; deviation The larger the value, the smaller the second term. Multiplying the two exponential factors ensures that the drive characteristic matching score can only be close to a certain level when the speaker unit simultaneously matches the sound source requirements in both the time scale and gain adaptation dimensions. High values.

[0030] Step 4: Perform propagation calculations on each candidate power allocation scheme based on the room transfer function to obtain the arrival power prediction matrix for each audience location; Obtain pre-calibrated Room transfer function from a representative listener position to each speaker unit ,in Index the audience's location. For each sound source category. In each frequency band Generate multiple sets of candidate power allocation ratios ,in For candidate solution indexing, Indicates the type of sound source In frequency band Distributed to speaker units The power ratio satisfies ,in This represents the total number of loudspeaker units. For each candidate power allocation scheme, convolution propagation calculations are performed using the room transfer function from each loudspeaker unit to each listener's location to obtain the arrival power of each sound source category received at each listener's location: in, This refers to the total number of speaker units. To sum the loudspeaker unit indices for traversal, For sound source category In frequency band Energy demand forecasts Let be the power gain of the room transfer function, expressed as a dimensionless ratio. This is a dimensionless power distribution ratio. After Z-score normalization, it becomes a dimensionless quantity, therefore To achieve dimensionless normalized arrival power, all dimensions are consistent, ensuring the validity of the formula. The arrival power for all sound source categories, all frequency bands, and all listener locations is organized into an arrival power prediction matrix. .

[0031] It should be noted that the above method for generating candidate power allocation ratios is based on satisfying normalization constraints. Within the simplex space, the driving characteristic is used to match the fraction. The normalized distribution is used as the sampling center, and multiple candidate power allocation ratios are generated in its neighborhood according to the Latin hypercube sampling strategy.

[0032] Furthermore, the specific method of using the normalized distribution of driving characteristic matching scores as the sampling center is as follows: for each sound source category In each frequency band Match the driving characteristics of each speaker unit to a fraction. Normalize along the speaker unit dimension to obtain the sampling center weights of each speaker unit. ,in The loudspeaker unit indices are traversed using normalized summation, weighted by the sampling center. As the distribution center of the Latin hypercube sampling, multiple sets of candidate power allocation ratios that satisfy the normalization constraints are generated in its neighborhood, thereby concentrating the candidate schemes in the allocation region with higher driving characteristic matching scores and improving the efficiency of subsequent optimization solutions.

[0033] Step 5: Based on the arrival power prediction matrix, calculate the masking threshold matrix and perceived loudness estimate for each listener location under each candidate power allocation scheme; For each group of candidate power allocation schemes The arrival power of each sound source category at each listener's location. The total arrival power at each listener's location is obtained by superimposing the signals along the sound source dimension. For each audience position The simultaneous masking amount is obtained by performing frequency domain spread calculation on the total arriving power according to the psychoacoustic spread function. The temporal masking amount is calculated based on the power variation relationship between adjacent frames. The larger of the simultaneous masking and the temporal masking is taken as the overall masking threshold. The combined masking thresholds for each frequency band are organized into a masking threshold matrix. Based on total arrival power According to the psychoacoustic loudness calculation standard, the power of each frequency band is transformed into characteristic loudness and integrated along the frequency axis to obtain the perceived loudness estimate for each listener's location. .

[0034] It should be noted that the aforementioned psychoacoustic spread function is a function that weights and spreads the total arriving power in the frequency domain according to the frequency selectivity of the auditory filter, used to simulate the mutual masking effect between different frequency components on the basilar membrane of the human ear. Specifically, for the frequency index... Total arriving power at the location Simultaneous masking quantity By adjusting the power of each frequency component according to the extended weights of the auditory filter... We obtain the result by weighted summation, i.e. ,in For An auditory filter with a center frequency at a frequency The weight of the position, The frequency index is used for summation and traversal, reflecting the frequency. Power at frequency The contribution of the masking effect is determined based on the equivalent rectangular bandwidth auditory filter model. Temporal masking amount. The difference in total arriving power between the current frame and the previous frame is calculated using an exponential decay model, reflecting the forward masking effect of the strong signal in the previous frame on the weak signal in the current frame. ,in Represents an exponential function. This is the temporal masking amount for the previous frame. The time interval between adjacent frames. This is the preset time masking decay time constant. Simultaneously, the masking quantity characterizes the masking effect between frequency components at the same moment, while the time masking quantity characterizes the forward and backward masking effect of the strong signal from the previous frame on the weak signal in the current frame.

[0035] Furthermore, the above perceived loudness estimates Reflecting the candidate solutions The instantaneous perceived loudness state at each listener's location. Step 6 involves jointly optimizing the objective function. Minimizing this is done by considering the spatial variance of the perceived loudness estimates for each listener's location at the current frame time. That is, in the optimization solution for each frame, the spatial variance of the perceived loudness estimates for that frame is used. As input to the optimization objective, where The total number of representative audience positions makes the power allocation coefficient... The system dynamically updates each frame based on the current acoustic state, thereby achieving frame-by-frame adaptive spatial loudness equalization control in the time dimension.

[0036] Step 6: Solve for the optimal power allocation coefficients based on the joint optimization objective function; The joint optimization objective function is a weighted combination of minimizing the perceived loudness variance across all listener locations and maximizing the matching scores of the driving characteristics for each sound source category. The joint optimization objective function is defined as follows: in, for The variance of the perceived loudness estimate for each listener location The total number of representative audience seats. and These are preset weighting coefficients that control the priority of spatial loudness uniformity and drive matching quality, respectively. It should be noted that the perceived loudness estimate... Matching score with driving characteristics Since the dimensions and numerical ranges of the two terms are different, in order to ensure that the two terms have comparable numerical scales in the joint optimization objective function, the values ​​of the two terms must be compared before they are substituted into the joint optimization objective function. Mean normalization based on range is used to match the driving characteristic score. Because its value is already Therefore, no additional normalization is needed within the specified range; pre-set weighting coefficients can be used. and Further adjust the relative weights of the two items.

[0037] The constraints of the aforementioned joint optimization include: the total power allocated to each speaker unit in each frequency band does not exceed the available dynamic margin of that unit as an upper limit of the constraint, wherein the available dynamic margin of each speaker unit is determined by the difference between the maximum power handled in each frequency band and the currently allocated power, which is measured and stored in advance; and the peak perceived loudness at any listener position does not exceed a preset comfort upper limit. And the minimum perceived loudness is not lower than the preset audible lower limit. As a constraint boundary, the allocation ratio of each sound source category in each frequency band satisfies the normalization constraint. ,in This represents the total number of speaker units.

[0038] The alternating direction multiplier method is used to solve the above-mentioned constrained optimization problem. The inputs of the alternating direction multiplier method are the joint optimization objective function, the above constraints, and the initial point of iteration. The output is the joint optimal power allocation coefficient. In this process, the initial point of the iteration is selected from the candidate power allocation schemes generated in step 4 based on the matching fractional sampling of driving characteristics, which is the scheme with the optimal joint optimization objective function value, thereby reducing the number of iterations required to reach convergence.

[0039] Furthermore, the first term in the above joint optimization objective function Power allocation factor The dependency path is as follows: the power allocation coefficient is used to calculate the arrival power at each listener's location through propagation in step 4, and then the perceived loudness estimate at each location is obtained through psychoacoustic loudness conversion in step 5. Finally, the variance is calculated. (Second term) The power allocation coefficient is a linear function, where the drive characteristic matching fraction is... The calculations, pre-calculated in step 3, are used as fixed coefficients in the optimization process. The joint optimization objective function minimizes the first term and effectively maximizes the second term due to the prefix negative sign. The alternating direction multiplier method decomposes the original problem into several subproblems and solves them iteratively. In each iteration, closed-form updates are performed on the power allocation coefficients and auxiliary variables, while simultaneously updating the dual variables to gradually satisfy the constraints, until the joint optimization objective function value and constraint violation amount converge to within the preset accuracy threshold. The joint optimal power allocation coefficients are then output. .

[0040] Step 7: Calculate the actual received spectrum at each listener's location based on the joint optimal power allocation coefficient, and extract the location-specific perception sensitivity coefficient; Based on joint optimal power allocation coefficient and room transfer function Calculate the actual received spectrum for each listener location. For each listener location... The masking threshold matrix corresponding to this position The corresponding actual arrival power matrix is ​​concatenated along the channel dimension to generate a dual-channel auditory perception feature map. ,in This represents the number of frequency bands. It should be noted that the masking threshold matrix... Since the values ​​of the actual arriving power matrix differ from the numerical range of the two matrices, Z-score normalization is applied to both before concatenation to eliminate the impact of dimensional differences on subsequent convolution operations. The dual-channel auditory perception feature maps of each location are stacked along the spatial location dimension to form a multi-location auditory perception feature tensor. ,in This represents the total number of representative listener positions. The multi-position auditory perception feature tensor is input into the auditory masking perception convolutional neural network, which outputs the position-specific perception sensitivity coefficient for each position in each frequency band. ,in For frequency band indexing, location-specific sensing sensitivity coefficient The range of values ​​is It can be directly used as the weight input for the compression parameter modulation in the subsequent step 8, without the need for additional decoding or transformation.

[0041] It should be noted that the aforementioned auditory masking perception convolutional neural network consists of three components: a frequency band feature extraction layer, a cross-location spatial convolutional layer, and a sensitivity output layer. The frequency band feature extraction layer receives multi-location auditory perception feature tensors. For each location's dual-channel auditory perception feature map, a one-dimensional convolution operation along the frequency band dimension is performed independently to extract local frequency band masking features for each location. These local frequency band masking features are then passed to a cross-location spatial convolutional layer. The cross-location spatial convolutional layer performs a one-dimensional convolution operation along the spatial location dimension, with the convolutional kernel sliding between different listener locations to learn the spatial difference patterns and spatial gradation rules of the masking structure between adjacent and distant locations, outputting location-aware features containing spatial context information. The sensitivity output layer performs position-by-position fully connected operations and sigmoid activation operations on the location-aware features, outputting the location-specific perception sensitivity coefficients for each location in each frequency band. The range of values ​​is The larger the value, the more sensitive the auditory perception at the corresponding position in the corresponding frequency band is to changes in the signal.

[0042] In this embodiment of the application, in order to enable the cross-location spatial convolutional layer to capture both local and global spatial difference patterns, the cross-location spatial convolutional layer employs a combination of multi-scale convolutional kernels, including smaller convolutional kernels for extracting local masking differences between adjacent locations, and larger convolutional kernels for extracting global masking gradient trends across multiple locations. The outputs of each scale convolutional kernel are concatenated along the channel dimension and then fused through pointwise convolution.

[0043] Furthermore, the input to the aforementioned auditory masking perception convolutional neural network is a multi-location auditory perception feature tensor. The output is the position-specific sensing sensitivity coefficient of each location in each frequency band. The auditory masking perception convolutional neural network is pre-trained in a supervised manner on a training dataset containing multi-location masking thresholds, arrival power labels, and corresponding perception sensitivity labels. It predicts location-specific perception sensitivity coefficients for each location and frequency band. Compared with the actual labeled value The mean squared error between them is used as the loss function, i.e. ,in The network weights are updated using the Adam optimization algorithm as the perception sensitivity loss function.

[0044] Step 8: Generate adaptive dynamic compression gain values ​​based on location-specific sensing sensitivity coefficients; right Location-specific sensing sensitivity coefficient Based on the weighting factors at each position Perform spatial weighted fusion to generate a comprehensive perception sensitivity coefficient. : in, The total number of representative audience seats. To sum and iterate through the audience position indices, the weight factors for each position are... satisfy The compression parameters for each frequency band are dynamically modulated based on the overall sensing sensitivity coefficient: a lower compression ratio is used in frequency bands with a higher overall sensing sensitivity coefficient to retain more dynamic details, while a higher compression ratio is used in frequency bands with a lower overall sensing sensitivity coefficient to suppress signal fluctuations that are not significantly perceived. Specifically, the adaptive dynamic compression gain value for each frequency band... Determine it as follows: Assume the preset upper limit of the compression ratio is... The lower limit of the compression ratio is Then the frequency band The corresponding compression ratio is In other words, the higher the overall sensing sensitivity coefficient, the lower the compression ratio and the more fully dynamic the data is preserved; when obtaining the compression ratio of each frequency band... Then, the input power of the current frame for each frequency band is... With compression threshold When comparing, Exceed At that time, the adaptive dynamic compression gain value is based on Calculation, where The input power of the current frame for each frequency band. The preset compression start-up power thresholds for each frequency band are both in units of power, and their ratio is dimensionless, making the formula calculation valid; when No more than hour, That is, no compression is applied.

[0045] It should be noted that the weighting factors for each of the above positions... The setting method is to normalize the values ​​based on the spatial distribution density of each listener's location in the target listening area. Areas with higher listener density correspond to larger position weight factors, while areas with lower listener density correspond to smaller position weight factors.

[0046] In this embodiment of the application, to avoid auditory artifacts caused by drastic jumps in the adaptive dynamic compression gain value between adjacent frequency bands, the adaptive dynamic compression gain value of each frequency band is obtained... Subsequently, the adaptive dynamic compression gain value was also adjusted. A moving average smoothing process is performed along the frequency band dimension to make the adaptive dynamic compression gain values ​​of adjacent frequency bands transition smoothly.

[0047] Step 9: Generate the drive signal for each speaker unit based on the adaptive dynamic compression gain value and the joint optimal power allocation coefficient; Adaptive dynamic compression gain value The corresponding frequency bands of the mixed audio signal are applied, and dynamic compression processing is performed on each frequency band signal to obtain the compressed frequency band signal. This is based on the joint optimal power allocation coefficient. The compressed frequency band signal is gain-scaled according to the allocation ratio of the corresponding speaker unit in each frequency band to obtain the allocation signal of each speaker unit in each frequency band. For each speaker unit, the allocation signals of each frequency band are synthesized to generate the drive signal of that speaker unit. The drive signals for each speaker unit are output to their respective power amplification channels.

[0048] This implementation uses a source-separating convolutional neural network to decompose the mixed audio signal into independently estimated spectra for each source category. This allows the spectral residual convolutional neural network to independently analyze the spectral evolution characteristics of each source, outputting energy demand prediction vectors and transient feature descriptors for each source category. Because the attack time estimate, sustain time estimate, and attenuation slope estimate in the transient feature descriptor can distinguish the differentiated requirements of different sources within the same frequency band for the transient response characteristics of the speaker unit, the drive characteristic matching score can quantify the degree of fit between the physical response characteristics of each speaker unit and the transient requirements of each source category. This refines power allocation decisions from the overall frequency band energy level to the source level, avoiding the problem of confusion regarding the drive requirements of different sources within the same frequency band.

[0049] Furthermore, this implementation incorporates the room transfer function for each listener's location into the power allocation optimization process for propagation calculations. This ensures that the sound pressure distribution differences generated by each candidate power allocation scheme at each listener's location are predicted and evaluated before the optimization decision is made. The joint optimization objective function includes both a cross-location perceived loudness variance term and a drive characteristic matching fraction term, enabling the power allocation decision to consider both the sound source drive matching quality and spatial loudness uniformity. Therefore, the joint optimal power allocation coefficient comprehensively considers both sound source characteristics and spatial propagation characteristics, eliminating the problem of the objective function being disconnected between the loudspeaker power allocation stage and the dynamic compression stage.

[0050] Furthermore, the auditory masking perception convolutional neural network performs spatial differential perception analysis based on the actual received spectrum and masking threshold matrix of each listener position under the joint optimal power allocation coefficient correspondence scheme. The cross-position spatial convolutional layer extracts the spatial difference pattern of the masking structure between different listener positions through convolution operations in the position dimension, enabling the dynamic compression strategy to adaptively modulate according to the position-specific perception sensitivity coefficient differences of each position. Because the input of dynamic compression is based on acoustic propagation prediction data under the joint optimal power allocation coefficient correspondence result, the loudspeaker power allocation stage and the dynamic compression stage operate in coordination on the same acoustic prediction basis, so that the dynamic control of the sound system simultaneously tends towards the perception optimum at multiple listener positions.

[0051] The following is an example of an application of the present invention, such as Figure 2-9 As shown, the implementation process is as follows: An indoor concert venue is equipped with a line array system consisting of four speaker units (N=4), deployed at the front left of the stage (unit 1), the front right of the stage (unit 2), the left side of the audience area (unit 3), and the right side of the audience area (unit 4), corresponding to four independent power amplification channels. The performance includes four types of sound sources: lead vocals, electric guitar, bass, and drums (S=4). Five representative audience positions (P=5) were pre-marked at the venue, covering the front row center, front row side, middle row center, back row center, and back row side. The digital signal processing platform has pre-stored the physical response characteristic parameters of each speaker unit and the room transfer function data from each position to each speaker unit. The current processing frame is the 512th frame of the climax of the performance, with a sampling rate of 48kHz, a frame length of 1024 points, and a frame shift of 512 points.

[0052] In step 1, a short-time Fourier transform is performed on the mixed audio signal of frame 512 to obtain the mixed spectrum X(512,f). This mixed spectrum is input into a pre-trained source separation convolutional neural network. The network encoder extracts multi-scale spectral features through multi-layer convolution, and the backbone network performs one-dimensional convolutions along the time and frequency axes to learn the differential distribution patterns of vocals, electric guitar, bass, and drums in the time-frequency domain. The decoder restores the time-frequency resolution through transposed convolution and then outputs mask matrices for the four types of sound sources after sigmoid activation. Each mask matrix is ​​multiplied element-wise with the mixed spectrum to obtain the independent estimated spectrum of the four types of sound sources in the current frame. Taking the low-frequency band (f=80Hz) as an example, the mask value for vocals is 0.12, for electric guitar it is 0.31, for bass it is 0.74, and for drums it is 0.58, reflecting the dominant energy distribution of bass and drums in the low-frequency band.

[0053] Table 1. Source separation mask values ​​and independently estimated spectral amplitudes for frame 512. In step 2, the Mel frequency cepstral coefficients and logarithmic power spectra of the independently estimated spectra of the four types of sound sources are calculated respectively, and the spectral evolution diagrams of each sound source are generated by stacking them along the time axis using a sliding window (window length of 16 frames). The four spectral evolution diagrams are respectively input into a spectral residual convolutional neural network with shared weights. After being processed layer by layer by the input convolutional layer and multiple residual convolutional blocks, the energy demand prediction vector and transient feature descriptor are output by the energy demand branch and transient feature branch respectively. The attack time estimate of the human voice is 8ms, the sustain time estimate is 65ms, and the decay slope estimate is 0.42, reflecting the fast transient characteristics of human voice articulation; the attack time estimate of the drum kit is 3ms, with extremely fast decay; the sustain time estimate of the bass is 210ms, reflecting the continuous steady-state output characteristics. Each output component is Z-score normalized before being fed into step 3.

[0054] Table 2. Feature extraction output for each sound source category (raw values ​​before standardization) In step 3, the physical response characteristic parameters of the four speaker units are obtained. After Z-score normalization of the transient response time constant and frequency response gain of each unit, the driving characteristic matching score is calculated with the normalized transient characteristic descriptor output in step 2. Taking the matching of human voice and unit 1 in the mid-frequency range (1kHz) as an example, the normalized attack time estimate is 0.31, the normalized transient response time constant of unit 1 is 0.28, the normalized attenuation slope is 0.38, the normalized frequency response gain of unit 1 is 1.12, and the preset parameters β1=2.0, β2=1.5, γ0=0.45, then: The drum kit has an extremely short attack time (normalized value of -1.52), and its matching score with Unit 2 (high-frequency fast response unit with a normalized transient response time constant of -1.44) reaches 0.91, which is much higher than its matching score of 0.34 with Unit 3 (low-frequency filler unit), reflecting the differentiated adaptation of different sound sources to the physical characteristics of the loudspeaker.

[0055] Table 3. Matching scores of driving characteristics of each sound source category and each speaker unit in the mid-frequency range (1kHz). In step 4, the room transfer function from five pre-calibrated representative listener positions to four speaker units is obtained. Taking human voice in the mid-frequency band (1kHz) as an example, based on the sampling center weights after matching score normalization, 32 candidate power allocation ratios are generated using Latin hypercube sampling in the simplex space. For each candidate scheme, the human voice arrival power at the five listener positions is calculated according to the propagation calculation formula. Taking candidate scheme number Q017 as an example, its human voice power allocation in the mid-frequency band is (unit 1: 0.38, unit 2: 0.36, unit 3: 0.13, unit 4: 0.13). The arrival power at the front center position (position 1) is calculated as follows, where the normalized predicted value of human voice mid-frequency band energy demand is 0.72: Table 4 shows the mid-frequency arrival power (normalized dimensionless value) of each sound source at different listener locations under candidate scheme Q017. In step 5, for candidate scheme Q017, the arrival power of each sound source is superimposed along the sound source dimension to obtain the total arrival power at each location. Taking location 1 as an example: The total arriving power at each location is frequency-domain expanded using the psychoacoustic spread function to calculate the simultaneous masking. Combined with the temporal masking from the previous frame, the temporal masking is calculated using an exponential decay model (time constant is 50ms, frame shift corresponds to a time interval of 10.67ms). The larger of these two values ​​is used to determine the overall masking threshold. Location 1, being closer to the stage and having higher total arriving power, has a higher masking threshold than the back rows, reflecting the spatial differences in auditory masking structures at different locations. Based on the total arriving power converted using characteristic loudness transformation and integrated along the frequency axis, the perceived loudness estimates for each location are obtained. Location 1 has an estimate of 63.4 sone, while location 5 has only 38.7 sone, showing a significant difference and large variance, indicating that the spatial loudness uniformity of this candidate scheme needs optimization.

[0056] Table 5. Perceived loudness estimates and comprehensive masking thresholds (representative values ​​in the mid-frequency band) for each listener location under candidate scheme Q017. In step 6, the 32 candidate schemes are evaluated using a joint optimization objective function, and the scheme Q017 with the optimal objective function value is selected as the initial iteration point for the alternating direction multiplier method. Constraints include: the total power allocated to each unit and frequency band does not exceed the currently available dynamic margin (42W for unit 1, 45W for unit 2, 38W for unit 3, and 39W for unit 4); the peak perceived loudness at any location does not exceed 85 sones, and the lowest perceived loudness is not lower than 30 sones; and the allocation ratio of each sound source to each frequency band is normalized. In the joint optimization objective function, the variance of the perceived loudness estimates at the five locations is calculated after range-mean normalization, with λ1=0.6 and λ2=0.4. The alternating direction multiplier method converges after 23 iterations, outputting the joint optimal power allocation coefficients. After optimization, the vocals are concentrated in the mid-frequency range of units 1 and 2 (the allocation ratio is increased to 0.42 and 0.41), and the bass is concentrated in units 3 and 4 (the allocation ratio is increased to 0.47 and 0.44). The perceived loudness of the rear seats is significantly improved, and the variance of the perceived loudness of the five positions is reduced from 97.8 in the Q017 scheme to 31.4 in the optimized scheme.

[0057] Table 6 Joint Optimal Power Allocation Coefficients (Mid-Frequency Band 1kHz) In step 7, the actual received spectrum at each location is calculated based on the joint optimal power allocation coefficient and room transfer function. After Z-score normalization with the corresponding masking threshold matrix, the spectrum is concatenated along the channel dimension to generate dual-channel auditory perception feature maps for 5 locations (each map has a dimension of 2×32, with a total of 32 frequency bands). These maps are then stacked along the location dimension to form a multi-location auditory perception feature tensor (with a dimension of 5×2×32). The auditory masking perception convolutional neural network is input: the frequency band feature extraction layer independently performs one-dimensional convolution along the frequency band dimension for each location to extract local frequency band masking features at each location; the cross-location spatial convolutional layer uses a combination of multi-scale convolutional kernels with sizes of 2 and 5 to capture local masking differences between adjacent locations (such as the front row difference between location 1 and location 2) and global loudness gradient trends across multiple locations (such as the attenuation law from the front row to the back row); the sensitivity output layer outputs the location-specific perception sensitivity coefficients for each frequency band at each location after position-by-position fully connected and sigmoid activation. The front row positions have a higher sensitivity coefficient in the mid-to-high frequency band (position 1 has a mid-frequency sensitivity coefficient of 0.87), while the rear row positions have a higher sensitivity coefficient in the low frequency band (position 4 has a low-frequency sensitivity coefficient of 0.79), which reflects the combined effect of spatial position and masking structure.

[0058] In step 8, weighting factors are assigned based on the spatial distribution density of each position within the target listening area: Front row center position 1 has a weight of 0.28, front row side position 2 has a weight of 0.22, middle row center position 3 has a weight of 0.25, back row center position 4 has a weight of 0.15, and back row side position 5 has a weight of 0.10, with a sum of 1. Taking the mid-frequency band (1kHz band) as an example, the comprehensive perception sensitivity coefficient is calculated as follows: Assuming the maximum compression ratio Rmax = 4.0 and the minimum compression ratio Rmin = 1.5, then the mid-frequency compression ratio is: The current frame's mid-band input power is 1.83W, the compression threshold is 1.20W, the input power exceeds the threshold, and the adaptive dynamic compression gain value is: The low-frequency band has a comprehensive sensing sensitivity coefficient of 0.631 and a compression ratio of 2.422, indicating lower sensitivity and therefore stronger compression. The high-frequency band has a comprehensive sensing sensitivity coefficient of 0.812 and a compression ratio of 1.970, indicating higher sensitivity and therefore more complete dynamic retention. The adaptive dynamic compression gain values ​​for each frequency band are smoothed using a moving average along the frequency band dimension (window length of 3 frequency bands) to eliminate gain jumps between adjacent frequency bands.

[0059] Table 7. Comprehensive sensing sensitivity coefficients and adaptive dynamic compression parameters for each representative frequency band. In step 9, adaptive dynamic compression gain values ​​for each frequency band are applied to the corresponding frequency bands of the mixed audio signal to complete the dynamic compression process. Taking the mid-frequency band as an example, the amplitude of the compressed signal is scaled to 0.784 times the original value. Subsequently, based on the joint optimal power allocation coefficient, the gain of each frequency band signal after compression is scaled according to the allocation ratio of each speaker unit: the compressed mid-frequency signal of vocals is scaled to (0.42, 0.41, 0.09, 0.08) and then allocated to 4 units; the compressed mid-frequency signal of bass is allocated to (0.05, 0.04, 0.47, 0.44); and the mid-frequency signal of the drum kit is centrally allocated to units 1 and 2. For each speaker unit, the allocated signals of each frequency band are synthesized to generate the complete drive signal of that unit, which is then output to the corresponding power amplification channel to drive the speaker to produce sound. Units 3 and 4 have higher drive power because they handle a large proportion of bass signals, while units 1 and 2 have higher drive power in the mid-to-high frequency range because they handle vocal and drum signals. The drive power of each unit does not exceed the current available dynamic margin constraint.

[0060] The data flow throughout the implementation process reflects a clear, step-by-step refinement logic: Step 1 decomposes the mixed spectrum into four types of independently estimated spectra of sound sources; Step 2 extracts energy demand vectors and transient descriptors from the spectral evolution diagrams of each sound source, providing a source-level feature basis for subsequent matching and allocation; Step 3 uses the transient descriptors and loudspeaker physical parameters to calculate the driving characteristic matching score, forming a priori distribution to guide candidate scheme sampling; Step 4 maps candidate allocation schemes to the arrival power at each listener location based on the room transfer function; Step 5 further converts this into the masking threshold and perceived loudness of the perceptual domain, thus transforming the object... Step 6 converts sound pressure information into psychoacoustic quantities; under the framework of joint optimization objective function, with the matching score in step 3 and the perceived loudness variance in step 5 as dual objectives, outputs joint optimal power allocation coefficients; Step 7 constructs dual-channel perception features for each location based on the optimal allocation results and outputs location-specific sensitivity coefficients through an auditory masking perception network; Step 8 generates band-adaptive compression gain values ​​accordingly; Step 9 combines the optimal allocation coefficients and compression gain values ​​to apply to the signal, generating drive signals for each loudspeaker unit, so that power allocation and dynamic compression are coordinated on the same acoustic prediction basis.

[0061] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A method for dynamic sound control optimization based on convolutional neural networks, characterized in that, Includes the following steps: A short-time Fourier transform is performed on the mixed audio signal to obtain the mixed spectrum. The mixed spectrum is then input into a pre-trained sound source separation convolutional neural network to obtain the separation spectrum mask matrix for each sound source category. Based on the element-wise multiplication of the separation spectrum mask matrix with the mixed spectrum, the independent estimated spectrum for each sound source category is obtained. Feature extraction is performed on the independently estimated spectra of each sound source category to obtain the energy demand prediction vector and transient feature descriptor for each sound source category; The driving characteristic matching score between each sound source category and each speaker unit is calculated based on the transient feature descriptor and the physical response characteristic parameters of each speaker unit. Based on the room transfer function, propagation calculations are performed on multiple candidate power allocation schemes to obtain the arrival power prediction matrix for each audience location; Based on the arrival power prediction matrix, the masking threshold matrix and perceived loudness estimate of each audience position under each candidate power allocation scheme are calculated. The joint optimal power allocation coefficients are solved by taking the weighted combination of minimizing the perceived loudness variance across listener locations and maximizing the driving characteristic matching score as the joint optimization objective function. The actual received spectrum at each listener's location is calculated based on the joint optimal power allocation coefficient, and the location-specific perception sensitivity coefficient is extracted. Generate adaptive dynamic compression gain values ​​based on location-specific sensing sensitivity coefficients; The drive signal for each speaker unit is generated based on the adaptive dynamic compression gain value and the joint optimal power allocation coefficient.

2. The method for dynamic sound control optimization based on convolutional neural networks according to claim 1, characterized in that, The sound source separation convolutional neural network consists of an encoder, a separation backbone network, and a decoder. The encoder receives the mixed spectrum and extracts multi-scale spectral feature representations through multi-layer convolution operations. The separation backbone network is composed of multiple layers of time-series convolutional modules stacked together. Each layer of time-series convolutional modules performs one-dimensional convolution operations along the time axis and frequency axis, and outputs the separation features corresponding to each sound source category. The decoder receives the separation features of each sound source category, restores them to the same time-frequency resolution as the mixed spectrum through transposed convolution operation, and outputs a separation spectrum mask matrix with values ​​ranging from zero to one through the sigmoid activation function; the sound source separation convolutional neural network is pre-trained using the mean square error between the independent estimated spectrum of each sound source category and the corresponding real independent source spectrum as the loss function.

3. The method for dynamic sound control optimization based on convolutional neural networks according to claim 1, characterized in that, Feature extraction for the independently estimated spectra of each sound source category includes: calculating the Mel frequency cepstral coefficients and logarithmic power spectrum for each independently estimated spectrum of each sound source category; stacking the Mel frequency cepstral coefficients and logarithmic power spectra of each frame along the time axis in a sliding window manner to generate a spectral evolution map for each sound source category; inputting each spectral evolution map into a shared-weight spectral residual convolutional neural network to output the energy demand prediction vector and transient feature descriptor for each sound source category in each frequency band; the transient feature descriptor includes three components: attack time estimate, sustain time estimate, and decay slope estimate, which respectively characterize the rise time of the sound source from silence to peak, the duration of the sound source near the peak, and the decay rate of the sound source from the peak; before using the energy demand prediction vector and the transient feature descriptor to calculate the driving characteristic matching score, each component in the energy demand prediction vector and the transient feature descriptor is subjected to Z-score normalization.

4. The audio dynamic control optimization method based on convolutional neural networks according to claim 3, characterized in that, The spectral residual convolutional neural network consists of an input convolutional layer, multiple residual convolutional blocks, and an output branch layer. The input convolutional layer receives the spectral evolution map and extracts the initial feature map through two-dimensional convolution operations. Each residual convolutional block contains two layers of convolution operations and skip connections. After each convolutional layer, batch normalization and non-linear activation are performed. The skip connections directly add the input to the convolutional output. The output branch layer contains two parallel fully connected branches. The energy demand branch performs global average pooling and fully connected operations on the deep feature map to output an energy demand prediction vector. The transient feature branch performs global average pooling and fully connected operations on the deep feature map to output a transient feature descriptor. The spectral residual convolutional neural network is pre-trained using a weighted sum of the energy demand prediction error and the transient feature prediction error as the loss function.

5. The method for dynamic sound control optimization based on convolutional neural networks according to claim 1, characterized in that, Calculating the driving characteristic matching score between each sound source category and each speaker unit includes: obtaining the transient response time constant and frequency response gain of each speaker unit, and performing Z-score normalization respectively; for each sound source category and each speaker unit in each frequency band, the driving characteristic matching score is determined by the product of two exponential factors. The first exponential factor is negatively attenuated based on the absolute value of the deviation of the ratio of the estimated attack time to the transient response time constant from one, according to a preset attenuation coefficient, to measure the time scale matching degree between the sound source attack time and the speaker transient response time; the second exponential factor is negatively attenuated based on the absolute value of the deviation of the product of the estimated attenuation slope and the frequency response gain from a preset attenuation gain reference value, according to a preset attenuation coefficient, to measure the gain adaptation degree between the sound source attenuation characteristics and the speaker frequency response; the value range of the driving characteristic matching score is greater than zero and does not exceed one.

6. The method for dynamic sound control optimization based on convolutional neural networks according to claim 1, characterized in that, The propagation calculation of multiple candidate power allocation schemes based on the room transfer function includes: obtaining the room transfer function from each representative listener position to each loudspeaker unit; for each sound source category in each frequency band, within a simplex space that satisfies normalization constraints, using the distribution after normalization of the driving characteristic matching score as the sampling center, and generating multiple candidate power allocation ratios in its neighborhood according to the Latin hypercube sampling strategy; for each candidate power allocation scheme, multiplying the power allocation ratio of the sound source category in each frequency band with the corresponding predicted energy demand value and the power gain of the room transfer function, and summing along the loudspeaker unit dimension to obtain the arrival power of each sound source category received at each listener position, and organizing it into an arrival power prediction matrix.

7. The method for dynamic sound control optimization based on convolutional neural networks according to claim 1, characterized in that, The calculation of the masking threshold matrix and perceived loudness estimate for each listener position under each candidate power allocation scheme includes: superimposing the arrival power of each sound source category at each listener position along the sound source dimension to obtain the total arrival power; performing frequency domain expansion calculation on the total arrival power according to the psychoacoustic expansion function to obtain the simultaneous masking amount, wherein the psychoacoustic expansion function is weighted and summed on the power of each frequency component according to the expansion weight determined by the equivalent rectangular bandwidth auditory filter model; calculating the temporal masking amount based on the power change relationship between adjacent frames according to the exponential decay model; taking the larger value between the simultaneous masking amount and the temporal masking amount as the comprehensive masking threshold and organizing it into a masking threshold matrix; performing characteristic loudness conversion on the power of each frequency band according to the psychoacoustic loudness calculation standard based on the total arrival power and integrating along the frequency axis to obtain the perceived loudness estimate; the perceived loudness variance in the joint optimization objective function is calculated for the spatial variance of the perceived loudness estimate of each listener position at the corresponding time of the current frame, and the power allocation coefficient is dynamically updated in each frame according to the acoustic state at the current time.

8. The method for dynamic sound control optimization based on convolutional neural networks according to claim 1, characterized in that, Solving for the joint optimal power allocation coefficients includes: the joint optimization objective function is to minimize the variance of perceived loudness across all listener positions multiplied by a first weighting coefficient minus the sum of the products of the power allocation coefficients and drive characteristic matching scores of each loudspeaker unit in each frequency band of each sound source category multiplied by a second weighting coefficient; the perceived loudness estimate is normalized to the mean based on the range before being substituted into the joint optimization objective function; the constraints include that the total allocated power of each loudspeaker unit in each frequency band does not exceed the available dynamic margin determined by the difference between the maximum power that the unit can withstand in each frequency band and the currently allocated power, the peak perceived loudness at any listener position does not exceed the preset comfort upper limit and the minimum perceived loudness is not lower than the preset audible lower limit, and the allocation ratio of each sound source category in each frequency band satisfies the normalization constraint; the alternating direction multiplier method is used to solve the problem, and the initial point of iteration is taken as the scheme with the optimal joint optimization objective function value among the candidate power allocation schemes.

9. The method for dynamic sound control optimization based on convolutional neural networks according to claim 1, characterized in that, Extracting location-specific perception sensitivity coefficients includes: calculating the actual received spectrum for each listener location based on the joint optimal power allocation coefficient and room transfer function; for each listener location, the masking threshold matrix and the actual arrival power matrix are Z-score normalized and then concatenated along the channel dimension to generate a dual-channel auditory perception feature map, which is then stacked along the spatial location dimension to form a multi-location auditory perception feature tensor; the multi-location auditory perception feature tensor is input into an auditory masking perception convolutional neural network to output the location-specific perception sensitivity coefficients for each location in each frequency band; the auditory masking perception convolutional neural network consists of a frequency band feature extraction layer, a cross-location spatial convolutional layer, and a sensitivity output layer. The frequency band feature extraction layer independently performs a one-dimensional convolution operation along the frequency band dimension for each location to extract local frequency band masking features, and the cross-location spatial convolutional layer performs... One-dimensional convolutional operations are performed, and multi-scale convolutional kernel combinations are used to extract spatial difference patterns of masking structures between different listener positions. The sensitivity output layer outputs position-specific perception sensitivity coefficients through position-by-position fully connected operations and sigmoid activation operations. The adaptive dynamic compression gain value is generated based on the position-specific perception sensitivity coefficients, including: performing spatial weighted fusion on the position-specific perception sensitivity coefficients of each position based on the weight factor assigned by the spatial distribution density normalization to generate a comprehensive perception sensitivity coefficient; determining the compression ratio of each frequency band by linear modulation between the preset upper limit and lower limit of the compression ratio based on the comprehensive perception sensitivity coefficients; calculating the adaptive dynamic compression gain value according to the compression ratio for the portion of the current frame input power of each frequency band that exceeds the compression start power threshold; and performing moving average smoothing processing on the adaptive dynamic compression gain value along the frequency band dimension.

10. A sound dynamic control optimization system based on a convolutional neural network, used to execute the sound dynamic control optimization method based on a convolutional neural network as described in any one of claims 1 to 9, characterized in that, include: The sound source separation module is used to perform a short-time Fourier transform on the mixed audio signal to obtain the mixed spectrum, input the mixed spectrum into a pre-trained sound source separation convolutional neural network to obtain the separation spectrum mask matrix of each sound source category, and obtain the independent estimated spectrum of each sound source category based on the element-wise multiplication of the separation spectrum mask matrix and the mixed spectrum. The feature extraction module is used to extract features from the independently estimated spectra of each sound source category, and obtain the energy demand prediction vector and transient feature descriptor for each sound source category; The matching score calculation module is used to calculate the driving characteristic matching score between each sound source category and each speaker unit based on the transient feature descriptor and the physical response characteristic parameters of each speaker unit; The propagation calculation module is used to perform propagation calculations on multiple candidate power allocation schemes based on the room transfer function, and obtain the arrival power prediction matrix for each audience location. The perception calculation module is used to calculate the masking threshold matrix and perceived loudness estimate of each listener position under each candidate power allocation scheme based on the arrival power prediction matrix; The joint optimization module is used to solve for the joint optimal power allocation coefficients by taking the weighted combination of minimizing the perceived loudness variance across listener locations and maximizing the driving characteristic matching score as the joint optimization objective function. The perception sensitivity extraction module is used to calculate the actual received spectrum of each listener's location based on the joint optimal power allocation coefficient, and extract the location-specific perception sensitivity coefficient. The dynamic compression module is used to generate an adaptive dynamic compression gain value based on the location-specific sensing sensitivity coefficient. The drive signal generation module is used to generate drive signals for each speaker unit based on the adaptive dynamic compression gain value and the joint optimal power allocation coefficient.

Images