Sound playing control method and system based on voice sensor

Abstract

The invention relates to the technical field of sound equipment, discloses a sound equipment playing control method and system based on a voice sensor, and aims to solve the problem that audio experience is disjointed due to the fact that an existing sound equipment system cannot sense a continuously changing acoustic environment and a user state. According to the method, acoustic signals are collected in real time through a multi-channel voice sensor array, a three-dimensional position track, an emotional state label and a room impulse response function of an audience are extracted, a multi-dimensional dynamic scene state descriptor is constructed in a fusion mode, and sound field rendering parameters such as a direct sound / reflected sound energy ratio, reverberation time, an equilibrium curve and sound image diffusion degree are generated according to the descriptor. And dynamically synthesizing a personalized three-dimensional sound field matched with the current scene. The system comprises an acoustic signal acquisition module, a scene feature extraction module, a state fusion modeling module, a rendering parameter generation module and a dynamic sound field synthesis module. According to the method and the device, the conversion from passive instruction response to active environment adaptation is realized, and the audio immersion and the man-machine interaction naturalness are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of audio technology, specifically relating to an audio playback control method and system based on a voice sensor. Background Technology

[0002] With the rapid development of artificial intelligence and human-computer interaction technologies, smart speaker devices have been widely used in homes, vehicles, and public spaces, becoming an important entry point for users to access audio content and control smart home devices. Voice, as one of the most natural interaction methods, relies on voice sensors to control speaker playback, and has become the mainstream technology in the industry. This technology collects user voice commands, performs voice recognition and semantic understanding, and triggers corresponding playback operations such as play, pause, skip tracks, and adjust volume, significantly improving the convenience and immersive experience for users.

[0003] Among them, the audio playback control method based on voice sensors focuses on how to accurately and efficiently translate the user's voice intent into specific actions executed by the audio system. Its core lies in real-time acquisition of voice signals, robust recognition in noisy environments, accurate semantic parsing of commands, and dynamic coordination with the audio playback status. Ideally, the system should respond quickly to valid commands in complex acoustic environments, while suppressing false triggers, and intelligently adjusting its response strategy based on the current playback content and context.

[0004] In existing technologies, most voice-controlled speakers employ a two-tier architecture of a fixed wake word and independent command recognition, relying on cloud-based or locally pre-set models for voice processing. However, such solutions generally suffer from high response latency, low recognition rates in offline scenarios, and an inability to distinguish between valid commands and environmental speech interference. Especially in multi-person conversations, background music playback, or high-noise environments, voice sensors easily capture non-target sound sources, leading to misidentification or missed recognition, severely impacting user experience. Furthermore, existing systems typically lack the ability to perceive the speaker's current playback status (such as song type, playback progress, and volume level), making it difficult to achieve context-aware intelligent responses, such as automatically lowering the volume to respond to user questions while playing news, or proactively pausing playback after detecting continuous user silence. Therefore, there is an urgent need for a speaker playback control method and system that can deeply integrate voice perception, environmental understanding, and playback status feedback to address the comprehensive deficiencies of existing technologies in terms of accuracy, robustness, and intelligence. Summary of the Invention

[0005] To address the technical problem that existing audio playback control systems can only respond to discrete voice commands and cannot perceive and adapt to continuously changing acoustic environments and user states, resulting in a disconnect between the audio experience and the actual scenario, this invention provides an audio playback control method and system based on a voice sensor.

[0006] This invention constructs a multi-dimensional acoustic scene perception framework to analyze the listener's spatial location, vocal emotional state, and the acoustic transmission characteristics of their surrounding space in real time. It then integrates these dynamically changing scene features into a unified listener-environment state model. Based on this model, instead of simply adjusting playback parameters, this invention proactively and in real-time synthesizes a personalized three-dimensional sound field that highly matches the current listener state and spatial environment. This achieves deep coupling between audio content and auditory perception, fundamentally enhancing the immersiveness and naturalness of human-computer interaction.

[0007] On one hand, the present invention provides a method for controlling audio playback based on a voice sensor, comprising the following steps: S1: Periodically collects multi-channel raw acoustic signals within the physical space where the audio equipment is located through a sensor array consisting of multiple acoustic sensor units.

[0008] S2: Input the multi-channel raw acoustic signals into an acoustic scene feature extraction module, which performs the following operations in parallel: By using a multi-channel sound source localization algorithm, the three-dimensional spatial coordinates of one or more human voice sources in the space are analyzed, and the three-dimensional spatial coordinates are processed by time-series smoothing filtering to obtain the real-time location trajectory data of one or more listeners. The human voice feature analysis algorithm processes the human voice signal segment in the multi-channel raw acoustic signal to extract the acoustic prosodic feature parameter set that represents the speaker's emotional state. The acoustic prosodic feature parameter set includes fundamental frequency, formants, short-time energy, speech rate and Mel frequency cepstral coefficients. The acoustic prosodic feature parameter set is then mapped to a quantified emotional state label using a pre-set emotion classification model. By using a blind system identification algorithm, the audio content played by the audio equipment itself is used as a detection signal. Combined with the multi-channel acoustic signals collected by the sensor array, the room impulse response function from the audio equipment to each acoustic sensor unit is calculated in reverse and estimated in real time, thereby obtaining the transmission characteristic data of the spatial acoustic environment.

[0009] S3: Input the real-time location trajectory data, quantized emotional state labels, and room impulse response function into a listener-environment state fusion modeling module. The listener-environment state fusion modeling module constructs a joint state vector, which contains the location information, emotional classification information, and key acoustic parameters characterizing the room reverberation and reflection characteristics of all identified listeners. The Kalman filter is used to iteratively update the joint state vector to generate a continuous and stable multi-dimensional dynamic scene state descriptor.

[0010] S4: Input the multi-dimensional dynamic scene state descriptor into a sound field rendering parameter generation module. The sound field rendering parameter generation module contains a preset mapping rule library. This mapping rule library maps different combinations of multi-dimensional dynamic scene state descriptors into a set of specific sound field rendering target parameters. The sound field rendering target parameters include the direct sound to reflected sound energy ratio at the target listener's position, the target reverberation time, the multi-band equalizer gain curve, and the target sound image diffusion.

[0011] S5: The original audio signal to be played and the sound field rendering target parameters are input into a dynamic sound field synthesis module. The dynamic sound field synthesis module performs real-time digital signal processing on the original audio signal to be played based on the sound field rendering target parameters. The processing includes: Beamforming technology is used to focus audio energy onto the listener's location indicated by real-time location trajectory data; The room impulse response function and audio signal are processed through multi-channel convolution operations to match the target reverberation time; The spectral components of the audio signal are adjusted using the gain curve of a multi-band equalizer. Finally, the delay and phase relationship of each playback channel are adjusted through a virtual surround sound algorithm to achieve the target sound image diffusion. The final output is a multi-channel audio signal that has been modulated in real time and played by the speaker unit of the audio equipment.

[0012] Preferably, the sensor array is a planar circular array containing eight microelectromechanical system microphone units, which are evenly distributed along a circumference with a diameter of 100 mm. The signal-to-noise ratio of each microphone unit is not less than 65 dB, the sampling frequency is set to 48 kHz, and the sampling bit depth is 24 bits.

[0013] The multi-channel sound source localization algorithm is a steering response power algorithm based on generalized cross-correlation-phase transformation. It determines the azimuth, pitch, and distance of the sound source by calculating the cross-correlation function between all microphone unit pairs and searching for energy peaks in a preset three-dimensional spatial grid.

[0014] Furthermore, the voice feature analysis algorithm first employs a Gaussian mixture model-based speech activity detection algorithm to accurately segment voice signal segments from the multi-channel raw acoustic signal. Subsequently, the segmented voice signal segments are divided into 20ms frame lengths, and a Hamming window is applied. For each frame, its autocorrelation function is calculated to extract the fundamental frequency, linear predictive coding coefficients are calculated to determine the formant positions, the logarithm of the sum of squares is calculated to obtain short-time energy, and the number of syllables per unit time is calculated to determine the speech rate. Simultaneously, the first thirteen-dimensional Mel-frequency cepstral coefficients are extracted.

[0015] The emotion classification model is a support vector machine classifier that is pre-trained on a large amount of emotion-labeled speech data. Its input is a set of acoustic prosodic feature parameters, and its output is one of three emotion state labels: "calm", "excited", and "neutral".

[0016] Furthermore, the blind system identification algorithm adopts a multi-channel recursive least squares adaptive filtering algorithm. This algorithm takes the original audio signal played by the speaker as the reference input and the multi-channel signal collected by the sensor array as the desired output. By iteratively adjusting the coefficients of a set of finite impulse response filters, the mean square error between the filter output and the desired output is minimized. When the algorithm converges, the coefficients of the finite impulse response filter are the estimated room impulse response function.

[0017] Preferably, in the audience-environment state fusion modeling module, the state equation of the Kalman filter is used to predict the joint state vector at the next moment, while the observation equation is used to correct the predicted value using the real-time location, sentiment label, and room impulse response function collected at the current moment. The process noise covariance matrix and observation noise covariance matrix of the Kalman filter are set according to prior knowledge to balance the smoothness of state prediction with the response speed to real-time observation.

[0018] Furthermore, the mapping rule base in the sound field rendering parameter generation module specifies the following mapping relationships: When the multidimensional dynamic scene state descriptor indicates a single listener, the emotional state is "calm", and the room impulse response function shows a short reverberation time, a set of sound field rendering target parameters is generated to improve speech intelligibility and presence. Specifically, this involves setting a high direct sound to reflected sound energy ratio, shortening the target reverberation time, increasing the equalizer gain in the mid-to-high frequency band, and reducing sound image diffusion. When the descriptor indicates multiple listeners, the emotional state is "excited," and the room impulse response function shows a long reverberation time, a set of sound field rendering target parameters is generated to create a grand sense of space and atmosphere. Specifically, this involves setting a low direct sound to reflected sound energy ratio, extending the target reverberation time, increasing the equalizer gain in the low and high frequency bands to form a "V" shaped curve, and increasing the sound image diffusion.

[0019] On the other hand, the present invention provides an audio playback control system based on a voice sensor, comprising: The acoustic signal acquisition module is a planar circular array consisting of eight microelectromechanical system microphone units, configured to periodically acquire multi-channel raw acoustic signals within the physical space where the audio equipment is located. The acoustic scene feature extraction module, connected to the output of the acoustic signal acquisition module, integrates a sound source localization unit, a voice feature analysis unit, and a room impulse response estimation unit. The sound source localization unit is configured to execute a steering response power algorithm based on generalized cross-correlation-phase transform to analyze the three-dimensional spatial coordinates of the human voice source and output the listener's real-time position trajectory data. The voice feature analysis unit is configured to extract acoustic prosodic feature parameters such as fundamental frequency, formants, and short-time energy, and output quantified emotional state labels using a built-in support vector machine emotion classification model. The room impulse response estimation unit is configured to execute a multi-channel recursive least squares adaptive filtering algorithm to estimate the transmission characteristics data of the spatial acoustic environment in real time. The listener-environment state fusion modeling module has its input connected to the output of the sound source localization unit, the human voice feature analysis unit, and the room impulse response estimation unit, respectively. It internally constructs a joint state vector and configures a Kalman filter to smooth and predict the joint state vector in order to generate a continuous and stable multi-dimensional dynamic scene state descriptor. The sound field rendering parameter generation module has its input end connected to the output end of the listener-environment state fusion modeling module. It stores a preset mapping rule library and is configured to query and output a set of corresponding sound field rendering target parameters based on the multi-dimensional dynamic scene state descriptor. The parameters include the direct sound to reflected sound energy ratio, target reverberation time, multi-band equalizer gain curve, and target sound image diffusion. The dynamic sound field synthesis module has two input terminals. One input terminal is used to receive the original audio signal to be played, and the other input terminal is connected to the output terminal of the sound field rendering parameter generation module. Internally, it integrates a beamforming processor, a multi-channel convolutional processor, a parametric equalizer, and a virtual surround sound processor. It is configured to perform real-time audio effects processing and spatial rendering on the original audio signal according to the sound field rendering target parameters, and output the processed multi-channel audio signal to the speaker unit of the audio equipment.

[0020] Compared with the prior art, the advantages and positive effects of the present invention are as follows: 1. This invention acquires acoustic signals in real time using a multi-channel voice sensor array, integrates and extracts listener location, emotional state, and room acoustic characteristics to construct a multi-dimensional dynamic scene state descriptor, and dynamically synthesizes a personalized three-dimensional sound field highly matched to the current scene based on this descriptor. This system no longer relies on discrete voice commands but can proactively perceive and adapt to continuously changing listener states and acoustic environments, significantly improving the scene fit of audio output and the naturalness of human-computer interaction.

[0021] 2. This invention integrates multi-dimensional features such as sound source localization, emotion analysis, and room impulse response estimation with Kalman filtering to achieve smooth state tracking. The system can generate sound field parameters in real time, including the direct sound / reflected sound energy ratio, reverberation time, equalization curve, and sound image diffusion. Furthermore, it utilizes beamforming, multi-channel convolution, and virtual surround sound technologies to achieve dynamic synthesis of the sound field. This ensures that the audio output is highly coordinated with the listener's real-time state and environment in terms of spatial distribution, spectral characteristics, and reverberation effects, greatly enhancing the auditory immersion.

[0022] 3. This invention employs mature algorithms such as generalized cross-correlation-phase transform localization, recursive least squares adaptive system identification, and support vector machine emotion classification, combined with multi-core parallel processing and hardware acceleration mechanisms. It can operate stably in complex acoustic environments with high noise, multiple sound sources, and dynamic reverberation. The overall system latency is controlled within 50 milliseconds, supports multi-audience tracking and emotion state recognition, and possesses excellent real-time performance and scene adaptability. It can be widely applied in scenarios with high requirements for audio quality and natural interaction, such as home theaters, car audio systems, and virtual conferencing. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall technical solution architecture of the present invention. Detailed Implementation

[0024] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description of specific embodiments based on the present invention is provided in conjunction with the accompanying drawings and preferred embodiments.

[0025] Reference Figure 1 As shown, the present invention provides a method and system for controlling audio playback based on a voice sensor. Its core lies in constructing a multi-dimensional dynamic scene model that can perceive the listener's position, emotional state and spatial acoustic environment in real time, and actively synthesizing a personalized three-dimensional sound field that is highly matched with the current auditory scene based on the model.

[0026] A method for controlling audio playback based on a voice sensor includes the following steps: S1: Periodically collects multi-channel raw acoustic signals within the physical space where the audio equipment is located through a sensor array consisting of multiple acoustic sensor units.

[0027] The sensor array is specifically implemented as a planar circular array containing eight microelectromechanical system (MEMS) microphone units. All microphone units are uniformly distributed along a circumference with a diameter of 100 mm. The signal-to-noise ratio (SNR) of each microphone unit is no less than 65 dB, the sampling frequency is set to 48 kHz, and the sampling bit depth is 24 bits. The acquisition process is executed at a fixed period of 20 ms, corresponding to a sampling rate of 50 frames per second. Within each acquisition period, the eight microphone units synchronously output eight independent digital audio signals, each signal being a time series with a length of 960 sampling points.

[0028] The multi-channel raw acoustic signal includes not only ambient noise and background music, but also human voice signals emitted by one or more listeners. To ensure signal synchronization, all microphone units share the same clock source and achieve nanosecond-level time alignment through a hardware triggering mechanism. If a microphone unit loses data due to hardware failure or signal interruption, the system activates a redundancy check mechanism, uses the signals from adjacent microphone units to compensate through linear interpolation, and marks that channel as unreliable, reducing its weight or removing it directly in subsequent processing.

[0029] The acquired eight-channel signals are encapsulated into a three-dimensional tensor with dimensions of 8 channels × 960 sampling points / channel × 1 frame, which serves as the input data for the subsequent acoustic scene feature extraction module.

[0030] S2: Input the multi-channel raw acoustic signals into an acoustic scene feature extraction module. The acoustic scene feature extraction module performs three core operations in parallel, corresponding to sound source localization, human voice emotion analysis and room impulse response estimation, respectively.

[0031] S2.1: First, a sound source localization operation is performed, which employs a steering response power algorithm based on generalized cross-correlation-phase transform. Specifically, the system iterates through all possible microphone unit pair combinations, a total of 28 pairs, calculates the generalized cross-correlation function for each pair of signals, and applies phase transform weighting to suppress reverberation interference.

[0032] Subsequently, energy integration is performed within a pre-defined three-dimensional spatial search grid. This grid covers a spherical space with a radius of five meters centered on the audio equipment, with an azimuth resolution of 5°, a pitch resolution of 5°, and a distance resolution of 0.1m. By searching for the point of maximum turning response power within this three-dimensional spatial search grid, the initial three-dimensional spatial coordinates of the human voice source are determined.

[0033] To improve positioning stability, the system performs temporal smoothing filtering on the positioning results of 10 consecutive frames, using a first-order low-pass filter with a time constant set to 200ms, thereby outputting real-time location trajectory data for one or more listeners. If multiple sound sources are detected, the system clusters them based on energy intensity and spatial separation, tracking a maximum of three independent sound sources. When the sound source's moving speed exceeds a preset threshold (e.g., 2m / s), the filter automatically switches to high dynamic range mode, reducing the time constant to enhance response speed.

[0034] S2.2: Next, perform voice feature analysis. This operation first uses a Gaussian mixture model-based speech activity detection algorithm to jointly analyze the eight-channel signals and identify the effective time period containing human voices.

[0035] The speech activity detection algorithm calculates short-time energy and zero-crossing rate for each frame at 20ms / frame, and combines multi-channel spatial coherence characteristics to determine whether the frame belongs to a human voice activity segment. Once a human voice activity segment is detected, the system extracts the signal within that time period and divides it into frames with a frame length of 20ms and a frame shift of 10ms, and applies a Hamming window to each frame.

[0036] For each frame of human voice signal, the system performs the following feature extraction: The peak position of the autocorrelation function is calculated to determine the fundamental frequency, which is limited to the range of 70 Hz-300 Hz. Through 12th-order linear predictive coding analysis, the roots of the characteristic polynomial are solved, and the center frequencies and bandwidths of the first three formants are extracted. The logarithm of the sum of squares of the signal within the frame is calculated as the short-time energy. The number of syllable boundaries per unit time is counted to determine the speech rate. Syllable boundaries are obtained by detecting local minima of the energy envelope. At the same time, the first 13-dimensional Mel frequency cepstral coefficients are extracted. The calculation process includes pre-emphasis, Fourier transform, Mel filter bank mapping, and discrete cosine transform.

[0037] All the above features constitute an 18-dimensional acoustic prosodic feature parameter set, which is input into a pre-trained support vector machine (SVM) sentiment classification model. The sentiment classification model uses radial basis function kernels, and its decision boundary is obtained through training on a large number of speech datasets labeled "calm," "excited," and "neutral." The sentiment classification model outputs one of three sentiment state labels. If the distance between the input feature and any category center exceeds a preset threshold, the system determines it as "unknown sentiment" and maintains the previous valid sentiment label.

[0038] S2.3: Parallel execution of room impulse response estimation operation. This operation adopts a multi-channel recursive least squares adaptive filtering algorithm. The system uses the original audio signal currently played by the audio equipment as the reference input signal, and its sampling rate is consistent with that of the microphone array.

[0039] For each microphone channel, the system maintains a finite impulse response filter with a length of 4096 taps. In each acquisition cycle, the algorithm calculates the error between the filter output and the actual microphone signal, and updates the filter coefficients using a recursive least squares formula.

[0040] The forgetting factor is set to 0.999 to balance convergence speed and steady-state accuracy. When the mean square error change of 100 consecutive frames is less than the preset tolerance (e.g., -60dB), the algorithm is considered to have converged. At this time, the filter coefficients are the estimated values ​​of the room impulse response function from the audio playback point to the microphone unit. The system performs this process on each of the eight channels, and finally obtains eight sets of room impulse response functions, each set consisting of 4096 floating-point numbers, which together characterize the transmission characteristics of the spatial acoustic environment. If the signal-to-noise ratio of a certain channel is too low to converge, the impulse response of the adjacent channel is used for spatial interpolation.

[0041] S3: Input real-time location trajectory data, quantified emotional state labels, and room impulse response functions into a listener-environment state fusion modeling module.

[0042] The audience-environment state fusion modeling module first constructs a joint state vector, the structure of which is as follows: For each tracked listener, the data includes their three-dimensional coordinates (x, y, z), velocity vector (vx, vy, vz), emotional state label (represented by integer encoding), and an associated room impulse response index; in addition, it includes global acoustic parameters such as mean reverberation time (obtained by fitting decay curves of eight impulse responses), early reflection energy percentage, and background noise power spectral density.

[0043] The total dimension of the joint state vector changes dynamically according to the number of listeners, supporting a maximum of three listeners, at which point the dimension is 3×(6+1+1)+3=27.

[0044] Subsequently, the system is configured with a Kalman filter to iteratively update the joint state vector. The state equation adopts a uniform motion model, that is, the position at the next moment is equal to the current position plus the velocity multiplied by the time step, and the velocity remains unchanged; the observation equation maps the real-time data output by the current S2 step to the state space.

[0045] Specifically, location observations directly correspond to the coordinate components in the state vector; sentiment tags, as discrete observations, are softened by introducing virtual continuous variables; and the room impulse response function extracts its key statistics (such as the energy decay time constant) as continuous observation inputs.

[0046] The process noise covariance matrix Q of the Kalman filter is set as a diagonal matrix, where the variance of the position component is 0.01m², the variance of the velocity component is 0.001m² / s², and the emotional and acoustic parameter components are set empirically. The observation noise covariance matrix R is dynamically adjusted according to the actual accuracy of each sensor. For example, the variance of the position observation is increased when the positioning error is large.

[0047] The filter performs a prediction and update loop once per frame, outputting a continuous, smooth, and physically consistent multidimensional dynamic scene state descriptor. This descriptor contains not only the best estimate of the current state, but also its uncertainty covariance matrix, which is used for confidence assessment of subsequent decisions.

[0048] S4: Input the multi-dimensional dynamic scene state descriptor into a sound field rendering parameter generation module. The sound field rendering parameter generation module stores a preset mapping rule library, which is implemented by combining lookup tables and conditional logic.

[0049] The core logic of the rule base is based on three key dimensions: the number of listeners, the dominant emotional state (taking the emotional label of the listener with the strongest energy), and the global reverberation time for branch judgment.

[0050] When the system identifies a single listener, whose emotional state is "calm," and whose reverberation time is less than 0.4s, the rule base outputs the first set of sound field rendering target parameters. The direct sound to reflected sound energy ratio is set to 12dB, the target reverberation time is set to 0.3s, the multi-band equalizer gain curve is boosted by 3 to 6dB in the 500Hz to 4000Hz frequency band to form a mid-high frequency bulge, and the sound image diffusion is set to 15° to create a clear and focused near-field listening experience.

[0051] When the system identifies multiple listeners (two or more), their emotional state is "excited," and the reverberation time is greater than 0.6s, the rule base outputs the second set of parameters: the direct sound to reflected sound energy ratio is set to 3dB, the target reverberation time is set to 0.8s, the equalizer gain curve is boosted by 4dB in the frequency bands below 200Hz and above 8000Hz respectively to form a "V" shaped response, and the sound image diffusion is set to 60° to enhance the sense of spatial immersion and dynamic impact.

[0052] For other intermediate states, the rule base uses linear interpolation or fuzzy logic to smooth the transition. For example, when the emotional state is "neutral," all parameters take the median value between the two extremes; when there are two listeners and their emotions are inconsistent, the energy-weighted average emotional intensity is used as the basis for decision-making.

[0053] All sound field rendering target parameters are encapsulated into a structured data packet containing four fields: direct reflection ratio (floating-point number), target reverberation time (seconds), equalizer gain curve (an array of gain values ​​for 32 frequency points), and sound image diffusion (angle).

[0054] S5: Input the original audio signal to be played and the sound field rendering target parameters into a dynamic sound field synthesis module. The dynamic sound field synthesis module performs a series of real-time digital signal processing operations.

[0055] First, the beamforming processor calculates the geometric propagation delay from the speaker array to the target listener based on real-time position trajectory data and generates a corresponding complex weight vector. The beamforming processor employs a minimum variance distortionless response beamforming algorithm to ensure the formation of a main lobe in the target direction while simultaneously creating nulls in the interference direction. The audio signal is replicated to multiple virtual channels, each with corresponding delay and phase adjustment applied to achieve energy focusing.

[0056] Secondly, the multi-channel convolutional converter receives eight sets of original room impulse response functions and target reverberation time parameters. The system first performs dreverberation preprocessing on the original audio signal using inverse filtering technology to eliminate the acoustic effects of the current room; then, based on the target reverberation time, it selects or synthesizes a new set of impulse response functions from the pre-stored reverberation kernel library in real time, and performs fast convolution operation with the dreverberated signal to inject the target reverberation characteristics.

[0057] Secondly, the parametric equalizer performs spectral shaping on the audio signal based on the gain curve of the multi-band equalizer. The parametric equalizer uses a 32-channel parametric filter bank, with each channel being a second-order IIR filter with a fixed center frequency, and its gain is directly set by the corresponding frequency value output from the rule base.

[0058] Finally, the virtual surround sound processor adjusts the correlation between each speaker channel according to the target sound image diffusion. Specifically, it introduces a controllable delay difference and phase shift between the left and right channels and superimposes a filtered crosstalk signal to make the sound image form a diffusion area of ​​a specified width in front of the listener.

[0059] All of the above processing is performed in parallel in a pipeline on a dedicated audio digital signal processor, with the total processing delay controlled within 50ms. The processed multi-channel audio signal is output to the physical speaker unit of the audio equipment for playback, thereby achieving a personalized three-dimensional sound field that is highly matched with the current state of the listener and the spatial environment.

[0060] At the system level, the present invention provides an audio playback control system based on a voice sensor, including an acoustic signal acquisition module, an acoustic scene feature extraction module, an audience-environment state fusion modeling module, a sound field rendering parameter generation module, and a dynamic sound field synthesis module.

[0061] The acoustic signal acquisition module, namely the aforementioned eight-microphone circular array, has an I2S bus hardware interface that is directly connected to the audio codec of the main control processor.

[0062] The acoustic scene feature extraction module is deployed in software on an embedded multi-core processor, where sound source localization, human voice analysis and impulse response estimation run on independent processing cores and exchange data through shared memory.

[0063] The audience-environment state fusion modeling module is implemented as a state machine, whose Kalman filter runs at a fixed rate to ensure real-time state updates.

[0064] The sound field rendering parameter generation module exists in the form of a rule engine, and its mapping rule library is stored in non-volatile memory, supporting remote updates.

[0065] The dynamic sound field synthesis module is integrated into a high-performance audio DSP chip, featuring hardware-accelerated convolution and filtering units to ensure low latency and high fidelity in audio processing.

[0066] All modules are interconnected via an internal bus, forming a closed-loop perception-decision-execution system, thus fully realizing the technical solution of this invention.

[0067] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0068] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment includes only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for controlling audio playback based on a voice sensor, characterized in that, include: S1: Collects multi-channel raw acoustic signals within the physical space where the audio equipment is located through a sensor array consisting of multiple acoustic sensor units; S2: Input the multi-channel raw acoustic signals into the acoustic scene feature extraction module and perform the following operations in parallel: The real-time location trajectory data of the audience is obtained through a multi-channel sound source localization algorithm; By using a human voice feature analysis algorithm, the set of acoustic prosodic feature parameters is extracted and mapped to emotional state labels; The room impulse response function is estimated using a blind system identification algorithm; S3: Input real-time location trajectory data, emotional state labels, and room impulse response function into the audience-environment state fusion modeling module to construct a joint state vector, and use a Kalman filter to generate a multi-dimensional dynamic scene state descriptor; S4: Input the multi-dimensional dynamic scene state descriptor into the sound field rendering parameter generation module, and map it into sound field rendering target parameters according to the mapping rule library; S5: Input the original audio signal to be played and the sound field rendering target parameters into the dynamic sound field synthesis module, perform real-time digital signal processing, and output the modulated multi-channel audio signal.

2. The audio playback control method based on a voice sensor according to claim 1, characterized in that, In step S2, the multi-channel sound source localization algorithm includes: The signals between all microphone unit pairs in the multi-channel raw acoustic signal are processed by generalized cross-correlation-phase transformation, the steering response power is calculated, and the energy peak is searched in the preset three-dimensional spatial grid to determine the azimuth, pitch and distance of the sound source. The positioning results of multiple consecutive frames are processed by a first-order low-pass filter for temporal smoothing, and real-time position trajectory data is output.

3. The audio playback control method based on a voice sensor according to claim 1, characterized in that, In step S2, the human voice feature analysis algorithm includes: A speech activity detection algorithm is used to segment human voice signal segments from multi-channel raw acoustic signals; The human voice signal segment is segmented by frame length and a Hamming window is applied. The fundamental frequency, formant, short-time energy, speech rate and Mel frequency cepstral coefficients of each frame signal are calculated to form a set of acoustic prosodic feature parameters. The acoustic prosodic feature parameter set is input into the support vector machine emotion classification model, and one of the three emotion state labels, namely "calm", "excited" or "neutral", is output as the quantified emotion state label.

4. The audio playback control method based on a voice sensor according to claim 1, characterized in that, In step S2, the blind system identification algorithm includes: The original audio signal played by the audio equipment is used as the reference input, and the multi-channel acoustic signal collected by the sensor array is used as the desired output. For each microphone channel, run a multi-channel recursive least squares adaptive filtering algorithm to iteratively adjust the coefficients of the finite impulse response filter until convergence. The converged filter coefficients are used as the room impulse response function from the audio equipment to the corresponding microphone unit.

5. The audio playback control method based on a voice sensor according to claim 1, characterized in that, In step S3, constructing the joint state vector includes: For each tracked listener, a state subvector is constructed that includes three-dimensional coordinates, velocity vector, emotional state labels, and room impulse response index; Combine the state subvectors of all listeners with the global acoustic parameters into a joint state vector; A state equation is constructed based on a uniform motion model, and an observation equation is constructed based on key statistics of real-time location trajectory data, emotional state labels, and room impulse response function. In each processing cycle, a Kalman filter prediction and update step is performed, outputting a multi-dimensional dynamic scene state descriptor.

6. The audio playback control method based on a voice sensor according to claim 1, characterized in that, In step S4, the target parameters for sound field rendering include the ratio of direct sound to reflected sound energy at the target listener's location, the target reverberation time, the multi-band equalizer gain curve, and the target sound image diffusion. When the multi-dimensional dynamic scene state descriptor indicates a single listener, the emotional state is "calm" and the reverberation time is less than the preset threshold, a high direct sound to reflected sound energy ratio is set, the target reverberation time is shortened, the mid-to-high frequency band equalizer gain is increased and the sound image diffusion is reduced. When the multi-dimensional dynamic scene state descriptor indicates multiple listeners, the emotional state is "excited", and the reverberation time is greater than the preset threshold, a low direct sound to reflected sound energy ratio is set, the target reverberation time is extended, the low-frequency and high-frequency equalizer gain is increased to form a "V" shaped curve and increase the sound image diffusion.

7. The audio playback control method based on a voice sensor according to claim 1, characterized in that, In step S5, the dynamic sound field synthesis module performs real-time digital signal processing, including: Beamforming technology is used to focus audio energy onto the listener's location indicated by real-time location trajectory data; The room impulse response function and audio signal are processed through multi-channel convolution operations to match the target reverberation time; The spectral components of the audio signal are adjusted using the gain curve of a multi-band equalizer. The delay and phase relationship of each playback channel are adjusted by a virtual surround sound algorithm to achieve the target sound image diffusion.

8. The audio playback control method based on a voice sensor according to claim 7, characterized in that, The application of beamforming technology includes: The geometric propagation delay from the sound array to the target audience is calculated based on real-time location trajectory data; A minimum variance distortionless response beamforming algorithm is used to generate complex weight vectors. By applying corresponding delay and phase adjustment to the original audio signal, the audio energy is focused in the target direction.

9. The audio playback control method based on a voice sensor according to claim 7, characterized in that, Matching the target reverberation time through multi-channel convolution operations includes: The original audio signal is preprocessed to remove reverberation to eliminate the acoustic effects of the current room. The target room impulse response function is selected from a pre-stored reverberation kernel library or synthesized in real time based on the target reverberation time; The dereverberated audio signal is rapidly convolved with the target room impulse response function to inject the target reverberation characteristics.

10. A voice sensor-based audio playback control system, characterized in that, include: The acoustic signal acquisition module is used to acquire multi-channel raw acoustic signals within the physical space where the audio equipment is located through a sensor array consisting of multiple acoustic sensor units. The acoustic scene feature extraction module is used to perform sound source localization, human voice emotion analysis and room impulse response estimation in parallel from the multi-channel raw acoustic signals, and output real-time location trajectory data, quantized emotion state labels and spatial acoustic environment transmission characteristic data, respectively. The audience-environment state fusion modeling module is used to fuse real-time location trajectory data, emotional state labels and room impulse response functions into a joint state vector, and generate a multi-dimensional dynamic scene state descriptor through a Kalman filter. The sound field rendering parameter generation module is used to query a preset mapping rule library based on the multi-dimensional dynamic scene state descriptor and output a set of sound field rendering target parameters. The dynamic sound field synthesis module receives the original audio signal to be played and the target parameters for sound field rendering, and performs beamforming, multi-channel convolution, spectral equalization and virtual surround sound processing, and outputs the modulated multi-channel audio signal to the speaker unit of the audio equipment.

Images