A multi-audio acquisition terminal state intelligent audio-visual device adjustment method and system

Abstract

The application discloses a kind of multi-audio acquisition end state's wisdom audio-visual equipment adjustment method and system.Method includes: obtaining the initial spatial audio signal and initial acoustic impulse response of multiple audio acquisition end activation state in scene, and component analysis is carried out;Signal correlation degree of multiple audio acquisition end audio acquisition is evaluated, and according to signal correlation degree, reverberation tail signal and its time decay envelope are obtained;The energy distribution state between early reflection and late diffusion in reverberation tail signal is evaluated, and gain adjustment is carried out to reverberation tail signal, to obtain adjusted reverberation tail signal and its time decay envelope;Target spatial listening audio is generated, and the multi-audio acquisition end state adjustment of audio-visual equipment is completed.The application provides a kind of multi-audio acquisition end state's wisdom audio-visual equipment adjustment method and system, to solve the safety and the lower natural degree of hearing of the prior art in reverberation tail signal truncation, and the technical problem of poor audio output quality.

Description

Technical Field

[0001] This invention relates to the field of acoustics, and in particular to a method and system for adjusting the status of intelligent audio-visual equipment with multiple audio acquisition terminals. Background Technology

[0002] With the rapid development of smart audio-visual devices, their application in scenarios such as conference discussions, karaoke entertainment, and remote work is becoming increasingly widespread. One of the core technologies for achieving high-quality audio output from smart audio-visual devices is the multi-mode state adjustment method. Its goal is to adaptively truncate the reverberation tail signals of different audio acquisition channels in complex acoustic environments where conference modes and entertainment modes such as karaoke are mixed, thereby ensuring a balance between speech clarity and naturalness of sound.

[0003] Currently, the mainstream methods for adjusting the multi-mode status of smart audio-visual devices mainly include adjustment methods based on fixed acoustic thresholds and arbitration switching methods based on the priority of audio acquisition terminals. The adjustment method based on fixed acoustic thresholds primarily employs preset signal-to-noise ratio (SNR) threshold technology. However, the truncation strategy generated by this method is determined by a single SNR value. When multiple audio acquisition terminals pick up highly correlated signals from the same source, the degree of background noise superposition is far lower than the preset estimate, resulting in an overly aggressive truncation position. The reverberation tail is forcibly cut off, leading to extremely poor naturalness and failing to conform to the human ear's perception of spatial reverberation. Furthermore, in pursuit of ultimate speech clarity, the truncation action generated by the algorithm often ignores the energy distribution ratio between early reflections and later diffusion, resulting in a severe energy imbalance and insufficient sound fullness. In actual listening, this easily produces a dry and harsh auditory experience. Arbitration switching methods based on audio acquisition terminal priority mainly employ priority-based preemptive truncation technology. The timing of truncation of the reverberation tail signal is directly triggered by the audio acquisition terminal's switching command. This randomness leads to a severe misalignment between the truncation point and the physical laws of the reverberation attenuation envelope, resulting in abrupt breaks or lingering echoes, and poor smoothness and continuity. Therefore, the biggest drawback of existing multi-mode state adjustment methods lies in the low security and naturalness of reverberation tail signal truncation, resulting in poor audio output quality. Summary of the Invention

[0004] This invention provides a method and system for adjusting the status of intelligent audio-visual equipment with multiple audio acquisition terminals, in order to solve the technical problems of low safety and naturalness of sound in the reverberation tail signal truncation and poor audio output quality in the prior art.

[0005] The present invention discloses the following technical solutions: In a first aspect, embodiments of the present invention provide a method for adjusting the status of a smart audio-visual device with multiple audio acquisition terminals, the method comprising: The initial spatial audio signal and initial acoustic impulse response of multiple audio acquisition terminals in the scene are acquired under active state. The initial spatial audio signal and initial acoustic impulse response are processed by component analysis to separate the background noise superposition component generated by the superposition of multiple sound channels and the homologous speech component of the same sound. Based on the noise floor superposition component and the homologous speech component, the signal correlation of audio collected by multiple audio acquisition terminals is evaluated, and the reverberation tail signal and its time decay envelope are obtained from the initial spatial audio signal according to the signal correlation. Based on the reverberant tail signal and its time decay envelope, the energy distribution state between early reflection and late diffusion in the reverberant tail signal is evaluated, and the gain of the reverberant tail signal is adjusted to obtain the adjusted reverberant tail signal and its time decay envelope. Based on the homologous speech components, the adjusted reverberation tail signal and its time decay envelope, the target spatial auditory audio is generated to complete the state adjustment of the multi-audio acquisition end of the audio-visual equipment.

[0006] Optionally, the initial spatial audio signal and initial acoustic impulse response of multiple audio acquisition terminals in the scene are acquired under active states, and component analysis is performed on the initial spatial audio signal and initial acoustic impulse response to separate the background noise superposition component generated by the superposition of multiple sounds, and the homologous speech component of the same sound, including: Based on the initial spatial audio signal, the acoustic impulse response of the initial scene is identified; Frequency domain audio data is obtained by frequency domain mapping of the initial spatial audio signal and the initial scene acoustic impulse response. Component analysis is performed on the frequency domain audio data to separate the noise superposition component generated by the superposition of multiple sounds and the homologous speech component of the same sound.

[0007] Optionally, based on the noise floor superposition component and the homologous speech components, the signal correlation of audio acquired by multiple audio acquisition terminals is evaluated, including: Calculate the covariance matrix of the noise floor superposition component and the homologous speech component to obtain the cross-correlation matrix; The cross-correlation matrix is ​​subjected to singular value decomposition to obtain the largest eigenvalue; The audio acquisition terminal that picks up signals whose maximum eigenvalue exceeds the preset acoustic correlation threshold determines that the signals have the same origin attribute and determines the signal correlation degree.

[0008] Optionally, the reverberant tail signal and its time decay envelope are obtained from the initial spatial audio signal based on signal correlation, including: Based on signal correlation, the initial spatial audio signal is subjected to frequency domain masking to obtain a transient suppression signal; The non-direct sound components are separated from the transient suppression signal, and the energy curve of the reverberation tail signal is extracted. Based on the energy curve, the time decay envelope of the reverberation tail is obtained.

[0009] Optionally, based on the reverberant tail signal and its time decay envelope, the energy distribution state between early reflection and late diffusion in the reverberant tail signal is evaluated, and the gain of the reverberant tail signal is adjusted to obtain the adjusted reverberant tail signal and its time decay envelope, including: By performing time-frequency domain conversion on the time decay envelope using short-time Fourier transform, the early reflection energy distribution and the later diffusion energy distribution of the reverberation tail signal are obtained. Calculate the energy ratio between the early reflection energy distribution and the later diffusion energy distribution, analyze whether the extreme values ​​in the energy ratio exceed the preset energy distribution range, and obtain the imbalance assessment results; Based on the imbalance assessment results, the gain of the reverberation tail signal is adjusted to obtain the adjusted reverberation tail signal and its time decay envelope.

[0010] Optionally, based on the imbalance assessment results, the gain of the reverberation tail signal is adjusted to obtain the adjusted reverberation tail signal and its time decay envelope, including: Obtain the energy distribution ratio between early reflection and later diffusion from the imbalance assessment results; When the energy allocation ratio is greater than the preset allocation ratio threshold, the difference between the energy allocation ratio and the allocation ratio threshold is calculated to determine the gain coefficient; Gain adjustment of the reverberation tail signal is performed based on the gain coefficient. The real-time truncation position of the reverberation tail signal is determined by detecting the trigger time of the priority signal at the audio acquisition end. The priority signal at the audio acquisition end is obtained from the reverberation tail signal after gain adjustment. The reverberation tail signal is truncated based on the real-time truncation position to obtain the target time decay envelope.

[0011] Optionally, based on the homologous speech components, the adjusted reverberant tail signal, and their time decay envelope, target spatial auditory audio is generated to complete the state adjustment of the multi-audio acquisition end of the audio-visual equipment, including: Amplitude modulation of the time decay envelope yields a decayed reverberation component with spatial sound field characteristics. Based on the homologous speech components and the attenuated reverberation components, the time difference of arrival is calculated to obtain a phase-aligned signal containing azimuth information. The phase-aligned signal and the attenuation reverberation component are fused and superimposed to generate the target space listening audio, thus completing the multi-mode state adjustment of the audio-visual equipment.

[0012] Secondly, embodiments of the present invention provide a smart audio-visual device adjustment system for multiple audio acquisition terminal states, including: The acquisition module is used to acquire the initial spatial audio signal and initial acoustic impulse response of multiple audio acquisition terminals in the scene under active state, and to perform component analysis processing on the initial spatial audio signal and initial acoustic impulse response to separate the background noise superposition component generated by the superposition of multiple sounds and the homologous speech component of the same sound. The evaluation module is used to evaluate the signal correlation of audio collected by multiple audio acquisition terminals based on the noise floor superposition component and the homogeneous speech component, and to obtain the reverberation tail signal and its time decay envelope from the initial spatial audio signal according to the signal correlation. The gain module is used to evaluate the energy distribution state between early reflection and late diffusion in the reverberation tail signal based on the reverberation tail signal and its time decay envelope, and to adjust the gain of the reverberation tail signal to obtain the adjusted reverberation tail signal and its time decay envelope. The adjustment module is used to generate target spatial audio based on homologous speech components, the adjusted reverberation tail signal and its time decay envelope, and to complete the status adjustment of the multi-audio acquisition end of the audio-visual equipment.

[0013] Thirdly, another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the smart audio-visual device adjustment method as described above for any of the multiple audio acquisition terminal states.

[0014] Another embodiment of the present invention provides a computer program product, including a computer program or instructions, which, when executed by a device, implement the steps of a smart audio-visual device adjustment method for multiple audio acquisition terminal states as described above.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention decomposes complex mixed signals into background noise and homogeneous speech through independent component analysis, providing a clean signal source for subsequent accurate correlation assessment and reverberation processing. By extracting signal correlation, starting from the physical essence of whether multiple signals originate from the same source, it solves the contradiction of truncation strategies caused by traditional methods relying solely on signal-to-noise ratio, providing a decision basis for adaptive reverberation control. By evaluating the energy distribution of early reflections and later diffusion, it accurately identifies the auditory imbalance problem of the reverberation tail signal in entertainment mode, providing a quantitative target for targeted gain adjustment. Finally, by fusing the adjusted reverberation with clean speech, a target audio that combines conference clarity and entertainment richness is generated, achieving adaptive balance across multiple modes. Attached Figure Description

[0016] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a method for adjusting the status of a smart audio-visual device with multiple audio acquisition terminals, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the specific implementation process of a smart audio-visual device adjustment method for multiple audio acquisition terminal states provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the time attenuation envelope acquisition process of a smart audio-visual device adjustment method for multiple audio acquisition terminal states provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a smart audio-visual device adjustment device for multiple audio acquisition terminal states provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the invention, are intended to cover non-exclusive inclusion.

[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] See Figure 1To address the technical problems of low safety and naturalness of sound due to reverberation tail signal truncation in existing technologies, resulting in poor audio output quality, an embodiment of the present invention provides a method for adjusting the status of a smart audio-visual device with multiple audio acquisition terminals, comprising: S1. Acquire the initial spatial audio signal and initial acoustic impulse response of multiple audio acquisition terminals in the scene under active states, and perform component analysis processing on the initial spatial audio signal and initial acoustic impulse response to separate the background noise superposition component generated by the superposition of multiple sounds and the homologous speech component of the same sound.

[0022] The system first synchronously acquires initial spatial audio signals from multiple physical audio pickup nodes deployed in a smart audio-visual environment, assuming the multiple audio acquisition terminals are in an active state. To ensure precise alignment of the multiple signals on the time axis, the system performs high-precision clock synchronization on each pickup node based on a unified global clock source. Simultaneously, the system utilizes the mixed audio signals generated when the multiple audio acquisition terminals are activated to extract the initial scene acoustic impulse response of each pickup node, serving as acoustic prior information for subsequent signal separation. Based on this, the initial spatial audio signals and the initial scene acoustic impulse responses are mapped in the frequency domain to obtain frequency domain audio data. Subsequently, an independent component analysis (ICA) algorithm is used to perform blind source separation processing on the frequency domain audio data. Specifically, an initial acoustic mixing matrix is ​​first constructed using the initial scene acoustic impulse response, and this matrix is ​​introduced as a constraint into the iterative optimization process of ICA. The objective separation matrix is ​​iteratively solved with the goal of maximizing the non-Gaussianity of the output signal. The target separation matrix is ​​used to decouple the frequency domain audio data, separating the noise superposition component with Gaussian distribution and stable energy, and the homologous speech component with non-Gaussian distribution, exhibiting harmonic structure and short-term energy fluctuations. Finally, the separated homologous speech components are reconstructed through inverse frequency domain mapping to obtain a clean speech signal with noise removed, which is used for subsequent reverberation tail extraction and energy distribution analysis.

[0023] S2 evaluates the signal correlation of audio collected by multiple audio acquisition terminals based on the noise floor superposition component and the homologous speech component, and obtains the reverberation tail signal and its time decay envelope from the initial spatial audio signal according to the signal correlation.

[0024] After obtaining the noise floor superposition component and the homologous speech components, the system further evaluates the correlation between the signals picked up by the multi-channel active audio acquisition terminals. First, the homologous speech components corresponding to each audio acquisition terminal are extracted, and their cross-correlation matrices are calculated pairwise. Each element of this matrix represents the correlation coefficient between the two signals in the time domain. Singular value decomposition is performed on the cross-correlation matrix to extract its maximum eigenvalue. The maximum eigenvalue is compared with a preset acoustic correlation threshold. If the maximum eigenvalue exceeds the threshold, it is determined that the signals picked up by each audio acquisition terminal originate from the same speaker, i.e., they have homologous attributes, and the signal correlation degree is calculated accordingly. Based on the evaluated signal correlation degree, the system extracts the reverberation tail and its time decay envelope from the initial spatial audio signal. When the signal correlation degree is high, it indicates that the direct sound and early reflections are dominant. The system monitors the correlation degree sequence in 10-millisecond steps. When the relative decrease in correlation degree between adjacent windows exceeds 30% and the decrease slope of three consecutive windows is less than 0.05 per millisecond, that moment is marked as the starting point of the reverberation tail. Subsequently, a spatial covariance matrix was constructed using the multi-channel cross-correlation matrix, and the strongly directional homologous speech components were suppressed by inversion operations, thereby extracting the reverberant tail exhibiting diffuse reflection characteristics. The extracted reverberant tail signal was first subjected to full-wave rectification, and then processed by a low-pass smoothing filter with a cutoff frequency of 20Hz to eliminate high-frequency micro-fluctuations, resulting in a smooth time-decay envelope. This envelope intuitively characterizes the macroscopic trend of reverberant tail energy decay over time, providing a foundation for subsequent time-frequency domain transformation and energy distribution analysis.

[0025] S3. Based on the reverberant tail signal and its time decay envelope, evaluate the energy distribution state between early reflection and late diffusion in the reverberant tail signal, and adjust the gain of the reverberant tail signal to obtain the adjusted reverberant tail signal and its time decay envelope.

[0026] After obtaining the time decay envelope, it is transformed to the time-frequency domain using a short-time Fourier transform. The early reflection energy distribution and the later diffusion energy distribution are separated using the adaptive reverberation junction point as the boundary. The energy ratio sequence of the two is calculated, and extreme values ​​are extracted. These extreme values ​​are compared with a preset energy distribution interval; if the extreme value exceeds the interval, it is determined that the energy distribution is unbalanced. Based on the direction of the imbalance, the reverberation tail is divided into three independent frequency bands: low, medium, and high. The error between the early and late energy ratios of each frequency band and the target balance ratio is calculated, and the frequency band gain is dynamically adjusted to restore energy distribution balance. Simultaneously, the reverberation energy change during priority switching at the audio acquisition end is monitored, and the cutoff position is determined at the energy plateau point. Finally, a target time decay envelope that smoothly returns to zero according to an exponential law is generated.

[0027] S4 generates target spatial audio based on homologous speech components, adjusted reverberation tail signals and their time decay envelope, and completes the state adjustment of multiple audio acquisition terminals of audio-visual equipment.

[0028] The process involves multiplying the target time decay envelope point-by-point with the adjusted reverberation tail and performing amplitude modulation to ensure the reverberation energy conforms to the exponential decay characteristics of the target space. The decay constant is determined based on the reverberation time RT60. Simultaneously, the energy ratio of the direct sound to the reverberation sound is calculated based on the distance and orientation of the virtual sound source. Gain scaling is applied to the source speech components and the modulated reverberation tail according to this ratio. In practical applications, to prevent phase cancellation, the cross-correlation peak is typically estimated within the key frequency band of 200Hz to 2kHz, and the initial delay of the reverberation tail is fine-tuned to achieve phase alignment. Finally, the scaled and aligned components are linearly superimposed to generate target spatial audio with clear speech and realistic spatial perception, thus enabling adaptive state adjustment when switching between meeting and entertainment modes.

[0029] Furthermore, such as Figure 2 As shown, this application provides a specific implementation process for a smart audio-visual device adjustment method based on the states of multiple audio acquisition terminals. First, in a multi-mode audio-visual scenario where meetings and karaoke (i.e., entertainment mode) are combined, multiple microphones (i.e., audio acquisition terminals) are acquired through multiple physical pickup nodes, activating the initial spatial audio signal, and the initial scene acoustic impulse response of each pickup node is simultaneously identified. Then, based on the initial spatial audio signal and the initial scene acoustic impulse response, an independent component analysis algorithm is used to separate the noise floor component generated by the superposition of multiple pickups from the homologous speech component of the conference speaker. Next, the cross-correlation matrix of the noise floor superposition component and the homologous speech component is extracted. By analyzing whether the maximum eigenvalue exceeds the preset acoustic correlation threshold, the signal correlation of the signals picked up by the multi-channel active audio acquisition end is evaluated. Based on the signal correlation, the reverberation tail and its time decay envelope are extracted from the initial spatial audio signal. Then, the reverberation tail and time decay envelope are transformed in the time-frequency domain by short-time Fourier transform to obtain the early reflection energy distribution and the later diffusion energy distribution. The energy ratio of the early reflection energy distribution to the later diffusion energy distribution is then extracted, and the extreme values ​​in the energy ratio are analyzed to determine whether they exceed the preset energy distribution range, thus evaluating the energy distribution imbalance between early reflection and later diffusion in the karaoke reverberation tail signal. Based on this energy distribution imbalance, an adaptive filtering algorithm is used to adjust the frequency band gain of the reverberation tail, and the real-time truncation position of the reverberation tail signal when the audio acquisition end switches priorities is determined, resulting in the adjusted reverberation tail and the target time decay envelope. Finally, the adjusted reverberation tail, target time decay envelope, and homologous speech components are fused together to generate target spatial audio, thereby effectively eliminating the abruptness of trailing and the muffledness of speech during mode switching, achieving an adaptive balance between meeting clarity and karaoke immersion.

[0030] In one embodiment, the initial spatial audio signal and initial acoustic impulse response of multiple audio acquisition terminals in the scene are acquired under active states, and component analysis is performed on the initial spatial audio signal and initial acoustic impulse response to separate the background noise superposition component generated by the superposition of multiple sounds and the homologous speech component of the same sound. This includes steps S201 to S203, each step of which is as follows: S201 identifies the initial scene acoustic impulse response based on the initial spatial audio signal.

[0031] In the multi-channel audio acquisition state, the system synchronously acquires the initial spatial audio signals from each physical pickup node and aligns them using a unified clock source. To identify the initial acoustic impulse response, an adaptive blind system identification algorithm is employed. First, the multiple initial spatial audio signals are converted to the frequency domain using a short-time Fourier transform. A multi-channel cross-correlation model based on the cross-correlation characteristics of signals from different pickup nodes is constructed, and a cost function for the impulse response of the unknown room is established. Subsequently, the filter weight coefficients are iteratively updated using either the normalized minimum mean square error algorithm or the recursive least squares algorithm under continuous input of multiple signals. When the normalized residual energy is lower than a preset residual energy threshold, the iteration stops, and the current filter weights are extracted as the initial acoustic impulse response. To improve recognition robustness, when strong nonlinear distortion is detected in a certain signal, the update step size of that filter is dynamically reduced, and its weight in the global cost function is decreased to avoid local abnormal signals affecting the overall recognition accuracy.

[0032] S202, the initial spatial audio signal and the initial scene acoustic impulse response are mapped in the frequency domain to obtain frequency domain audio data.

[0033] In this process, the continuous time-domain signal is framed and windowed using a short-time Fourier transform. In specific applications, a Hamming window with a frame length of 32 milliseconds and a frame shift of 16 milliseconds is typically used. The corresponding frequency domain representation is then obtained through a fast Fourier transform. The initial spatial audio signal is transformed to obtain a frequency domain audio data matrix, and the initial scene acoustic impulse response is transformed to obtain a frequency domain impulse response vector. The purpose of frequency domain mapping is to transform the time-domain convolution relationship into a frequency-domain multiplication relationship, facilitating efficient decoupling of the signal using algorithms such as independent component analysis, while reducing computational complexity and improving separation accuracy.

[0034] S203 performs component analysis on the frequency domain audio data to separate the noise superposition component generated by the superposition of multiple sounds and the homologous speech component of the same sound.

[0035] In this process, an initial acoustic mixing matrix is ​​constructed using the initial acoustic impulse response of the scene, serving as a constraint introduced into the iterative process of independent component analysis. The objective function is to maximize the non-Gaussianity of the output signal, iteratively solving for the target separation matrix. This separation matrix is ​​used to linearly decouple the frequency-domain audio data, yielding multiple independent frequency-domain components. Based on the statistical characteristics of each component, those exhibiting Gaussian distribution and stable energy are identified as noise floor superposition components; those exhibiting non-Gaussian distribution, harmonic structure, and short-term energy fluctuations are identified as homologous speech components. Finally, the separated homologous speech components are reconstructed into a clean time-domain speech signal through inverse frequency-domain mapping.

[0036] This embodiment first identifies the initial acoustic impulse response of the scene, then performs frequency domain mapping between the initial spatial audio signal and the impulse response, and finally performs independent component analysis on the frequency domain audio data, achieving accurate separation of the noise floor superposition component and the homologous speech component. Its advantages lie in using the room acoustic impulse response as a priori constraint, effectively overcoming the problem of traditional blind source separation algorithms easily getting trapped in local optima in strong reverberation environments, and significantly improving the separation accuracy of noise floor and speech. Simultaneously, frequency domain mapping reduces computational complexity, making the separation process more efficient. The final output of clean homologous speech components eliminates the signal-to-noise ratio decrease caused by noise floor superposition when multiple audio acquisition terminals are used concurrently, preserving the naturalness and intelligibility of the speech, and providing a high-quality audio foundation for subsequent reverberation tail extraction and spatial auditory reconstruction.

[0037] In one embodiment, the signal correlation of audio collected by multiple audio acquisition terminals is evaluated based on the noise floor superposition component and the homologous speech component, including steps S301 to S303, each step of which is as follows: S301, calculate the covariance matrix of the noise floor superposition component and the homologous speech component to obtain the cross-correlation matrix.

[0038] In this process, after separating the background noise superposition component and the homologous speech component, the system constructs a multi-channel signal vector from the homologous speech components corresponding to each audio acquisition terminal. The covariance matrix of this vector is then calculated, i.e., cross-correlation is performed on the signals of each pair of channels to obtain the cross-correlation matrix. Specifically, let the homologous speech components of the i-th and j-th channels be... and Calculate their cross-covariance as follows:

[0039] In the formula, and The mean is used. The calculation results of pairwise combinations of all channels form an N×N cross-correlation matrix, where N is the number of activated audio acquisition terminals. The elements of this matrix are the correlation coefficients in the time domain between the source speech components picked up by the i-th audio acquisition terminal and the j-th audio acquisition terminal, providing a basis for subsequent extraction of the maximum feature value and evaluation of signal correlation.

[0040] S302, perform singular value decomposition on the cross-correlation matrix to obtain the largest eigenvalue.

[0041] After obtaining the cross-correlation matrix among multiple source speech components, the system performs singular value decomposition on this matrix. Let the cross-correlation matrix be R, and the singular value decomposition is expressed as:

[0042] In the formula, It is a diagonal matrix. and The matrix is ​​unitary, with its main diagonal elements being singular values ​​arranged in descending order. Extracting the main diagonal elements yields a sequence of singular values. For a real symmetric cross-correlation matrix, the singular values ​​are equal to the absolute values ​​of the eigenvalues; the maximum value is taken as the largest eigenvalue. This largest eigenvalue quantifies the energy concentration of the dominant correlation component in multiple homologous speech components and is used to compare with a preset acoustic correlation threshold to evaluate the homologous properties and signal correlation of signals picked up by multiple audio acquisition terminals.

[0043] S303 determines the signal correlation degree by identifying audio acquisition signals whose maximum feature value exceeds the preset acoustic correlation threshold and determining that they have the same source attribute.

[0044] In this process, after obtaining the maximum eigenvalue of the cross-correlation matrix, the system compares it with a preset acoustic correlation threshold. In this embodiment, the typical value of the preset acoustic correlation threshold is 0.72, which is based on statistical calibration of single-source, dual-source, and noise samples in a typical conference room environment. If the maximum eigenvalue is greater than the preset acoustic correlation threshold, it is determined that the signals picked up by the multiple audio acquisition terminals originate from the same speaker, i.e., they have the same source attribute. Based on this, the signal correlation degree is calculated. The calculation formula is as follows:

[0045] In the formula, The largest eigenvalue, A preset acoustic correlation threshold is set. The signal correlation degree is limited to between 0 and 1. When the maximum eigenvalue is less than or equal to the preset acoustic correlation threshold, the signal correlation degree is 0. The larger the value, the stronger the spatial coherence of the signals and the more significant their common-source properties. This correlation degree serves as a control parameter for subsequent adaptive filter time window adjustment and reverberation tail extraction, directly affecting the aggressiveness of reverberation cutoff.

[0046] This embodiment calculates the cross-correlation matrix by plotting the covariance matrix of the noise floor superposition component and the homologous speech components. It then performs singular value decomposition on the cross-correlation matrix to extract the maximum eigenvalue, and compares this maximum eigenvalue with a preset acoustic correlation threshold. This allows for accurate determination of whether signals picked up by multiple audio acquisition terminals possess homologous attributes and quantifies signal correlation. The beneficial effect lies in using the maximum eigenvalue to reflect the energy concentration of the dominant correlation component in multiple homologous speech components. This effectively distinguishes coherent signals from the same speaker from incoherent signals from different sound sources or environmental noise, avoiding the mechanical judgment of aggressively truncating reverberation when the signal-to-noise ratio decreases. When a high correlation is determined, the system can moderately retain the reverberation tail signal to maintain a natural listening experience; conversely, it quickly truncates it to ensure speech clarity. This achieves adaptive control of the reverberation attenuation envelope in scenarios combining meetings and entertainment, resolving the contradiction between signal-to-noise ratio changes and signal correlation when multiple audio acquisition terminals are activated, and significantly improving audio quality and user experience during multi-mode switching.

[0047] In one embodiment, obtaining the reverberation tail signal and its time decay envelope from the initial spatial audio signal based on signal correlation includes steps S401 to S403, each step being as follows: S401, based on signal correlation, performs frequency domain masking on the initial spatial audio signal to obtain a transient suppression signal.

[0048] First, a short-time Fourier transform is performed on the initial spatial audio signal to obtain the time-frequency matrix. A masking threshold is dynamically set based on signal correlation; higher correlation indicates stronger dominance of the same source speech, resulting in relatively stable steady-state noise in the frequency domain, which can be more aggressively suppressed. Specifically, the steady-state noise energy background of each time-frequency unit is estimated, and the ratio of signal energy to this background is compared with a correlation-weighted masking threshold. Time-frequency units below the threshold are set to zero or attenuated. This frequency-domain masking preserves speech components with transient fluctuations and reverberation tail details while filtering out stable steady-state noise, outputting a transient suppression signal for subsequent accurate extraction of the reverberation tail.

[0049] S402 separates the non-direct sound components from the transient suppression signal and extracts the energy curve of the reverberation tail signal.

[0050] In this process, after obtaining the transient suppression signal, the system further separates the non-direct sound components to extract the reverberation tail. Based on the constructed multi-channel cross-correlation matrix, the spatial covariance matrix is ​​calculated and inverted to suppress the homologous speech components with strong directionality, mainly direct sound and early reflections, thereby stripping away the non-direct sound components that exhibit diffuse reflection and non-directional characteristics, which are the reverberation tail. For the extracted reverberation tail signal, its short-time energy is calculated. In this embodiment, specifically, using a sliding window with a frame length of 10-20 milliseconds and a frame shift of 5-10 milliseconds, the sum of squares or root mean square values ​​of the signal amplitude are calculated frame by frame to obtain a discrete sequence of energy changes over time. This sequence is then subjected to full-wave rectification and low-pass smoothing filtering to fit a smooth energy curve, i.e., the energy curve of the reverberation tail, for subsequent time-frequency conversion and energy distribution analysis.

[0051] S403, based on the energy curve, obtains the time decay envelope of the reverberation tail.

[0052] First, the energy curve is fully rectified to eliminate positive and negative amplitude fluctuations. Then, a low-pass smoothing filter is applied to filter out high-frequency micro-fluctuations and inter-frame jumps, resulting in a smooth energy decay curve. This curve reflects the macroscopic trend of reverberation tail energy decaying over time, i.e., the time decay envelope. The time decay envelope can be approximated as an exponential decay form, with its decay rate related to the room reverberation time RT60. The extracted time decay envelope provides fundamental characteristics for subsequent time-frequency domain transformation, energy division of early reflections and later diffusion, and reverberation gain adjustment.

[0053] This embodiment obtains a transient suppression signal by performing frequency domain masking on the initial spatial audio signal based on signal correlation. Then, it extracts the energy curve of the reverberation tail by separating non-direct sound components from the transient suppression signal, and finally obtains a smooth time decay envelope based on the energy curve. Its advantages lie in dynamically controlling the aggressiveness of frequency domain masking using signal correlation. This effectively suppresses steady-state noise while preserving transient speech details at high correlation, and avoids over-suppression causing reverberation distortion at low correlation. By suppressing strongly directional homologous speech components, it separates non-direct sound components with diffuse reflection characteristics, avoiding interference from direct sound and early reflections on reverberation tail extraction, thus improving the accuracy of reverberation tail detection. The final time decay envelope truly reflects the macroscopic trend of room energy decay, providing reliable feature input for subsequent energy division of early reflections and later diffusion, reverberation tail signal imbalance assessment, and adaptive gain adjustment. This achieves adaptive optimization of reverberation tail extraction in multi-audio acquisition scenarios, ensuring a dynamic balance between meeting clarity and entertainment immersion.

[0054] Furthermore, such as Figure 3As shown, this application provides a specific implementation process for obtaining the time decay envelope in a smart audio-visual device adjustment method with multiple audio acquisition terminal states. First, the picked-up signals from multiple active audio acquisition terminals and the initial spatial audio signal are acquired, and the background noise superposition component and homologous speech components are extracted from the audio acquisition terminal picked-up signals. Then, the cross-correlation matrix is ​​obtained by calculating the covariance matrix of the background noise superposition component and the homologous speech components. Next, the cross-correlation matrix is ​​processed using a singular value decomposition algorithm, and the main diagonal elements of the singular value matrix of the cross-correlation matrix are extracted to obtain the maximum eigenvalue. By comparing the maximum eigenvalue with a preset acoustic correlation threshold, if it exceeds the threshold, it is determined that the audio acquisition terminal picked-up signals have homologous attributes, and the signal correlation degree is determined accordingly. Based on the signal correlation degree, a frequency domain masking operation is performed on the initial spatial audio signal to filter out the steady-state noise components in the initial spatial audio signal, obtaining a transient suppression signal. Finally, by separating the non-direct sound components from the transient suppression signal, the reverberation tail is extracted, and the time decay envelope is obtained based on the energy change curve of the reverberation tail over time, thus providing accurate acoustic characteristic basis for subsequent adaptive truncation control of the reverberation tail.

[0055] In one embodiment, based on the reverberation tail signal and its time decay envelope, the energy distribution state between early reflection and late diffusion in the reverberation tail signal is evaluated, and the gain of the reverberation tail signal is adjusted to obtain the adjusted reverberation tail signal and its time decay envelope, including steps S501 to S503, each step as follows: S501 performs time-frequency domain conversion on the time decay envelope using short-time Fourier transform to obtain the early reflection energy distribution and late diffusion energy distribution of the reverberation tail signal.

[0056] In this process, after obtaining the time decay envelope, the system processes it together with the reverberation tail signal. First, a short-time Fourier transform is applied to the reverberation tail signal. In this embodiment, a Hamming window with a frame length of 32 milliseconds and a frame shift of 16 milliseconds is used to convert the time-domain signal into a time-frequency matrix, obtaining the energy distribution of each time-frequency unit. Simultaneously, the time decay envelope is used as an energy weighting factor. Then, an adaptive reverberation junction point is calculated based on the room volume and sound absorption characteristics. Using this junction point as a boundary, the time-frequency matrix is ​​divided into two segments along the time axis: the energy accumulation before the junction point yields the early reflection energy distribution; the energy accumulation after the junction point yields the later diffusion energy distribution. This completes the time-frequency domain energy division of the reverberation tail signal.

[0057] S502, calculate the energy ratio of the early reflection energy distribution and the later diffusion energy distribution, analyze whether the extreme values ​​in the energy ratio exceed the preset energy distribution range, and obtain the imbalance assessment result.

[0058] After obtaining the early reflection energy distribution and the later diffusion energy distribution, the system calculates the energy ratio of the two frame-by-frame or frequency band to obtain an energy ratio sequence. Extreme values, i.e., local maxima or minima, are extracted from this sequence. These extreme values ​​are compared with a preset energy distribution range. If the extreme value is below the lower limit of the range, it indicates that the later diffusion energy is relatively too large, resulting in a heavy and muddy reverberation tail signal. If the extreme value is above the upper limit of the range, it indicates that the early reflection energy is too strong, resulting in a dry and lacking enveloping sound. Once the extreme value exceeds the range, it is determined to be an energy distribution imbalance. The system generates an imbalance assessment result based on the direction and degree of deviation. The imbalance assessment result includes an imbalance indicator and an imbalance degree, providing a basis for subsequent adaptive gain adjustment.

[0059] S503, based on the imbalance assessment results, adjusts the gain of the reverberation tail signal to obtain the adjusted reverberation tail signal and its time decay envelope.

[0060] In this embodiment, the reverberation tail signal is divided into three independent sub-bands: low frequency (20-250Hz), mid frequency (250Hz-4kHz), and high frequency (4kHz-16kHz), with each sub-band configured with an adaptive filter. The error between the energy ratio of early reflection and late diffusion in the current sub-band and the target balance ratio is calculated, and the gain of that band is dynamically adjusted: reduced by 1-3dB when late diffusion is too large, and increased by 1-2dB when early reflection is insufficient. The filter coefficients are updated in real time according to the minimum mean square error criterion. Simultaneously, the reverberation energy change during priority switching at the audio acquisition end is monitored, and the cutoff position is determined at the energy plateau point. A 5-10 ms cosine square fade-out window is applied to achieve smooth cutoff of the reverberation tail signal. The reverberation tail after gain adjustment and cutoff processing is fused with the exponential decay profile to obtain the target time decay envelope that smoothly returns to zero over time.

[0061] This embodiment transforms the time decay envelope to the time-frequency domain using a short-time Fourier transform. It divides the early reflection energy distribution and the later diffusion energy distribution using the adaptive reverberation junction as the boundary, calculates the energy ratio between the two, and extracts the extreme values. These extreme values ​​are then compared with a preset energy distribution interval to obtain a quantified imbalance assessment result. Based on this imbalance assessment, adaptive band gain adjustment is applied to the reverberation tail signal to ultimately obtain the target time decay envelope. Its advantages lie in the precise separation of the early reflection and later diffusion energy components using time-frequency domain analysis. By comparing the extreme values ​​with the preset interval, the specific direction and degree of energy distribution imbalance in the reverberation tail signal can be objectively and quantitatively identified, avoiding the arbitrariness of subjective experience judgment. Furthermore, an adaptive filtering algorithm is used to implement differentiated gain adjustment for the low-frequency, mid-frequency, and high-frequency sub-bands, dynamically returning the energy ratio to the target balance range. Simultaneously, the priority switching at the audio acquisition end determines the smooth cutoff position, generating a target time decay envelope that conforms to the exponential decay law and lacks abruptness. This solution effectively solves the problem of reverberation signal distortion caused by concurrent activation and mode switching of multiple audio acquisition terminals in entertainment mode. It achieves an adaptive balance between meeting clarity and entertainment immersion, significantly improving the audio output quality and user experience of smart audio-visual devices in multi-mode mixed use scenarios.

[0062] In one embodiment, based on the imbalance assessment result, the gain of the reverberation tail signal is adjusted to obtain the adjusted reverberation tail signal and its time decay envelope, including steps S601 to S604, each step as follows: S601, obtain the energy distribution ratio of early reflection to late diffusion from the imbalance assessment results.

[0063] The global energy allocation ratio is obtained by dividing the sum of early reflection energy in the time-frequency domain by the sum of late diffusion energy. Alternatively, the allocation ratio for each sub-band can be calculated separately. This energy allocation ratio directly reflects the relative strength of the early sharpness component and the late atmospheric component in the reverberation tail signal. This ratio and its deviation from the preset target balance ratio can be directly obtained from the imbalance assessment results, serving as the error input for subsequent frequency band gain adjustment.

[0064] S602, when the energy allocation ratio is greater than the preset allocation ratio threshold, calculate the difference between the energy allocation ratio and the allocation ratio threshold, and determine the gain coefficient.

[0065] The system obtains the early reflection and late diffusion energy distribution ratio for the current sub-band and compares it with a preset distribution ratio threshold. If the energy distribution ratio is greater than the threshold, it indicates that the early reflection energy is relatively excessive or the late diffusion energy is relatively insufficient, requiring gain adjustment. The difference between the energy distribution ratio and the threshold is calculated, reflecting the degree to which the energy distribution deviates from the target balance. Based on the magnitude of the difference, the adaptive gain coefficient for that frequency band is determined according to a preset mapping relationship. Specifically, when the difference is small, the gain reduction is small; when the difference is large, the gain reduction is correspondingly larger. The gain coefficient is fed into the adaptive filter of the corresponding frequency band to adjust the band gain at the reverberation tail, thereby bringing the energy distribution ratio back to the target balance range.

[0066] S603 adjusts the gain of the reverberation tail signal based on the gain coefficient. It determines the real-time truncation position of the reverberation tail signal by detecting the trigger time of the priority signal at the audio acquisition end. The priority signal at the audio acquisition end is obtained from the reverberation tail signal after gain adjustment.

[0067] In this system, after adjusting the gain of each frequency band of the reverberation tail based on the gain coefficient, the system further processes the priority switching scenario of the audio acquisition end. The priority signal of the high-priority audio acquisition end is acquired, and its trigger time is detected in real time. Simultaneously, in this embodiment, the energy attenuation state of the reverberation tail signal after adjustment by the current low-priority audio acquisition end is monitored. Specifically, short-time energy is calculated with a frame length of 10 milliseconds and a frame shift of 5 milliseconds, and the energy derivative is calculated for adjacent frames. When three consecutive frames satisfy the condition that the absolute value of the energy derivative is less than 0.05 times the peak energy of the current frame, this position is determined as an energy plateau point and used as a candidate truncation position. Based on the time difference between the initial energy of the high-priority audio acquisition end and the masking effect, the candidate truncation position is corrected for time offset, ultimately determining the real-time truncation position. A 5-10 millisecond cosine square fade-out window is applied at this position to smoothly truncate the reverberation tail signal.

[0068] S604 performs truncation processing on the reverberation tail signal based on the real-time truncation position to obtain the target time decay envelope.

[0069] Specifically, a cosine squared fade-out window with a length of 5-10 milliseconds is applied before the cutoff position.

[0070] In the formula, The number of sample points is the window length. The reverberation tail signal is multiplied point by point with the fade-out window to achieve a smooth energy transition to zero, avoiding abrupt cutoff. After truncation, the amplitude curve from the reverberation start point to the truncation point is extracted and smoothed with a low-pass filter to obtain the energy profile that monotonically decreases with time and smoothly returns to zero at the truncation point, which is the target time decay envelope. This envelope retains the exponential decay trend that conforms to the acoustic characteristics of the target space, while also incorporating the real-time control requirements during audio acquisition switching, providing a reverberation tail that sounds natural for subsequent fusion with homologous speech components.

[0071] This embodiment obtains the energy distribution ratio between early reflections and later diffusion from the imbalance assessment results. When the distribution ratio is greater than a preset threshold, the difference is calculated to determine the gain coefficient. Based on this, the frequency band gain of the reverberation tail signal is adjusted, and the trigger time of the priority signal at the audio acquisition end is detected to determine the real-time truncation position. Finally, based on this truncation position, the reverberation tail is truncated to obtain the target time decay envelope. Its beneficial effect is that the quantization deviation between the energy distribution ratio and the preset threshold is directly mapped to an adaptive gain coefficient, giving the reverberation tail adjustment a clear direction and amplitude basis, avoiding the timbre distortion or over-balance that may be caused by blind gain adjustment. Meanwhile, by detecting the triggering moment of the priority signal at the audio acquisition end, the real-time truncation position is determined at the energy level stagnation point in combination with the masking effect, and a fade-out window is applied for smooth truncation processing. This effectively eliminates the abrupt sense of discontinuity caused by directly cutting off the reverberation tail signal of the low-priority audio acquisition end when switching audio acquisition end priorities, making the switching process conform to the masking hearing characteristics of the human ear. The final target time decay envelope not only meets the energy balance requirements of early reflection and late diffusion in entertainment mode, but also achieves seamless switching when the audio acquisition end prioritizes arbitration in conference mode. Thus, it significantly improves the real-time performance, naturalness, and listening comfort of the reverberation tail signal control in multi-mode concurrent scenarios of smart audio-visual devices.

[0072] In one embodiment, a target spatial auditory audio is generated based on homologous speech components, the adjusted reverberation tail signal, and its time decay envelope, thereby completing the state adjustment of the multi-audio acquisition terminal of the audio-visual equipment. This includes steps S701 to S703, each of which is as follows: S701 modulates the time decay envelope to obtain a decayed reverberation component with spatial sound field characteristics.

[0073] In this process, each frame of the adjusted reverberant tail signal is multiplied point-by-point with the gain value at the corresponding moment in the target time decay envelope, either in the time or frequency domain. The target time decay envelope is an energy profile curve that monotonically decreases with time and smoothly returns to zero at the cutoff point; its decay rate is determined based on the reverberation time RT60 of the target space. Through this multiplicative modulation, the energy distribution of the reverberant tail signal is forced into a decay pattern that conforms to the acoustic characteristics of the target space, thus obtaining a decayed reverberant component with spatial sound field characteristics. The energy of this component decays smoothly with an exponential law, laying the foundation for subsequent fusion with homologous speech components.

[0074] S702 calculates the time difference of arrival based on the same source speech components and the attenuated reverberation components, and obtains a phase-aligned signal containing azimuth information.

[0075] This includes determining the target sound source's location in the virtual space, including distance. Azimuth angle, used to calculate the theoretical time difference between the arrival points of the same speech components (representing direct sound) and the attenuated reverberation components (representing post-reverberation). ( The system calculates the arrival time difference by adding the interaural time difference introduced by the head-related transfer function (HAT). Then, in actual signal processing, the system performs cross-correlation analysis on the source speech component and the attenuated reverberation component within the key frequency band of 200Hz-2kHz, estimating the actual cross-correlation peak position to accurately calculate the arrival time difference. Based on this time difference, a fine-tuned time delay is applied to the attenuated reverberation component to keep it in phase aligned with the source speech component. After time difference compensation, a phase-aligned signal containing azimuth information is obtained, which is used for subsequent linear superposition with the attenuated reverberation component, avoiding comb filtering effects caused by phase inconsistency.

[0076] The S703 fuses and superimposes the phase alignment signal and the attenuation reverberation component to generate target spatial listening audio, thus completing the multi-mode state adjustment of audio-visual equipment.

[0077] First, the energy ratio of direct sound to reverberant sound is calculated based on the distance and orientation of the sound source in the target virtual space. The phase-aligned signal and the attenuated reverberation component are then gain-scaled according to this ratio. Next, the two scaled components are linearly superimposed in the time domain, and the superimposed signal is smoothed to avoid inter-frame abrupt changes. The final output is target-space audio with clear direct speech and natural spatial reverberation. This audio maintains high speech intelligibility in conference mode and presents a rich sense of presence in entertainment mode, thus achieving adaptive adjustment across multiple modes of the smart audio-visual device.

[0078] This embodiment obtains a decaying reverberation component with spatial sound field characteristics by amplitude modulation of the target time decay envelope. Then, based on the same source speech component and the decaying reverberation component, the arrival time difference is calculated and phase alignment is performed to obtain a phase-aligned signal containing directional information. Finally, the two components are scaled and fused according to their energy ratios to generate the target spatial audio. Its beneficial effect is that amplitude modulation ensures that the energy decay at the reverberation tail precisely matches the exponential decay law of the target space, thereby reproducing a realistic room acoustic atmosphere. Through arrival time difference calculation and phase alignment, the comb filtering effect caused by phase inconsistency between direct sound and reverberation sound during superposition is effectively avoided, ensuring the naturalness and clarity of the sound. Simultaneously, the energy ratio of direct sound and reverberation sound is adaptively adjusted according to the distance and direction of the virtual sound source, giving the final output audio a clear sense of spatial direction and presence. This method ensures high speech intelligibility in conference mode and provides a rich reverberation envelope in entertainment mode, thus enabling adaptive adjustment of the audio state of smart audio-visual devices during multi-mode switching, significantly improving the user's immersive listening experience.

[0079] Based on the same inventive concept, this application also provides a smart audio-visual device adjustment system for realizing the aforementioned multi-audio acquisition terminal states. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations in one or more embodiments of the smart audio-visual device adjustment device for multi-audio acquisition terminal states provided below can be found in the limitations of the smart audio-visual device adjustment method for multi-audio acquisition terminal states described above, and will not be repeated here.

[0080] In one exemplary embodiment, such as Figure 4 As shown, a smart audio-visual device adjustment system with multiple audio acquisition terminal states is provided, including: The acquisition module 801 is used to acquire the initial spatial audio signal and initial acoustic impulse response of multiple audio acquisition terminals in the scene under active state, and to perform component analysis processing on the initial spatial audio signal and initial acoustic impulse response to separate the background noise superposition component generated by the superposition of multiple sounds and the homologous speech component of the same sound. Evaluation module 802 is used to evaluate the signal correlation of audio collected by multiple audio acquisition terminals based on the noise floor superposition component and the homologous speech component, and to obtain the reverberation tail signal and its time decay envelope from the initial spatial audio signal according to the signal correlation. Gain module 803 is used to evaluate the energy distribution state between early reflection and late diffusion in the reverberation tail signal based on the reverberation tail signal and its time decay envelope, and to adjust the gain of the reverberation tail signal to obtain the adjusted reverberation tail signal and its time decay envelope. The adjustment module 804 is used to generate target spatial audio based on homologous speech components, the adjusted reverberation tail signal and its time decay envelope, and to complete the state adjustment of the multi-audio acquisition end of the audio-visual equipment.

[0081] In one embodiment, a computer program product is provided, including a computer program or instructions, which, when executed by a device, implement the steps of a smart audio-visual device adjustment method for multiple audio acquisition terminal states as described above.

[0082] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0083] This technical solution deeply integrates multiple physical pickup nodes, independent component analysis algorithms, and adaptive filtering technology, and correlates them with room acoustic impulse response and reverberation energy distribution models to construct a reverberation tail signal adaptive truncation and control system specifically designed for mixed-use scenarios of meetings and entertainment. Its core advantage lies in overcoming the limitations of existing technologies that rely solely on fixed acoustic thresholds or signal-to-noise ratios for mechanical truncation. It achieves dynamic and intelligent evaluation and control of the reverberation tail attenuation envelope in complex multi-audio acquisition environments. The system can actively adjust the reverberation truncation position and frequency band gain based on signal correlation and the early reflection / late diffusion energy ratio, observing in real time the impact of different adjustments on speech clarity and spatial naturalness. This provides adaptive audio processing for multi-mode audio-visual equipment, achieving a leap from passive truncation to active balancing. Meanwhile, this scheme fully leverages the advantages of independent component analysis in blind source separation, short-time Fourier transform in time-frequency domain energy analysis, and adaptive filtering in frequency band gain adjustment, solving the problems of abrupt reverberation tail signals, muffled speech, and lack of spatial realism in traditional schemes when priority switching occurs at the audio acquisition end.

[0084] For the device embodiments, since they basically correspond to the method embodiments, the relevant details can be found in the descriptions of the method embodiments. The device embodiments described above are merely illustrative; components described as separate parts may or may not be physically separate, and components shown as units may or may not be physical units, meaning they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0085] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention in detail. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for adjusting the status of a smart audio-visual device with multiple audio acquisition terminals, characterized in that, include: The initial spatial audio signal and initial acoustic impulse response of multiple audio acquisition terminals in the scene are acquired under active state. The initial spatial audio signal and initial acoustic impulse response are processed by component analysis to separate the background noise superposition component generated by the superposition of multiple sound channels and the homologous speech component of the same sound. Based on the noise floor superposition component and the homologous speech component, the signal correlation of audio collected by multiple audio acquisition terminals is evaluated, and the reverberation tail signal and its time decay envelope are obtained from the initial spatial audio signal according to the signal correlation. Based on the reverberant tail signal and its time decay envelope, the energy distribution state between early reflection and late diffusion in the reverberant tail signal is evaluated, and the gain of the reverberant tail signal is adjusted to obtain the adjusted reverberant tail signal and its time decay envelope. Based on the homologous speech components, the adjusted reverberation tail signal and its time decay envelope, the target spatial auditory audio is generated to complete the state adjustment of the multi-audio acquisition end of the audio-visual equipment.

2. The method for adjusting the status of a smart audio-visual device with multiple audio acquisition terminals as described in claim 1, characterized in that, The system acquires the initial spatial audio signals and initial acoustic impulse responses of multiple audio acquisition terminals in the scene under active states, and performs component analysis on the initial spatial audio signals and initial acoustic impulse responses to separate the background noise superposition component generated by the superposition of multiple sounds, as well as the homologous speech components of the same sound, including: Based on the initial spatial audio signal, the acoustic impulse response of the initial scene is identified; Frequency domain audio data is obtained by frequency domain mapping of the initial spatial audio signal and the initial scene acoustic impulse response. Component analysis is performed on the frequency domain audio data to separate the noise superposition component generated by the superposition of multiple sounds and the homologous speech component of the same sound.

3. The method for adjusting the status of a smart audio-visual device with multiple audio acquisition terminals as described in claim 1, characterized in that, Based on the noise floor superposition component and homologous speech components, the signal correlation of audio collected from multiple audio acquisition terminals is evaluated, including: Calculate the covariance matrix of the noise floor superposition component and the homologous speech component to obtain the cross-correlation matrix; The cross-correlation matrix is ​​subjected to singular value decomposition to obtain the largest eigenvalue; The audio acquisition terminal that picks up signals whose maximum eigenvalue exceeds the preset acoustic correlation threshold determines that the signals have the same origin attribute and determines the signal correlation degree.

4. The method for adjusting the status of a smart audio-visual device with multiple audio acquisition terminals as described in claim 3, characterized in that, The reverberant tail signal and its time decay envelope are obtained from the initial spatial audio signal based on signal correlation, including: Based on signal correlation, the initial spatial audio signal is subjected to frequency domain masking to obtain a transient suppression signal; The non-direct sound components are separated from the transient suppression signal, and the energy curve of the reverberation tail signal is extracted. Based on the energy curve, the time decay envelope of the reverberation tail signal is obtained.

5. The method for adjusting the status of a smart audio-visual device with multiple audio acquisition terminals as described in claim 1, characterized in that, Based on the reverberation tail signal and its time decay envelope, the energy distribution state between early reflection and late diffusion in the reverberation tail signal is evaluated, and the gain of the reverberation tail signal is adjusted to obtain the adjusted reverberation tail signal and its time decay envelope, including: By performing time-frequency domain conversion on the time decay envelope using short-time Fourier transform, the early reflection energy distribution and the later diffusion energy distribution of the reverberation tail signal are obtained. Calculate the energy ratio between the early reflection energy distribution and the later diffusion energy distribution, analyze whether the extreme values ​​in the energy ratio exceed the preset energy distribution range, and obtain the imbalance assessment results; Based on the imbalance assessment results, the gain of the reverberation tail signal is adjusted to obtain the adjusted reverberation tail signal and its time decay envelope.

6. The method for adjusting the status of a smart audio-visual device with multiple audio acquisition terminals as described in claim 5, characterized in that, Based on the imbalance assessment results, the gain of the reverberation tail signal is adjusted to obtain the adjusted reverberation tail signal and its time decay envelope, including: Obtain the energy distribution ratio between early reflection and later diffusion from the imbalance assessment results; When the energy allocation ratio is greater than the preset allocation ratio threshold, the difference between the energy allocation ratio and the allocation ratio threshold is calculated to determine the gain coefficient; Gain adjustment of the reverberation tail signal is performed based on the gain coefficient. The real-time truncation position of the reverberation tail signal is determined by detecting the trigger time of the priority signal at the audio acquisition end. The priority signal at the audio acquisition end is obtained from the reverberation tail signal after gain adjustment. The reverberation tail signal is truncated based on the real-time truncation position to obtain the target time decay envelope.

7. The method for adjusting the status of a smart audio-visual device with multiple audio acquisition terminals as described in claim 1, characterized in that, Based on the homologous speech components, the adjusted reverberation tail signal, and its time decay envelope, target spatial auditory audio is generated to complete the status adjustment of the multi-audio acquisition end of the audio-visual equipment, including: Amplitude modulation of the time decay envelope yields a decayed reverberation component with spatial sound field characteristics. Based on the homologous speech components and the attenuated reverberation components, the time difference of arrival is calculated to obtain a phase-aligned signal containing azimuth information. The phase-aligned signal and the attenuation reverberation component are fused and superimposed to generate the target space listening audio, thus completing the multi-mode state adjustment of the audio-visual equipment.

8. A smart audio-visual equipment adjustment system with multiple audio acquisition terminal states, characterized in that the system... include: The acquisition module is used to acquire the initial spatial audio signal and initial acoustic impulse response of multiple audio acquisition terminals in the scene under active state, and to perform component analysis processing on the initial spatial audio signal and initial acoustic impulse response to separate the background noise superposition component generated by the superposition of multiple sounds and the homologous speech component of the same sound. The evaluation module is used to evaluate the signal correlation of audio collected by multiple audio acquisition terminals based on the noise floor superposition component and the homogeneous speech component, and to obtain the reverberation tail signal and its time decay envelope from the initial spatial audio signal according to the signal correlation. The gain module is used to evaluate the energy distribution state between early reflection and late diffusion in the reverberation tail signal based on the reverberation tail signal and its time decay envelope, and to adjust the gain of the reverberation tail signal to obtain the adjusted reverberation tail signal and its time decay envelope. The adjustment module is used to generate target spatial audio based on homologous speech components, the adjusted reverberation tail signal and its time decay envelope, and to complete the status adjustment of the multi-audio acquisition end of the audio-visual equipment.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions, which, when executed by a communication device, implement the intelligent audio-visual device adjustment method for the state of multiple audio acquisition terminals as described in any one of claims 1-7.

10. A computer program product, comprising a computer program or instructions, characterized in that, A smart audio-visual device adjustment method that implements the state of multiple audio acquisition terminals as described in any one of claims 1-7 when a computer program or instruction is executed by a communication device.

Images