Intelligent sensor-based microphone management control system and method

Abstract

The application discloses a microphone management control system and method based on intelligent sensing, and belongs to the technical field of microphone control, and comprises the following steps: acquiring multi-modal sensing data and determining a target sound pickup channel; acquiring node self-sensing data, judging whether there is a physical shielding event between each microphone node, updating a beamforming weight vector according to a judgment result, and generating real-time dynamic spatial topology data; constructing a causal diagram for reasoning, generating multiple candidate causal microphone control decisions; calculating the collection quality scores of the candidate causal microphone control decisions, selecting the optimal control strategy based on the collection quality scores, and executing the optimal control strategy. The application constructs a parameterized causal diagram covering the whole link of a sound source, propagation and collection, and realizes accurate quantitative positioning of sound pickup abnormal causes in combination with counterfactual reasoning, thereby providing clear causal numerical support for the generation of a control strategy and effectively avoiding signal distortion problems caused by blind parameter adjustment.

Description

Technical Field

[0001] This invention belongs to the field of microphone control technology, specifically relating to a microphone management and control system and method based on intelligent sensing. Background Technology

[0002] In scenarios such as conference systems, smart classrooms, and intelligent voice interaction, microphone arrays are the core devices for achieving high-quality sound pickup over long distances; existing microphone array control solutions mainly suffer from the following technical shortcomings:

[0003] Firstly, existing beamforming algorithms assume that all nodes in the array are in an ideal state of normal unobstructed operation. They lack real-time perception of the physical obstruction status of array nodes and dynamic compensation mechanism for beam weights. When people move around indoors or objects are moved, causing some array elements to be obstructed, it will cause problems such as beam pointing offset and array gain attenuation, resulting in a significant decrease in sound pickup quality.

[0004] Secondly, the diagnosis of abnormal sound pickup signals often relies on simple signal amplitude threshold judgment. It can only identify whether an abnormality has occurred, but cannot quantitatively distinguish different abnormal causes such as abnormal equipment status, environmental noise interference, and changes in the state of the sound source. It lacks structured causal reasoning ability, which makes subsequent control strategies such as gain adjustment and beam adjustment lack precise basis and is prone to blind adjustment.

[0005] Third, the effectiveness verification and parameter optimization of existing beamforming schemes rely heavily on on-site testing and debugging of physical arrays, making it impossible to quickly quantify and evaluate multiple candidate adjustment schemes. Furthermore, there is a lack of a multi-dimensional comprehensive quality evaluation system that takes into account signal cleanliness, speech clarity, and spatial directivity, resulting in low scheme iteration efficiency and difficulty in efficiently adapting to dynamically changing indoor sound field environments.

[0006] Therefore, there is an urgent need for a microphone management and control system and method based on intelligent sensing to solve the above problems. Summary of the Invention

[0007] The purpose of this invention is to provide a microphone management and control system and method based on intelligent sensing, so as to solve the problems faced in the above-mentioned background art.

[0008] The objective of this invention can be achieved through the following technical solutions:

[0009] A microphone management and control method based on intelligent sensing, the control method comprising:

[0010] S1. Acquire multimodal sensing data, predict the speaking time and urgency of each speaker based on the multimodal sensing data; pre-activate the microphone node at the corresponding position within a preset time window before the speaking time, and arbitrate the speaking intentions of multiple people based on the speaking urgency to determine the target sound pickup channel.

[0011] S2. Obtain the node self-sensing data of each microphone node in the target pickup channel, determine whether there is a physical occlusion event between each microphone node based on the node self-sensing data, and update the beamforming weight vector according to the judgment result to generate real-time dynamic spatial topology data.

[0012] S3. Obtain the audio signal amplitude sequence of the target pickup channel, construct a causal graph, and combine it with real-time dynamic spatial topology data to perform inference and generate multiple candidate causal microphone control decisions.

[0013] S4. Construct a digital twin acoustic field model based on real-time dynamic spatial topology data, input each candidate causal microphone control decision into the digital twin acoustic field model for forward extrapolation, obtain the acquisition quality score corresponding to each candidate causal microphone control decision, select the optimal control strategy based on the acquisition quality score and execute it.

[0014] Furthermore, methods for predicting the speaking time and urgency of each speaker include:

[0015] Multimodal sensing data includes sound pressure level time series data, facial orientation angle time series data, mouth opening and closing time series data, torso tilt angle time series data, and indoor temperature and humidity time series data;

[0016] Time alignment and normalization were performed on the sound pressure level time series data, facial orientation angle time series data, mouth opening and closing time series data, trunk tilt angle time series data, and indoor temperature and humidity time series data respectively to obtain the aligned multimodal feature sequence.

[0017] Multimodal feature sequences are input into a temporal fusion network, which includes a long short-term memory (LSM) encoding layer, a cross-modal attention interaction layer, and a fully connected prediction layer. The LSM encoding layer performs temporal encoding on each modality feature sequence to obtain the latent state vector of each modality. The cross-modal attention interaction layer uses the latent state vector corresponding to the mouth opening and closing temporal data as the query vector, and uses the latent state vectors of other modalities as the key and value vectors to calculate the cross-modal attention weights and perform weighted fusion to obtain the fused latent state vector. The fully connected prediction layer performs joint regression based on the fused latent state vector and the speaker's historical speaking weight vector to output the predicted value of the speaking time and the speaking urgency score.

[0018] Furthermore, the method for determining the target pickup channel is as follows:

[0019] Obtain the predicted speaking time, speaking urgency score, and historical speaking weight vector for each speaker; map the predicted speaking time of each speaker to a unified timeline, and pre-construct the speaking time window for each speaker;

[0020] Iterate through the speaking time windows of each speaker. When the speaking time windows of any two speakers intersect, group the speakers corresponding to the intersecting speaking time windows into the same set of conflicting speakers. When the speaking time window of the same speaker intersects with the speaking time windows of multiple other speakers, merge the same speaker and all other speakers with intersecting windows into the same set of conflicting speakers, thus obtaining the set of conflicting speakers for each speaker.

[0021] For each speaker in the conflict speaker set, a comprehensive arbitration score is calculated based on the urgency score of the speech and the weight vector of the historical speech. The speaker with the highest comprehensive arbitration score is selected as the target sound pickup object, and the microphone node corresponding to the target sound pickup object is determined as the target sound pickup channel.

[0022] Furthermore, the method for determining whether there is a physical occlusion event between microphone segments is as follows:

[0023] The node self-sensing data includes ambient background sound data continuously collected by each microphone node and ultrasonic ranging pulse data transmitted between nodes.

[0024] The ambient background sound data is processed by framing and windowing, and the Mel frequency cepstral coefficient vector of each frame is extracted. The Mel frequency cepstral coefficient vectors of multiple consecutive frames are concatenated into an acoustic fingerprint matrix. The acoustic fingerprint matrix is ​​then encoded by local sensitive hashing to generate an acoustic fingerprint hash.

[0025] Combine any two different microphone nodes to construct a microphone node pair, detect the pulse arrival time of the ultrasonic ranging pulse data, calculate the sound wave propagation delay between each microphone node pair, acquire the temperature data in the ambient background sound data, calculate the current sound speed based on the temperature data, and calculate the geometric distance between the microphone node pairs based on the sound wave propagation delay and the current sound speed.

[0026] The geometric distance between microphone node pairs is compared with the corresponding preset nominal installation distance to obtain the acoustic propagation delay matrix between nodes;

[0027] Calculate the Hamming distance of the acoustic fingerprint hash between microphone node pairs. When the Hamming distance exceeds a preset fingerprint threshold, it is determined that the acoustic fingerprint is abnormal. Extract the geometric distance corresponding to the microphone node pairs in the acoustic propagation delay matrix between nodes. When the difference between the geometric distance and the nominal installation distance exceeds a preset distance threshold, it is determined that the propagation delay is abnormal.

[0028] When both acoustic fingerprint anomaly and propagation delay anomaly are present, it is determined that there is a physical obstruction event between the microphone node pairs.

[0029] Furthermore, the method for generating real-time dynamic spatial topology data is as follows:

[0030] Traverse the physical occlusion determination results. When a physical occlusion event is determined to exist, mark the geometric distance of the corresponding microphone node pair as invalid distance and replace the invalid distance with the corresponding nominal installation distance to obtain the corrected geometric distance. When no physical occlusion event is determined to exist, retain the corresponding geometric distance to obtain the corrected geometric distance.

[0031] The relative coordinates of each microphone node with respect to a preset reference node are calculated based on the corrected geometric distance. The preset reference node is the microphone node with the highest signal quality score among all microphone nodes. The signal quality score is calculated from the signal-to-noise ratio of the ambient background sound data.

[0032] A spatial topology graph is constructed based on relative coordinates. The nodes of the spatial topology graph are each microphone node, the edges are each pair of microphone nodes, and the edge weights are the corrected geometric distances between each pair of microphone nodes.

[0033] Obtain the current azimuth angle of the sound source, and calculate the initial beamforming weight vector for each microphone node based on the current azimuth angle of the sound source and the spatial topology map.

[0034] When a physical occlusion event is found in the physical occlusion determination result, the initial beamforming weight vector of the occluded microphone node corresponding to the physical occlusion event is set to zero, and the corresponding weight increment is allocated to the adjacent microphone nodes that are adjacent to the occluded microphone node and are not occluded according to a preset ratio to obtain the updated beamforming weight vector.

[0035] When no physical occlusion event is found in the physical occlusion determination result, the initial beamforming weight vector is used as the updated beamforming weight vector; the updated beamforming weight vector is written into the node attribute field of each corresponding microphone node in the spatial topology graph to obtain real-time dynamic spatial topology data.

[0036] Furthermore, the method for generating multiple candidate causal microphone control decisions is as follows:

[0037] The time-domain feature vector and frequency-domain feature vector are extracted based on the amplitude sequence of the audio signal, and the time-domain feature vector and frequency-domain feature vector are concatenated to form an audio feature vector; the device status index of each microphone node is calculated based on the signal quality score of each microphone node in the real-time dynamic spatial topology data; and the air vibration propagation attenuation coefficient of each microphone node pair is calculated based on the corrected geometric distance of each microphone node pair in the real-time dynamic spatial topology data.

[0038] Construct a causal graph and a set of directed edges. Based on the set of directed edges, use a structural causal model to parameterize the causal graph to obtain a parameterized causal graph.

[0039] Acquire the observation sequence of the microphone signal node. When the decrease in the current frame amplitude value of the observation sequence relative to the previous frame amplitude value exceeds a preset drop threshold, trigger counterfactual reasoning.

[0040] An intervention path set is constructed, and a do-intervention operation is performed on each intervention path in the intervention path set to calculate the counterfactual prediction value under each intervention path. The absolute difference between the counterfactual prediction value corresponding to each intervention path and the actual observation value of the microphone signal node in the current frame is taken as the causal effect strength of the corresponding intervention path. The intervention state value of the fixed node corresponding to the intervention path with the smallest causal effect strength is taken as the candidate causal microphone control decision. Multiple candidate causal microphone control decisions are generated by traversing all intervention paths.

[0041] Furthermore, the node set of the causal graph includes: speaker intention node, vocal cord vibration node, air vibration node, microphone signal node, ambient noise node, and device status node; wherein the observed value of the speaker intention node is the amplitude mutation rate in the audio feature vector, the observed value of the vocal cord vibration node is the main frequency component in the audio feature vector, the observed value of the air vibration node is the air vibration propagation attenuation coefficient, the observed value of the microphone signal node is the audio signal amplitude sequence, the observed value of the ambient noise node is the spectral entropy in the audio feature vector, and the observed value of the device status node is the device status index.

[0042] The set of directed edges includes: a first directed edge pointing from the speaker intention node to the vocal cord vibration node; a second directed edge pointing from the vocal cord vibration node to the air vibration node; a third directed edge pointing from the air vibration node to the microphone signal node; a fourth directed edge pointing from the ambient noise node to the microphone signal node; a fifth directed edge pointing from the device status node to the microphone signal node; and a sixth directed edge pointing from the speaker intention node to the microphone signal node.

[0043] Furthermore, the method for selecting and executing the optimal control strategy is as follows:

[0044] Each candidate causal microphone control decision includes a corresponding candidate gain coefficient and candidate beam pointing angle;

[0045] A digital twin acoustic field model is constructed based on the relative coordinates of each microphone node and the corrected geometric distance between each microphone node pair in real-time dynamic spatial topology data. The digital twin acoustic field model uses relative coordinates as spatial boundary conditions and corrected geometric distance as medium propagation parameters.

[0046] Obtain the current sound source location coordinates, input the current sound source location coordinates, candidate gain coefficients and candidate beam pointing angles in each candidate causal microphone control decision into the digital twin acoustic field model, perform forward extrapolation with a preset time step, and obtain the modulated sound pressure received value.

[0047] The propagation delay, reflection loss, and scattering loss of the sound pressure wavefront represented by the air vibration node are calculated from the current sound source location coordinates to each microphone node. The propagation delay, reflection loss, and scattering loss are superimposed on the modulated sound pressure received value to obtain the predicted acquisition signal sequence of each microphone node.

[0048] The acquisition quality score of each candidate causal microphone control decision is calculated based on the predicted acquisition signal sequence. The candidate causal microphone control decision with the highest acquisition quality score is selected as the optimal control strategy. The corresponding candidate gain coefficient and candidate beam pointing angle in the optimal control strategy are written into the node attribute field of each corresponding microphone node in the real-time dynamic spatial topology data. Within a preset time window after the speaker is detected to be preparing to speak, the physical pointing of each microphone node is adjusted based on the corresponding candidate beam pointing angle, and the signal amplification factor of each microphone node is adjusted based on the corresponding candidate gain coefficient.

[0049] A microphone management and control system based on intelligent sensing, wherein the system is used to implement the microphone management and control method based on intelligent sensing described above.

[0050] The beneficial effects of this invention are:

[0051] This invention achieves automatic compensation for beam performance in occlusion scenarios through a multi-dimensional joint array element occlusion determination mechanism and a dynamic weight redistribution strategy. This effectively corrects beam pointing distortion and gain loss caused by occlusion, improving the stability and robustness of array sound pickup in dynamic scenarios. Simultaneously, by constructing a parameterized causal graph covering the entire sound source, propagation, and acquisition chain, and combining counterfactual reasoning, it achieves precise quantitative localization of the causes of sound pickup anomalies, providing clear causal numerical support for the generation of control strategies and effectively avoiding signal distortion problems caused by blind parameter tuning. Furthermore, relying on the path-based extrapolation capabilities of the digital twin acoustic field, a multi-dimensional positive comprehensive acquisition quality evaluation system is constructed. This system can quickly complete the quantitative evaluation of the effects of multiple candidate schemes without physical debugging, significantly improving the optimization efficiency of beam schemes and the adaptability to dynamic sound field environments. Overall, it improves the sound pickup quality and system response efficiency of microphone arrays in complex indoor scenarios.

[0052] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] In one embodiment, a microphone management and control method based on intelligent sensing is disclosed, such as... Figure 1 As shown, the method mainly includes:

[0057] S1. Acquire multimodal sensing data, predict the speaking time and urgency of each speaker based on the multimodal sensing data; pre-activate the microphone node at the corresponding position within a preset time window before the speaking time, and arbitrate the speaking intentions of multiple people based on the speaking urgency to determine the target sound pickup channel.

[0058] The methods for predicting the speaking time and urgency of each speaker include: multimodal sensing data including sound pressure level time series data, facial orientation angle time series data, mouth opening and closing time series data, torso tilt angle time series data, and indoor temperature and humidity time series data.

[0059] Time alignment and normalization were performed on the sound pressure level time series data, facial orientation angle time series data, mouth opening and closing time series data, trunk tilt angle time series data, and indoor temperature and humidity time series data respectively to obtain the aligned multimodal feature sequence.

[0060] Multimodal feature sequences are input into a temporal fusion network, which includes a long short-term memory (LSM) encoding layer, a cross-modal attention interaction layer, and a fully connected prediction layer. The LSM encoding layer performs temporal encoding on each modality feature sequence to obtain the latent state vector of each modality. The cross-modal attention interaction layer uses the latent state vector corresponding to the mouth opening and closing temporal data as the query vector, and uses the latent state vectors of other modalities as the key and value vectors to calculate the cross-modal attention weights and perform weighted fusion to obtain the fused latent state vector. The fully connected prediction layer performs joint regression based on the fused latent state vector and the speaker's historical speaking weight vector to output the predicted value of the speaking time and the speaking urgency score.

[0061] The method for determining the target audio pickup channel is as follows: obtain the predicted speaking time, speaking urgency score, and historical speaking weight vector of each speaker; map the predicted speaking time of each speaker to a unified time axis, and pre-construct the speaking time window of each speaker;

[0062] Iterate through the speaking time windows of each speaker. When the speaking time windows of any two speakers intersect, group the speakers corresponding to the intersecting speaking time windows into the same set of conflicting speakers. When the speaking time window of the same speaker intersects with the speaking time windows of multiple other speakers, merge the same speaker and all other speakers with intersecting windows into the same set of conflicting speakers, thus obtaining the set of conflicting speakers for each speaker.

[0063] For each speaker in the conflict speaker set, a comprehensive arbitration score is calculated based on the urgency score of the speech and the weight vector of the historical speech. The speaker with the highest comprehensive arbitration score is selected as the target sound pickup object, and the microphone node corresponding to the target sound pickup object is determined as the target sound pickup channel.

[0064] Specifically, the multimodal sensing data is collected collaboratively by multiple types of sensors deployed in conference rooms or lecture venues, including:

[0065] Sound pressure level time series data: acquired by acoustic sensors (such as MEMS microphone arrays) at a sampling frequency of 16kHz, characterizing the sequence of changes in the instantaneous sound pressure amplitude of sound signals in the environment over time;

[0066] Facial orientation angle temporal data: acquired by visual sensors (such as wide-angle cameras or depth cameras) at a frame rate of 30Hz, representing the time sequence of the deflection angle of the speaker's facial normal vector relative to the conference room reference coordinate system; facial orientation angle is obtained by extracting facial key points through face detection algorithms (such as MTCNN or RetinaFace), and then calculating the angle between the facial normal vector and the x-axis of the reference coordinate system.

[0067] Mouth opening and closing time series data: synchronously acquired by a visual sensor at a frame rate of 30Hz, representing the change sequence of a normalized measure of the vertical distance between the upper and lower lips and the lip width ratio over time; mouth opening and closing is obtained by dividing the Euclidean distance between the midpoint of the upper lip and the midpoint of the lower lip in the facial key points by the distance between the corners of the mouth.

[0068] Trunk tilt time series data: acquired by inertial sensors (such as an IMU module worn on the speaker's chest or a vision-based human posture estimation algorithm) at a sampling frequency of 30Hz, representing the time sequence of the tilt angle of the speaker's trunk axis relative to the vertical direction.

[0069] Indoor temperature and humidity time series data: acquired by environmental sensors (such as DHT11 or SHT30 temperature and humidity sensors) at a sampling frequency of 1Hz, characterizing the change sequence of ambient temperature and relative humidity in the conference room over time.

[0070] Due to the differences in sampling frequencies among the various modal sensor data, time alignment is required. After alignment, the data for each modality are normalized. The sound pressure level time series data are standardized using Z-score (subtracting the mean and dividing by the standard deviation), the facial orientation angle time series data are mapped to the [-1,1] interval using trigonometric functions, the mouth opening and closing angle time series data are normalized to the [0,1] interval using Min-Max, the torso tilt angle time series data are processed using sine mapping, and the indoor temperature and humidity time series data are normalized according to their respective historical statistical ranges. The normalized modal data are then concatenated into an aligned multimodal feature sequence.

[0071] Multimodal feature sequences are input into a temporal fusion network, which includes a Long Short-Term Memory (LSTM) encoding layer, a cross-modal attention interaction layer, and a fully connected prediction layer. The LSTM encoding layer comprises five parallel LSTM sub-networks, each receiving modal features from the aligned multimodal feature sequence as input. Each LSTM sub-network contains 128 hidden units, encoding the temporal features through gating mechanisms (input gate, forget gate, output gate) and outputting the latent state vectors for each modality. The cross-modal attention interaction layer uses the latent state vector corresponding to the lip opening / closing time-series data as the query vector, the key matrix formed by concatenating the latent state vectors of the other modalities as the key vector, and the value matrix corresponding to the key matrix as the value vector. It calculates the cross-modal attention weights and performs weighted fusion to obtain the fused latent state vector. The attention weights for each modality are predetermined based on experience and will not be described in detail here. The fully connected prediction layer contains two parallel fully connected sub-networks: a speech timing prediction network and a speech urgency prediction network. The speech timing prediction network contains two fully connected layers (64 and 32 hidden units respectively), and the output layer is a single neuron with a ReLU activation function, outputting the speech timing prediction value. The speech urgency prediction network contains two fully connected layers (64 and 32 hidden units respectively), and the output layer is a single neuron with a sigmoid activation function, outputting a speech urgency score (range [0,1]), the larger the value, the higher the speech urgency. The input of the fully connected prediction layer is a concatenated vector of the hidden state vector and the speaker's historical speech weight vector. The speaker's historical speech weight vector... The value is calculated from the number of times the speaker spoke (n_speak), the total speaking time (t_speak), and the number of times the speaker was interrupted (n_interrupt) within a preset historical time window (e.g., the past 30 minutes), as shown in the following expression:

[0072]

[0073] Where α, β, and γ are weighting coefficients preset based on experience, satisfying α+β+γ=1. The above-mentioned number of speeches n_speak, total speaking time t_speak, and number of interruptions n_interrupt are all values ​​mapped to [0,1] after normalization.

[0074] After obtaining the predicted speaking times, speaking urgency scores, and historical speaking weight vectors for all speakers, the target audio pickup channel is determined through conflict detection and iterative arbitration. The specific process is as follows:

[0075] The predicted speaking times of each speaker are mapped to a unified time axis. A speaking time window is constructed with the predicted speaking times as the center point and the preset speaking duration as the interval length. The left boundary of the window is the predicted speaking time minus half of the preset speaking duration, and the right boundary is the predicted speaking time plus half of the preset speaking duration. In this embodiment, the preset speaking duration is set to 10 seconds.

[0076] Iterate through the speaking time windows of all speakers. When the speaking time windows of any two speakers intersect, they are grouped into the same conflicting speaker set. If the time window of the same speaker intersects with the windows of multiple other speakers, then the speaker is merged with all speakers with intersecting time windows into the same conflicting set, resulting in several non-overlapping conflicting speaker sets.

[0077] For each speaker in the conflicting speaker set, score them based on the urgency of their speech. Weight vector of historical speech The formula for calculating the overall arbitration score is as follows:

[0078]

[0079] in, This is a dynamic weighting coefficient, whose value increases with the number of times the speaker with the highest number of interruptions in the conflicting speaker set is interrupted. The initial value is 0.6, and it increases by 1 for each additional interruption. Increase by 0.05, with a maximum of 0.9; to ensure that speakers who are frequently interrupted receive higher priority.

[0080] The speaker with the highest comprehensive arbitration score is selected as the target sound pickup object, and its corresponding microphone node is the target sound pickup channel. For speakers who are not selected in the set, their predicted speaking time value is shifted backward by a preset delay increment, the time window is reconstructed, and the conflict detection step is returned until there are no conflicting speakers or the maximum number of iterations is reached. In this embodiment, the preset delay increment is set to 3 seconds, and the maximum number of iterations is set to 5.

[0081] After determining the target pickup channel, the microphone node at the corresponding position is pre-activated within a preset time window before the predicted speaking time, completing the feedforward preparation of the pickup channel. In this embodiment, the preset advance time window is 2 seconds.

[0082] This approach integrates multimodal sensor data such as sound pressure level, facial orientation, lip opening and closing, torso tilt angle, and temperature and humidity to build a cross-modal attention temporal fusion network with lip shape features as the query vector. This network can accurately predict the speaking time and urgency of each speaker in advance, enabling pre-activation of microphone nodes. This effectively solves the problems of missing words at the beginning of a speech and delayed sound pickup response in traditional audio threshold triggering schemes. At the same time, a dynamic weighted conflict arbitration mechanism that takes into account the urgency of speaking and the rights of past speakers is constructed. It avoids frequent interruptions to speakers through iterative delayed arbitration and dynamically tilts priority towards speakers who have been interrupted more often. This not only improves the smoothness of interaction and speaking order in multi-person speaking scenarios but also ensures the fairness of meeting discussions and significantly reduces the probability of accidental triggering of speeches.

[0083] S2. Obtain the node self-sensing data of each microphone node in the target pickup channel, determine whether there is a physical occlusion event between each microphone node based on the node self-sensing data, and update the beamforming weight vector according to the judgment result to generate real-time dynamic spatial topology data.

[0084] The method for determining whether there is a physical obstruction event between each microphone node is as follows: the node self-sensing data includes the ambient background sound data continuously collected by each microphone node and the ultrasonic ranging pulse data transmitted between nodes.

[0085] The ambient background sound data is processed by framing and windowing, and the Mel frequency cepstral coefficient vector of each frame is extracted. The Mel frequency cepstral coefficient vectors of multiple consecutive frames are concatenated into an acoustic fingerprint matrix. The acoustic fingerprint matrix is ​​then encoded by local sensitive hashing to generate an acoustic fingerprint hash.

[0086] Combine any two different microphone nodes to construct a microphone node pair, detect the pulse arrival time of the ultrasonic ranging pulse data, calculate the sound wave propagation delay between each microphone node pair, acquire the temperature data in the ambient background sound data, calculate the current sound speed based on the temperature data, and calculate the geometric distance between the microphone node pairs based on the sound wave propagation delay and the current sound speed.

[0087] The geometric distance between microphone node pairs is compared with the corresponding preset nominal installation distance to obtain the acoustic propagation delay matrix between nodes;

[0088] Calculate the Hamming distance of the acoustic fingerprint hash between microphone node pairs. When the Hamming distance exceeds a preset fingerprint threshold, it is determined that the acoustic fingerprint is abnormal. Extract the geometric distance corresponding to the microphone node pairs in the acoustic propagation delay matrix between nodes. When the difference between the geometric distance and the nominal installation distance exceeds a preset distance threshold, it is determined that the propagation delay is abnormal.

[0089] When both acoustic fingerprint anomaly and propagation delay anomaly are present, it is determined that there is a physical obstruction event between the microphone node pairs.

[0090] The method for generating real-time dynamic spatial topology data is as follows: traverse the physical occlusion determination results. When a physical occlusion event is determined to exist, mark the geometric distance of the corresponding microphone node pair as invalid distance and replace the invalid distance with the corresponding nominal installation distance to obtain the corrected geometric distance. When no physical occlusion event is determined to exist, retain the corresponding geometric distance to obtain the corrected geometric distance.

[0091] The relative coordinates of each microphone node with respect to a preset reference node are calculated based on the corrected geometric distance. The preset reference node is the microphone node with the highest signal quality score among all microphone nodes. The signal quality score is calculated from the signal-to-noise ratio of the ambient background sound data.

[0092] A spatial topology graph is constructed based on relative coordinates. The nodes of the spatial topology graph are each microphone node, the edges are each pair of microphone nodes, and the edge weights are the corrected geometric distances between each pair of microphone nodes.

[0093] Obtain the current azimuth angle of the sound source, and calculate the initial beamforming weight vector for each microphone node based on the current azimuth angle of the sound source and the spatial topology map.

[0094] When a physical occlusion event is found in the physical occlusion determination result, the initial beamforming weight vector of the occluded microphone node corresponding to the physical occlusion event is set to zero, and the corresponding weight increment is allocated to the adjacent microphone nodes that are adjacent to the occluded microphone node and are not occluded according to a preset ratio to obtain the updated beamforming weight vector.

[0095] When no physical occlusion event is found in the physical occlusion determination result, the initial beamforming weight vector is used as the updated beamforming weight vector; the updated beamforming weight vector is written into the node attribute field of each corresponding microphone node in the spatial topology graph to obtain real-time dynamic spatial topology data.

[0096] Specifically, the target sound pickup channel contains multiple microphone nodes arranged in an array. The node self-sensing data includes ambient background sound data continuously collected by each microphone node and ultrasonic ranging pulse data exchanged between nodes.

[0097] Ambient background noise data: Each microphone node continuously collects ambient background noise during non-speaking periods at a sampling rate of 16kHz, a single frame duration of 25ms, and a frame shift of 10ms, which is used to extract acoustic fingerprints and calculate signal-to-noise ratio; at the same time, the ambient background noise data integrates real-time temperature values ​​collected by temperature sensors for sound speed correction.

[0098] Ultrasonic ranging pulse data: Each microphone node sequentially transmits an ultrasonic pulse signal with a frequency of 40kHz, and the other nodes receive it synchronously, forming bidirectional ultrasonic propagation data between the nodes. The pulse transmission period is 100ms, which is used to calculate the geometric distance between the nodes.

[0099] The ambient background sound data is processed to generate acoustic fingerprint hashes for each microphone node. The specific process is as follows:

[0100] The ambient background sound data is processed by framing, with a frame length of 25ms (corresponding to 400 sampling points) and a frame shift of 10ms (corresponding to 160 sampling points). A Hamming window is superimposed on each frame signal to suppress spectral leakage.

[0101] The spectrum is obtained by performing a fast Fourier transform on the windowed signal of each frame. After filtering by the Mel filter bank, the logarithm is taken, and then the first 12-dimensional Mel frequency cepstral coefficients are extracted by discrete cosine transform to form a 1×12 single-frame MFCC vector.

[0102] The MFCC vectors of 10 consecutive frames are concatenated in chronological order to form a 12×10 acoustic fingerprint matrix, which characterizes the environmental acoustic features of the node's location.

[0103] The acoustic fingerprint matrix is ​​encoded using the Locality Sensitive Hash (LSH) algorithm, which maps the high-dimensional matrix into a 64-bit binary hash code, i.e., acoustic fingerprint hash. The hash encoding uses the random projection method, which performs binarization mapping on the feature vector through a preset random hyperplane to ensure that similar features have a high probability of hash collision.

[0104] By combining any two different microphone nodes to construct microphone node pairs, and calculating the geometric distance between each microphone node pair based on ultrasonic ranging pulse data, an acoustic propagation time delay matrix between nodes is constructed. The specific process is as follows:

[0105] For each microphone node, the arrival time of the ultrasonic pulse signal is detected by cross-correlation: the transmitted standard pulse signal and the received signal are cross-correlated, and the time corresponding to the peak value of the cross-correlation is the arrival time of the pulse. The difference between the transmission time and the arrival time is calculated to obtain the sound wave propagation delay between the node pair.

[0106] Extract the real-time temperature value T (unit: degrees Celsius) from the ambient background sound data, and calculate the current sound speed c in the current environment based on the empirical formula for sound speed. The expression is:

[0107]

[0108] Based on the sound wave propagation delay and the current sound speed, the geometric distance between microphone node pairs is calculated by multiplying the current sound speed by the sound wave propagation delay.

[0109] The geometric distance of all microphone node pairs is compared with the corresponding preset nominal installation distance to construct an N×N inter-node acoustic propagation delay matrix (N is the total number of microphone nodes), and the matrix elements are the geometric distances of the corresponding node pairs; wherein, the nominal installation distance is the reference distance between node pairs measured by ultrasonic ranging in a standard environment without obstruction during the system initialization phase, and is pre-stored in the system configuration.

[0110] Combining acoustic fingerprint differences and propagation delay deviations as dual-dimensional features, the following judgment rules are used to jointly determine whether physical occlusion events exist between node pairs:

[0111] Calculate the Hamming distance between the acoustic fingerprint hashes of any two microphone nodes. When the Hamming distance exceeds the preset fingerprint threshold (based on experience), it is determined that the node pair has an acoustic fingerprint anomaly. Since physical occlusion will change the sound field environment at the node, the difference in background acoustic features between the two nodes will increase, and the Hamming distance will increase.

[0112] Extract the geometric distance of the corresponding node pair in the acoustic propagation delay matrix, calculate the absolute difference between it and the nominal installation distance, and determine that the node pair has an abnormal propagation delay when the difference exceeds the preset distance threshold (preset according to experience); obstructions will cause sound waves to propagate along the diffraction path, and the propagation distance will increase, which is manifested as the geometric distance measurement value being higher than the nominal value.

[0113] A physical occlusion event is determined to exist between microphone nodes only when both acoustic fingerprint anomaly and propagation delay anomaly are present simultaneously; single-dimensional anomalies are not considered occlusions to avoid misjudgment.

[0114] Based on the physical occlusion determination results, the node distance parameters are corrected, beamforming weights are calculated, and real-time dynamic spatial topology data is generated. The specific process is as follows:

[0115] Traverse all node pairs for physical occlusion determination results: if a physical occlusion event is determined, mark the geometric distance of the node pair as invalid distance, replace the invalid distance with the corresponding nominal installation distance, and obtain the corrected geometric distance; if no physical occlusion event is determined, retain the corresponding geometric distance as the corrected geometric distance.

[0116] The node with the highest signal quality score among all microphone nodes is selected as the preset reference node. The signal quality score is calculated from the signal-to-noise ratio (SNR) of the ambient background sound data. Specifically: First, a continuous ambient background sound data segment collected by the microphone node during idle periods when there is no effective speech is extracted as sample data. The sample data is then framed and windowed, and spectral transformation is performed frame by frame to obtain the frequency domain energy distribution of each frame. In the frequency domain, the system's preset speech working frequency band is defined, and the total energy of the speech working frequency band and the total energy of the background noise outside the frequency band are calculated in each frame. The ratio of the two is converted to decibels to obtain the instantaneous SNR of a single frame. The arithmetic mean of the instantaneous SNR of all frames is taken to obtain the average SNR corresponding to the node. Finally, the average SNR is normalized to a value range of 0-1, which is the signal quality score of the microphone node. The higher the average SNR, the larger the signal quality score, and the better the acquisition performance and signal quality of the corresponding microphone node.

[0117] Based on the corrected geometric distance of all node pairs, a multidimensional scaling method is used to calculate the two-dimensional relative coordinates of each microphone node relative to a preset reference node, thereby obtaining the relative coordinates of each node. The preset reference node is the microphone node with the highest signal quality score among all microphone nodes. The specific steps are as follows:

[0118] The first step is to construct a distance square matrix. Let the total number of microphone nodes be M. Based on the corrected geometric distance of all node pairs, generate an M-row, M-column symmetric distance square matrix. The value in the i-th row and j-th column of the matrix is ​​the square of the corrected geometric distance between the i-th microphone node and the j-th microphone node. The diagonal elements of the matrix (the distance between a node and itself) are all 0.

[0119] The second step is to perform a double-centering process on the squared distance matrix. First, calculate the row mean of each row, the column mean of each column, and the global mean of the entire matrix. Then, perform a centering operation on each element in the matrix: subtract the row mean of the corresponding row and the column mean of the corresponding column from the original element value, and add the global mean to obtain the centered inner product matrix. This matrix represents the inner product relationship between the vectors of each node and is used to convert the distance information into a vector relationship of solvable coordinates.

[0120] The third step is to solve for the initial coordinates by eigenvalue decomposition. The inner product matrix is ​​decomposed into M eigenvalues ​​and corresponding M eigenvectors. The eigenvalues ​​are sorted from largest to smallest, and the two largest positive eigenvalues ​​are selected. The square root of each eigenvalue is then multiplied element-wise by the corresponding eigenvector to obtain an initial coordinate matrix of dimension M×2. Each row of the matrix corresponds to the two-dimensional initial coordinates of a microphone node. Since the microphone array is deployed in the same plane, the first two eigenvalues ​​are sufficient to represent most of the spatial location information.

[0121] The fourth step is to align the reference nodes and generate relative coordinates. Using the selected reference node as a reference, the initial two-dimensional coordinates of all nodes are translated as a whole, so that the coordinates of the reference node are adjusted to the origin (0,0). The coordinates of all nodes after translation are the relative coordinates of each microphone node relative to the reference node.

[0122] A spatial topology graph is constructed based on relative coordinates. The vertices of the topology graph are each microphone node, the edges are each pair of microphone nodes, and the edge weight is the corrected geometric distance of the corresponding node pair. At the same time, each node contains attribute fields such as signal quality score and node coordinates.

[0123] The current azimuth angle of the sound source is obtained (derived from the relative position of the speaker and the center of the array). Based on the current azimuth angle and the spatial topology, the initial beamforming weight vector of each microphone node is calculated using the minimum variance distortionless response algorithm. This ensures that the main lobe of the beam is aligned with the direction of the sound source while suppressing signals from the direction of interference. The minimum variance distortionless response algorithm for calculating beamforming weights is a mature existing technology in the field of array signal processing. This algorithm constructs an array steering vector based on the current azimuth angle of the sound source and minimizes output interference and noise power under the constraint of ensuring that the signal from the desired sound source direction passes through without distortion. The initial beamforming weight vector of each microphone node is then obtained by solving this problem. This will not be described in detail here.

[0124] If a physical occlusion event exists, the initial beamforming weight vector of the occluded microphone node corresponding to the physical occlusion event is set to zero, and the total weight of the occluded nodes (i.e., the weight increment, which is the sum of the initial beamforming weight values ​​of all microphone nodes identified as physically occluded) is calculated. This weight increment is then distributed to the neighboring microphone nodes that are adjacent to the occluded node and are not occluded, according to the signal quality score ratio of the neighboring nodes, to obtain the updated beamforming weight vector. If no physical occlusion event exists, the initial beamforming weight vector is directly used as the updated weight vector.

[0125] The updated beamforming weight vector is written into the attribute fields of each corresponding node in the spatial topology graph, ultimately forming structured data including node coordinates, correction distance, signal quality, and beam weight, i.e., real-time dynamic spatial topology data.

[0126] This approach, by jointly determining physical occlusion events using both acoustic fingerprint hashing and ultrasonic ranging delay, effectively eliminates interference factors such as temperature drift and environmental noise, significantly improving the accuracy of occlusion detection. Simultaneously, it corrects node geometric distance parameters based on occlusion determination results and distributes the weights of occluded nodes to neighboring unoccluded nodes according to signal quality proportions, compensating for beam pointing distortion and gain loss caused by occlusion. This ensures the stability of beamforming and sound pickup in dynamic scenarios such as personnel movement and object displacement. Furthermore, the generated structured real-time dynamic spatial topology data uniformly carries node coordinates, acoustic parameters, and beam weights, providing standardized global reuse data for subsequent causal inference and digital twin modeling. It also allows the microphone array to adapt to changes in the sound field environment without repeated manual calibration, greatly improving the system's environmental robustness and operational efficiency.

[0127] S3. Obtain the audio signal amplitude sequence of the target pickup channel, construct a causal graph, and combine it with real-time dynamic spatial topology data to perform inference and generate multiple candidate causal microphone control decisions.

[0128] The method for generating multiple candidate causal microphone control decisions is as follows: extract time-domain feature vectors and frequency-domain feature vectors based on the audio signal amplitude sequence, and concatenate the time-domain feature vectors and frequency-domain feature vectors into an audio feature vector; calculate the device status index of each microphone node based on the signal quality score of each microphone node in the real-time dynamic spatial topology data; and calculate the air vibration propagation attenuation coefficient of each microphone node pair based on the corrected geometric distance of each microphone node pair in the real-time dynamic spatial topology data.

[0129] Construct a causal graph and a set of directed edges. Based on the set of directed edges, use a structural causal model to parameterize the causal graph to obtain a parameterized causal graph.

[0130] Acquire the observation sequence of the microphone signal node. When the decrease in the current frame amplitude value of the observation sequence relative to the previous frame amplitude value exceeds a preset drop threshold, trigger counterfactual reasoning.

[0131] Construct an intervention path set, perform do-intervention operations on each intervention path in the intervention path set, and calculate the counterfactual prediction value under each intervention path; take the absolute difference between the counterfactual prediction value corresponding to each intervention path and the actual observation value of the microphone signal node in the current frame as the causal effect strength of the corresponding intervention path, take the intervention state value of the fixed node corresponding to the intervention path with the smallest causal effect strength as the candidate causal microphone control decision, and traverse all intervention paths to generate multiple candidate causal microphone control decisions;

[0132] The node set of the cause-effect graph includes: speaker intention node, vocal cord vibration node, air vibration node, microphone signal node, ambient noise node, and device status node; among which, the observed value of the speaker intention node is the amplitude mutation rate in the audio feature vector, the observed value of the vocal cord vibration node is the main frequency component in the audio feature vector, the observed value of the air vibration node is the air vibration propagation attenuation coefficient, the observed value of the microphone signal node is the audio signal amplitude sequence, the observed value of the ambient noise node is the spectral entropy in the audio feature vector, and the observed value of the device status node is the device status index.

[0133] The set of directed edges includes: a first directed edge from the speaker intention node to the vocal cord vibration node; a second directed edge from the vocal cord vibration node to the air vibration node; a third directed edge from the air vibration node to the microphone signal node; a fourth directed edge from the ambient noise node to the microphone signal node; a fifth directed edge from the device status node to the microphone signal node; and a sixth directed edge from the speaker intention node to the microphone signal node.

[0134] Specifically, based on the audio signal amplitude sequence of the target pickup channel and real-time dynamic spatial topology data, three types of features are extracted for assigning values ​​to causal graph nodes, as follows:

[0135] Audio feature vector: Based on the amplitude sequence of the audio signal, time-domain and frequency-domain features are extracted and concatenated to form an audio feature vector. The time-domain feature vector includes the amplitude mean, amplitude variance, and amplitude abrupt change rate. The amplitude abrupt change rate is the ratio of the difference between the amplitude of the current frame and the amplitude of the previous frame to the amplitude of the previous frame, representing the degree of abrupt change in the signal amplitude. The frequency-domain feature vector includes the dominant frequency component and the spectral entropy. The dominant frequency component is the frequency value with the highest energy in the signal spectrum. The spectral entropy is used to characterize the flatness of the spectrum; the higher the value, the more complex the noise components.

[0136] Device Status Index: Based on the signal quality scores of each microphone node in the real-time dynamic spatial topology data, the device status index is calculated. It is the ratio of the signal quality score of the current node to the preset benchmark signal quality score (the factory calibration value). The closer the device status index is to 1, the better the device status.

[0137] Air vibration propagation attenuation coefficient: The air vibration propagation attenuation coefficient is calculated based on the corrected geometric distance between each microphone node pair in real-time dynamic spatial topology data. The spherical wave attenuation model is adopted, and the expression is:

[0138]

[0139] in The air attenuation coefficient is taken as 0.005 at room temperature and pressure. This is the corrected geometric distance between microphone nodes. The larger the corrected geometric distance, the stronger the sound wave propagation attenuation and the smaller the air vibration propagation attenuation coefficient.

[0140] A causal graph and a set of directed edges are constructed. Based on the set of directed edges, a structural causal model is used to parameterize the causal graph, resulting in a parameterized causal graph, as follows:

[0141] Definition of the node set in a cause-effect graph: A cause-effect graph contains 6 nodes, and the physical meaning and observations of each node are as follows:

[0142] Speaker Intent Node: The observed value is the amplitude mutation rate in the audio feature vector, which represents the intensity of the speaker's vocal intention.

[0143] Vocal cord vibration node: The observed value is the dominant frequency component in the audio feature vector, which characterizes the vibration characteristics of the sound source;

[0144] Air vibration node: The observed value is the air vibration propagation attenuation coefficient, which characterizes the attenuation characteristics of the sound wave propagation medium;

[0145] Microphone signal node: The observed value is the audio signal amplitude sequence, which is the final observed sound pickup result;

[0146] Environmental noise node: The observed value is the spectral entropy in the audio feature vector, which characterizes the complexity of the environmental background noise;

[0147] Device Status Node: The observed value is the device status index, which represents the working status of the microphone hardware.

[0148] Definition of a set of directed edges: Six directed edges are defined to represent the causal relationships between nodes, as follows:

[0149] First directed edge: Speaker intention node → Vocal cord vibration node, representing the speaking intention driving vocal cord vibration to produce sound;

[0150] The second directed edge: vocal cord vibration node → air vibration node, representing the sound waves generated by vocal cord vibration and propagation in the air;

[0151] The third directed edge: air vibration node → microphone signal node, representing the sound wave propagating to the microphone and being converted into an electrical signal;

[0152] The fourth directed edge: ambient noise node → microphone signal node, representing ambient noise superimposed on the microphone signal;

[0153] The fifth directed edge: Device status node → Microphone signal node, representing the impact of device hardware status on signal acquisition quality;

[0154] The sixth directed edge: Speaker intention node → Microphone signal node, representing the direct control effect of the speaker's volume adjustment and other behaviors on the signal.

[0155] Based on a set of directed edges, a structural causal model is used to parameterize the causal graph. The model assumes that the state value of each node is the sum of a linear combination of the state values ​​of its parent nodes and independent noise terms. The general expression is:

[0156]

[0157] in, The state of the child node. for The set of parent nodes, parent node To child nodes Causal weight coefficient, The exogenous noise term is independent of the parent node and follows a normal distribution. All causal weight coefficients are solved using the maximum likelihood estimation method based on historical observation data to obtain a parameterized causal graph. The maximum likelihood estimation method for solving the causal weight coefficients is an existing technique for parameter estimation of structural causal models. It is based on multiple sets of historical node state observation data. Under the assumption that the exogenous noise term follows a normal distribution, the causal weight coefficients corresponding to each directed edge are obtained by maximizing the log-likelihood function of the observed samples. This will not be described in detail here.

[0158] The system monitors the observation sequence of microphone signal nodes in real time. When the decrease in the amplitude of the current frame relative to the amplitude of the previous frame exceeds a preset drop threshold, it is determined to be an abnormal signal drop, triggering counterfactual reasoning. In this embodiment, the drop threshold is set to 30% of the amplitude of the previous frame.

[0159] Constructing a set of intervention paths, this embodiment preferably constructs three intervention paths:

[0160] First intervention path (equipment anomaly hypothesis): Fix the equipment status node to the normal state value (value is 1), and keep the environmental noise node and speaker intention node unchanged in the observed state value, in order to verify whether the sudden drop in signal is caused by equipment anomaly.

[0161] The second intervention path (noise interference hypothesis): The environmental noise node is fixed as a silent state value (value is 0), and the device state node and the speaker intention node are kept unchanged in the observed state value, which is used to verify whether the signal drop is caused by environmental noise interference.

[0162] The third intervention path (speech cessation hypothesis): The speaker intention node is fixed as the continuous speaking state value (taking the amplitude change rate of the frame before the sudden drop), and the device state node and the environmental noise node are kept unchanged in the observed state value, which is used to verify whether the signal drop is caused by the speaker stopping speaking.

[0163] For each intervention pathway, a do-intervention operation was performed, the causal effect strength was calculated, and candidate control decisions were generated. The specific process is as follows:

[0164] For each intervention path, the state value of the fixed node is forcibly replaced with the corresponding intervention state value, while keeping the causal dependencies of the other nodes unchanged. The updated state value is propagated layer by layer from the parent node to the child node along the directed edge, and the counterfactual prediction value of the microphone signal node under the intervention path is calculated. The absolute difference between the counterfactual prediction value corresponding to each intervention path and the actual observation value of the microphone signal node in the current frame is taken as the causal effect strength of the corresponding intervention path. The smaller the causal effect strength, the closer the hypothesis corresponding to the intervention path is to the real cause of the anomaly.

[0165] Finally, the intervention state value corresponding to the intervention path with the smallest causal effect is used as the core basis to generate corresponding candidate causal microphone control decisions; traversing all intervention paths, three types of candidate decisions are generated, each containing a corresponding candidate gain coefficient and candidate beam pointing angle:

[0166] Equipment malfunction response: Switch to the backup microphone node and simultaneously adjust the beam pointing angle and gain compensation coefficient;

[0167] Noise interference response decisions: Increase noise suppression gain, narrow beam main lobe width, and adjust beam pointing angle to avoid noise sources;

[0168] The decision to stop speaking is as follows: Maintain the current gain coefficient and candidate beam pointing angle, and enter standby waiting state.

[0169] This approach, by constructing a multi-node causal graph covering the entire chain from sound source and air propagation to signal acquisition, fully characterizes various influencing factors and causal transmission paths of the picked-up signal, providing a structured reasoning framework for signal anomaly analysis. Simultaneously, relying on a parameterized structural causal model and a do-intervention counterfactual reasoning mechanism, it can quantitatively calculate the causal effect intensity of different triggers such as equipment malfunction, noise interference, and changes in speaking state, accurately pinpointing the root cause of sudden signal amplitude drops, thereby significantly improving the accuracy of anomaly localization. Furthermore, it generates multiple candidate control decisions corresponding one-to-one with the causes of anomalies, effectively avoiding signal distortion caused by blind gain adjustment, improving the targeting and accuracy of control strategies, and providing ample space for comparison of optimal strategies for subsequent digital twin acoustic fields.

[0170] S4. Construct a digital twin acoustic field model based on real-time dynamic spatial topology data, input each candidate causal microphone control decision into the digital twin acoustic field model for forward extrapolation, obtain the acquisition quality score corresponding to each candidate causal microphone control decision, select the optimal control strategy based on the acquisition quality score and execute it.

[0171] The method for selecting and executing the optimal control strategy is as follows: each candidate causal microphone control decision includes a corresponding candidate gain coefficient and candidate beam pointing angle.

[0172] A digital twin acoustic field model is constructed based on the relative coordinates of each microphone node and the corrected geometric distance between each microphone node pair in real-time dynamic spatial topology data. The digital twin acoustic field model uses relative coordinates as spatial boundary conditions and corrected geometric distance as medium propagation parameters.

[0173] Obtain the current sound source location coordinates, input the current sound source location coordinates, candidate gain coefficients and candidate beam pointing angles in each candidate causal microphone control decision into the digital twin acoustic field model, perform forward extrapolation with a preset time step, and obtain the modulated sound pressure received value.

[0174] The propagation delay, reflection loss, and scattering loss of the sound pressure wavefront represented by the air vibration node are calculated from the current sound source location coordinates to each microphone node. The propagation delay, reflection loss, and scattering loss are superimposed on the modulated sound pressure received value to obtain the predicted acquisition signal sequence of each microphone node.

[0175] The acquisition quality score of each candidate causal microphone control decision is calculated based on the predicted acquisition signal sequence. The candidate causal microphone control decision with the highest acquisition quality score is selected as the optimal control strategy. The corresponding candidate gain coefficient and candidate beam pointing angle in the optimal control strategy are written into the node attribute field of each corresponding microphone node in the real-time dynamic spatial topology data. Within a preset time window after the speaker is detected to be preparing to speak, the physical pointing of each microphone node is adjusted based on the corresponding candidate beam pointing angle, and the signal amplification factor of each microphone node is adjusted based on the corresponding candidate gain coefficient.

[0176] Specifically, based on the relative coordinates of microphone nodes and the corrected geometric distance between node pairs in real-time dynamic spatial topology data, a digital twin acoustic field model combining ray tracing and wave simulation is constructed. The model architecture employs a hybrid modeling approach combining high-frequency ray tracing and wave simulation. The ray tracing module calculates the specular reflection path and energy attenuation of high-frequency sound waves (greater than 1 kHz), suitable for propagation calculations in the main speech frequency bands. The wave simulation module calculates the diffraction and scattering effects of low-frequency sound waves (less than or equal to 1 kHz) using the finite difference method, compensating for the insufficient accuracy of ray tracing in the low-frequency band. The spatial range of the acoustic simulation is defined using the relative coordinates of the microphone nodes as spatial boundary conditions. The corrected geometric distance between each node pair is used as the basic parameter for medium propagation, while pre-loaded environmental acoustic parameters such as wall absorption coefficient and air humidity attenuation coefficient are also loaded. The model can input control quantities such as sound source location, beam parameters, and gain coefficient, and output the predicted acquisition signal sequence and sound field distribution of each microphone node.

[0177] The current sound source location coordinates are obtained (derived from the speaker's facial orientation and the time difference of arrival). The candidate gain coefficients and candidate beam pointing angles from each candidate causal microphone control decision are input into the digital twin acoustic field model, and forward extrapolation is performed with a preset time step of 10ms. The specific process is as follows:

[0178] Based on the candidate beam pointing angle, the theoretical sound pressure level (SPL) received value of each microphone node is calculated using a beam pattern function. Then, the theoretical SPL received value is linearly amplitude modulated using a candidate gain coefficient to obtain the modulated SPL received value. Calculating the theoretical SPL received value based on the candidate beam pointing angle using a beam pattern function and then obtaining the modulated SPL received value through linear amplitude modulation using a gain coefficient is an existing technique in the field of array acoustic signal processing. For example, the SPL response amplitude of each microphone node can be obtained by substituting the candidate beam pointing angle into the array pattern function, and then the amplitude can be linearly scaled using a candidate gain coefficient to obtain the modulated SPL received value, which will not be described in detail here.

[0179] By employing a hybrid calculation combining ray tracing and wave simulation, the propagation delay, reflection loss, and scattering loss of the sound pressure wavefront, represented by the air vibration nodes, from the current sound source location coordinates to each microphone node are calculated.

[0180] Propagation delay: The propagation time of the sound pressure wavefront along its direct path, obtained by dividing the straight-line distance between the sound source location coordinates and the microphone node by the speed of sound;

[0181] Reflection loss: The energy attenuation of a sound pressure wavefront (sound wave) after reflection by reflective surfaces such as walls and tabletops is calculated from the number of reflections and the corresponding absorption coefficient. By tracing the propagation path of each sound pressure wavefront through ray tracing, the total number of reflections of the sound wave hitting reflective surfaces such as walls and tabletops along the path is counted, as well as the preset energy absorption coefficient of the reflective surface for each reflection. For a single reflection, the proportion of remaining energy after the sound wave is reflected is 1 minus the absorption coefficient of the corresponding reflective surface. The proportions of remaining energy corresponding to all reflections along the entire path are multiplied sequentially to obtain the total energy retention proportion after reflection along the path. Finally, the total energy retention proportion is taken as the common logarithm and multiplied by 10 to convert it to decibels. The resulting attenuation is the reflection loss of the propagation path. The more reflections and the higher the absorption coefficient of the reflective surface, the greater the reflection loss.

[0182] Scattering loss: The energy attenuation of a sound pressure wave after being scattered by an obstacle, calculated from the scattering coefficient related to the size and frequency of the obstruction. First, obstacles in the sound wave propagation path are identified, and their equivalent geometric dimensions are obtained. The corresponding sound wave wavelength is then calculated based on the current sound wave frequency. According to the ratio of the obstacle size to the sound wave wavelength, a preset frequency-related scattering coefficient table is matched (this table is based on the fundamental physical laws of sound wave scattering, combined with the equivalent size range of typical indoor obstacles and measured scattering attenuation data of sound waves at different frequencies, pre-calibrated according to the correspondence between sound wave frequency and the equivalent size of the obstacle, pre-calibrating and summarizing the remaining energy proportion corresponding to a single scattering under each working condition), to obtain the remaining energy proportion of the sound wave corresponding to a single scattering. The number of scattering events along the propagation path is counted, and the remaining energy proportion of the sound wave corresponding to a single scattering is multiplied by the number of scattering events to obtain the total scattering energy retention proportion, which is also converted to attenuation in decibels, representing the scattering loss of that path. The closer the obstacle size is to the sound wave wavelength and the more scattering events occur, the greater the scattering loss.

[0183] Finally, the received modulated sound pressure values ​​are superimposed with propagation delay, reflection loss, and scattering loss to obtain the predicted acquisition signal sequence for each microphone node.

[0184] Based on the predicted acquisition signal sequence, the acquisition quality score for each candidate causal microphone control decision is calculated. The acquisition quality score is a weighted composite index of three indicators: signal-to-noise ratio (SNR), direct sound ratio, and spatial diversity. The weights of SNR, direct sound ratio, and spatial diversity are set based on their importance to speech pickup clarity and acquisition quality, combined with expert experience. Specifically:

[0185] The sound pressure signal output from the straight, non-reflective propagation path from the sound source to the microphone node is obtained. The time-domain sampled values ​​of the sound pressure signal are squared and summed to obtain the direct sound energy. The sound pressure signals output from all propagation paths that are reflected once or multiple times by reflective surfaces such as walls and desktops in the model are superimposed. The squared sums of the superimposed signals are obtained to obtain the environmental reflected sound energy. All stray sound propagation paths that are scattered by obstacles and reach the microphone node are traversed. The sound pressure amplitude of each scattering path reaching the microphone is obtained. The squared sums of the sound pressure amplitudes of all scattering paths are obtained to obtain the energy corresponding to the scattered stray sound. The energy corresponding to the scattered stray sound is added to the noise energy of the model to obtain the noise energy. The direct sound energy is divided by the sum of the environmental reflected sound energy and the noise energy, and normalized and mapped to the [0,1] interval to obtain the signal-to-noise ratio. A higher signal-to-noise ratio indicates higher speech clarity and intelligibility, and stronger resistance to environmental interference.

[0186] Obtain the direct sound energy of the main propagation path and the total received energy of all propagation paths. Divide the direct sound energy of the main propagation path by the total received energy to obtain the direct sound ratio. The higher the value, the higher the speech intelligibility.

[0187] The cross-correlation peak value of the predicted acquisition signal sequences of adjacent microphone nodes is obtained (the predicted acquisition time-domain signal sequences of the same length and sampling rate corresponding to two spatially adjacent microphone nodes are taken, cross-correlation operation is performed on the two to obtain the cross-correlation function curve, and the maximum value of the cross-correlation function curve is taken as the cross-correlation peak value). The spatial diversity is obtained by subtracting the normalized cross-correlation peak value from 1. The larger the spatial diversity value, the lower the waveform overlap of the signals acquired by adjacent nodes, the stronger the spatial sampling independence of the array, and the more accurate the beam spatial pointing selectivity.

[0188] Finally, the signal-to-noise ratio, the proportion of direct sound, and the spatial diversity are weighted and summed to obtain the acquisition quality score. The higher the acquisition quality score, the better the overall sound pickup effect meets the requirements of high-quality speech acquisition.

[0189] Finally, the candidate causal microphone control decision with the highest acquisition quality score is selected as the optimal control strategy. The corresponding candidate gain coefficient and candidate beam pointing angle in the optimal control strategy are written into the node attribute field of each corresponding microphone node in the real-time dynamic spatial topology data as the corresponding optimal gain coefficient and optimal beam pointing angle. At the same time, within the preset feedforward time window after detecting the speaker's preparation to speak (in this embodiment, the preset feedforward time window is 500ms), control commands are sent to the microphone array: adjust the physical pointing (electronically controlled steering microphone) or digital beam pointing of each microphone node based on the optimal beam pointing angle, and adjust the signal amplification factor of each microphone node based on the optimal gain coefficient, thus completing the implementation of the control strategy.

[0190] In this way, by constructing a unified and positive quantitative evaluation system from three dimensions—signal cleanliness, speech clarity, and spatial directional selectivity—and calculating the acquisition quality score by weighting, the pickup effect of different candidate control decisions can be objectively and comprehensively measured. This provides a clear numerical basis for the selection of the optimal beamforming scheme, effectively avoids the subjectivity and bias of empirical parameter tuning, and improves the pickup quality of the microphone array and the system response efficiency.

[0191] In one embodiment, a microphone management and control system based on intelligent sensing is also disclosed, which is used to implement the microphone management and control method based on intelligent sensing described above.

[0192] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0193] All formulas in this manual are dimensionless and calculated numerically. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0194] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A microphone management and control method based on intelligent sensing, characterized in that, The control method includes: S1. Acquire multimodal sensing data, predict the speaking time and urgency of each speaker based on the multimodal sensing data; pre-activate the microphone node at the corresponding position within a preset time window before the speaking time, and arbitrate the speaking intentions of multiple people based on the speaking urgency to determine the target sound pickup channel. S2. Obtain the node self-sensing data of each microphone node in the target pickup channel, determine whether there is a physical occlusion event between each microphone node based on the node self-sensing data, and update the beamforming weight vector according to the judgment result to generate real-time dynamic spatial topology data. S3. Obtain the audio signal amplitude sequence of the target pickup channel, construct a causal graph, and combine it with real-time dynamic spatial topology data to perform inference and generate multiple candidate causal microphone control decisions. S4. Construct a digital twin acoustic field model based on real-time dynamic spatial topology data, input each candidate causal microphone control decision into the digital twin acoustic field model for forward extrapolation, obtain the acquisition quality score corresponding to each candidate causal microphone control decision, select the optimal control strategy based on the acquisition quality score and execute it.

2. The microphone management and control system method based on intelligent sensing according to claim 1, characterized in that, Methods for predicting the speaking time and urgency of each speaker include: Multimodal sensing data includes sound pressure level time series data, facial orientation angle time series data, mouth opening and closing time series data, torso tilt angle time series data, and indoor temperature and humidity time series data; Time alignment and normalization were performed on the sound pressure level time series data, facial orientation angle time series data, mouth opening and closing time series data, trunk tilt angle time series data, and indoor temperature and humidity time series data respectively to obtain the aligned multimodal feature sequence. Multimodal feature sequences are input into a temporal fusion network, which includes a long short-term memory (LSM) encoding layer, a cross-modal attention interaction layer, and a fully connected prediction layer. The LSM encoding layer performs temporal encoding on each modality feature sequence to obtain the latent state vector of each modality. The cross-modal attention interaction layer uses the latent state vector corresponding to the mouth opening and closing temporal data as the query vector, and uses the latent state vectors of other modalities as the key and value vectors to calculate the cross-modal attention weights and perform weighted fusion to obtain the fused latent state vector. The fully connected prediction layer performs joint regression based on the fused latent state vector and the speaker's historical speaking weight vector to output the predicted value of the speaking time and the speaking urgency score.

3. The microphone management and control method based on intelligent sensing according to claim 2, characterized in that, The method for determining the target pickup channel is as follows: Obtain the predicted speaking time, speaking urgency score, and historical speaking weight vector for each speaker; map the predicted speaking time of each speaker to a unified timeline, and pre-construct the speaking time window for each speaker; Iterate through the speaking time windows of each speaker. When the speaking time windows of any two speakers intersect, group the speakers corresponding to the intersecting speaking time windows into the same set of conflicting speakers. When the speaking time window of the same speaker intersects with the speaking time windows of multiple other speakers, merge the same speaker and all other speakers with intersecting windows into the same set of conflicting speakers, thus obtaining the set of conflicting speakers for each speaker. For each speaker in the conflict speaker set, a comprehensive arbitration score is calculated based on the urgency score of the speech and the weight vector of historical speeches; The speaker with the highest overall arbitration score is selected as the target audio pickup target, and the microphone node corresponding to the target audio pickup target is determined as the target audio pickup channel.

4. The microphone management and control method based on intelligent sensing according to claim 3, characterized in that, The method for determining whether there is a physical occlusion event between microphone segments is as follows: The node self-sensing data includes ambient background sound data continuously collected by each microphone node and ultrasonic ranging pulse data transmitted between nodes. The ambient background sound data is processed by framing and windowing, and the Mel frequency cepstral coefficient vector of each frame is extracted. The Mel frequency cepstral coefficient vectors of multiple consecutive frames are concatenated into an acoustic fingerprint matrix. The acoustic fingerprint matrix is ​​then encoded by local sensitive hashing to generate an acoustic fingerprint hash. Combine any two different microphone nodes to construct a microphone node pair, perform pulse arrival time detection on ultrasonic ranging pulse data, and calculate the sound wave propagation delay between each microphone node pair; Acquire temperature data from ambient background sound data, and calculate the current speed of sound based on the temperature data; The geometric distance between microphone node pairs is calculated based on the sound wave propagation delay and the current sound speed. The geometric distance between microphone node pairs is compared with the corresponding preset nominal installation distance to obtain the acoustic propagation delay matrix between nodes; Calculate the Hamming distance of the acoustic fingerprint hash between microphone node pairs. When the Hamming distance exceeds a preset fingerprint threshold, it is determined to be an acoustic fingerprint anomaly. Extract the geometric distance between microphone node pairs in the acoustic propagation delay matrix between nodes. When the difference between the geometric distance and the nominal installation distance exceeds the preset distance threshold, it is determined to be an abnormal propagation delay. When both acoustic fingerprint anomaly and propagation delay anomaly are present, it is determined that there is a physical obstruction event between the microphone node pairs.

5. The microphone management and control method based on intelligent sensing according to claim 4, characterized in that, The method for generating real-time dynamic spatial topology data is as follows: Traverse the physical occlusion determination results. When a physical occlusion event is determined to exist, mark the geometric distance of the corresponding microphone node pair as invalid distance and replace the invalid distance with the corresponding nominal installation distance to obtain the corrected geometric distance. When it is determined that there is no physical occlusion event, the corresponding geometric distance is retained to obtain the corrected geometric distance; The relative coordinates of each microphone node with respect to a preset reference node are calculated based on the corrected geometric distance. The preset reference node is the microphone node with the highest signal quality score among all microphone nodes. The signal quality score is calculated from the signal-to-noise ratio of the ambient background sound data. A spatial topology graph is constructed based on relative coordinates. The nodes of the spatial topology graph are each microphone node, the edges are each pair of microphone nodes, and the edge weights are the corrected geometric distances between each pair of microphone nodes. Obtain the current azimuth angle of the sound source, and calculate the initial beamforming weight vector for each microphone node based on the current azimuth angle of the sound source and the spatial topology map. When a physical occlusion event is found in the physical occlusion determination result, the initial beamforming weight vector of the occluded microphone node corresponding to the physical occlusion event is set to zero, and the corresponding weight increment is allocated to the adjacent microphone nodes that are adjacent to the occluded microphone node and are not occluded according to a preset ratio to obtain the updated beamforming weight vector. When no physical occlusion event is found in the physical occlusion determination result, the initial beamforming weight vector is used as the updated beamforming weight vector; the updated beamforming weight vector is written into the node attribute field of each corresponding microphone node in the spatial topology graph to obtain real-time dynamic spatial topology data.

6. The microphone management and control method based on intelligent sensing according to claim 5, characterized in that, The method for generating multiple candidate causal microphone control decisions is as follows: The time-domain feature vector and frequency-domain feature vector are extracted based on the amplitude sequence of the audio signal, and the time-domain feature vector and frequency-domain feature vector are concatenated to form an audio feature vector; the device status index of each microphone node is calculated based on the signal quality score of each microphone node in the real-time dynamic spatial topology data; and the air vibration propagation attenuation coefficient of each microphone node pair is calculated based on the corrected geometric distance of each microphone node pair in the real-time dynamic spatial topology data. Construct a causal graph and a set of directed edges. Based on the set of directed edges, use a structural causal model to parameterize the causal graph to obtain a parameterized causal graph. Acquire the observation sequence of the microphone signal node. When the decrease in the current frame amplitude value of the observation sequence relative to the previous frame amplitude value exceeds a preset drop threshold, trigger counterfactual reasoning. An intervention path set is constructed, and a do-intervention operation is performed on each intervention path in the intervention path set to calculate the counterfactual prediction value under each intervention path. The absolute difference between the counterfactual prediction value corresponding to each intervention path and the actual observation value of the microphone signal node in the current frame is taken as the causal effect strength of the corresponding intervention path. The intervention state value of the fixed node corresponding to the intervention path with the smallest causal effect strength is taken as the candidate causal microphone control decision. Multiple candidate causal microphone control decisions are generated by traversing all intervention paths.

7. The microphone management and control method based on intelligent sensing according to claim 6, characterized in that, The node set of the cause-effect graph includes: speaker intention node, vocal cord vibration node, air vibration node, microphone signal node, ambient noise node, and device status node; among which, the observed value of the speaker intention node is the amplitude mutation rate in the audio feature vector, the observed value of the vocal cord vibration node is the main frequency component in the audio feature vector, the observed value of the air vibration node is the air vibration propagation attenuation coefficient, the observed value of the microphone signal node is the audio signal amplitude sequence, the observed value of the ambient noise node is the spectral entropy in the audio feature vector, and the observed value of the device status node is the device status index. The set of directed edges includes: a first directed edge pointing from the speaker intention node to the vocal cord vibration node; a second directed edge pointing from the vocal cord vibration node to the air vibration node; a third directed edge pointing from the air vibration node to the microphone signal node; a fourth directed edge pointing from the ambient noise node to the microphone signal node; a fifth directed edge pointing from the device status node to the microphone signal node; and a sixth directed edge pointing from the speaker intention node to the microphone signal node.

8. The microphone management and control method based on intelligent sensing according to claim 7, characterized in that, The method for selecting and executing the optimal control strategy is as follows: Each candidate causal microphone control decision includes a corresponding candidate gain coefficient and candidate beam pointing angle; A digital twin acoustic field model is constructed based on the relative coordinates of each microphone node in real-time dynamic spatial topology data and the corrected geometric distance between each pair of microphone nodes. The digital twin acoustic field model uses relative coordinates as spatial boundary conditions and modified geometric distance as medium propagation parameters; Obtain the current sound source location coordinates, input the current sound source location coordinates, candidate gain coefficients and candidate beam pointing angles in each candidate causal microphone control decision into the digital twin acoustic field model, perform forward extrapolation with a preset time step, and obtain the modulated sound pressure received value. The propagation delay, reflection loss, and scattering loss of the sound pressure wavefront represented by the air vibration node are calculated from the current sound source location coordinates to each microphone node. The propagation delay, reflection loss, and scattering loss are superimposed on the modulated sound pressure received value to obtain the predicted acquisition signal sequence of each microphone node. The acquisition quality score for each candidate causal microphone control decision is calculated based on the predicted acquisition signal sequence. The candidate causal microphone control decision with the highest acquisition quality score is selected as the optimal control strategy. The corresponding candidate gain coefficient and candidate beam pointing angle in the optimal control strategy are written into the node attribute field of each corresponding microphone node in the real-time dynamic spatial topology data. Within a preset time window after the speaker is detected to be preparing to speak, the physical pointing of each microphone node is adjusted based on the corresponding candidate beam pointing angle, and the signal amplification factor of each microphone node is adjusted based on the corresponding candidate gain coefficient.

9. A microphone management and control system based on intelligent sensing, characterized in that, The system is used to implement the microphone management and control method based on intelligent sensing as described in any one of claims 1-8.

Images