An AI-based intelligent conference voice interaction system

Abstract

This invention discloses an AI-based intelligent conference voice interaction system, relating to the field of intelligent voice interaction technology. The system acquires multi-channel voice signals through a distributed voice sensor array and performs time-of-arrival analysis to generate a sound source azimuth sequence. It then corrects abnormal angle values ​​in the sound source azimuth sequence by combining the pressure distribution map of a pressure-sensing grid on the conference table, obtaining the spatial positioning coordinates of the sound source. Based on these coordinates, it extracts the formant frequency offset of the corresponding channel to generate a speaker's laryngeal tension coefficient. Using the laryngeal tension coefficient, it adjusts the smoothing window length of the fundamental frequency curve of the voice signal to obtain the denoised voice fundamental frequency trajectory. Based on the temporal correspondence between this fundamental frequency trajectory and the local pressure change peaks in the pressure distribution map, it determines the speech direction and generates a conference speaking right switching signal.

Description

Technical Field

[0001] This invention relates to the field of intelligent voice interaction technology, specifically to an AI-based intelligent conference voice interaction system. Background Technology

[0002] In multi-person conference scenarios, existing solutions utilize microphone arrays for sound source localization, calculating the speaker's location by the time difference of arrival (TDOA) of speech in each channel. However, indoor environments suffer from multipath reflections, reverberation, and background noise. Location estimation based solely on TDOA is prone to deviations from the actual sound source, resulting in numerous outliers in the angle sequence. Directly applying this to spatial localization can lead to frequent shifts in speaker coordinates or incorrectly locking onto empty seats. For speaking rights management, conventional methods rely on speech energy detection, silence duration, or manual control. These methods only utilize the amplitude or temporal characteristics of acoustic signals and cannot detect changes in the speaker's posture or stress levels. In fundamental frequency analysis, speech fundamental frequency curves are typically processed using a fixed-length smoothing window, failing to consider changes in laryngeal tension caused by emotions or emphasis. This results in residual spike noise on the fundamental frequency trajectory or excessive smoothing that loses intonation details, affecting the capture of intonation features such as rising intonation. These shortcomings lead to inaccurate timing or incorrect targeting of speaking rights in scenarios involving multiple speakers taking turns, interruptions, or remote collaboration. Therefore, it is necessary to solve the problem of how to use additional physical quantities to correct the location of abnormal sound sources caused by environmental interference. At the same time, it is necessary to solve the problem of how to combine the tension of the speaker's vocal organs with the pressure change trend of the body on the table to accurately determine the direction of the speech and achieve adaptive switching of speaking rights. Summary of the Invention

[0003] To address the issues of sound source localization being susceptible to environmental interference and abnormal angles in existing conference voice interaction, and the lack of fusion perception of body pressure and laryngeal state during speaking power switching, an AI-based intelligent conference voice interaction system is provided. This system corrects the spatial localization of sound sources through pressure distribution, dynamically adjusts fundamental frequency processing and determines speech direction by combining formant shift and the temporal relationship of pressure changes, and generates reliable speaking power switching signals, thereby improving the real-time performance and accuracy of multi-person conference interaction.

[0004] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides an AI-based intelligent conference voice interaction system, including a voice sensor array distributed in the conference venue and a pressure-sensing grid set on the conference table. The voice sensor array collects multi-channel voice signals, and generates a sound source azimuth sequence by analyzing the arrival time difference of each channel signal; the pressure-sensing grid outputs a pressure distribution map reflecting the contact state between the participant's upper limbs and the table. The processing unit corrects abnormal angle values ​​in the sound source azimuth sequence based on the pressure distribution map, obtains the spatial positioning coordinates of the sound source, and extracts the formant frequency offset of the voice signal from the sensor channel pointed to by the coordinates, generating a laryngeal tension coefficient characterizing the speaker's laryngeal tension. Using this coefficient, the smoothing window length of the fundamental frequency curve of the voice signal is adaptively adjusted, and after denoising processing, a smooth voice fundamental frequency trajectory that retains intonation details is obtained. The system combines the temporal correspondence between the starting point of the intonation rise in the fundamental frequency trajectory and the peak value of local pressure change in the pressure distribution map to determine the current speaker's speech direction, thereby generating a conference speaking right switching signal to achieve accurate and natural automatic control of speaking rights.

[0005] As a technical solution of this invention, in the process of generating the sound source azimuth sequence, cross-correlation calculation is performed on the signals received by each pair of adjacent sensors in the voice sensor array. The peak position of the cross-correlation function is used as the arrival time difference, and the sound source plane incident angle is calculated in combination with the geometric spacing of the pair of sensors. The least-squares fitting is performed on the obtained plane incident angles of all sensors in the array to obtain a smooth and stable sound source azimuth sequence. When correcting with the pressure distribution map, continuous grid areas with pressure values ​​exceeding a preset contact threshold are extracted from the pressure distribution map, and their geometric centers are marked as candidate sitting pressure center points. The direction vector deviation of each angle value in the sound source azimuth sequence relative to the pressure center point is calculated, and abnormal angle values ​​with deviations exceeding the angle deviation threshold are removed. The remaining angle values ​​are used to perform spatial triangulation to obtain the spatial positioning coordinates of the sound source. This process eliminates the sound source angle jump caused by environmental reflection or moving noise by using the human sitting posture pressure characteristics, significantly improving the positioning reliability.

[0006] Preferably, when extracting the formant frequency offset, the system uses the sensor channel pointed to by the spatial positioning coordinates of the sound source as the target channel, performs linear predictive coding analysis on the speech signal of that channel to obtain the frequency values ​​of the first three formants, and calculates the difference between these values ​​and the pre-stored reference formant frequencies of the speaker in a neutral vocal state to obtain the formant frequency offset. The absolute values ​​of these three offsets are then summed and divided by the number of formants to generate the laryngeal tension coefficient. The reference formant frequencies of each speaker are stored in the local cache of the meeting according to their identity, and are initialized and updated before the meeting begins through a short silent vocal test to ensure the accuracy of the personalized physiological characteristic baseline. The resulting laryngeal tension coefficient can accurately reflect the speaker's vocal exertion state, providing a reliable basis for subsequent processing.

[0007] Furthermore, when generating the denoised speech fundamental frequency trajectory, the smoothing window length is set to be inversely proportional to the laryngeal tension coefficient. When the speaker's laryngeal tension increases and fundamental frequency fluctuations intensify, the smoothing window length is automatically reduced to preserve details of rapid pitch changes while suppressing noise; conversely, the window is increased to filter out occasional interference in smooth speech. Under this adjusted smoothing window, the fundamental frequency curve is subjected to median filtering to remove spike noise points. Then, the filtered fundamental frequency curve is resampled into an equal-time interval sequence along the time axis, ultimately obtaining a clean and continuous speech fundamental frequency trajectory, effectively improving the adaptability of fundamental frequency detection to noisy conference environments.

[0008] This invention utilizes multimodal information to determine the direction of speech and generate a signal for switching speaking rights. The system detects the initial moment when the rate of increase of the fundamental frequency value in the speech trajectory exceeds the rise rate threshold, marking this as the starting point of the rising tone. Simultaneously, it retrieves local pressure change peaks appearing in the pressure distribution map within a predetermined time window before and after this moment, extracting the pressure growth direction vector corresponding to each peak. Simultaneously, it constructs a reference direction vector with the spatial positioning coordinates of the sound source as the starting point and the geometric center of the conference table as the ending point, calculating the dot product between the pressure growth direction vector and the reference direction vector. When the dot product is positive, the speech direction is determined to be pointing towards others, generating a switching signal to transfer speaking rights to the seat corresponding to the direction indicated by the pressure growth direction vector. When the dot product is negative, the speech direction is determined to be pointing towards oneself, generating a hold signal to maintain the current speaking rights and locking the current speaking channel. This scheme, through the coordinated analysis of body leaning or hand gesture pressure direction and rising tone, makes the switching of speaking rights more in line with natural conference interaction habits, reducing misjudgments.

[0009] For scenarios where multiple speakers intend to speak simultaneously, the system also has a function to identify discourse power conflicts. When there are two or more sound sources whose spatial positioning coordinates correspond to the same target coordinate, the number of speech signals pointing to that target coordinate is counted. When the number exceeds the threshold for the number of conflicting speakers, the overlap duration of the fundamental frequency trajectories of each speech signal is extracted, and the overlap ratio between each pair is calculated. If the maximum value of the overlap ratio exceeds the overlap threshold, the current meeting state is marked as a discourse power conflict state, and a waiting prompt signal is generated, thereby orderly guiding the speaking order and avoiding the chaos caused by multiple people vying to speak at the same time.

[0010] Furthermore, the system dynamically adjusts microphone sensitivity based on the laryngeal tension coefficient. It calculates the sliding average of the laryngeal tension coefficient over a predetermined time period and maps it to a gain adjustment factor. This gain adjustment factor is multiplied by the microphone preamplifier's baseline gain value to obtain a dynamic gain value, which controls the preamplifier's amplification. When the dynamic gain value exceeds an upper threshold, the low-frequency gain is further reduced to compensate for the near-field low-frequency amplification effect caused by the speaker's increased voice volume, ensuring clear and natural sound pickup regardless of the emotional intensity during speeches, debates, or other situations.

[0011] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0012] By extracting continuous grid regions with pressure values ​​exceeding a preset contact threshold from the pressure distribution map, and using the geometric center of these regions as candidate sitting pressure center points, the deviation of the direction vector of each angle in the sound source azimuth sequence relative to this center point is calculated. Abnormal angle values ​​with deviations exceeding a preset angle deviation threshold are eliminated, and spatial triangulation is performed on the remaining angle values ​​to obtain the spatial positioning coordinates of the sound source. This method of constraining the direction of the sound source using the pressure distribution on the conference table can filter out false angle information caused by non-direct paths such as wall reflections and seat obstructions, making the spatial positioning coordinates closely related to the speaker's actual seating area and upper body tilt. The positioning point is less susceptible to disturbances by transient environmental noise, and the continuity of the coordinates is significantly enhanced. After locking the target sensor channel based on the spatial positioning coordinates of the sound source, linear predictive coding analysis is performed on the speech signal of that channel to extract the frequency values ​​of the first three formants. The difference is calculated with the pre-stored reference formant frequencies of the speaker in a neutral vocal state to obtain the formant frequency offset. The absolute values ​​of the offsets are accumulated and divided by the number of formants to generate the laryngeal tension coefficient. The length of the fundamental frequency curve smoothing window is inversely proportional to this coefficient. When vocal tension increases, the window automatically shortens to preserve the details of rapid, subtle changes in the fundamental frequency caused by laryngeal muscle tension; when tension decreases, the window automatically lengthens to suppress random noise fluctuations. After adaptive median filtering, the signal is resampled into an equal-time interval sequence to generate a denoised speech fundamental frequency trajectory. The starting point when the fundamental frequency value rise rate in this trajectory exceeds the rise rate threshold is detected and marked as the starting point of the pitch rise. Simultaneously, local pressure change peaks appearing in the pressure distribution map within a predetermined time window before and after the starting point are retrieved, and the pressure growth direction vector of the peak is extracted. A reference direction vector is constructed with the spatial positioning coordinates of the sound source as the starting point and the geometric center of the conference table as the ending point. The pressure growth direction vector is then multiplied by this reference direction vector. The sign of the dot product determines whether the speech direction is pointing to others or to oneself. When pointing to others, a switching signal is generated to transfer the speaking right to the seat indicated by the pressure growth direction; when pointing to oneself, a hold signal is generated to maintain the current speaking right. This method of speech inflection, which combines changes in vocal organ tension with dynamic trends in desktop pressure, allows the fundamental frequency curve to be smoothly processed and adapted to the speaker's physiological state. It removes noise while retaining the instantaneous characteristic of rising intonation. The determination of speech direction no longer relies solely on pitch changes but also incorporates the dimension of body pressure bias. This effectively distinguishes between the speaker's internal emphasis and intention to direct others, reducing erroneous switching caused by fluctuations in a single vocal feature. In multi-person debates or natural continuation of speech, the switching of speaking rights is more in line with the actual rhythm of communication, and the interaction process is smoother. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0014] Figure 1 This is a schematic diagram of the structure of an AI-based intelligent meeting voice interaction system;

[0015] Figure 2 This is a flowchart of a sound source spatial localization method based on the deviation analysis between the sound source azimuth angle and the seated pressure center.

[0016] Figure 3 It is a diagram showing the sound source azimuth sequence correction and abnormal angle removal;

[0017] Figure 4 It is a denoised speech fundamental frequency trajectory diagram based on laryngeal tension modulation;

[0018] Figure 5 This is a comparison chart of the baseband trajectories of multiple voice signals. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] See Figure 1 This invention provides an AI-based intelligent conference voice interaction system. The system includes: acquiring multi-channel voice signals through a distributed array of voice sensors deployed at the conference venue, performing time-of-arrival difference analysis on each channel's voice signal to generate a sound source azimuth sequence; correcting abnormal angle values ​​in the sound source azimuth sequence based on the sound source azimuth sequence and a pressure distribution map output by a preset pressure-sensing grid on the conference table to obtain the sound source spatial positioning coordinates; extracting the formant frequency offset from the corresponding channel's voice signal based on the sound source spatial positioning coordinates, and generating a speaker's laryngeal tension coefficient based on the formant frequency offset; adjusting the smoothing window length of the fundamental frequency curve in the voice signal using the laryngeal tension coefficient to generate a denoised voice fundamental frequency trajectory; determining the current speaker's speech direction based on the temporal correspondence between the voice fundamental frequency trajectory and the local pressure change peaks in the pressure distribution map, and generating a conference speaking right switching signal based on the speech direction.

[0021] Example 1:

[0022] In specific implementation, please refer to Figure 2 The method involves collecting multi-channel audio signals using a distributed array of audio sensors deployed at the conference venue, and performing time difference of arrival analysis on each channel's audio signals to generate a sound source azimuth sequence. This is achieved by performing cross-correlation calculations on the audio signals received by each pair of adjacent sensors in the audio sensor array. Let the signal received by one of the two adjacent sensors at time t be... The signal received by the other sensor at time t is Calculate the cross-correlation function In the cross-correlation function Search for the maximum value point on the waveform to obtain the time delay value corresponding to the maximum value point. The arrival time difference is extracted as the value between this pair of adjacent sensors. The above cross-correlation calculation process is repeated on all adjacent sensor pairs to obtain a set of arrival time differences.

[0023] Based on the arrival time difference of each pair of adjacent sensors and the geometric distance between them, the plane incidence angle of the sound source relative to the pair of adjacent sensors is calculated using the expression:

[0024]

[0025] in, This represents the angle of incidence of the sound source plane calculated for this pair of adjacent sensors. This indicates the speed at which sound travels through the air. The value is 340 m / s. This represents the time difference of arrival for this pair of adjacent sensors. This represents the straight-line distance between the geometric center points of this pair of adjacent sensors. Through the above calculations, each pair of adjacent sensors outputs a planar incident angle.

[0026] A set of angle observations is formed by combining all the plane incident angles calculated from all adjacent sensor pairs in the speech sensor array. Least square fitting is then performed on this set to generate a sound source azimuth sequence. Specifically, using the true azimuth of the sound source as the unknown, observation equations are established based on the coordinates of each sensor pair and the observed plane incident angles, forming an overdetermined system of equations. The least squares method is used to solve the overdetermined system of equations, minimizing the sum of squared residuals of all observation equations to obtain the optimal sound source azimuth estimate for the current moment. This process is repeated for each frame of speech signal, and the optimal sound source azimuth estimates for each frame are arranged in chronological order to form a sound source azimuth sequence.

[0027] In some implementations, continuous grid regions with pressure values ​​exceeding a preset contact threshold are extracted from the pressure distribution map output by a preset pressure-sensing grid on the conference table. The preset contact threshold is set to 0.5 Newtons, which is greater than the background noise fluctuation amplitude of the pressure-sensing grid when there is no contact with the table, but less than the typical lower limit of pressure generated when an adult naturally places their hand on the conference table. All grid cells in the pressure-sensing grid with output pressure values ​​greater than 0.5 Newtons are marked as pressure-bearing cells, and a connected component labeling algorithm is used to aggregate all adjacent pressure-bearing cells into continuous grid regions. The geometric mean of the spatial coordinates of all pressure-bearing cells within each continuous grid region is calculated, and the coordinate point corresponding to this geometric mean is marked as a candidate sitting pressure center point. When multiple continuous grid regions exist, multiple candidate sitting pressure center points are obtained.

[0028] The process of calculating the direction vector deviation of each angle value in the sound source azimuth sequence relative to the candidate sitting pressure center point is as follows: A first direction vector is constructed, starting from the reference center point of the speech sensor array and ending at the candidate sitting pressure center point. Then, a second direction vector is constructed, starting from the reference center point of the speech sensor array and taking the direction indicated by the currently calculated angle value in the sound source azimuth sequence as the ray direction, and selecting a distal endpoint on this ray. The spatial angle between the first and second direction vectors is calculated; this spatial angle is the direction vector deviation. This calculation is performed for each angle value in the sound source azimuth sequence to obtain the corresponding direction vector deviation sequence.

[0029] Each deviation in the direction vector deviation sequence is compared with a preset angle deviation threshold. The preset angle deviation threshold is set to 15 degrees, based on the following: Under normal sitting posture and desktop interaction scenarios, the deviation between the speaker's arm pressure center and the true direction of the sound source is usually less than 15 degrees. Angle values ​​exceeding 15 degrees are highly likely to be abnormal values ​​caused by multipath reflection or sudden noise interference. Abnormal angle values ​​with direction vector deviations exceeding 15 degrees are removed, and angle values ​​with direction vector deviations less than or equal to 15 degrees are retained as valid angle values.

[0030] Spatial triangulation was performed on all retained valid angle values ​​to obtain the spatial location coordinates of the sound source. The spatial triangulation calculation used the three-dimensional installation coordinates of each sensor in the speech sensor array within the conference room as the reference point. A spatial ray was constructed along the direction corresponding to each valid angle value, and multiple spatial rays approximately intersected in space. The least squares estimation method was used to find the nearest intersection point of the spatial rays, and the obtained three-dimensional coordinates were used as the spatial location coordinates of the sound source. These spatial location coordinates indicate the current speaker's position in the conference room space and are used to point to the corresponding sensor channel.

[0031] See Figure 3In the figure, the horizontal axis represents the time frame number, and the vertical axis represents the azimuth angle of the sound source, in degrees. The blue dashed curve represents the original azimuth angle sequence obtained by cross-correlation calculation and least-squares fitting of the arrival time differences of adjacent sensor pairs in the speech sensor array. This sequence contains several anomalous angle values, which are represented by prominent peaks in the curve. The orange crosses mark the locations of these anomalous angle values, reflecting the anomalous azimuth angle estimation results caused by multipath reflection or sudden noise interference.

[0032] The red solid line curve represents the sound source azimuth sequence after correcting for abnormal angle values ​​by combining the original sound source azimuth sequence with the pressure distribution map output from the conference table pressure sensing grid. The corrected azimuth sequence curve is smooth and continuous, eliminating the sharp peaks in the original sequence, and the overall trend conforms to the actual motion trajectory of the sound source.

[0033] Example 2:

[0034] In practical implementation, the sensor channel pointed to by the spatial positioning coordinates of the sound source is taken as the target channel. The process of extracting the first three formant frequencies by performing linear predictive coding analysis on the speech signal of the target channel is as follows: The speech signal obtained from the target channel is a discrete-time signal. A first-order digital pre-emphasis processing is applied to this speech signal, and the transfer function of the pre-emphasis filter is... Among them, the pre-weighting coefficient The value is set to 0.97. The pre-emphasized speech signal is divided into short frames of 25 milliseconds each, with a sliding step of 10 milliseconds between adjacent frames. Each short frame is multiplied by a Hamming window function, the expression of which is: , , The number of sample points per frame is determined by the sampling rate and frame duration. For each windowed frame, its autocorrelation function is calculated, with a lag order of 12. The Levinson-Dubin recursive algorithm is used to solve for the 12th-order linear prediction coefficients from the autocorrelation function. A full-pole model of the vocal tract transmission is constructed from the linear prediction coefficients. The roots of the denominator polynomial of this full-pole model are solved, and the frequencies corresponding to the three smallest conjugate roots are taken as the first, second, and third formant frequencies of the current frame. This process is repeated for each frame of the speech signal to obtain the time series of the three formant frequencies.

[0035] In one processing method, the first formant frequency, second formant frequency, and third formant frequency of each frame are subtracted from the reference first formant frequency, reference second formant frequency, and reference third formant frequency corresponding to the speaker's neutral vocal state, respectively, to obtain the first formant frequency offset. Second resonant frequency shift and the frequency shift of the third resonance peak The reference formant frequencies are stored in the local conference cache according to the speaker's identity. The local conference cache is the solid-state storage area of ​​the conference control host. The speaker's identity is entered through the conference check-in system and bound to facial recognition information or seat number. In the preparatory stage before the conference begins, the system guides each speaker to complete a short silent speech test. Specifically, after a prompt tone, the speaker is asked to read a 5-second passage of a prescribed text in a natural and relaxed tone and volume. The test audio is collected, and the first, second, and third formant frequencies of all valid frames of the test audio are extracted. The arithmetic mean of each is calculated and recorded as the reference first formant frequency, reference second formant frequency, and reference third formant frequency, respectively. These are then associated with the speaker's identity and stored in the local conference cache, completing the initialization update.

[0036] The laryngeal tension coefficient is calculated using the following formula:

[0037]

[0038] in, This indicates the laryngeal tension coefficient. This represents the absolute value of the frequency shift of the first resonant peak. This represents the absolute value of the frequency shift of the second resonant peak. This represents the absolute value of the frequency shift of the third resonance peak, with the denominator 3 being the number of resonance peaks.

[0039] Set smooth window length With laryngeal tension coefficient They are inversely proportional. When the laryngeal tension coefficient increases, the smoothing window length decreases; when the laryngeal tension coefficient decreases, the smoothing window length increases. Specifically, a mapping relationship is used. Determine the smooth window length value, where This represents the maximum smoothing window length, with a value of 21 sampling points. This represents the sensitivity adjustment factor, with a value of 2.0. This indicates a floor operation. The maximum smoothing window length of 21 sampling points is set based on the fundamental frequency trajectory smoothing window range commonly used in speech fundamental frequency analysis. This provides a strong smoothing effect when the laryngeal tension coefficient is 0, while the window is shortened to a few sampling points when the tension is high to preserve intonation details.

[0040] Based on the adjusted smoothing window length, median filtering is applied to the fundamental frequency curve of the speech signal. The fundamental frequency curve is calculated frame-by-frame using an autocorrelation fundamental frequency extraction algorithm, with each frame's fundamental frequency value corresponding to a fundamental frequency point. The adjusted smoothing window length is used... As the window size of the median filter, during the median filtering process, for the k-th fundamental frequency point in the fundamental frequency curve, the values ​​before and after it are taken. The fundamental frequency points together form a set of fundamental frequency values ​​within the window. This set of fundamental frequency values ​​is sorted by numerical value, and the median is taken as the filtered fundamental frequency value of the k-th fundamental frequency point. When the window length... When the value is even, the average of the two middle values ​​is taken as the median. The above median filtering operation is then performed on all fundamental frequency points of the fundamental frequency curve to filter out spike noise points in the fundamental frequency curve caused by glottal pulse detection errors or environmental noise.

[0041] The median-filtered fundamental frequency curve is resampled along the time axis into an equal-time sequence to generate the denoised speech fundamental frequency trajectory. The resampling process uses linear interpolation with fixed resampling time intervals. New time grid points are generated at millisecond intervals, and the fundamental frequency value corresponding to each new time grid point is calculated. For new time grid points occurring between two original fundamental frequency points, linear interpolation is performed based on the time and fundamental frequency values ​​of the two adjacent original fundamental frequency points. For grid points outside the original fundamental frequency time range, the nearest valid fundamental frequency value is assigned. The resampled, equally timed fundamental frequency value sequence is the denoised speech fundamental frequency trajectory.

[0042] See Figure 4 In the graph, the horizontal axis represents time in seconds, ranging from 0 to 10 seconds; the vertical axis represents the fundamental frequency of the speech in Hertz, ranging from approximately 70 Hertz to 180 Hertz. The legend shows two curves: the blue dashed curve is the original fundamental frequency curve, and the red solid curve is the denoised speech fundamental frequency trajectory after adjusting the smoothing window length based on the laryngeal tension coefficient as described in Example 2.

[0043] As can be seen from the figure, the original fundamental frequency curve exhibits strong high-frequency spike noise and irregular fluctuations, with some fundamental frequency points showing abnormal jumps. The highest peak exceeds 160 Hz, and the lowest point is close to 70 Hz, indicating the presence of noise and detection errors in the signal. The red denoised fundamental frequency trajectory closely follows the overall fundamental frequency change trend, effectively smoothing out abnormal spikes and sudden noise points, making the fundamental frequency curve smooth and continuous, and better preserving the subtle variation characteristics of the fundamental frequency.

[0044] Example 3:

[0045] In practice, the moment when the rate of increase of the fundamental frequency value in the speech fundamental frequency trajectory exceeds the rise rate threshold is detected and marked as the starting point of the pitch rise. The speech fundamental frequency trajectory is a sequence of fundamental frequency values ​​arranged along the time axis, and the fundamental frequency value at the u-th time point is denoted as... The time interval between two adjacent time points is the resampling time interval. The value is 5 milliseconds. The rise rate of the fundamental frequency is obtained by first-order backward difference calculation, and the rise rate at time point u is... The calculation method is as follows The rise rate threshold was set at 50 Hz / second. This value is based on the physiological characteristics of fundamental frequency changes during the natural rise in human speech. Generally, the rate of fundamental frequency change during the rise in tone at the end of a declarative sentence or in a question is above 50 Hz / second, while the rate of fundamental frequency fluctuation during the transition between normal syllables is below this threshold. For the rise rate sequence... A point-by-point scan is performed. When the rise rate values ​​of P consecutive time points are all greater than 50 Hz / second, the earliest time point among these P time points is marked as the starting point of the pitch rise. The value of P is 4, corresponding to a duration of 20 milliseconds, in order to eliminate instantaneous spike interference.

[0046] The system synchronously retrieves local pressure change peaks within a predetermined time window before and after the onset of pitch rise in the pressure distribution map. The predetermined time window is set to 100 milliseconds before and after the onset of pitch rise, with a width of 200 milliseconds, covering the typical neuromuscular transmission delay range between vocal organ tension and hand pressure movements. Within each frame of the pressure distribution map within the time window, the pressure values ​​of all pressure-sensing grid cells are scanned, the temporal gradient of pressure in each grid cell is calculated, and grid regions where pressure values ​​simultaneously exhibit local maxima in both the temporal and spatial dimensions are identified. The geometric center of these grid regions is used as the geometric center of the grid region containing the local pressure change peak.

[0047] Starting from the spatial location coordinates of the sound source and ending at the geometric center of the grid region where the peak of local pressure change occurs, a pressure growth direction vector is constructed. The three-dimensional components of the spatial location coordinates of the sound source are denoted as... The three-dimensional component of the geometric center of the grid region where the local pressure change peak occurs is denoted as... The vector of the direction of pressure growth Represented as .

[0048] Using the spatial location coordinates of the sound source as the origin and the geometric center point of the conference table as the reference endpoint, a direction vector is constructed connecting the spatial location coordinates of the sound source. The three-dimensional components of the geometric center point of the conference table in the conference room's spatial coordinate system are denoted as... The direction vector of the line connecting the spatial coordinates of the sound source Represented as .

[0049] Calculate the vector of pressure growth direction Direction vector of the line connecting the spatial location coordinates of the sound source dot product between, dot product value The calculation formula is:

[0050]

[0051] in, This represents the result of the dot product operation between the pressure growth direction vector and the direction vector of the line connecting the spatial location coordinates of the sound source. This represents the three-dimensional coordinates of the geometric center of the grid region where the peak of the local pressure change is located. Three-dimensional coordinates representing the spatial location coordinates of a sound source. The three-dimensional coordinates of the geometric center point of the conference table.

[0052] If the dot product value A positive value indicates that the angle between the pressure growth direction vector and the direction vector of the line connecting the spatial positioning coordinates of the sound source is less than 90 degrees. The pressure growth direction points towards an area outside the geometric center of the conference table, i.e., towards other people's seats. The direction of the speech is determined to be pointing towards others, generating a switching signal to transfer the speaking right to the seat corresponding to the direction indicated by the pressure growth direction vector. The switching signal includes an identifier for the target seat, which is determined by calculating the distance between the extension of the pressure growth direction vector and the preset spatial coordinates of each seat; the seat with the smallest distance is the target seat. If the dot product value... A negative value indicates that the angle between the pressure growth direction vector and the direction vector of the line connecting the spatial positioning coordinates of the sound source is greater than 90 degrees. The pressure growth direction is towards the rear, i.e., towards the speaker's own body. The speech direction is determined to be pointing towards the speaker, generating a hold signal to maintain the current speaking right and outputting the hold signal to the conference control terminal to lock the current speaking channel. After receiving the hold signal, the conference control terminal rejects speaking requests from other seats for a preset lock time of 2 seconds. If a new speaking request is received after the lock time, the speech direction determination will be re-performed.

[0053] Example 4:

[0054] In practice, when two or more sound source spatial location coordinates correspond to speech directions that all point to the same target coordinate, the number of speech signals pointing to the same target coordinate is counted. Speech direction is determined by the dot product value. The sign of the signal and the direction of the extension of the pressure growth vector are determined. The target coordinates refer to the spatial coordinates of each participant's seat in the pre-set meeting room. Taking the current moment as the analysis time point, the spatial location coordinates of all sound sources in speaking mode at that analysis time point are obtained. Each sound source spatial location coordinate corresponds to one speech signal and one speech direction determination result. The speech signals marked as pointing to others in the speech direction determination results of each sound source spatial location coordinate are extracted, and the target coordinates pointed to by each speech signal pointing to others are recorded. The number of speech signals pointing to the same target coordinate is calculated by traversing and counting each target coordinate in the meeting room. When a target coordinate is pointed to by two or more speech signals at the same time, the target coordinate and the speech signals pointing to the target coordinate enter the discourse power conflict detection process.

[0055] When the number of voice signals pointing to the same target coordinate exceeds the conflict threshold, the overlap duration of the fundamental frequency trajectories in each voice signal is extracted. The conflict threshold is set to 3, based on the general situation in a meeting where two people simultaneously initiating a conversation with the same person constitutes a power struggle, and three or more people simultaneously pointing to the same target significantly escalate the conflict. Setting the conflict threshold to 3 can filter out normal alternations between two people. All voice signals pointing to the same target coordinate are numbered as voice signal 1, voice signal 2, ..., voice signal M, where M is the total number of voice signals pointing to the same target coordinate. Each voice signal corresponds to a denoised voice fundamental frequency trajectory, which consists of a time point sequence and a fundamental frequency value sequence. For any two voice signals, the overlap duration is defined as the length of the overlapping interval where the fundamental frequency trajectories of the two voice signals simultaneously have valid fundamental frequency values ​​on the time axis. A valid fundamental frequency value exists at a certain time point in the fundamental frequency trajectory if the fundamental frequency value at that time point is greater than zero and the fundamental frequency value is not the default value filled by silence segments. For the m-th and n-th speech signals, starting from the beginning of the common interval of the time axes of the two speech signals, the resampling time interval is... To iterate through the common time interval with a step size, the cumulative length of the continuous time period during which the fundamental frequency values ​​of the two speech signals are simultaneously valid is calculated. This cumulative length is the overlap duration of the fundamental frequency trajectories of the m-th and n-th speech signals, denoted as [missing information]. .

[0056] Calculate the overlap ratio between each pair of audio signals. The overlap ratio between the m-th and n-th audio signals. The calculation formula is:

[0057]

[0058] in, This indicates the overlap ratio between the m-th and n-th speech signals. This represents the overlap duration of the fundamental frequency trajectories of the m-th and n-th speech signals, in milliseconds. This represents the total duration of the m-th speech signal, in milliseconds. This represents the total duration of the nth audio signal, in milliseconds. This represents the smaller of the total duration of the m-th speech signal and the total duration of the n-th speech signal. The total duration refers to the total time from the first occurrence of a valid fundamental frequency value to the last occurrence of a valid fundamental frequency value in a single speech signal. The overlap ratio is calculated for all pairwise combinations of the M speech signals pointing to the same target coordinate, resulting in a total of... The percentage of overlapping values.

[0059] If the maximum value among all overlap percentages exceeds the overlap threshold, the current meeting status is marked as a speaking power conflict, and a waiting prompt signal is generated. The overlap threshold is set to 0.7, based on the fact that when the overlap percentage of two audio signals exceeds 0.7, it indicates that the audio signal with the shorter total speaking time overlaps with the other audio signal for more than 70% of its time period. This situation almost never occurs in normal, orderly turn-by-turn speaking, implying a significant conflict of speaking up or speaking simultaneously. The maximum value among all calculated overlap percentages is taken; if the maximum value is greater than 0.7, it is determined that multiple audio signals pointing to the same target coordinate constitute a speaking power conflict. The waiting prompt signal is generated by the conference control host and includes a conflict status identifier, the target coordinate, and a list of spatial location coordinates of the sound sources corresponding to each participating audio signal. The waiting prompt signal is pushed to the display interface of the conference control terminal, notifying the host of the current speaking power conflict and injecting a prompt tone into the microphone channel of each conflicting speaker. The notification tone is a short, single-frequency tone lasting 300 milliseconds and with a frequency of 800 Hz, with a volume 6 decibels lower than the normal speech sound pressure level, in order to avoid further interference with the meeting.

[0060] See Figure 5 In the graph, the horizontal axis represents time (in seconds), and the vertical axis represents the fundamental frequency of the speech signal (in Hertz). The legend corresponds to the fundamental frequency trajectories of speech signal 1, speech signal 2, and speech signal 3, represented by a solid blue line, a dashed red line, and a dotted orange line, respectively. All three fundamental frequency trajectories begin to show effective fundamental frequency values ​​around 0.6 seconds, with values ​​exceeding 100 Hertz, indicating that the three speech signals have begun to produce sound. The production range of speech signal 1 is approximately 0.6 to 3.8 seconds, with a fundamental frequency fluctuation between approximately 130 and 150 Hertz, exhibiting a relatively stable trend. The production range of speech signal 2 is approximately 0.9 to 4 seconds, with a fundamental frequency generally higher than speech signal 1, between approximately 140 and 165 Hertz, and a slightly higher fundamental frequency fluctuation, showing periodic fluctuations. The speech signal 3 has a range of approximately 1.2 to 4.5 seconds, the highest fundamental frequency, and a fluctuation range of approximately 150 Hz to 180 Hz. The fundamental frequency trajectory also exhibits a clear and rapid periodic change.

[0061] The fundamental frequency trajectories of the three speech signals overlap to some extent in their phonation intervals on the time axis, particularly in the interval between 1.2 seconds and 3.8 seconds. All three signals simultaneously possess valid fundamental frequency values, satisfying the prerequisite for the overlapping analysis of the fundamental frequency trajectories of multiple speech signals in Example 4. The different fluctuation characteristics of the fundamental frequency trajectories reflect the acoustic characteristics and intonation changes of different speakers, which is beneficial for subsequent detection and determination of discourse power conflicts through the calculation of the overlapping duration of the fundamental frequency trajectories.

[0062] Example 5:

[0063] In practice, the sliding average of the laryngeal tension coefficient within a predetermined time period is calculated. The predetermined time period is set to 2 seconds because the changes in laryngeal muscle tension during human speech are constrained by physiological regulatory mechanisms, and its fluctuation period is typically greater than 2 seconds. Using a 2-second sliding time window can smooth out instantaneous measurement noise and respond promptly to trend changes in tension. The laryngeal tension coefficient corresponding to each frame of speech signal within a continuous time period is stored in a first-in-first-out (FIFO) buffer queue with a queue capacity of [missing information]. By sampling interval The milliseconds and the predetermined time period of 2 seconds determine the value. Frame. Each new frame corresponds to a change in the laryngeal tension coefficient. Upon arrival, add it to the rear of the queue and remove the earliest laryngeal tension coefficient from the front of the queue. Calculate the arithmetic mean of all laryngeal tension coefficient values ​​in the queue to obtain the moving average of the laryngeal tension coefficients. .

[0064] The moving average of the laryngeal tension coefficient is mapped to a gain adjustment factor. Gain adjustment factor The calculation formula is:

[0065]

[0066] in, This represents the gain adjustment factor, which is dimensionless and its value ranges from 1 upwards. This represents the gain mapping coefficient, with a value of 0.8. This value is based on the fact that in a conference room acoustic environment, the gain adjustment range of a microphone preamplifier typically needs to cover dynamic changes of ±10 dB, while the laryngeal tension coefficient sliding average... In actual measurements, the distribution is in the 0 to 3 Hz range, and the setting is... This ensures that the gain adjustment factor is between 1 and 3.4, corresponding to a gain change of approximately 10.6 dB, which perfectly matches the linear adjustment capability of the preamplifier. This represents the sliding average value of the laryngeal tension coefficient, in Hertz, calculated using the sliding window method described above.

[0067] The dynamic gain value is obtained by multiplying the gain adjustment factor by the current microphone's preamplifier's reference gain value. The preset fixed amplification factor for the circuit, expressed in decibels. The value is 40 dB, which translates to approximately 100 times the linear gain. Dynamic gain value. Expressed as a linear multiple .

[0068] The preamplifier's gain is controlled by a dynamic gain value, and the gain in the low-frequency range is reduced when the dynamic gain value exceeds an upper threshold. The preamplifier's gain is achieved by a digital potentiometer or a variable gain amplifier, whose control register receives the decibel value corresponding to the dynamic gain value. The upper threshold is set to 60 dB, based on the gain value corresponding to the maximum linear output amplitude specified in the preamplifier's datasheet; exceeding 60 dB will cause harmonic distortion. At a given decibel level, the preamplifier's gain is directly set to 60 dB. Simultaneously, a first-order high-pass filter is applied to the acquired digital audio signal, with the cutoff frequency set to 200 Hz. Frequency components below 200 Hz are attenuated at a slope of 6 dB per octave. The magnitude of the low-frequency gain reduction is... Decibels, uniformly reducing this difference in the frequency band below 200 Hz. When the dynamic gain value does not exceed the upper limit threshold, directly set the preamplifier to... The gain value in decibels is not subject to any additional processing on the frequency band.

[0069] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An AI-based intelligent conference voice interaction system, characterized in that, The system includes: Multi-channel voice signals are collected by a distributed array of voice sensors deployed at the conference venue, and the time difference of arrival of each channel voice signal is analyzed to generate a sound source azimuth sequence. Based on the sound source azimuth sequence and the pressure distribution map output by the preset pressure sensing grid on the conference table, the abnormal angle values ​​in the sound source azimuth sequence are corrected to obtain the spatial positioning coordinates of the sound source. Based on the spatial positioning coordinates of the sound source, the formant frequency offset in the corresponding channel speech signal is extracted, and the speaker's laryngeal tension coefficient is generated according to the formant frequency offset. The length of the smoothing window of the fundamental frequency curve in the speech signal is adjusted using the laryngeal tension coefficient to generate the denoised speech fundamental frequency trajectory; Based on the temporal correspondence between the fundamental frequency trajectory of the speech and the peak values ​​of local pressure changes in the pressure distribution map, the current speaker's speech direction is determined, and a signal for switching the right to speak in the conference is generated based on the speech direction.

2. The AI-based intelligent conference voice interaction system according to claim 1, characterized in that, Multi-channel speech signals are acquired by a distributed array of speech sensors deployed at the conference venue. Time difference of arrival analysis is performed on the speech signals from each channel to generate a sound source azimuth sequence, specifically including: Cross-correlation calculation is performed on the voice signals received by each pair of adjacent sensors in the voice sensor array, and the peak position of the cross-correlation function is extracted as the arrival time difference. Based on the arrival time difference and the geometric spacing between the adjacent sensors, the plane incident angle of the sound source relative to the pair of sensors is calculated; The least-squares fit is performed on the calculated plane incident angles of all sensors in the voice sensor array to generate the sound source azimuth sequence.

3. The AI-based intelligent conference voice interaction system according to claim 2, characterized in that, Based on the sound source azimuth sequence and the pressure distribution map output by the preset pressure sensing grid on the conference table, abnormal angle values ​​in the sound source azimuth sequence are corrected to obtain the spatial positioning coordinates of the sound source, specifically including: Extract continuous grid regions from the pressure distribution map where the pressure value exceeds a preset contact threshold, and mark the geometric center of the continuous grid regions as candidate sitting posture pressure center points; Calculate the direction vector deviation of each angle value in the sound source azimuth sequence relative to the candidate sitting posture pressure center point; Abnormal angle values ​​whose direction vector deviation exceeds a preset angle deviation threshold are removed, and spatial triangulation is performed on the remaining angle values ​​to obtain the spatial positioning coordinates of the sound source.

4. The AI-based intelligent conference voice interaction system according to claim 3, characterized in that, Based on the spatial location coordinates of the sound source, the formant frequency shift in the corresponding channel speech signal is extracted, and the speaker's laryngeal tension coefficient is generated based on the formant frequency shift, specifically including: Using the sensor channel pointed to by the spatial positioning coordinates of the sound source as the target channel, linear predictive coding analysis is performed on the speech signal of the target channel to extract the frequency values ​​of the first three formants. The frequency offset of the first three formants is calculated by comparing the frequency values ​​of the first three formants with the pre-stored reference formant frequencies of the speaker in a neutral vocal state. The laryngeal tension coefficient is generated by summing the absolute values ​​of the resonant frequency offsets and dividing by the number of resonant peaks.

5. The AI-based intelligent conference voice interaction system according to claim 4, characterized in that, The reference formant frequencies are stored in the local cache of the conference according to the speaker's identity, and are initialized and updated through a short silent speech test before the conference begins.

6. The AI-based intelligent conference voice interaction system according to claim 4, characterized in that, The process of adjusting the smoothing window length of the fundamental frequency curve in the speech signal using the laryngeal tension coefficient to generate a denoised speech fundamental frequency trajectory specifically includes: The length of the smoothing window is set to be inversely proportional to the laryngeal tension coefficient; when the laryngeal tension coefficient increases, the length of the smoothing window is reduced. Based on the adjusted smoothing window length, the fundamental frequency curve of the speech signal is subjected to median filtering to remove peak noise points in the fundamental frequency curve; The fundamental frequency curve after median filtering is resampled into an equal time interval sequence along the time axis to generate the denoised speech fundamental frequency trajectory.

7. The AI-based intelligent conference voice interaction system according to claim 1, characterized in that, Based on the temporal correspondence between the fundamental frequency trajectory of the speech and the peak values ​​of local pressure changes in the pressure distribution map, the current speaker's speech direction is determined, and a signal for switching speaking rights is generated based on the speech direction, specifically including: The moment when the rise rate of the fundamental frequency value in the speech fundamental frequency trajectory exceeds the rise rate threshold is detected and marked as the starting point of the pitch rise; Synchronously retrieve the local pressure change peaks that appear in the pressure distribution map within a predetermined time window before and after the starting point of the tone rise, and extract the pressure growth direction vector of the peak. A reference direction vector is constructed with the spatial positioning coordinates of the sound source as the starting point and the geometric center point of the conference table as the ending point. The pressure growth direction vector is multiplied by the reference direction vector. The sign of the result determines whether the speech direction is pointing towards oneself or towards others, and a corresponding signal for switching the speaking rights in the meeting is generated.

8. The AI-based intelligent conference voice interaction system according to claim 7, characterized in that, The pressure growth direction vector is multiplied by the reference direction vector. The sign of the result determines whether the speaking direction is pointing towards oneself or others, and a corresponding signal for switching speaking rights is generated. Specifically, this includes: Starting from the spatial positioning coordinates of the sound source, and ending at the geometric center of the grid region where the local pressure change peak is located in the pressure distribution map, the pressure growth direction vector is constructed. Using the spatial positioning coordinates of the sound source as the origin and the geometric center point of the conference table as the reference endpoint, construct the direction vector of the line connecting the spatial positioning coordinates of the sound source; Calculate the dot product between the pressure growth direction vector and the direction vector connecting the spatial positioning coordinates of the sound source. If the dot product is positive, determine that the speech direction is pointing towards others, and generate a switching signal to transfer the speaking right to the seat corresponding to the direction pointed to by the pressure growth direction vector. If the dot product is negative, determine that the speech direction is pointing towards oneself, generate a hold signal to maintain the current speaking right, and output the hold signal to the conference control terminal to lock the current speaking channel.

9. The AI-based intelligent conference voice interaction system according to claim 1, characterized in that, The system also includes identifying power struggles in the meeting based on the direction of the discourse, specifically including: When there are two or more sound source spatial positioning coordinates that correspond to speech direction that both point to the same target coordinate, count the number of speech signals that point to the same target coordinate. When the number of voice signals exceeds the threshold for the number of conflicting individuals, the overlap duration of the fundamental frequency trajectory in each voice signal is extracted, and the overlap ratio between each pair is calculated. If the maximum value of the overlap ratio exceeds the overlap threshold, the current meeting status is marked as a power struggle state, and a waiting prompt signal is generated.

10. The AI-based intelligent conference voice interaction system according to claim 1, characterized in that, The system also includes dynamic adjustment of microphone sensitivity triggered by the laryngeal tension coefficient, specifically including: Calculate the moving average of the laryngeal tension coefficient over a predetermined time period, and map the moving average to a gain adjustment factor; Multiply the gain adjustment factor by the reference gain value of the current microphone's preamplifier to obtain the dynamic gain value; The amplification factor of the preamplifier is controlled according to the dynamic gain value, and the gain of the low-frequency band is reduced when the dynamic gain value exceeds the upper limit threshold.

Images