Human voice recognition method and system based on multi-person conference record

Abstract

The invention discloses a human voice recognition method and system based on multi-person conference records, and relates to the technical field of human voice recognition, and the method comprises the steps: collecting multi-person conference audio signals through a microphone based on a multi-person conference; framing processing is carried out according to the audio signals, and the short-time energy and the zero crossing rate of the audio signals in the conference are calculated. According to the method, through filtering processing, background noise can be remarkably suppressed, the definition and the discernibility of voice signals are improved, voice activities are analyzed through framing processing, short-time energy calculation and zero crossing rate calculation on audio signals, the accuracy and the efficiency of voice recognition and processing in conference records can be remarkably improved, and the voice recognition and processing efficiency is improved. Time domain signals can be accurately converted to a frequency domain through FFT, basic attributes of voice signals are accurately captured, overlapped voices of the voice signals are further recognized and classified in combination with data collection of a multi-microphone array, conference recognition accuracy is improved, and semantic features are combined.

Description

Technical Field

[0001] The present invention relates to the technical field of human voice recognition, and in particular to a human voice recognition method and system based on multi-person conference records. Background Art

[0002] Offline multi-person meetings can achieve efficient information exchange and decision-making through face-to-face communication. They are a common form of discussion and communication in companies and organizations. However, due to the complexity and length of the discussion content, it is usually necessary to record the meeting in order to summarize the results of the meeting. Traditional meeting record methods mainly rely on manual recording, which is not only inefficient but also prone to missing key information. In order to improve the efficiency and accuracy of meeting records, speech recognition technology is widely used in meeting record systems.

[0003] However, traditional speech recognition technology is mainly developed for single-person speech. When faced with complex conference scenarios with multiple people, it is often difficult to accurately distinguish the voices of different speakers, and it is unable to effectively deal with background noise and speech overlap. As a result, the existing speech recognition system has a relatively simple means of detecting and analyzing speech activities, and is unable to fully capture and utilize the various features in the speech signal, resulting in low recognition accuracy. Summary of the invention

[0004] The present invention aims to provide a method and system for voice recognition based on multi-person conference records to solve the above problems.

[0005] The present invention is achieved through the following technical solutions:

[0006] A human voice recognition method and system based on multi-person conference records, including:

[0007] Based on multi-person conference, audio signals of multi-person conference are collected through microphone;

[0008] Perform frame processing based on the audio signal, calculate the short-time energy and zero-crossing rate of the audio signal in the meeting, and analyze the voice activity;

[0009] Based on the voice activity audio signal, the time domain signal is converted to the frequency domain, discrete cosine transform is performed and voiceprint features are captured;

[0010] Based on the microphone array, the speech is separated according to the calculated probability, the semantic features are extracted by converting the audio signal into text, and the speaker characteristics are identified by combining the voiceprint feature clustering;

[0011] Construct a visual scatter plot to analyze the clustering results, mark the speaker ID, generate a meeting report and send it.

[0012] As a preferred solution of the human voice recognition method based on multi-person conference records of the present invention, wherein: the multi-person conference audio signal is collected through a microphone based on the multi-person conference, including:

[0013] Use conference recording microphones to collect audio signals from multi-person conferences;

[0014] The collected audio signal is subjected to noise reduction and gain control, and the background noise is removed by using the Wiener filter, which is expressed as:

[0015]

[0016] in represents the filtered signal, P s (f) represents the signal power spectrum, P N (f) represents the noise power spectrum, and X(f) represents the original audio signal.

[0017] As a preferred solution of the human voice recognition method based on multi-person conference records of the present invention, wherein: the frame processing is performed according to the audio signal, the short-time energy and zero crossing rate of the audio signal in the conference are calculated, and the voice activity is analyzed, including:

[0018] The audio signal is processed by dividing into frames, each frame contains N sampling points;

[0019] Apply the Hanning window function to each frame of audio signal;

[0020] The short-term energy is calculated to determine the period of speech activity, expressed as:

[0021]

[0022] Where E(n) represents the short-time energy of the nth frame, nT represents the starting sampling point of the nth frame, m represents the mth sampling point in the frame, and T represents the offset of the starting position of each frame relative to the previous frame;

[0023] For each frame of speech signal, the zero crossing rate of each frame is calculated, which is expressed as:

[0024]

[0025] Where Z(n) represents the zero crossing rate of the nth frame, and sgn(x) represents the sign function;

[0026] Selecting the average value of the short-time energy of the audio signal as the short-time energy threshold, selecting the average value of the zero-crossing rate of the audio signal as the zero-crossing rate threshold, if the calculated Z(n) is less than or equal to the zero-crossing rate threshold, and the calculated E(n) is greater than or equal to the short-time energy threshold, it is determined that voice activity is contained;

[0027] The audio signals determined to contain voice activities are grouped into a voice activity set.

[0028] As a preferred solution of the human voice recognition method based on multi-person conference records of the present invention, wherein: based on the voice activity audio signal, the time domain signal is converted into the frequency domain, a discrete cosine transform is performed and the voiceprint features are captured, including:

[0029] The speech signal is pre-emphasized based on the audio signal of the speech activity set, which is expressed as:

[0030] y(t)=x(t)-αx(t-1);

[0031] Where y(t) represents the t-th audio signal of pre-emphasis, x(t) represents the t-th audio signal containing voice activity, and α represents the pre-emphasis coefficient;

[0032] The pre-emphasized audio signal is divided into different frames, each frame includes N sampling points, and the Hamming window function is applied to each frame of the audio signal, which is expressed as:

[0033] y b (j) = y(bH+j)·w(j);

[0034] where y b (j) represents the signal value of the jth windowed sampling point in the bth frame, H represents the offset of the starting position of each frame relative to the previous frame, bH represents the starting sampling point of the bth frame, and b represents the bth sampling point in the frame;

[0035] The windowed signal of each frame is transformed into a frequency domain signal by fast Fourier transform, which is expressed as:

[0036]

[0037] Where X b (k) represents the kth frequency component of the bth frame frequency domain signal, i represents an imaginary unit, and J represents the number of sampling points contained in each frame;

[0038] Map the frequency domain signal X(k) to the Mel frequency scale and apply the Mel filter, expressed as:

[0039]

[0040] Where X mel (d) represents the output of the d-th Mel filter, H d (k) represents the response value of the dth Mel filter at frequency k;

[0041] Calculate the logarithmic energy of each Mel filter output, and perform discrete cosine transform based on the logarithmic energy to calculate MFCC, expressed as:

[0042]

[0043] Where MFCC(c) represents the cth Mel-frequency cepstral coefficient, and D represents the total number of Mel filters.

[0044] As a preferred solution of the human voice recognition method based on multi-person conference records of the present invention, wherein: the voice separation based on the microphone array according to the calculated probability includes:

[0045] The time information of the audio signal is aligned and unified based on the calculated MFCC (c) as the voiceprint feature;

[0046] Use multiple microphones to form a microphone array and calculate beamforming, expressed as:

[0047]

[0048] Where y(t) represents the output signal after beamforming, w r represents the weight of the i-th microphone, x r represents the audio signal received by the rth microphone, τ r represents the time delay of the rth microphone, and R represents the total number of microphones;

[0049] The MFCC features extracted from each microphone are fused to form a microphone feature vector, which is expressed as:

[0050] F m =[F1,F2,…,F r ];

[0051] where F m represents the microphone feature vector, F r represents the MFCC voiceprint feature of the rth microphone;

[0052] Construct a Gaussian mixture model GMM to model the feature vector, expressed as:

[0053]

[0054] Where p(xλ) represents the probability density function of the input feature vector, w k represents the kth Gaussian component weight, μ k is the mean of the kth Gaussian component, κ k is the covariance of the kth Gaussian component, K represents the total number of sound categories;

[0055] Use the expectation maximization EM algorithm to train GMM and calculate the responsibility value of each data point belonging to each Gaussian component under the previous parameters, expressed as:

[0056]

[0057] where r nk Indicates the responsibility value that the nth data point belongs to k;

[0058] Then update the parameters according to the maximization step and complete w respectively. k , μ k and μ k Parameter update of

[0059] For each time frame, the MFCC feature vector x t , calculate the probability of belonging to each Gaussian component under the trained GMM model, expressed as:

[0060]

[0061] where r tk Indicates the probability that the MFCC feature vector of the tth frame belongs to k;

[0062] The maximum a posteriori probability method MAP is used to classify each time frame, expressed as:

[0063]

[0064] where k * Indicates the most likely sound category k to which the audio signal of the tth frame belongs;

[0065] The audio signal that completes the speech classification is composed into voiceprint features.

[0066] As a preferred solution of the human voice recognition method based on multi-person conference records of the present invention, wherein: the extraction of semantic features based on the conversion of audio signals into text and the identification of speaker features in combination with voiceprint feature clustering include:

[0067] The filtered audio signal is converted into text through automatic speech recognition ASR technology. The deep semantic feature vector is extracted from the converted text using the BERT model and fused with the classified voiceprint features and semantic features, expressed as:

[0068] F z =[F M ,F Y ];

[0069] where F z represents the comprehensive feature vector, F M Indicates the voiceprint feature, F Y Represents the semantic features extracted by BERT;

[0070] The comprehensive feature vector is Z-score normalized, and the normalized comprehensive feature vector is clustered using the K-means algorithm to identify different speakers in the meeting, which is expressed as:

[0071]

[0072] Where S represents the objective function of clustering, q represents the number of clusters, and C i represents all the data of the i-th cluster, x represents the data point of the comprehensive feature vector, μ i represents the center of the ith cluster.

[0073] As a preferred solution of the human voice recognition method based on multi-person conference records of the present invention, wherein: the construction of a visual scatter plot to analyze the clustering results, and marking the speaker ID, generating a conference report and sending the report includes,

[0074] It means initializing the speaker ID according to the cluster calculation, assigning the speaker ID to each cluster center, traversing the data points in each cluster, setting the speaker ID of the data point to the corresponding cluster ID, and using Python's matplotlib and seaborn libraries to draw a cluster scatter plot. In the scatter plot, the data points are distinguished by color according to their clusters, and the center position of each cluster is marked;

[0075] Based on speaker ID labeling and cluster analysis, summarize the frequency, duration, sentiment analysis and key topics of each speaker's speech, use Python and Pandas libraries to process and analyze the clustered speech data, extract the key information of each speaker, use Jupyter Notebook to design a report template, integrate text descriptions, data statistics and charts, and generate a detailed meeting report;

[0076] Send the generated reports via SFTP, use PGP for encrypted transmission, and store the data and generated reports in a cloud server.

[0077] Another object of the present invention is to provide a system for voice recognition method based on multi-person conference records, which comprises:

[0078] Audio signal acquisition module, which collects audio signals of multi-person conferences through microphones;

[0079] The voice activity detection module processes the collected audio signal by frames and calculates the short-time energy and zero-crossing rate to analyze the voice activity;

[0080] The voiceprint feature extraction module converts the time domain signal into the frequency domain based on the audio signal of speech activity, calculates the probability that each audio frame belongs to each Gaussian component, performs speech separation, and extracts voiceprint features;

[0081] Semantic feature extraction module, which converts audio signals into text and extracts semantic features from the text

[0082] The speaker recognition module combines voiceprint features and semantic features and uses a clustering algorithm to identify different speakers.

[0083] Visualization module, builds a visual scatter plot to analyze clustering results;

[0084] The report generation module generates a meeting report based on the audio signal and speaker identification of the meeting.

[0085] A computer device comprises: a memory and a processor; the memory stores a computer program, and the processor implements the steps of the above-mentioned method for human voice recognition based on multi-person conference records when executing the computer program.

[0086] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for human voice recognition based on multi-person conference records.

[0087] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0088] 1. The present invention can significantly suppress background noise and improve the clarity and discernibility of speech signals through filtering processing. By performing frame processing on audio signals and calculating short-time energy and zero-crossing rate to analyze speech activities, the accuracy and efficiency of speech recognition and processing in conference records can be significantly improved.

[0089] 2. The present invention can accurately convert the time domain signal into the frequency domain through FFT, accurately capture the basic properties of the voice signal, and further identify and classify the overlapping voices of the voice signal in combination with the data collection of the multi-microphone array, thereby improving the accuracy of conference recognition and combining semantic features;

[0090] 3. The present invention uses the information of speech content to enhance the recognition ability of speakers through repeated allocation and updating of cluster centers by clustering algorithms, thereby improving the accuracy and robustness of speaker recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0092] Figure 1 The flowchart of the human voice recognition method based on multi-person conference records is shown in FIG.

[0093] Figure 2This is a structural diagram of a voice recognition system based on multi-person conference records. DETAILED DESCRIPTION

[0094] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and drawings. The schematic implementation modes and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention. It should be noted that the present invention is already in the actual development and use stage.

[0095] Example 1

[0096] Reference Figure 1 , which is the first embodiment of the present invention, and provides a method for human voice recognition based on multi-person conference records. The method for human voice recognition based on multi-person conference records includes:

[0097] S1, based on a multi-person conference, collects audio signals of the multi-person conference through a microphone;

[0098] Preferably, collecting audio signals of a multi-person conference through a microphone based on the multi-person conference includes:

[0099] Use conference recording microphones to collect audio signals from multi-person conferences;

[0100] The collected audio signal is subjected to noise reduction and gain control, and the background noise is removed by using the Wiener filter, which is expressed as:

[0101]

[0102] in represents the filtered signal, P s (f) represents the signal power spectrum, P N (f) represents the noise power spectrum, and X(f) represents the original audio signal.

[0103] Through filtering processing, background noise can be significantly suppressed and the clarity and discernibility of voice signals can be improved. Especially in noisy conference environments, interference noises such as air conditioning noise and keyboard tapping can be effectively removed, making the speaker's voice clearer. At the same time, the Wiener filter can not only remove noise, but also retain and enhance useful voice signal components, so that the quality of the signal in a noisy environment can be guaranteed. By collecting and processing audio signals from multi-person conferences, especially using Wiener filters for noise reduction and gain control, the quality of audio signals can be significantly improved, thereby improving the accuracy and robustness of subsequent voice processing and analysis.

[0104] S2, performs frame processing according to the audio signal, calculates the short-time energy and zero-crossing rate of the audio signal in the meeting, and analyzes the voice activity;

[0105] Preferably, the audio signal is framed, the short-time energy and zero-crossing rate of the audio signal in the conference are calculated, and the voice activity is analyzed, including:

[0106] The audio signal is processed by dividing into frames, each frame contains N sampling points;

[0107] Apply the Hanning window function w(m) to each frame of audio signal, expressed as:

[0108]

[0109] Where w(m) represents the window function value of the mth sampling point, and N represents the number of sampling points contained in each frame;

[0110] The short-term energy is calculated to determine the period of speech activity, expressed as:

[0111]

[0112] Where E(n) represents the short-time energy of the nth frame, nT represents the starting sampling point of the nth frame, x(nT+m) represents the value of the mth sampling point of the nth frame, m represents the mth sampling point in the frame, and T represents the offset of the starting position of each frame relative to the previous frame;

[0113] For each frame of speech signal, the zero crossing rate of each frame is calculated, which is expressed as:

[0114]

[0115] Where Z(n) represents the zero crossing rate of the nth frame, sgn(x) represents the sign function, which returns 1 when x is greater than or equal to zero and returns -1 when x is less than zero;

[0116] Selecting the average value of the short-time energy of the audio signal as the short-time energy threshold, selecting the average value of the zero-crossing rate of the audio signal as the zero-crossing rate threshold, if the calculated Z(n) is less than or equal to the zero-crossing rate threshold, and the calculated E(n) is greater than or equal to the short-time energy threshold, it is determined that voice activity is contained;

[0117] The audio signals determined to contain voice activities are grouped into a voice activity set.

[0118] By framing the audio signal and calculating the short-time energy and zero-crossing rate to analyze voice activity, the accuracy and efficiency of voice recognition and processing in conference records can be significantly improved. The Hanning window function can smooth the boundaries of each frame signal, reduce spectrum leakage, and improve the accuracy of frequency domain analysis. Framing and windowing can effectively separate the noise in the voice signal to ensure that subsequent analysis is more accurate. By calculating the short-time energy, the time period of voice activity can be accurately identified to ensure that every moment of speaking is captured. Since the short-time energy can effectively distinguish between voice signals and background noise, voice activity can still be accurately detected in a noisy environment. The zero-crossing rate can reflect the frequency of change of the voice signal and help distinguish different types of sounds. Combining the short-time energy and the zero-crossing rate, the voice activity period can be more accurately identified, especially when processing voice signals with high-frequency components. Finally, by combining the dual judgment of short-time energy and zero-crossing rate, the possibility of misjudgment is greatly reduced and the reliability of voice activity detection is improved.

[0119] S3, based on the voice activity audio signal, converts the time domain signal into the frequency domain, performs discrete cosine transform and captures the voiceprint features;

[0120] Preferably, based on the voice activity audio signal, the time domain signal is converted into the frequency domain, a discrete cosine transform is performed and the voiceprint features are captured, including:

[0121] The speech signal is pre-emphasized based on the audio signal of the speech activity set, which is expressed as:

[0122] y(t)=x(t)-αx(t-1);

[0123] Where y(t) represents the t-th audio signal with pre-emphasis, x(t) represents the t-th audio signal containing voice activity, and α represents the pre-emphasis coefficient, which is usually 0.95;

[0124] The pre-emphasized audio signal is divided into different frames, each frame includes N sampling points, and the Hamming window function is applied to each frame of the audio signal, which is expressed as:

[0125]

[0126] y b (j) = y(bH+j)·w(j);

[0127] Where w(j) represents the window function value of the jth sampling point, y b (j) represents the signal value of the jth windowed sampling point in the bth frame, H represents the offset of the starting position of each frame relative to the previous frame, bH represents the starting sampling point of the bth frame, y(bH+j) represents the value of the jth sampling point of the bth frame in the pre-emphasized signal, and b represents the bth sampling point in the frame;

[0128] The windowed signal of each frame is transformed into a frequency domain signal by fast Fourier transform, which is expressed as:

[0129]

[0130] Where X b (k) represents the kth frequency component of the bth frame frequency domain signal, i represents an imaginary unit, and J represents the number of sampling points contained in each frame;

[0131] Map the frequency domain signal X(k) to the Mel frequency scale and apply the Mel filter, expressed as:

[0132]

[0133] Where X mel (d) represents the output of the d-th Mel filter, H d (k) represents the response value of the dth Mel filter at frequency k;

[0134] Calculate the logarithmic energy of each Mel filter output, and perform discrete cosine transform based on the logarithmic energy to calculate MFCC, expressed as:

[0135]

[0136] Where MFCC(c) represents the cth Mel frequency cepstral coefficient, and D represents the total number of Mel filters;

[0137] The speech signal is preprocessed, framed, windowed, fast Fourier transformed, Mel frequency filtered and log energy transformed, and finally converted into Mel frequency cepstral coefficients (MFCC) through discrete cosine transform (DCT). These coefficients capture the basic properties of speech signals such as pitch, timbre and intensity.

[0138] Pre-emphasis processing can enhance the high-frequency components in the speech signal and weaken the low-frequency parts, so as to better retain and analyze the detailed information in the speech and improve the expressiveness of the high-frequency signal. It can also reduce the distortion that may occur in the transmission and processing of the speech signal to ensure the signal quality. The Hamming window function can smooth the signal boundary, reduce spectrum leakage, improve the accuracy of frequency domain analysis, and make the spectrum components more accurate. FFT can accurately convert the time domain signal to the frequency domain to capture the spectrum characteristics of the signal and provide high-quality frequency domain data for subsequent feature extraction. The Mel frequency filter can simulate the human ear's perception characteristics of different frequencies, so that the extracted features are more in line with the human ear's hearing characteristics and improve the effectiveness of speech recognition and analysis. In this way, the basic attributes of the speech signal, such as pitch, timbre and sound intensity, can be effectively captured. These attributes are very useful for distinguishing different speakers and improving the accuracy of speaker recognition. It can not only effectively improve the quality of the speech signal, but also accurately capture and distinguish the speech characteristics of different speakers, providing a solid technical guarantee for multi-person conference recording and speech recognition.

[0139] S4, based on the microphone array, performs speech separation according to the calculated probability, extracts semantic features according to the conversion of audio signals into text, and identifies speaker features by combining voiceprint feature clustering;

[0140] Furthermore, speech separation is performed based on the microphone array according to the calculated probability, including:

[0141] The time information of the audio signal is aligned and unified based on the calculated MFCC (c) as the voiceprint feature;

[0142] Use multiple microphones to form a microphone array and calculate beamforming, expressed as:

[0143]

[0144] Where y(t) represents the output signal after beamforming, w r represents the weight of the i-th microphone, x r represents the audio signal received by the rth microphone, τ r represents the time delay of the rth microphone, and R represents the total number of microphones;

[0145] The MFCC features extracted from each microphone are fused to form a microphone feature vector, which is expressed as:

[0146] F m =[F1,F2,…,F r ];

[0147] where F m represents the microphone feature vector, F r represents the MFCC voiceprint feature of the rth microphone;

[0148] Construct a Gaussian mixture model GMM to model the feature vector, expressed as:

[0149]

[0150] Where p(xλ) represents the probability density function of the input feature vector, w k represents the kth Gaussian component weight, Ν(xμ k ,κ k ) represents the normal distribution of the kth Gaussian component, μ k is the mean of the k-th Gaussian component, representing the average MFCC feature of the k-th sound category, κ k is the covariance of the k-th Gaussian component, which represents the relationship between the variability and characteristic of MFCC in the k-th sound category, and K represents the total number of sound categories;

[0151] Use the expectation maximization EM algorithm to train GMM and calculate the responsibility value of each data point belonging to each Gaussian component under the previous parameters, expressed as:

[0152]

[0153] where r nk Indicates the responsibility value that the nth data point belongs to k;

[0154] Then update the parameters according to the maximization step, expressed as:

[0155]

[0156] Where N k Represents the number of valid data points of the kth Gaussian component, N o Indicates the total number of data points;

[0157] Complete w k , μ k and μ k Parameter update of

[0158] For each time frame, the MFCC feature vector x t , calculate the probability of belonging to each Gaussian component under the trained GMM model, expressed as:

[0159]

[0160] where r tk Indicates the probability that the MFCC feature vector of the tth frame belongs to k;

[0161] The maximum a posteriori probability method MAP is used to classify each time frame, expressed as:

[0162]

[0163] where k * Indicates the most likely sound category k to which the audio signal of the tth frame belongs;

[0164] The audio signal that has completed the speech classification is formed into a voiceprint feature;

[0165] Calculate the feature vector x for each time frame t The probability of different Gaussian components is used to classify and separate. This probability-based method can make full use of the statistical characteristics of speech signals, improve the accuracy of separation, and calculate the responsibility value r. nk It can represent the probability that each time frame belongs to different Gaussian components, which is the key decision basis in the separation process. By calculating the maximum responsibility value of each time frame, it can be assigned to the most likely speaker, achieving effective speech separation and avoiding the overlap of audio signals.

[0166] Through the combination of multiple Gaussian distributions of GMM, complex speech feature distribution can be effectively represented, so as to distinguish between multiple speakers. The EM algorithm and the maximum a posteriori probability method ensure the efficiency and accuracy of parameter estimation and speech separation, which is suitable for real-time processing scenarios. The probability-based separation method maximizes the integrity and continuity of the original speech signal, and has strong anti-noise ability. Through GMM training and speech separation, the separation of multi-speaker speech signals can be achieved efficiently and accurately, helping to identify different speakers and classify speech signals. At the same time, data collection based on microphone arrays can make the audio collection of meetings more sufficient and can adapt to the different number of participants in different meetings, thereby improving the adaptability of voice recognition for meeting audio. Further, through speech separation, the situation when multiple people speak simultaneously can be effectively identified, thereby improving the accuracy and precision of meeting records.

[0167] Preferably, converting the audio signal into text to extract semantic features, combining voiceprint feature clustering to identify speaker features, includes:

[0168] The filtered audio signal is converted into text through automatic speech recognition ASR technology. Google Speech-to-Text service can be used. The BERT model is used to extract deep semantic feature vectors from the converted text and merge them with the classified voiceprint features and semantic features, which are expressed as:

[0169] F z =[F M ,F Y ];

[0170] where F z represents the comprehensive feature vector, FM Indicates the voiceprint feature, F Y Represents the semantic features extracted by BERT;

[0171] The comprehensive feature vector is Z-score normalized, and the normalized comprehensive feature vector is clustered using the K-means algorithm to identify different speakers in the meeting, which is expressed as:

[0172]

[0173] Where S represents the loss function of the clustering result, q represents the number of clusters, and C i represents all the data of the i-th cluster, x represents the data point of the comprehensive feature vector, μ i Represents the center of the i-th cluster, which is the average position of all points in the cluster;

[0174] In the calculation, for each data point of the feature vector, its relationship with each cluster center μ is calculated. i distance, and assign x to the cluster C represented by the nearest cluster center i , and then recalculate the center μ of the cluster i is the average position of all points, and the assignment and update steps are repeated until the cluster center μ i If the position no longer changes significantly, stop updating and complete the aggregation calculation.

[0175] By converting speech into text and extracting semantic features, the system can not only analyze the speaker's voiceprint features, but also use the information of the speech content to enhance the recognition ability of the speaker. The BERT model can capture the contextual relationship and semantic information in the text, supplement the semantic content that is difficult to capture by voiceprint features alone, and improve the recognition accuracy. By fusing features from different sources, the system can better distinguish different speakers. Even in the case of similar speech features, semantic features can provide additional distinguishing information. The K-means algorithm has high computational efficiency and is suitable for processing large-scale data. By repeatedly assigning and updating cluster centers, different speakers can be effectively identified. The combination of standardization and clustering ensures that the feature points of each speaker are accurately clustered, which significantly improves the accuracy and robustness of speaker recognition.

[0176] S5, construct a visual scatter plot to analyze the clustering results, mark the speaker ID, generate a meeting report and send the report;

[0177] Preferably, a visual scatter plot is constructed to analyze the clustering results, and the speaker ID is marked, and a meeting report is generated and sent, including:

[0178] It means initializing the speaker ID according to the cluster calculation, assigning the speaker ID to each cluster center, traversing the data points in each cluster, setting the speaker ID of the data point to the corresponding cluster ID, and using Python's matplotlib and seaborn libraries to draw a cluster scatter plot. In the scatter plot, the data points are distinguished by color according to their clusters, and the center position of each cluster is marked;

[0179] Based on speaker ID labeling and cluster analysis, summarize the frequency, duration, sentiment analysis and key topics of each speaker's speech, use Python and Pandas libraries to process and analyze the clustered speech data, extract the key information of each speaker, use Jupyter Notebook to design a report template, integrate text descriptions, data statistics and charts, and generate a detailed meeting report;

[0180] Send the generated reports via SFTP, use PGP for encrypted transmission, and store the data and generated reports in a cloud server.

[0181] Through traversal and labeling, we ensure that each data point is correctly assigned to the corresponding speaker ID, thereby improving the accuracy and consistency of the data. Through color differentiation and center labeling, the scatter plot can intuitively display the distribution of different clusters and speakers, which is convenient for analyzing and understanding the clustering results. Through the intuitive display of clustering results, users can more easily find problems and formulate corresponding strategies to improve the accuracy and efficiency of decision-making. Through detailed statistics and analysis, we can accurately record the frequency and duration of each speaker's speech to ensure the accuracy of the meeting minutes. Through the design of a reasonable report template, we ensure that the report structure is clear, the content is coherent, and it is easy to read and understand. Through PGP encrypted transmission, we ensure that the report is not intercepted and read by unauthorized third parties during the transmission process, thereby improving the security of data transmission.

[0182] Example 2, reference Figure 2 , which is the second embodiment of the present invention, and which is different from the previous embodiment, provides a system for voice recognition method based on multi-person conference records, including:

[0183] Audio signal acquisition module, which collects audio signals of multi-person conferences through microphones;

[0184] The voice activity detection module processes the collected audio signal by frames and calculates the short-time energy and zero-crossing rate to analyze the voice activity;

[0185] The voiceprint feature extraction module converts the time domain signal into the frequency domain and extracts the voiceprint features based on the audio signal of the speech activity;

[0186] Semantic feature extraction module, which converts audio signals into text and extracts semantic features from the text

[0187] The speaker recognition module combines voiceprint features and semantic features and uses a clustering algorithm to identify different speakers.

[0188] Visualization module, builds a visual scatter plot to analyze clustering results;

[0189] The report generation module generates a meeting report based on the audio signal and speaker identification of the meeting.

[0190] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0191] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0192] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0193] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0194] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for human voice recognition based on multi-person conference records, comprising: Based on multi-person conference, audio signals of multi-person conference are collected through microphone; Perform frame processing based on the audio signal, calculate the short-time energy and zero-crossing rate of the audio signal in the meeting, and analyze the voice activity; Based on the voice activity audio signal, the time domain signal is converted to the frequency domain, discrete cosine transform is performed and voiceprint features are captured; Based on the microphone array, the speech is separated according to the calculated probability, the semantic features are extracted by converting the audio signal into text, and the speaker characteristics are identified by combining the voiceprint feature clustering; Construct a visual scatter plot to analyze the clustering results, mark the speaker ID, generate a meeting report and send it.

2. The method for human voice recognition based on multi-person conference records according to claim 1, characterized in that: The method of collecting audio signals of a multi-person conference through a microphone based on a multi-person conference includes: Use conference recording microphones to collect audio signals from multi-person conferences; The collected audio signal is subjected to noise reduction and gain control, and the background noise is removed by using the Wiener filter, which is expressed as: in represents the filtered signal, P s (f) represents the signal power spectrum, P N (f) represents the noise power spectrum, and X(f) represents the original audio signal.

3. The method for human voice recognition based on multi-person conference records according to claim 2 is characterized in that: The method of performing frame processing according to the audio signal, calculating the short-time energy and zero-crossing rate of the audio signal in the conference, and analyzing the voice activity includes: The audio signal is processed by dividing into frames, each frame contains N sampling points; Apply the Hanning window function to each frame of audio signal; The short-term energy is calculated to determine the period of speech activity, expressed as: Where E(n) represents the short-time energy of the nth frame, nT represents the starting sampling point of the nth frame, m represents the mth sampling point in the frame, and T represents the offset of the starting position of each frame relative to the previous frame; For each frame of speech signal, the zero crossing rate of each frame is calculated, which is expressed as: Where Z(n) represents the zero crossing rate of the nth frame, and sgn(x) represents the sign function; Selecting the average value of the short-time energy of the audio signal as the short-time energy threshold, selecting the average value of the zero-crossing rate of the audio signal as the zero-crossing rate threshold, if the calculated Z(n) is less than or equal to the zero-crossing rate threshold, and the calculated E(n) is greater than or equal to the short-time energy threshold, it is determined that voice activity is contained; The audio signals determined to contain voice activities are grouped into a voice activity set.

4. The method for human voice recognition based on multi-person conference records according to claim 3 is characterized in that: In the method, based on the voice activity audio signal, the time domain signal is converted into the frequency domain, a discrete cosine transform is performed and the voiceprint features are captured, including: The speech signal is pre-emphasized based on the audio signal of the speech activity set, which is expressed as: y(t)=x(t)-αx(t-1); Where y(t) represents the t-th audio signal of pre-emphasis, x(t) represents the t-th audio signal containing voice activity, and α represents the pre-emphasis coefficient; The pre-emphasized audio signal is divided into different frames, each frame includes N sampling points, and the Hamming window function is applied to each frame of the audio signal, which is expressed as: y b (j)=y(bH+j)·w(j); where y b (j) represents the signal value of the jth windowed sampling point in the bth frame, H represents the offset of the starting position of each frame relative to the previous frame, bH represents the starting sampling point of the bth frame, and b represents the bth sampling point in the frame; The windowed signal of each frame is transformed into a frequency domain signal by fast Fourier transform, which is expressed as: Where X b (k) represents the kth frequency component of the bth frame frequency domain signal, i represents an imaginary unit, and J represents the number of sampling points contained in each frame; Map the frequency domain signal X(k) to the Mel frequency scale and apply the Mel filter, expressed as: Where X mel (d) represents the output of the d-th Mel filter, H d (k) represents the response value of the dth Mel filter at frequency k; Calculate the logarithmic energy of each Mel filter output, and perform discrete cosine transform based on the logarithmic energy to calculate MFCC, expressed as: Where MFCC(c) represents the cth Mel-frequency cepstral coefficient, and D represents the total number of Mel filters.

5. The method for human voice recognition based on multi-person conference records according to claim 3 is characterized in that: The method performs speech separation based on the microphone array according to the calculated probability, include, The time information of the audio signal is aligned and unified based on the calculated MFCC (c) as the voiceprint feature; Use multiple microphones to form a microphone array and calculate beamforming, expressed as: Where y(t) represents the output signal after beamforming, w r represents the weight of the i-th microphone, x r represents the audio signal received by the rth microphone, τ r represents the time delay of the rth microphone, and R represents the total number of microphones; The MFCC features extracted from each microphone are fused to form a microphone feature vector, which is expressed as: F m =[F1,F2,…,F r ]; where F m represents the microphone feature vector, F r represents the MFCC voiceprint feature of the rth microphone; Construct a Gaussian mixture model GMM to model the feature vector, expressed as: Where p(x|λ) represents the probability density function of the input feature vector, w k represents the kth Gaussian component weight, μ k is the mean of the kth Gaussian component, κ k is the covariance of the kth Gaussian component, K represents the total number of sound categories; Use the expectation maximization EM algorithm to train GMM and calculate the responsibility value of each data point belonging to each Gaussian component under the previous parameters, expressed as: where r nk Indicates the responsibility value that the nth data point belongs to k; Then update the parameters according to the maximization step and complete w respectively. k , μ k and μ k Parameter update of For each time frame, the MFCC feature vector x t , calculate the probability of belonging to each Gaussian component under the trained GMM model, expressed as: where r tk Indicates the probability that the MFCC feature vector of the tth frame belongs to k; The maximum a posteriori probability method MAP is used to classify each time frame, expressed as: where k * Indicates the most likely sound category k to which the audio signal of the tth frame belongs; The audio signal that completes the speech classification is composed into voiceprint features.

6. The method for human voice recognition based on multi-person conference records according to claim 5, characterized in that: The method of converting the audio signal into text to extract semantic features and clustering the voiceprint features to identify speaker features includes: The filtered audio signal is converted into text through automatic speech recognition ASR technology. The deep semantic feature vector is extracted from the converted text using the BERT model and fused with the classified voiceprint features and semantic features, expressed as: F z =[F M ,F Y ]; where F z represents the comprehensive feature vector, F M Indicates the voiceprint feature, F Y Represents the semantic features extracted by BERT; The comprehensive feature vector is Z-score normalized, and the normalized comprehensive feature vector is clustered using the K-means algorithm to identify different speakers in the meeting, which is expressed as: Where S represents the objective function of clustering, q represents the number of clusters, and C i represents all the data of the i-th cluster, x represents the data point of the comprehensive feature vector, μ i represents the center of the ith cluster.

7. The method for human voice recognition based on multi-person conference records according to claim 5, characterized in that: The construction of a visual scatter plot to analyze the clustering results, annotate the speaker ID, generate a meeting report and send the report, including: It means initializing the speaker ID according to the cluster calculation, assigning the speaker ID to each cluster center, traversing the data points in each cluster, setting the speaker ID of the data point to the corresponding cluster ID, and using Python's matplotlib and seaborn libraries to draw a cluster scatter plot. In the scatter plot, the data points are distinguished by color according to their clusters, and the center position of each cluster is marked; Based on speaker ID labeling and cluster analysis, summarize the frequency, duration, sentiment analysis and key topics of each speaker's speech, use Python and Pandas libraries to process and analyze the clustered speech data, extract the key information of each speaker, use Jupyter Notebook to design a report template, integrate text descriptions, data statistics and charts, and generate a detailed meeting report; Send the generated reports via SFTP, use PGP for encrypted transmission, and store the data and generated reports in a cloud server.

8. A system based on the method for human voice recognition based on multi-person conference records according to any one of claims 1 to 7, characterized in that: include, Audio signal acquisition module, which collects audio signals of multi-person conferences through microphones; The voice activity detection module processes the collected audio signal by frames and calculates the short-time energy and zero-crossing rate to analyze the voice activity; The voiceprint feature extraction module converts the time domain signal into the frequency domain based on the audio signal of speech activity, calculates the probability that each audio frame belongs to each Gaussian component, performs speech separation, and extracts voiceprint features; Semantic feature extraction module, which converts audio signals into text and extracts semantic features from the text The speaker recognition module combines voiceprint features and semantic features and uses a clustering algorithm to identify different speakers. Visualization module, builds a visual scatter plot to analyze clustering results; The report generation module generates a meeting report based on the audio signal and speaker identification of the meeting.

9. A computer device comprising: Memory and processor; The memory stores a computer program, characterized in that when the processor executes the computer program, the steps of the method for human voice recognition based on multi-person conference records described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for human voice recognition based on multi-person conference records described in any one of claims 1 to 7 are implemented.

Images