A digital conference data processing method and system

Abstract

The present application provides a digital conference data processing method and system, including: extracting speech feature parameters such as speech pause duration, speech pause frequency, stress word occurrence position, and stress word occurrence frequency in an audio stream to obtain a speech feature vector; performing time series analysis on the speech feature vector to identify key feature data of intonation increase amplitude, intonation decrease amplitude, and intonation stability change in the speech to obtain a tone type label sequence; using a recurrent neural network model, taking the speech feature vector and the tone type label sequence as input, and taking the tone representation vector as output; parsing the tone representation vector based on a business negotiation tone-emotion mapping model, and if the emotional fluctuation amplitude of the participants exceeds a threshold, a mark of the appearance of a key decision node or the introduction of a controversial topic is triggered.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a digital conference data processing method and system. Background Art

[0002] In business negotiation meetings, how to accurately understand and analyze the rich information contained in voice data is a complex and critical technical challenge. During the meeting, the tone, emotions and attitudes of the participants often change dynamically over time. These subtle changes may indicate important turning points in the negotiation process. However, traditional speech analysis methods have difficulty capturing these subtle changes in tone, let alone linking them to the meeting topic, the emotions of the participants and the overall atmosphere. How to extract valuable speech features from continuous audio streams and associate them with high-level semantic information such as tone and emotion is an urgent problem to be solved. In addition, the key turning points in the meeting process are often hidden in massive data. How to identify these turning points in time and respond accordingly is also an important challenge. At the same time, different types of meetings may present different voice patterns and emotional characteristics. How to build a general analysis model that can adapt to specific scenarios is the key to achieving high-precision meeting speech analysis. This series of interrelated technical problems together constitute the core challenge of intelligent analysis of speech data in business negotiation meetings. Summary of the invention

[0003] The present invention provides a digital conference data processing method, which mainly includes:

[0004] Extract speech feature parameters of speech pause duration, speech pause frequency, stressed word occurrence position, and stressed word occurrence frequency in the audio stream to obtain a speech feature vector;

[0005] Perform time series analysis on the speech feature vector to identify key feature data of intonation increase, intonation decrease, and intonation stability changes in the speech, and obtain a tone type label sequence;

[0006] A recurrent neural network model is used, which takes speech feature vectors and tone type label sequences as input and tone representation vectors as output;

[0007] The tone representation vector is parsed based on the business negotiation tone-conference topic mapping model. If the tone-topic mapping changes, it is determined that the conference topic has switched, triggering the conference process stage transition marker.

[0008] The tone representation vector is analyzed based on the business negotiation tone-emotion mapping model. If the emotional fluctuation of the participants exceeds the threshold, a marker is triggered indicating the emergence of a key decision node or the introduction of a controversial topic.

[0009] Based on the business negotiation tone-meeting atmosphere mapping model, the tone representation vector is analyzed to evaluate the meeting atmosphere. If the change in the meeting atmosphere exceeds the threshold, the meeting is judged to have entered the quarrel stage and the corresponding warning is triggered.

[0010] Cluster analysis is performed on the analysis results between tone and topic, emotion, and atmosphere to identify the turning points of conference topic switching, emotional fluctuations of participants, and sudden changes in conference atmosphere, and trigger turning point markers.

[0011] Dynamically adjust data parameters according to turning point markers, including adjusting speech feature parameters, updating the category system of the tone type library, and fine-tuning the parsing rules of the tone-theme / emotion / atmosphere mapping model.

[0012] The present invention provides a digital conference data processing system, which mainly includes:

[0013] Speech feature extraction module, used to extract speech feature parameters such as speech pause duration, stressed words, etc.

[0014] The timing analysis module is used to analyze the amplitude and smoothness of intonation and generate a sequence of tone type labels;

[0015] A recurrent neural network module, which is used to input a speech feature vector and a tone label sequence and output a tone representation vector;

[0016] Tone analysis module, used to analyze tone representation vectors and determine the switching of meeting topics, emotional fluctuations, and changes in meeting atmosphere;

[0017] Cluster analysis module, used to perform cluster analysis on the analysis results and identify turning points of conference topic switching, emotional fluctuations, and sudden changes in atmosphere;

[0018] The data parameter dynamic adjustment module is used to adjust the speech feature parameters and update the parsing rules according to the turning point markers.

[0019] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0020] The present invention discloses a digital conference data processing method and system. The method pre-processes audio stream data through a deep learning model, extracts speech feature parameters and normalizes them. Then, a convolutional neural network is used to perform time series analysis, identify the characteristics of intonation changes, and classify and label them in combination with a preset tone type library. A recurrent neural network with an attention mechanism is then used to fuse speech features and tone labels, and output a comprehensive tone representation vector. Through a pre-established mapping model, the tone representation vector is analyzed to determine the switching of conference topics, emotional fluctuations of participants, and changes in conference atmosphere. Dynamic clustering analysis is performed using sliding time window technology to identify important turning points and trigger warnings. The present invention can also dynamically adjust analysis parameters according to turning points to achieve adaptive optimization and improve the understanding and analysis accuracy of voice data in business negotiation meetings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The present invention is a flowchart of a digital conference data processing method.

[0022] Figure 2 The figure is a schematic diagram of a digital conference data processing method and system of the present invention.

[0023] Figure 3 This is another schematic diagram of a digital conference data processing method and system of the present invention. DETAILED DESCRIPTION

[0024] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0025] like Figure 1 -3. A digital conference data processing system in this embodiment may specifically include:

[0026] Step S101, extracting speech feature parameters such as speech pause duration, speech pause frequency, stressed word occurrence position, and stressed word occurrence frequency in the audio stream to obtain a speech feature vector.

[0027] The input audio stream data is obtained, and the short-time energy analysis and zero-crossing rate analysis methods are used in turn to perform time domain processing on the audio stream data, and speech pause duration and speech pause frequency features are extracted therefrom; according to the speech pause duration and speech pause frequency features, a pitch detection method is used to calculate a fundamental frequency curve, and the occurrence position and frequency of stressed words are determined; the speech pause features and stressed word features are normalized by a minimum-maximum scaling method to obtain a standardized speech feature vector; the standardized speech feature vector is input into a pre-trained feedforward neural network, and a high-dimensional feature representation is obtained by calculating an activation function and a weight matrix; the high-dimensional feature representation calculates the Euclidean distance or cosine similarity, compares it with the standard features in the pre-trained speech feature model, and selects the feature with the smallest distance or the highest similarity as the final speech feature representation result.

[0028] Specifically, the input audio stream data is obtained, and the audio signal is processed in the time domain using short-time energy analysis and zero-crossing rate analysis to extract the speech pause duration and speech pause frequency features. For the characteristics of stressed vocabulary, the pitch detection algorithm is used to calculate the fundamental frequency curve to determine the location and frequency of stressed vocabulary. According to the extracted speech feature parameters, the minimum-maximum scaling method is used for normalization processing, and each feature parameter is mapped to the [0, 1] interval to obtain a standardized speech feature vector. A pre-trained feedforward neural network is loaded, which includes an input layer, two hidden layers and an output layer. The number of input layer nodes is the same as the dimension of the feature vector. The first hidden layer contains 64 nodes, the second hidden layer contains 32 nodes, and the number of output layer nodes is the same as the number of predefined speech feature categories. The standardized speech feature vector is input into the feedforward neural network, and the high-dimensional feature representation is obtained through activation function and weight matrix calculation. The high-dimensional features output by the feedforward neural network are processed, and the Euclidean distance or cosine similarity is calculated. The features are compared with the standard features in the pre-trained speech feature model, and the features with the smallest distance or the highest similarity are selected as the final speech feature representation results. The input audio stream data is processed and framed using a 20-ms time window and a 10-ms step size. The short-time energy and zero-crossing rate of each frame are calculated, and the thresholds are set to 0.01 and 0.02 respectively. When the short-time energy and zero-crossing rate of multiple consecutive frames are lower than the threshold, it is determined to be a speech pause. The start and end time of the pause are recorded, and the pause duration and frequency are calculated. For the extraction of accent vocabulary features, the autocorrelation method is used to calculate the pitch, with a sampling frequency of 16kHz, a window length of 30 milliseconds, and an overlap of 50%. The position and frequency of the accent vocabulary are determined by calculating the local peak of the fundamental frequency curve. The extracted feature parameters are normalized using the formula x_normalized = (x-x_min) / (x_max-x_min), where x is the original feature value, x_min and x_max are the minimum and maximum values ​​of the feature, respectively, and x_normalized is the normalized value of the feature. For speech pause duration, x_min is set to 0 seconds and x_max is set to 2 seconds; for pause frequency, x_min is set to 0 times / minute and x_max is set to 30 times / minute; for stress word position, x_min is set to 0 seconds and x_max is set to the total length of the audio; for stress word frequency, x_min is set to 0 times / minute and x_max is set to 100 times / minute. The normalized feature vector is input into a pre-trained feedforward neural network, which has 4 nodes in the input layer, corresponding to 4 feature parameters, 64 nodes in the first hidden layer, using the ReLU activation function, 32 nodes in the second hidden layer, also using the ReLU activation function, and 10 nodes in the output layer, corresponding to 10 predefined speech feature categories. The high-dimensional feature representation is obtained through forward propagation calculation.Finally, the Euclidean distance between the high-dimensional feature and the standard feature in the pre-trained speech feature model is calculated, and the category with the smallest distance is selected as the final speech feature representation result.

[0029] Step S102, performing time series analysis on the speech feature vector, identifying key feature data of intonation rise amplitude, intonation fall amplitude, and intonation smoothness change in the speech, and obtaining a tone type label sequence.

[0030] Obtain a standardized speech feature vector, and perform a time series analysis on the speech feature vector. Use a two-dimensional convolutional neural network to process the feature vector, the two-dimensional convolutional neural network includes a preset number of convolutional layers, each layer uses a convolution kernel of a preset size, and a ReLU activation function is used between convolutional layers. Perform a maximum pooling operation on the local features extracted by the two-dimensional convolutional neural network, and the pooling window size and step size of the maximum pooling operation are preset values. According to the maximum pooling operation, obtain key feature data of intonation changes. Read the tone type feature template from the pre-established tone type library, and use the Euclidean distance calculation method to perform similarity matching on the key feature data and the tone type feature template. If the similarity matching result is greater than a preset threshold, determine the tone label of the current speech segment. Use a long short-term memory network to process a speech segment sequence, the long short-term memory network includes an LSTM layer, and the number of hidden units is a preset value. By performing an argmax operation on the output of the long short-term memory network, the tone type of each time step is obtained to form a complete tone type label sequence.

[0031] Specifically, the standardized speech feature vector is obtained, and the time series analysis is performed on the speech feature vector. The feature vector is processed using a two-dimensional convolutional neural network. The network contains three convolutional layers, each layer uses a 3x3 convolution kernel, and the ReLU activation function is used between the convolutional layers to extract local features in the time dimension and frequency dimension, and identify the key feature data of the intonation increase amplitude, intonation decrease amplitude, and intonation stability change in the speech. The local features extracted by the two-dimensional convolutional neural network are subjected to a maximum pooling operation, with a pooling window size of 2x2 and a step size of 2, to reduce the feature dimension, improve the robustness of the feature, and obtain the key feature data of intonation change. The tone type feature template is read from the pre-established tone type library. The template is obtained by statistical learning of a large amount of annotated data and contains typical feature representations of different tone types. The Euclidean distance calculation method is used to perform similarity matching between the pooled key feature data and the tone type feature template, and the tone type with the highest similarity is selected as the tone label of the current speech segment. According to the speech timing information, the long short-term memory network is used to process the speech fragment sequence. The network contains an LSTM layer with 128 hidden units. The input is the feature representation of each speech fragment, and the output is the corresponding tone type probability distribution. By performing argmax operation on the output of the long short-term memory network, the most likely tone type at each time step is obtained, forming a complete tone type label sequence, and obtaining preliminary speech tone classification results. The standardized speech feature vector is processed, assuming that the input is 100 frames, and each frame has 40-dimensional features. A two-dimensional convolutional neural network is used. The first layer uses 32 3x3 convolution kernels with a step size of 1 and a padding of 1; the second layer uses 64 3x3 convolution kernels with a step size of 1 and a padding of 1; the third layer uses 128 3x3 convolution kernels with a step size of 1 and a padding of 1. After each convolution layer, the ReLU activation function is used to extract local features in the time and frequency dimensions. The convolution output is subjected to 2x2 maximum pooling with a step size of 2 to obtain a 50x20x128 feature map. Read 10 feature templates of tone types from the pre-established tone type library, each template is a 128-dimensional vector. Calculate the Euclidean distance between the feature map and the template, and select the type with the smallest distance as the tone label of the current frame. Use a long short-term memory network to process the label sequence. The network contains an LSTM layer with 128 hidden units. The input is a 128-dimensional feature vector for each frame, and the output is a 10-dimensional probability distribution. Perform an argmax operation on the output of 50 time steps to obtain the final tone type label sequence. For example, the input speech "The weather is really good today" may be processed to obtain a tone type sequence of [calm, calm, up, up, calm], indicating that the speaker has a significant upturn in the tone of the words "weather" and "true".

[0032] Step S103, adopting a recurrent neural network model, taking the speech feature vector and the tone type label sequence as input, and taking the tone representation vector as output.

[0033] Acquire a speech feature vector and a tone type label sequence, convert the tone type label sequence into a dense vector representation through an embedding layer, and obtain an embedding vector that matches the dimension of the speech feature vector. Use a bidirectional gated recurrent unit to perform time series processing on the speech feature vector and the embedding vector, extract context-related features of the two modal data, and obtain a hidden state sequence. For the hidden state sequence, use a multi-head self-attention mechanism to calculate the correlation between different time steps, calculate the attention weight by scaling the dot product attention, assign an attention weight to each time step, and obtain a weighted feature representation. According to the attention weight, fuse the speech features and tone type information to obtain a fused feature. Input the fused feature into the multi-layer fully connected network, use the activation function in each layer, and output the last layer as a comprehensive tone representation vector of fixed dimension.

[0034] Specifically, the speech feature vector and tone type label sequence are obtained, and the tone type label sequence is converted into a dense vector representation through the embedding layer. The embedding layer dimension is set to 128, and the normal distribution is used for initialization to obtain an embedding vector that matches the dimension of the speech feature vector. The speech feature vector and the embedding vector are processed sequentially using a bidirectional gated recurrent unit. The number of hidden units is set to 256 and the number of layers is set to 2. The context-related features of the two modal data are extracted to obtain a hidden state sequence. For the hidden state sequence, an 8-head self-attention mechanism is used to calculate the correlation between different time steps. The dimension of each head is 32. The attention weight is calculated by scaling the dot product attention. The attention weight is assigned to each time step to obtain a weighted feature representation. The speech features and tone type information are fused using the attention weight, and the weighted feature representation is input into a three-layer fully connected network. The number of neurons in the first layer is 512, the second layer is 256, and the third layer is 128. Each layer uses the ReLU activation function, and the output of the last layer is a comprehensive tone representation vector of fixed dimension.

[0035] Taking a 10-second speech as an example, 100 frames of speech feature vectors are extracted, each with 40 dimensions, and the corresponding 100 tone type labels are obtained at the same time. The tone type label is converted into a dense vector representation through a 128-dimensional embedding layer, and the embedding weight is initialized using a normal distribution with a mean of 0 and a standard deviation of 0.01. The 40-dimensional speech feature vector and the 128-dimensional tone type embedding vector are concatenated to obtain a 168-dimensional input vector sequence. The input sequence is processed using a bidirectional gated recurrent unit, with 256 hidden units, 2 layers of stacking, and 256 dimensions for the forward and backward directions, to obtain a 100 time step, 512-dimensional hidden state sequence. An 8-head self-attention mechanism is applied to the hidden state sequence, with the query, key, and value vector dimensions of each head being 64. The attention score is calculated by scaling the dot product, and the attention weight is obtained by softmax normalization. The outputs of the 8 heads are concatenated and transformed linearly to obtain a 512-dimensional weighted feature representation. Finally, through a three-layer fully connected network, with 512-256-128 neurons respectively, each layer uses the ReLU activation function to output a 128-dimensional comprehensive tone representation vector. For example, for the sentence "The weather is really good today", you may get a 128-dimensional vector like [0.8, 0.2, 0.6, ...], where different dimensions reflect the comprehensive information of speech features and tone types, such as intonation changes, emotional intensity, etc.

[0036] Step S104, parsing the tone representation vector based on the business negotiation tone-conference topic mapping model, if the tone-topic mapping changes, it is determined that the conference topic is switched, triggering a conference process stage transition mark.

[0037] Acquire a tone representation vector, classify the tone representation vector into a predefined business negotiation tone category through a pre-trained support vector machine, and obtain the current tone category label. According to the current tone category label, query the probability distribution of the conference topic related to the current tone from the pre-established tone-topic probability distribution table. Use the sliding window method to calculate the JS divergence of the conference topic probability distribution in adjacent time windows. If the JS divergence value exceeds the preset threshold, it is determined that the conference topic has switched. If it is determined that the conference topic has switched, the conference process stage transition mark is triggered, and the topic with the highest probability in the conference topic probability distribution is used as the new topic to update the current conference topic status. Use an online learning method to update the tone-topic probability distribution table with the newly added tone-topic correspondence after each meeting to maintain the dynamic adaptability of the mapping model.

[0038] Specifically, the tone representation vector is obtained, and the vector is classified by a pre-trained support vector machine. The radial basis function kernel is used, the penalty parameter C is set to 1.0, the gamma parameter is set to 0.1, and it is trained using 10,000 labeled data. The tone representation vector is mapped to the pre-defined 10 business negotiation tone categories to obtain the current tone category label. According to the tone category label, the pre-established tone-topic probability distribution table is queried. The table is a 10x20 matrix, the rows represent 10 tone categories, the columns represent 20 meeting topics, and the matrix element values ​​represent the probability corresponding to the tone-topic. It is obtained from 1,000 meeting data through frequency statistics to obtain the probability distribution of the meeting topic related to the current tone. The sliding window method is used, the window size is set to 5 minutes, the step size is 1 minute, and the JS divergence between the topic probability distribution in the current time window and the topic probability distribution in the previous time window is calculated. If the JS divergence value exceeds the preset threshold of 0.3, it is determined that the meeting topic has switched. If the conference topic is determined to be switched, the conference process stage transition mark is triggered, and the topic with the highest probability in the topic probability distribution is used as the new topic to update the current conference topic status, and the new topic status is stored in the conference process record. Using an online learning method, after each meeting, the newly added tone-topic correspondence is used to update the tone-topic probability distribution table to maintain the dynamic adaptability of the mapping model. In a business negotiation, the system receives a 128-dimensional tone representation vector [0.23, 0.15, 0.42, ...]. The support vector machine uses a radial basis function kernel, C = 1.0, gamma = 0.1, to map the vector to the "strong" tone category. Query the 10x20 tone-topic probability distribution table, and the "strong" tone corresponds to the row [0.05, 0.15, 0.30, 0.20, ...], which indicates the probability of appearing under different topics. Use a 5-minute sliding window with a step size of 1 minute to calculate the current window

[0039] The JS divergence of [0.05, 0.18, 0.32, 0.19, ...] and the previous window [0.06, 0.16, 0.28, 0.22, ...] is 0.35. Since 0.35 is greater than the threshold 0.3, the topic is determined to have switched. The "price negotiation" corresponding to the highest probability 0.32 is selected as the new topic, and the meeting status is updated and recorded. After the meeting, the newly added 50 tone-topic pairs are added to the original 10,000 data and the probability distribution table is recalculated. For example,

[0040] The probability of "strong" tone under the "price negotiation" topic was updated from 0.30 to 0.31, reflecting the latest tone-topic correlation trend.

[0041] Step S105, analyzing the tone representation vector based on the business negotiation tone-emotion mapping model, if the emotional fluctuation amplitude of the participants exceeds a threshold, triggering a mark indicating the appearance of a key decision node or the introduction of a controversial topic.

[0042] The tone representation vector is classified according to a pre-trained random forest classifier to obtain the current emotion category label; the random forest classifier includes a preset number of decision trees, has a preset maximum depth, and is trained using a preset number of labeled data. Query a pre-established tone-emotion intensity mapping table; the tone-emotion intensity mapping table is a matrix of preset dimensions, with rows representing a preset number of emotion categories and columns representing a preset number of intensity levels; obtain the emotion intensity value corresponding to the current emotion category label. The sliding window method is used to calculate the difference between the emotion intensity value in the current time window and the emotion intensity value in the previous time window; if the difference exceeds a preset threshold, it is determined that the emotion fluctuation is significant. If it is detected that the emotion fluctuation amplitude of individual participants exceeds the preset threshold, the duration of the emotion fluctuation is determined; if the duration of the emotion fluctuation exceeds the preset duration, it is determined to be a key decision node; if the duration of the emotion fluctuation does not exceed the preset duration, it is determined to be a controversial topic. Calculate the average and standard deviation of the emotion intensity of all participants; if the standard deviation exceeds the preset threshold, it is determined that the overall meeting atmosphere is tense, triggering the corresponding meeting status mark.

[0043] Specifically, the tone representation vector is obtained and classified by a pre-trained random forest classifier, which contains 100 decision trees with a maximum depth of 10 and is trained using 10,000 labeled data. The tone representation vector is mapped to the pre-defined 8 emotion categories to obtain the current emotion category label. According to the emotion category label, the pre-established tone-emotion intensity mapping table is queried. The table is an 8x10 matrix, with rows representing 8 emotion categories and columns representing 10 intensity levels. The matrix element value represents the probability corresponding to the tone-emotion intensity, which is obtained from 1,000 meeting data through frequency statistics to obtain the emotion intensity value related to the current tone. The sliding window method is used, with the window size set to 3 minutes and the step size set to 30 seconds. The difference between the emotion intensity value in the current time window and the emotion intensity value in the previous time window is calculated. If the difference exceeds the preset threshold of 0.2, it is determined that the emotion fluctuation is significant. If it is detected that the emotional fluctuation amplitude of individual participants exceeds the preset threshold, a key decision node or controversial topic mark is triggered. The duration of emotional fluctuation is judged by the duration of more than 2 minutes. It is determined to be a key decision node, otherwise it is determined to be a controversial topic, the meeting status record is updated, and the new status information is stored in the meeting process database. A comprehensive analysis of the emotional state of multiple participants is performed to calculate the average and standard deviation of the emotional intensity of all participants. If the standard deviation exceeds 0.5, it is determined that the overall meeting atmosphere is tense and the corresponding meeting status mark is triggered. In a business negotiation, the system receives a 128-dimensional tone representation vector [0.35, 0.12, 0.48, ...]. The random forest classifier uses 100 decision trees with a maximum depth of 10 to map the vector to the "excited" emotion category. Query the 8x10 tone-emotion intensity mapping table, the corresponding row of "excited" emotion is [0.02, 0.05, 0.15, 0.25, 0.30, 0.15, 0.05, 0.02, 0.01, 0], and select the intensity value 6 corresponding to the highest probability 0.30. Use a 3-minute sliding window with a step size of 30 seconds to calculate the difference between the current window intensity value 6 and the previous window value 4 as 2. Since 2 is greater than the threshold of 0.2, it is determined that the emotion fluctuation is significant. The system continuously monitors the fluctuation and finds that the duration is 2 minutes and 30 seconds, which exceeds the 2-minute threshold. Therefore, it is determined to be a key decision node and triggers the corresponding mark. At the same time, the emotion intensity values ​​of the five participants are calculated [6, 5, 7, 4, 6], with an average of 5.6 and a standard deviation of 0.98. Since the standard deviation of 0.98 is greater than 0.5, the system determines that the overall meeting atmosphere is tense and triggers the meeting status mark. All this information is recorded in the meeting process database for subsequent analysis and decision support.

[0044] Step S106, analyzing the tone representation vector based on the business negotiation tone-meeting atmosphere mapping model, evaluating the meeting atmosphere, and if the change range of the meeting atmosphere exceeds a threshold, determining that the meeting has entered a quarreling stage, triggering a corresponding warning.

[0045] The tone representation vector is obtained, and the tone representation vector is classified by using a pre-trained K-nearest neighbor algorithm, wherein the K value of the K-nearest neighbor algorithm is set to a preset value, and the K-nearest neighbor algorithm uses the Euclidean distance as a distance measurement method; according to the classification result of the K-nearest neighbor algorithm, the current conference atmosphere category label is obtained. According to the conference atmosphere category label, the atmosphere intensity value related to the current tone is obtained from a pre-established tone-atmosphere intensity mapping table, wherein the tone-atmosphere intensity mapping table is a matrix, wherein the rows of the matrix represent the atmosphere category, the columns of the matrix represent the intensity level, and the element values ​​of the matrix represent the probability corresponding to the tone-atmosphere intensity. An exponentially weighted moving average method is used, wherein the smoothing factor of the exponentially weighted moving average method is set to a preset value, and the calculation cycle of the exponentially weighted moving average method is set to a preset time; the change amplitude between the current atmosphere intensity value and the historical atmosphere intensity value is calculated by the exponentially weighted moving average method; if the change amplitude exceeds a preset threshold, it is determined that the conference atmosphere has undergone a sudden change. The tone representation vectors of multiple participants are obtained; and the average value of the tone representation vectors of the multiple participants is calculated to obtain a representative vector of the overall conference atmosphere. If it is determined that the meeting atmosphere has suddenly changed and the change exceeds a preset threshold, the meeting is triggered to enter the quarrel stage mark; an emotional warning signal for the conference host is generated, and the warning signal includes two indicators: the degree of atmosphere mutation and the duration; the warning signal is sent to the conference host through the instant messaging module of the conference management system; the warning signal is stored in the real-time status database of the conference.

[0046] Specifically, the tone representation vector is obtained, and the vector is classified by the pre-trained K nearest neighbor algorithm. The K value is set to 5. The Euclidean distance is used as the distance measurement method. The tone representation vector is mapped to the pre-defined 10 meeting atmosphere categories to obtain the current meeting atmosphere category label. According to the meeting atmosphere category label, the pre-established tone-atmosphere intensity mapping table is queried. The table is a 10x10 matrix. The rows represent 10 atmosphere categories, the columns represent 10 intensity levels, and the matrix element value represents the probability corresponding to the tone-atmosphere intensity. The atmosphere intensity value related to the current tone is obtained. The exponentially weighted moving average method is used, the smoothing factor is set to 0.2, and the calculation period is 5 minutes. The change range between the current atmosphere intensity value and the historical atmosphere intensity value is calculated. If the change range exceeds the preset threshold of 0.3, it is determined that the meeting atmosphere has changed suddenly. The tone representation vectors of multiple participants are combined, and the average value of the tone vectors of all participants is calculated as the representative vector of the overall meeting atmosphere. If it is determined that the meeting atmosphere has suddenly changed and the change exceeds the preset threshold of 0.3, the meeting is triggered to enter the quarrel stage, and an emotional warning signal for the meeting host is generated. The warning signal includes two indicators: the degree of atmosphere change and the duration. It is sent to the meeting host through the instant messaging module of the conference management system, and the warning information is stored in the real-time status database of the meeting. In a business negotiation, the system received 128-dimensional tone representation vectors of 5 participants, which are

[0047] [0.4, 0.2, 0.6, ...], [0.5, 0.3, 0.5, ...], [0.6, 0.1, 0.7, ...],

[0048] [0.3, 0.4, 0.4, ...] and [0.7, 0.2, 0.8, ...]. The K nearest neighbor algorithm (K = 5) classifies these vectors, uses the Euclidean distance calculation to obtain the nearest 5 neighbors, and determines the current meeting atmosphere category as "tense" through majority voting. Query the 10x10 tone-atmosphere intensity mapping table, and the "tense" category corresponds to the row

[0049] [0.05, 0.10, 0.15, 0.25, 0.20, 0.15, 0.05, 0.03, 0.01, 0.01], select the intensity value 4 corresponding to the highest probability of 0.25. Using a smoothing factor of 0.2 and a 5-minute exponentially weighted moving average method, the average intensity value of the previous period is calculated to be 2.5, and the current intensity value 4 has a change of 0.6 compared with it. Since 0.6 is greater than the preset threshold of 0.3, the system determines that the atmosphere of the meeting has changed suddenly. At the same time, the average value of the tone vectors of the five participants [0.5, 0.24, 0.6, ...] is calculated as the representative vector of the overall meeting atmosphere. Based on the mutation judgment and the overall atmosphere assessment, the system triggers the meeting to enter the quarrel stage mark and generates an early warning signal, including two indicators: the degree of atmosphere mutation 0.6 and the duration of 3 minutes. The early warning information is sent to the conference host through the instant messaging module and recorded in the real-time status database for subsequent analysis and intervention.

[0050] Step S107, cluster analysis is performed on the analysis results between tone and theme, emotion, and atmosphere, to identify turning points of conference theme switching, emotional fluctuations of participants, and sudden changes in conference atmosphere, and trigger turning point occurrence marks.

[0051] Obtain the continuous analysis results of tone-topic, tone-emotion, and tone-atmosphere, and use a fixed-size sliding time window to segment the analysis results; wherein the window size of the sliding time window is a preset value. For the analysis results in each time window, apply the K-means clustering algorithm; wherein the K-means clustering algorithm is initialized using K-means++, and the number of clusters is set to three, corresponding to the three dimensions of topic, emotion, and atmosphere respectively. Calculate the Euclidean distance between the cluster centers of adjacent time windows, and determine the potential turning point according to the preset threshold; conduct a comprehensive evaluation of the identified potential turning points, and if at least two dimensions have a turning point within the preset time, it is determined to be a key turning point event; wherein the turning point includes topic switching, emotional fluctuations, and sudden changes in atmosphere. Generate an early warning signal containing the turning type, occurrence time, and severity, and send the early warning signal to the conference host terminal through the instant messaging module of the conference management system; wherein the early warning signal is triggered when the comprehensive score calculated according to the preset weight exceeds the threshold.

[0052] Specifically, we obtain continuous tone-topic, tone-emotion, and tone-atmosphere analysis results, and use a fixed-size sliding time window to segment the analysis results. The window size is set to 5 minutes and the sliding step is 1 minute. The analysis results of each dimension are processed independently. The topic dimension is encoded with one-hot encoding, and the emotion and atmosphere dimensions are represented numerically. The K-means clustering algorithm is applied to the analysis results in each time window. K-means++ is used for initialization. The number of clusters is set to 3, corresponding to the three dimensions of topic, emotion, and atmosphere, and the cluster center of each window is obtained. The Euclidean distance between the cluster centers of adjacent time windows is calculated. The threshold of the topic dimension is set to 0.5, the threshold of the emotion dimension is set to 0.3, and the threshold of the atmosphere dimension is set to 0.4. If the distance of any dimension exceeds the corresponding threshold, it is determined to be a potential turning point, and the time point and the corresponding dimension information are recorded. A comprehensive evaluation is conducted on the identified potential turning points. If at least two dimensions have a turning point within 2 minutes, it is determined to be a key turning event. Different types of turning points are classified and weighted, with the weight of topic switching being 0.5, the weight of emotional fluctuation being 0.3, and the weight of atmosphere mutation being 0.2. When the comprehensive score calculated based on the weights exceeds 0.6, the corresponding mark and warning signal are triggered. The warning signal includes three indicators: turning point type, occurrence time, and severity. It is sent to the conference host through the instant messaging module of the conference management system, and the event information is recorded in the conference process database.

[0053] In a business negotiation lasting 90 minutes, the system obtains the analysis results of tone-topic, tone-emotion and tone-atmosphere every minute. A sliding window of 5 minutes is used, sliding once every 1 minute, and a total of 86 time windows are generated. For the topic dimension, 5-dimensional one-hot encoding is used to represent 5 possible topics; the intensity of the emotion and atmosphere dimensions is represented by a value of 1-10. At the 45th minute, the K-means clustering algorithm (initialized using K-means++) clusters the current window data and obtains the topic center [0, 1, 0, 0, 0], the emotion center 6.5, and the atmosphere center 7.2. Compared with the cluster center of the previous window, the Euclidean distance of the topic dimension is 1.414 (exceeding the threshold of 0.5), the emotion dimension changes by 0.8 (exceeding the threshold of 0.3), and the atmosphere dimension changes by 0.3 (not exceeding the threshold of 0.4). The system determines that this is a potential turning point and records the time and change dimension. In the following 2 minutes, the system detects another significant change in the emotion dimension. During the comprehensive evaluation, topic switching (weight 0.5) and emotional fluctuations (weight 0.3) occurred at the same time, and the calculated comprehensive score was 0.8, which exceeded the trigger threshold of 0.6. The system generated an early warning signal, including the transition type of "topic switching + emotional fluctuations", the occurrence time "45 minutes" and the severity "high", which was sent to the meeting host through the instant messaging module, and this key transition event was recorded in the meeting process database to provide a basis for subsequent analysis and decision-making.

[0054] According to the characteristics of turning points of different tone types, multiple window sizes are set, and dynamic clustering analysis of multiple scales is performed simultaneously through parallel computing to obtain clustering result sets at different time granularities. A weighted fusion algorithm is applied to the multi-scale clustering result set. According to the characteristic distribution of different types of turning points in historical data, a weight coefficient is assigned to the clustering result of each scale, and a comprehensive clustering result is obtained through weighted summation.

[0055] Acquire the tone data stream and use multiple groups of preset window sizes to perform sliding window processing to obtain multiple groups of time series data; perform dynamic clustering analysis based on the multiple groups of time series data, and the dynamic clustering analysis is implemented through the Spark distributed computing platform to obtain multiple clustering result sets with different time granularities; extract turning point feature distribution data from the historical database, calculate the occurrence frequency of each type of turning point under the multiple window sizes based on this, and normalize the frequency value to obtain the weight coefficient; for multiple clustering result sets, use the weighted fusion algorithm to perform weighted summation using the weight coefficient to obtain a comprehensive clustering result; perform quality assessment on the comprehensive clustering result, if the assessment result is lower than the preset threshold, adjust the weight coefficient and re-execute the weighted fusion algorithm until the assessment result reaches the preset threshold or the number of iterations reaches the upper limit.

[0056] Specifically, according to the characteristics of different tone turning points in historical data, three window sizes are set, namely 3 minutes, 5 minutes and 10 minutes, corresponding to fast, medium and slow tone changes. The input tone data stream is processed by sliding window to generate three sets of time series data. The Spark distributed computing platform is used to perform dynamic clustering analysis on the three sets of time series data at the same time. The K-means clustering algorithm is used, and the number of clusters is set to 5 to obtain three clustering result sets with different time granularities. The turning point feature distribution data of the past 1000 meetings are extracted from the historical database, and the occurrence frequency of each type of turning point under three window sizes is calculated. The frequency value is mapped to the interval [0, 1] using min-max normalization as the weight coefficient. The weighted fusion algorithm is applied to the three clustering result sets. The calculated weight coefficient is used to obtain the comprehensive clustering result through the weighted summation method. The specific steps are to convert the cluster label of each time point into one-hot encoding, multiply it with the corresponding weight, and then add the results of the three scales to obtain the final tone turning point recognition result. The silhouette coefficient is used to evaluate the quality of the comprehensive clustering results. If the silhouette coefficient is lower than 0.5, the weight coefficient is fine-tuned by the gradient descent method and the weighted fusion is re-performed until the silhouette coefficient reaches the preset threshold or the number of iterations reaches the upper limit. In a business negotiation lasting 120 minutes, the system collects tone data every 10 seconds, generating a total of 720 data points. Using sliding windows of 3 minutes, 5 minutes, and 10 minutes, 708, 696, and 660 time series samples are generated, respectively. Five worker nodes in the Spark cluster process these samples in parallel, and each node is responsible for one clustering category. The K-means clustering algorithm performs 50 iterations at each time granularity to obtain three groups of clustering results. Historical data shows that the rapid tone changes captured by the 3-minute window account for 30%, the moderate changes in the 5-minute window account for 50%, and the slow changes in the 10-minute window account for 20%. After min-max normalization, the weight coefficients are 0.6, 1.0, and 0.4, respectively. During the fusion process, the cluster labels at each time point are converted into a 5-dimensional one-hot vector, multiplied by the weights and added together to obtain a 5-dimensional comprehensive vector. For example, at the 30th minute, the cluster labels of the three scales are 2, 3, and 1, respectively. After being converted into a one-hot vector and multiplied by the weights, we get

[0057] [0, 0.6, 0, 0, 0], [0, 0, 1.0, 0, 0], and [0.4, 0, 0, 0, 0], after adding, we get

[0058] [0.4, 0.6, 1.0, 0, 0]. The calculated silhouette coefficient is 0.45, which is lower than the threshold of 0.5. The system adjusts the weights through the gradient descent method. After 10 iterations, the weights are updated to 0.65, 0.95, and 0.45, and the silhouette coefficient is increased to 0.52, obtaining the final result of tone turning point recognition.

[0059] If a turning point event occurs at a certain moment, the moment is marked as a critical turning point and a warning signal is triggered. The warning level is determined based on the turning amplitude and duration attributes, and a turning point marking box is generated. The color of the marking box is set according to the warning level, and the associated theme and emotional information are added to the marking box. When the mouse hovers over the marking box, the attribute information of the turning point is displayed, and clicking the marking box can jump to the corresponding voice clip.

[0060] Obtain the timestamp of the turning event occurrence time, and the timestamp is used to mark the key turning point; determine the warning level of the turning event according to the preset turning amplitude threshold and duration threshold; if the turning amplitude is less than the first preset threshold and the duration is less than the second preset threshold, determine the warning level to be low; if the turning amplitude is greater than or equal to the first preset threshold and less than the third preset threshold, or the duration is greater than or equal to the second preset threshold and less than the fourth preset threshold, determine the warning level to be medium; if the turning amplitude is greater than or equal to the third preset threshold, or the duration is greater than or equal to the fourth preset threshold, determine the warning level to be high; set the mark box color according to the warning level, and create a rectangular mark box object; extract the theme and emotional information corresponding to the key turning point from the speech analysis result, the theme is obtained by extracting keywords through the TF-IDF algorithm, and the emotional information is obtained by matching the emotional dictionary; add the theme and the emotional information to the attributes of the mark box object; bind the mouse event processing function to the mark box object; when a mouse hover event is detected, display the detailed attribute information of the mark box object; when a mouse click event is detected, call the audio player API according to the timestamp to locate and play the corresponding voice clip.

[0061] Specifically, the timestamp of the turning point event is obtained, and the moment is marked as a key turning point. According to the preset turning amplitude threshold and duration threshold, the warning level of the turning event is calculated. The specific calculation method is that the turning amplitude is less than 20% and the duration is less than 1 minute, which is a low level, the turning amplitude is 20%-50% or the duration is 1-3 minutes, which is a medium level, and the turning amplitude is greater than 50% or the duration is greater than 3 minutes. The warning signal containing the timestamp and warning level is generated. The color of the marking box is set according to the warning level, the low level is green, the medium level is yellow, and the high level is red. A rectangular marking box object is created. The position of the marking box is determined according to the timeline position, and the size is dynamically adjusted according to the duration. The theme and emotional information corresponding to the key turning point are extracted from the speech analysis results. The TF-IDF algorithm is used to extract keywords as the theme. The emotional dictionary is used to match the emotional tendency, and this information is added to the attributes of the marking box object. At the same time, the detailed attribute information such as the turning amplitude and duration is stored in the extended attributes of the marking box object. The mouse event processing function is bound to the marker box object. When a mouse hover event is detected, the detailed attribute information of the marker box is displayed. When a mouse click event is detected, the audio player API is called according to the timestamp to locate and play the corresponding voice clip. For the case where multiple turning points overlap, a cascading display method is used to distinguish different turning points by transparency. The paging loading technology is used to load a fixed number of turning point data each time to improve the processing performance of a large amount of turning point data. In a business negotiation lasting 90 minutes, the system detected a turning event at the 37th minute, and the timestamp was recorded as 2220 seconds. The turning amplitude was calculated to be 45%, and the duration was 2 minutes and 30 seconds. According to the preset rules, it was determined to be a medium-level warning. The system generates a yellow marker box, which is set at 3700 pixels on the timeline and 150 pixels wide (corresponding to 2.5 minutes). The TF-IDF algorithm is used to analyze the speech text of this period, and the keywords "price", "discount", and "competitor" are extracted as the theme information. The emotional dictionary matching results show that negative emotional words account for 70%, which is determined to be negative emotions. This information is added to the properties of the marker box object. When the user hovers over the marker box, detailed information is displayed: "Time: 37 minutes, duration: 2.5 minutes, turning point: 45%, theme: price negotiation, emotion: negative". The user clicks the marker box, calls the audio player API, and starts playing the voice clip from 2220 seconds. At the 60th minute, another turning point appears, overlapping with the previous marker box. The system sets the new marker box to 70% transparency and displays it in an overlay. To improve performance, 10 turning point data can be loaded each time, and the next batch of data is dynamically loaded when the user scrolls.

[0062] Step S108, dynamically adjusting data parameters according to the turning point markers, including adjusting speech feature parameters, updating the category system of the tone type library, and fine-tuning the parsing rules of the tone-theme / emotion / atmosphere mapping model.

[0063] According to the turning point mark of the conference process, the voice data before and after the turning point is obtained, the change amplitude of the voice feature parameters is calculated, and the difference value between the feature vectors is obtained. If the difference value exceeds the preset threshold, the parameters of the voice feature extraction algorithm are adjusted, including adjusting the audio sampling rate from the first frequency to the second frequency, adjusting the frame length from the first length to the second length, and adjusting the frame shift from the first interval to the second interval. The distribution of the tone type at the turning point is statistically analyzed, and the chi-square test is used to determine the significant change in the frequency of use of the tone type. If the test result is significant, the category system of the tone type library is updated. In view of the changes in the tone-topic, tone-emotion, and tone-atmosphere mapping relationships before and after the turning point, the mapping model is fine-tuned using the stochastic gradient descent algorithm, and the model weight parameters are updated. According to the adjusted voice feature parameters, the updated tone type library and the fine-tuned mapping model, the historical conference data is reprocessed to calculate the change value of the analysis accuracy. If the change value exceeds the preset accuracy improvement threshold, the new parameters and model are applied to the real-time analysis process.

[0064] Specifically, according to the turning point mark of the conference process, the voice data before and after the turning point are extracted, the change range of the voice feature parameters is calculated, and the difference between the feature vectors is calculated using the Euclidean distance. If the change range exceeds the preset threshold of 0.5, the corresponding voice feature extraction algorithm parameters are dynamically adjusted, including adjusting the audio sampling rate from 16kHz to 22kHz, the frame length from 25ms to 30ms, and the frame shift from 10ms to 15ms. The distribution of tone types at the turning point is statistically analyzed, and the chi-square test is used to determine the significant change in the frequency of use of tone types. The significance level is set to 0.05. If the test result is significant, the category system of the tone type library is updated, including adding new tone types, merging types with a similarity of more than 90%, or deleting types with a frequency of use of less than 1%. In view of the changes in the mapping relationship between tone-topic, tone-emotion, and tone-atmosphere before and after the turning point, the stochastic gradient descent algorithm is used to fine-tune the mapping model, with the learning rate set to 0.01, 100 samples are processed in each batch, the model weight parameters are updated, and the parsing rules of the model are optimized. Based on the adjusted speech feature parameters, updated tone type library and fine-tuned mapping model, the historical meeting data is reprocessed and the change in analysis accuracy is calculated using the F1 score. If the accuracy is improved by more than 5%, the new parameters and model are applied to the real-time analysis process. In order to deal with possible conflicts during parameter adjustment and model update, a parameter version control mechanism is set up to record the timestamp and change content of each update. If performance degradation occurs, it will automatically roll back to the previous stable version. By regularly evaluating system performance, continuously optimizing parameters and models, forming an adaptive optimization feedback loop mechanism, and performing a comprehensive optimization process every 24 hours. In a business negotiation lasting 3 hours, an important turning point was detected in the 78th minute. The speech data of 5 minutes before and after the turning point was extracted, and the Euclidean distance of the MFCC feature vector was calculated to obtain 0.62, which exceeded the preset threshold of 0.5. The audio sampling rate was increased from 16kHz to 22kHz, the frame length was increased from 25ms to 30ms, and the frame shift was adjusted from 10ms to 15ms. Subsequently, a chi-square test was conducted on the tone type at the turning point. The χ2 value was 12.7, the degree of freedom was 4, and the p value was 0.013, which was less than the significance level of 0.05, indicating that the distribution of tone types had changed significantly. The "cautious optimism" tone type can be added, the "anger" and "excitement" types with a similarity of 92% can be merged, and the "indifference" type with a usage frequency of only 0.8% can be deleted. In view of the change in the tone-topic mapping relationship, the stochastic gradient descent algorithm was used to fine-tune the model. The learning rate was set to 0.01, and 100 samples were processed per batch. After 50 iterations, the loss function value dropped from 0.42 to 0.31. The F1 score of the most recent 1,000 historical data was re-analyzed, and the increase was more than 5% from 0.78 to 0.83. The new parameters and model were then applied to the real-time analysis process. Seven parameter updates were recorded throughout the process, of which one was rolled back due to performance degradation.In the routine optimization after 24 hours, 6,480 new data were comprehensively evaluated, and the model parameters were further fine-tuned, which improved the F1 score to 0.85, demonstrating the ability of continuous adaptive optimization.

[0065] According to the quantitative indicators of speech feature changes, a clustering algorithm is used to re-cluster the samples in the existing tone type library. The optimal number of clusters is determined by calculating the ratio of intra-cluster distance to inter-cluster distance, and the category system of tone types is dynamically updated to obtain a new tone type division scheme.

[0066] The invention obtains the quantitative index of speech feature change, extracts the pitch by the short-time Fourier transform method, calculates the volume according to the root mean square energy, obtains the speaking speed by phoneme recognition, and represents the timbre by the Mel frequency cepstral coefficient; extracts the feature vector for the samples in the tone type library to obtain the high-dimensional feature matrix; performs cluster analysis on the high-dimensional feature matrix by the agglomerative hierarchical clustering algorithm, and uses the Euclidean distance as the distance measurement method; calculates the ratio of the intra-class distance to the inter-class distance in the cluster analysis result, and determines that the number of clusters corresponding to the maximum value of the silhouette coefficient is the optimal number of clusters; performs the final clustering of the tone type samples according to the optimal number of clusters, and assigns a new tone type label to each cluster cluster by using the characteristics of the cluster center sample, and integrates it into the existing system by updating the configuration file; obtains the mutual information and adjusts the Rand index to determine whether the effectiveness of the new tone type division scheme exceeds the preset threshold; if the effectiveness of the new tone type division scheme exceeds the preset threshold, the dynamic update of the category system of the tone type library is triggered.

[0067] Specifically, the quantitative indicators of speech feature changes are obtained, including using short-time Fourier transform to extract pitch, calculating root mean square energy to obtain volume, obtaining speech speed through phoneme recognition, and using Mel frequency cepstral coefficients to represent timbre. Feature vectors are extracted from all samples in the existing tone type library to generate a high-dimensional feature matrix. The agglomerative hierarchical clustering algorithm is used to perform cluster analysis on the feature matrix, and the Euclidean distance is used as the distance measurement method. The number of clusters is gradually increased from 2 to 20, and the intra-class distance and inter-class distance of each clustering result are calculated. The intra-class distance is the average distance from the sample to the cluster center, and the inter-class distance is calculated using the average link method. The ratio of the intra-class distance to the inter-class distance under different cluster numbers is calculated, and the silhouette coefficient curve is plotted to find the cluster number corresponding to the maximum silhouette coefficient, which is determined as the optimal cluster number. According to the optimal cluster number, the tone type samples are finally clustered, and new tone type labels are assigned to each cluster cluster based on the characteristics of the cluster center samples. The category system of the tone type library is updated, and a new tone type division scheme is generated. The effectiveness of the new tone type classification scheme is evaluated using mutual information and adjusted Rand index. If the evaluation score exceeds the preset threshold of 0.8, a dynamic update is triggered. The dynamic update frequency is set to once a week, and the new tone type classification scheme is integrated into the existing system through the configuration file update method to achieve continuous optimization of the tone type system. In a business negotiation lasting 3 hours, the system extracts speech features every 5 minutes, and a total of 36 sets of feature data are obtained. The pitch is extracted using short-time Fourier transform, with a frame length of 25ms and a frame shift of 10ms to obtain the fundamental frequency curve; the root mean square energy is calculated to obtain the volume, with a window size of 100ms; the speech rate is measured by the phoneme recognizer in phonemes / second; and the 13-dimensional Mel-frequency cepstral coefficients are extracted to represent the timbre. These features are combined into a 52-dimensional vector, and feature extraction is performed on the existing 1000 tone type samples to generate a 1000x52 feature matrix. The agglomerative hierarchical clustering algorithm is used, and the sample similarity is measured using the Euclidean distance, and the number of clusters is iterated from 2 to 20. Each iteration calculates the intra-class distance, which is the average distance from the sample to the cluster center and the inter-class distance (using the average link method). Plotting the silhouette coefficient curve, it is found that when the number of clusters is 7, the silhouette coefficient reaches a maximum value of 0.68. According to this optimal number of clusters, the samples are finally clustered to obtain 7 new tone types. Based on the characteristics of the cluster center samples, labels such as "calm", "excited", "hesitant", etc. are assigned to these 7 types. The new scheme is evaluated using mutual information and adjusted Rand index, with a score of 0.85, which exceeds the preset threshold of 0.8 and triggers a dynamic update. The update process is automatically executed at 3 a.m. every Monday, and the new tone type division scheme is integrated into the existing system by modifying the configuration file, realizing the continuous optimization of the tone type system.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A digital conference data processing method, characterized in that: The method comprises: Extract speech feature parameters of speech pause duration, speech pause frequency, stressed word occurrence position, and stressed word occurrence frequency in the audio stream to obtain a speech feature vector; Perform time series analysis on the speech feature vector to identify key feature data of intonation increase, intonation decrease, and intonation stability changes in the speech, and obtain a tone type label sequence; A recurrent neural network model is used, which takes speech feature vectors and tone type label sequences as input and tone representation vectors as output; The tone representation vector is parsed based on the business negotiation tone-conference topic mapping model. If the tone-topic mapping changes, it is determined that the conference topic has switched, triggering the conference process stage transition marker. The tone representation vector is analyzed based on the business negotiation tone-emotion mapping model. If the emotional fluctuation of the participants exceeds the threshold, a marker is triggered indicating the emergence of a key decision node or the introduction of a controversial topic. Based on the business negotiation tone-meeting atmosphere mapping model, the tone representation vector is analyzed to evaluate the meeting atmosphere. If the change in the meeting atmosphere exceeds the threshold, the meeting is judged to have entered the quarrel stage and the corresponding warning is triggered. Cluster analysis is performed on the analysis results between tone and topic, emotion, and atmosphere to identify the turning points of conference topic switching, emotional fluctuations of participants, and sudden changes in conference atmosphere, and trigger turning point markers. Dynamically adjust data parameters according to turning point markers, including adjusting speech feature parameters, updating the category system of the tone type library, and fine-tuning the parsing rules of the tone-theme / emotion / atmosphere mapping model; The method of parsing the tone representation vector based on the business negotiation tone-conference topic mapping model and determining that the conference topic is switched if the tone-topic mapping changes, triggering the conference process phase transition mark, includes: Acquire a tone representation vector, and classify the tone representation vector into a predefined business negotiation tone category using a pre-trained support vector machine to obtain a current tone category label; According to the current tone category label, querying the conference topic probability distribution related to the current tone from a pre-established tone-topic probability distribution table; The sliding window method is used to calculate the JS divergence of the probability distribution of conference topics in adjacent time windows. If the JS divergence value exceeds a preset threshold, it is determined that the conference topic has switched; If it is determined that the conference theme is switched, the conference process phase transition mark is triggered, and the theme with the highest probability in the conference theme probability distribution is used as the new theme to update the current conference theme status; By adopting an online learning method, after each meeting, the newly added tone-topic correspondence is used to update the tone-topic probability distribution table to maintain the dynamic adaptability of the mapping model.

2. A digital conference data processing method according to claim 1, characterized in that: The method of extracting speech feature parameters of speech pause duration, speech pause frequency, stress word occurrence position, and stress word occurrence frequency in the audio stream to obtain a speech feature vector includes: Acquire input audio stream data, and perform time domain processing on the audio stream data using short-time energy analysis and zero-crossing rate analysis methods in sequence, to extract speech pause duration and speech pause frequency features; According to the speech pause duration and speech pause frequency characteristics, a pitch detection method is used to calculate a fundamental frequency curve to determine the occurrence position and frequency of the stressed vocabulary; Normalizing the speech pause features and the stress vocabulary features by using a minimum-maximum scaling method to obtain a standardized speech feature vector; Inputting the standardized speech feature vector into a pre-trained feedforward neural network, and obtaining a high-dimensional feature representation through activation function and weight matrix calculation; The high-dimensional feature representation calculates the Euclidean distance or cosine similarity, compares it with the standard features in the pre-trained speech feature model, and selects the feature with the smallest distance or the highest similarity as the final speech feature representation result.

3. A digital conference data processing method according to claim 1, characterized in that: The time series analysis of the speech feature vector is performed to identify key feature data of the intonation increase amplitude, intonation decrease amplitude, and intonation stability change in the speech, and obtain the tone type label sequence, including: Acquire a standardized speech feature vector, and perform time series analysis on the speech feature vector; The feature vector is processed using a two-dimensional convolutional neural network, wherein the two-dimensional convolutional neural network includes a preset number of convolutional layers, each layer uses a convolution kernel of a preset size, and a ReLU activation function is used between convolutional layers; Performing a maximum pooling operation on the local features extracted by the two-dimensional convolutional neural network, wherein the pooling window size and step size of the maximum pooling operation are preset values; Obtaining key feature data of intonation changes according to the maximum pooling operation; Reading a tone type feature template from a pre-established tone type library, and performing similarity matching between the key feature data and the tone type feature template using a Euclidean distance calculation method; If the similarity matching result is greater than a preset threshold, determining the tone label of the current speech segment; Processing the speech segment sequence using a long short-term memory network, wherein the long short-term memory network includes an LSTM layer and the number of hidden units is a preset value; By performing an argmax operation on the output of the long short-term memory network, the tone type of each time step is obtained to form a complete tone type label sequence.

4. A digital conference data processing method according to claim 1, characterized in that: The recurrent neural network model is adopted, with the speech feature vector and the tone type label sequence as input and the tone representation vector as output, including: Acquire a speech feature vector and a tone type label sequence, convert the tone type label sequence into a dense vector representation through an embedding layer, and obtain an embedding vector that matches the dimension of the speech feature vector; Using a bidirectional gated recurrent unit to perform time-series processing on the speech feature vector and the embedding vector, extracting context-related features of the two modal data, and obtaining a hidden state sequence; For the hidden state sequence, a multi-head self-attention mechanism is used to calculate the correlation between different time steps, and the attention weight is calculated by scaling the dot product attention, and the attention weight is assigned to each time step to obtain a weighted feature representation; The speech feature and the tone type information are fused according to the attention weight to obtain a fused feature; The fused features are input into a multi-layer fully connected network, each layer uses an activation function, and the last layer outputs a comprehensive tone representation vector of fixed dimension.

5. A digital conference data processing method according to claim 1, characterized in that: The business negotiation tone-emotion mapping model is based on analyzing the tone representation vector. If the emotional fluctuation of the participants exceeds the threshold, a mark indicating the appearance of a key decision node or the introduction of a controversial topic is triggered, including: Classify the tone representation vector according to the pre-trained random forest classifier to obtain the current emotion category label; The random forest classifier includes a preset number of decision trees, has a preset maximum depth, and is trained using a preset number of labeled data; Query the pre-established tone-emotion intensity mapping table; The tone-emotion intensity mapping table is a matrix of preset dimensions, with rows representing a preset number of emotion categories and columns representing a preset number of intensity levels; Obtaining an emotion intensity value corresponding to the current emotion category label; The sliding window method is used to calculate the difference between the emotion intensity value in the current time window and the emotion intensity value in the previous time window; If the difference exceeds a preset threshold, it is determined that the mood fluctuation is significant; If it is detected that the emotional fluctuation of individual participants exceeds the preset threshold, the duration of the emotional fluctuation is determined; If the emotional fluctuation lasts longer than a preset time, it is determined to be a critical decision node; If the emotional fluctuation does not last longer than the preset time, it is determined to be a controversial topic; Calculate the mean and standard deviation of the emotional intensity of all participants; If the standard deviation exceeds a preset threshold, it is determined that the overall meeting atmosphere is tense, and a corresponding meeting status mark is triggered.

6. A digital conference data processing method according to claim 1, characterized in that: The business negotiation tone-meeting atmosphere mapping model is used to analyze the tone representation vector and evaluate the meeting atmosphere. If the change in the meeting atmosphere exceeds a threshold, it is determined that the meeting has entered the quarrel stage and a corresponding warning is triggered, including: Acquire the tone representation vector, and classify the tone representation vector using a pre-trained K nearest neighbor algorithm, wherein a K value of the K nearest neighbor algorithm is set to a preset value, and the K nearest neighbor algorithm uses Euclidean distance as a distance measurement method; According to the classification result of the K nearest neighbor algorithm, a current meeting atmosphere category label is obtained; According to the conference atmosphere category label, an atmosphere intensity value related to the current tone is obtained from a pre-established tone-atmosphere intensity mapping table, wherein the tone-atmosphere intensity mapping table is a matrix, wherein the rows of the matrix represent the atmosphere categories, the columns of the matrix represent the intensity levels, and the element values ​​of the matrix represent the corresponding probabilities of the tone-atmosphere intensity; An exponentially weighted moving average method is used, wherein the smoothing factor of the exponentially weighted moving average method is set to a preset value, and the calculation period of the exponentially weighted moving average method is set to a preset time; Calculate the change between the current atmosphere intensity value and the historical atmosphere intensity value by using the exponentially weighted moving average method; If the change exceeds the preset threshold, it is determined that the meeting atmosphere has changed suddenly; Obtain tone representation vectors of multiple participants; Calculating the average of the tone representation vectors of the multiple participants to obtain a representative vector of the overall conference atmosphere; If it is determined that the atmosphere of the meeting has changed suddenly and the change exceeds a preset threshold, the meeting is marked as entering the quarrel stage; Generate a conference host's emotional warning signal, which includes two indicators: the degree of sudden change in atmosphere and the duration; Sending the warning signal to the conference host via the instant messaging module of the conference management system; The early warning signal is stored in the conference real-time status database.

7. A digital conference data processing method according to claim 1, characterized in that: The cluster analysis of the analysis results between tone and theme, emotion, and atmosphere is performed to identify the turning points of conference theme switching, emotional fluctuations of participants, and sudden changes in conference atmosphere, and trigger turning point markers, including: Obtaining continuous analysis results of tone-topic, tone-emotion and tone-atmosphere, and segmenting the analysis results using a sliding time window of a fixed size; Wherein, the window size of the sliding time window is a preset value; Applying a K-means clustering algorithm to the analysis results within each of the time windows; The K-means clustering algorithm is initialized using K-means++, and the number of clusters is set to three, corresponding to the three dimensions of theme, emotion, and atmosphere; Calculate the Euclidean distance between cluster centers of adjacent time windows and determine potential turning points based on preset thresholds; Comprehensively evaluate the identified potential turning points. If turning points occur in at least two dimensions within the preset time, they are considered key turning points. The turning points include topic switching, mood swings and sudden changes in atmosphere; Generate an early warning signal including the type of transition, the time of occurrence and the severity, and send the early warning signal to the conference host terminal through the instant messaging module of the conference management system; The warning signal is triggered when the comprehensive score calculated based on the preset weight exceeds the threshold; It also includes: setting multiple window sizes according to the characteristics of turning points of different tone types, performing dynamic clustering analysis of multiple scales simultaneously through parallel computing, and obtaining clustering result sets at different time granularities; applying a weighted fusion algorithm to the multi-scale clustering result set, assigning a weight coefficient to the clustering result of each scale according to the characteristic distribution of different types of turning points in historical data, and obtaining a comprehensive clustering result by weighted summation; if a turning event occurs at a certain moment, the moment is marked as a key turning point, triggering an early warning signal; determining the early warning level according to the attributes of the turning amplitude and duration, generating a turning point marking box, the color of the marking box is set according to the early warning level, and the associated theme and emotional information are added to the marking box; when the mouse hovers over the marking box, the attribute information of the turning point is displayed, and clicking the marking box can jump to the corresponding voice clip; The method comprises the following steps: setting multiple window sizes according to the characteristics of turning points of different tone types, and performing dynamic clustering analysis of multiple scales simultaneously by parallel computing to obtain clustering result sets at different time granularities; applying a weighted fusion algorithm to the multi-scale clustering result set, assigning a weight coefficient to the clustering result of each scale according to the characteristic distribution of different types of turning points in historical data, and obtaining a comprehensive clustering result by weighted summation, specifically comprising: obtaining a tone data stream and performing sliding window processing with multiple sets of preset window sizes to obtain multiple sets of time series data; performing dynamic clustering analysis according to the multiple sets of time series data, and the dynamic clustering analysis is performed by The method is implemented through the Spark distributed computing platform to obtain a plurality of clustering result sets of different time granularities; the turning point feature distribution data is extracted from the historical database, and the occurrence frequency of each type of turning point under a plurality of window sizes is calculated based on the turning point feature distribution data, and the frequency value is normalized to obtain the weight coefficient; for a plurality of clustering result sets, a weighted fusion algorithm is used to perform weighted summation using the weight coefficient to obtain a comprehensive clustering result; the comprehensive clustering result is quality evaluated, and if the evaluation result is lower than a preset threshold, the weight coefficient is adjusted and the weighted fusion algorithm is re-executed until the evaluation result reaches the preset threshold or the number of iterations reaches the upper limit; When a turning point event occurs at a certain moment, the moment is marked as a critical turning point and a warning signal is triggered; the warning level is determined according to the turning amplitude and duration attributes, a turning point marking box is generated, the color of the marking box is set according to the warning level, and the associated theme and emotional information are added to the marking box; when the mouse hovers over the marking box, the attribute information of the turning point is displayed, and clicking the marking box can jump to the corresponding voice clip, specifically including: obtaining the timestamp of the turning point event, and the timestamp is used to mark the critical turning point; judging the warning level of the turning point event according to the preset turning amplitude threshold and duration threshold; if the turning amplitude is less than the first preset threshold and the duration is less than the second preset threshold, the warning level is determined to be low; if the turning amplitude is greater than or equal to the first preset threshold and less than the third preset threshold, or the duration is greater than or equal to the third preset threshold, If the turning amplitude is greater than or equal to the third preset threshold, or the duration is greater than or equal to the fourth preset threshold, the warning level is determined to be intermediate; if the turning amplitude is greater than or equal to the third preset threshold, or the duration is greater than or equal to the fourth preset threshold, the warning level is determined to be advanced; the color of the mark box is set according to the warning level, and a rectangular mark box object is created; the subject and emotional information corresponding to the key turning point are extracted from the speech analysis results, the subject is obtained by extracting keywords through the TF-IDF algorithm, and the emotional information is obtained by matching the emotional dictionary; the subject and the emotional information are added to the properties of the mark box object; the mouse event processing function is bound to the mark box object; when a mouse hover event is detected, the detailed attribute information of the mark box object is displayed; when a mouse click event is detected, the audio player API is called according to the timestamp to locate and play the corresponding voice clip.

8. A digital conference data processing method according to claim 1, characterized in that: The method of dynamically adjusting data parameters according to turning point markers includes adjusting speech feature parameters, updating the category system of the tone type library, and fine-tuning the parsing rules of the tone-theme / emotion / atmosphere mapping model, including: According to the turning point mark of the conference process, the voice data before and after the turning point is obtained, the change range of the voice feature parameters is calculated, and the difference value between the feature vectors is obtained; If the difference value exceeds a preset threshold, adjusting the parameters of the speech feature extraction algorithm, including adjusting the audio sampling rate from the first frequency to the second frequency, adjusting the frame length from the first length to the second length, and adjusting the frame shift from the first interval to the second interval; The distribution of tone types at the turning points was statistically analyzed, and the chi-square test was used to determine the significant changes in the frequency of tone types. If the test result is significant, the category system of the tone type library is updated; According to the changes in the mapping relationships between tone-topic, tone-emotion and tone-atmosphere before and after the turning point, the stochastic gradient descent algorithm is used to fine-tune the mapping model and update the model weight parameters; Reprocess the historical conference data based on the adjusted speech feature parameters, updated tone type library, and fine-tuned mapping model, and calculate the change in analysis accuracy; If the change value exceeds the preset accuracy improvement threshold, the new parameters and models are applied to the real-time analysis process; it also includes: according to the quantitative index of speech feature change, a clustering algorithm is used to re-cluster the samples in the existing tone type library, and the optimal number of clusters is determined by calculating the ratio of the intra-class distance and the inter-class distance, and the category system of the tone type is dynamically updated to obtain a new tone type classification scheme, which specifically includes: obtaining the quantitative index of speech feature change, extracting the pitch by the short-time Fourier transform method, calculating the volume according to the root mean square energy, obtaining the speaking speed by phoneme recognition, and using the Mel frequency cepstral coefficient to represent the timbre; extracting feature vectors for the samples in the tone type library to obtain a high-dimensional feature matrix; and A cluster analysis is performed on the high-dimensional feature matrix using an agglomerative hierarchical clustering algorithm, and the Euclidean distance is used as a distance measurement method; the ratio of the intra-class distance to the inter-class distance in the cluster analysis result is calculated, and the number of clusters corresponding to the maximum value of the silhouette coefficient is determined to be the optimal number of clusters; the tone type samples are finally clustered according to the optimal number of clusters, and a new tone type label is assigned to each cluster cluster using the characteristics of the cluster center sample, and the label is integrated into the existing system by updating the configuration file; the mutual information is obtained and the Rand index is adjusted to determine whether the effectiveness of the new tone type division scheme exceeds a preset threshold; if the effectiveness of the new tone type division scheme exceeds the preset threshold, the dynamic update of the category system of the tone type library is triggered.

9. A digital conference data processing system, characterized in that: The digital conference data processing method according to any one of claims 1 to 8 is applied, wherein the system comprises: Speech feature extraction module, used to extract speech feature parameters such as speech pause duration, stressed words, etc. The timing analysis module is used to analyze the amplitude and smoothness of intonation and generate a sequence of tone type labels; A recurrent neural network module, which is used to input a speech feature vector and a tone label sequence and output a tone representation vector; Tone analysis module, used to analyze tone representation vectors and determine the switching of meeting topics, emotional fluctuations, and changes in meeting atmosphere; Cluster analysis module, used to perform cluster analysis on the analysis results and identify turning points of conference topic switching, emotional fluctuations, and sudden changes in atmosphere; The data parameter dynamic adjustment module is used to adjust the speech feature parameters and update the parsing rules according to the turning point markers.

Images