Meeting minutes processing method and system using AI

Abstract

The present invention relates to the field of intelligent conference processing technology, and relates to a conference minutes processing method and system using AI, including real-time noise suppression of collected conference audio streams and associated text data through a noise suppression algorithm, and cross-modal alignment of the noise-reduced data using a cross-modal alignment algorithm; inserting domain-specific attention heads into the attention layer of a pre-trained Transformer model to construct a domain-enhanced speech recognition model to convert audio into a text sequence with speaker labels; using a heterogeneous graph neural network to construct a structured topic evolution graph; extracting key decision nodes in the structured topic evolution graph based on a reinforcement learning strategy to generate a final conference summary document. In the decoding stage, the fusion ratio of the acoustic model and the language model is dynamically adjusted based on real-time acoustic confidence, thereby improving the recognition rate of professional vocabulary and effectively reducing the frequent transcription errors and poor semantic coherence of terminology in professional meetings.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent conference processing, and in particular to a conference minutes processing method and system using AI. Background Art

[0002] Meetings are often held within companies, and meeting minutes are recorded after the meeting so that people who did not attend the meeting can read the minutes to understand the content of the meeting, and people who attended the meeting can read it again to review the content of the meeting. Meeting minutes are used to record key information such as meeting discussion content, decision-making matters, and task division in a structured manner. However, the content of meeting minutes recorded in text is relatively large. With the development of AI intelligence, AI can achieve real-time transcription and automatically generate complete meeting minutes, realizing real-time recording and summary of meeting content.

[0003] However, due to the lack of targeted optimization of industry terminology, the misrecognition rate of domain vocabulary is high, term transcription errors occur frequently in professional meetings, and the language model weights cannot be dynamically adjusted according to the context, resulting in poor semantic coherence. Therefore, a meeting minutes processing method and system using AI are provided. Summary of the Invention

[0004] The purpose of the present invention is to provide a meeting minutes processing method and system using AI to solve the problems of lack of targeted optimization of industry terminology, resulting in a high misrecognition rate of domain vocabulary, frequent term transcription errors in professional meetings, and the inability to dynamically adjust the language model weights according to the context, resulting in poor semantic coherence.

[0005] To achieve the above objectives, the present invention provides a method for processing meeting minutes using AI, comprising the following steps:

[0006] S1. Use multi-source sensors to collect conference audio streams and associated text data, perform real-time noise suppression on the collected conference audio streams and associated text data through noise suppression algorithms, and use cross-modal alignment algorithms to perform cross-modal alignment on the noise-reduced data to obtain fusion features. ;

[0007] S2. Deploy a hybrid CTC / Attention end-to-end ASR model and insert domain-specific attention heads into the attention layer of the pre-trained Transformer model to build a domain-enhanced speech recognition model for converting audio into text sequences with speaker labels. ;

[0008] S3, based on fusion features , text sequences with speaker labels and PPT chapter structure By analyzing the semantic roles and temporal relationships of the text, a heterogeneous graph neural network is used to construct a structured topic evolution map. ;

[0009] S4. Extracting structured topic evolution graphs based on reinforcement learning strategies The key decision nodes in the meeting are used to generate the final meeting summary document.

[0010] As a further improvement of the present technical solution, in S1, the multi-source sensor includes an audio sensor, a visual sensor, and a text input source;

[0011] The audio sensor captures the speaker's audio directionally through a beamforming algorithm;

[0012] The text input source is connected to the PPT / PDF document parser in real time to extract text content and page turning event timestamps.

[0013] As a further improvement of the present technical solution, in S1, the specific steps involved in performing real-time noise suppression on the collected data using a noise suppression algorithm are:

[0014] The original audio signal is denoised by spectral subtraction to obtain the denoised amplitude spectrum;

[0015] The amplitude spectrum and phase spectrum after spectral subtraction are input into the deep neural network for refinement and noise reduction to obtain the denoised audio signal. .

[0016] As a further improvement of the present technical solution, in S1, the cross-modal alignment algorithm performs cross-modal alignment on the denoised data, and the specific steps involved are:

[0017] Use SlideTimer algorithm to parse PPT page turning log and extract chapter time collection ;

[0018] Detect silent segments through Silero-VAD and segment them into paragraph boundary sets ;

[0019] The chapter time set and paragraph boundary set are constructed into a candidate pairing set. The Hungarian algorithm is used to filter the legal timestamp pairing sets in the candidate pairing set to achieve coarse-grained time alignment.

[0020] The output is coarse-grained aligned chapter-audio paragraph pairs;

[0021] Training audio encoders via a contrastive learning framework With text encoder , and construct a unified feature space:

[0022] ;

[0023] Where, The loss value used to measure the cross-modal alignment of audio and text; Represents an audio encoder; Represents a text encoder; Represents an audio clip Feature vector extracted by audio encoder; Represents a text fragment Feature vector extracted by text encoder; Represents negative sample text, which refers to the audio clip Unmatched text fragments; Indicates the preset threshold;

[0024] Among them, the associated audio clip With text snippets In the embedding space, it satisfies:

[0025] ,and ;

[0026] Where, Represents the audio codec for the audio segment The encoding result of Represents the text encoder to the text segment The encoding result of express and The cosine similarity of Represents the preset boundary margin, which is used to enhance the discrimination between positive and negative samples;

[0027] Indicates that all Unmatched text fragments All satisfied ;

[0028] Based on the coarse-grained aligned chapter-audio paragraph pairs, the similarity between the chapter title and the audio paragraph is calculated;

[0029] An improved dynamic time warping algorithm is used to align sentence fragments, combined with a semantic cost function to compensate for local timing jitter.

[0030] Use CRF sequence tagging to identify professional terms and label the transfer matrix Dynamic updates;

[0031] Dynamically calculate weights based on real-time audio feature vectors and text feature vectors ;

[0032] Through the audio projection matrix and the text projection matrix Unify audio / text features to 512 dimensions to generate fusion features .

[0033] As a further improvement of this technical solution, the specific steps involved in constructing the domain-enhanced speech recognition model are:

[0034] Deploy a hybrid CTC / Attention end-to-end ASR model;

[0035] Dynamically adjust the language model weights during the decoding phase through a context-adaptive mechanism;

[0036] Build a gated network that includes acoustic confidence evaluation and receive the posterior probability distribution output by the acoustic model in real time ;

[0037] Analyze the vocabulary distribution of the current speech segment through the pre-trained domain term detector to generate a domain relevance score ;

[0038] The fusion features As the input of the gating network, the acoustic confidence feature is introduced at the same time and domain relevance scores , calculate the dynamic language model weight coefficient ;

[0039] Perform context-aware model fusion to generate final decoding probabilities ;

[0040] At the same time, K domain-specific attention heads are inserted into the attention layer of the pre-trained Transformer model to form a hybrid attention structure and construct a learnable domain mask matrix , and adopts a multi-task transfer learning strategy to optimize the model parameters and transcribe the audio features into text sequences.

[0041] As a further improvement of this technical solution, the specific steps involved in converting the audio into the initial text with speaker tags are:

[0042] Parallel extraction of denoised audio signals MFCC features and FBANK features , the bimodal features are concatenated into ;

[0043] Will As the input of the GMM-UBM model, the speaker embedding vector is obtained by maximum a posteriori probability estimation ;

[0044] Based on t-SNE dimensionality reduction and DBSCAN clustering algorithm, the voiceprint feature vector is collected Divided into Speaker clusters, output voiceprint cluster labels ;

[0045] Clustering paragraph text based on each voiceprint and fusion features , use Bi-LSTM+CRF model to calculate the semantic role probability distribution The semantic roles are cross-validated with the voiceprint clustering results, and the corresponding speaker identity is finally marked for each text segment to generate the final speaker label. ;

[0046] Introducing semantic similarity constraints and fusion features in dynamic time warping The temporal features of Align with PPT chapter information to generate text sequences with speaker labels ;

[0047] Among them, the text sequence with speaker labels is:

[0048]

[0049] Where, Represents text content; Indicates a timestamp; Speaker label that combines voiceprint and semantics.

[0050] As a further improvement of this technical solution, the specific steps involved in constructing a structured topic evolution graph in S3 are:

[0051] The fusion features , text sequences with speaker labels and PPT chapter structure As input, the PPT chapter structure Extracted by PPT / PDF document parser;

[0052] For text sequences with speaker labels Perform multimodal feature extraction to extract semantic, acoustic and temporal features, and fuse semantic role embedding with time interval information to generate a semantic representation vector for the text ;

[0053] The fusion features Through linear projection and semantic representation vector Splicing to generate joint features ;

[0054] The nested Dirichlet process is used to model the topic nodes of the conference content, mining the parent-child topic hierarchy in the conference text and generating topic nodes. ;

[0055] From text sequences with speaker labels Extract Speakers, forming a speaker node , and record the speaker's corresponding speaking frequency and voiceprint confidence;

[0056] According to the input PPT chapter structure , extract chapter nodes ;

[0057] The complete node set ;

[0058] Based on the semantic role embedding vector and the topic node embedding vector, a role-topic association scoring model is constructed to quantify the association strength between different semantic roles and specific topics, and dynamically generate contribution relationship edges based on the speaking frequency and topic popularity. ;

[0059] By constructing a temporal adjacency matrix , quantitative issues and topics The transfer relationship between them is combined with the time graph attention network to dynamically generate evolutionary relationship edges. ;

[0060] For each chapter node With each topic node , calculate chapter nodes and topic nodes The semantic similarity of the relationship is used as the inclusion relationship edge. The weight of the inclusion relationship edge ;

[0061] The complete edge set ;

[0062] Based on heterogeneous graph neural network for complete node set and the complete edge set Perform multi-layer message delivery and node updates, introduce timestamps, and generate a structured topic evolution graph .

[0063] As a further improvement of this technical solution, in S4, the specific steps involved in generating the final meeting summary document are:

[0064] Each topic node in the graph and its semantic features, temporal features and speaker contributions are used as state input;

[0065] Use deep Q-learning to select the topic node that contributes most to the summary from the graph ;

[0066] For each key decision node, the topic text slot value is determined by the term distribution model, the speaker slot value is determined by the maximum contribution edge weight, the decision content slot value is generated by the SRL model, and the chapter nodes are aligned by cosine similarity to output the time interval slot value and chapter slot value;

[0067] The above slot values ​​are spliced ​​in the order of the preset template to form the final meeting summary document.

[0068] On the other hand, the present invention provides a meeting minutes processing system using AI, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor executes the computer program to implement any one of the above-mentioned meeting minutes processing methods using AI.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] 1. This AI-based meeting minutes processing method and system uses a hybrid CTC / Attention end-to-end ASR model, combined with domain-specific attention heads and a dynamic language model weight adjustment mechanism, to perform targeted optimization of industry terminology. During the decoding phase, the fusion ratio of the acoustic model and language model is dynamically adjusted based on real-time acoustic confidence, improving the recognition rate of professional vocabulary and effectively reducing the frequent transcription errors and poor semantic coherence in professional meetings.

[0071] 2. This AI-based meeting minutes processing method and system uses a heterogeneous graph neural network to construct a structured topic evolution graph, integrate topic nodes, speaker contributions, and PPT chapter information, dynamically quantify the association strength between semantic roles and topics, and achieve a multi-dimensional structured representation of meeting content, facilitating the extraction and tracing of key information.

[0072] At the same time, based on the reinforcement learning strategy, key decision nodes are extracted from the graph, and combined with the term distribution model, semantic role labeling and time alignment, a meeting summary containing key slots such as topics, speakers, decision content, time intervals, etc. is automatically generated to ensure that the content is accurate and consistent with the actual discussion logic. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 The figure is a flow chart of the overall method of the present invention. DETAILED DESCRIPTION

[0074] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0075] Example 1:

[0076] See also Figure 1 As shown, this embodiment provides a method for processing meeting minutes using AI, including the following steps:

[0077] S1. Use multi-source sensors to collect conference audio streams and associated text data, perform real-time noise suppression on the collected conference audio streams and associated text data through noise suppression algorithms, and use cross-modal alignment algorithms to perform cross-modal alignment on the noise-reduced data to obtain fusion features. ;

[0078] In this embodiment, the multi-source sensor includes an audio sensor, a visual sensor, and a text input source;

[0079] The audio sensor captures the speaker's audio directionally through a beamforming algorithm;

[0080] The text input source is connected to the PPT / PDF document parser in real time to extract text content and page turning event timestamps.

[0081] Specifically, the specific steps involved in performing real-time noise suppression on the collected data using the noise suppression algorithm are:

[0082] The original audio signal is de-noised by spectral subtraction to obtain the de-noised amplitude spectrum;

[0083] The amplitude spectrum and phase spectrum after spectral subtraction are input into the deep neural network for refinement and noise reduction to obtain the denoised audio signal. ;

[0084] Real-time noise reduction is achieved by a noise suppression algorithm that combines spectral subtraction with a deep neural network. The noise suppression coefficient α(t) is calculated through a dynamic prediction model consisting of 5 convolutional layers and a GRU layer.

[0085] In this embodiment, the cross-modal alignment algorithm performs cross-modal alignment on the denoised data, and the specific steps involved are:

[0086] Use SlideTimer algorithm to parse PPT page turning log and extract chapter time collection ; Used to obtain the structured time tag of the text modality for subsequent global alignment;

[0087] Detect silent segments through Silero-VAD and segment them into paragraph boundary sets ; Used to divide the continuous audio stream into semantically complete paragraphs to facilitate alignment with text chapters;

[0088] The chapter time set and the paragraph boundary set are constructed into a candidate pairing set. The Hungarian algorithm is used to filter the legal timestamp pairing sets in the candidate pairing set to achieve coarse-grained time alignment:

[0089] Perform Hungarian algorithm matching Yes, and satisfied ; ( = 5 seconds), achieving coarse-grained time alignment to eliminate global timing misalignment between PPT presentations and speech;

[0090] The output is coarse-grained aligned chapter-audio paragraph pairs, providing spatiotemporally correlated positive samples for encoder training.

[0091] Furthermore, the audio encoder is trained using a contrastive learning framework. With text encoder , and construct a unified feature space to generate an encoder that can measure cross-modal similarity, providing a similarity calculation basis for hierarchical alignment:

[0092] ;

[0093] Where, The loss value used to measure the cross-modal alignment of audio and text; Represents an audio encoder; Represents a text encoder; Represents an audio clip 256-dimensional feature vector extracted by the audio encoder; Represents a text fragment A 256-dimensional feature vector extracted by a text encoder (such as BERT-Base); Represents negative sample text, which refers to the audio clip Unmatched text fragments; Represents a preset threshold (usually 0.2–0.3), requiring that the similarity of positive sample pairs is at least higher than that of negative sample pairs , otherwise it triggers loss;

[0094] Among them, the associated audio clip With text snippets In the embedding space, it satisfies:

[0095] ,and ;

[0096] Where, Represents the audio codec for the audio segment The encoding result of Represents the text encoder to the text segment The encoding result of express and The cosine similarity of ); Represents the preset boundary margin, which is used to enhance the discrimination between positive and negative samples. The value is 0.2-0.3; Indicates that all Unmatched text fragments All satisfied ;

[0097] Positive samples , from the same chapter-paragraph pair; negative samples , random sampling or hard negative mining in the same batch;

[0098] Map audio features (such as MFCC, FBANK) and visual features (such as facial key points, optical flow) to the same feature space through a unified encoder;

[0099] Audio-text pairs associated by the loss function are brought closer together in the embedding space, while unrelated pairs are moved away, forming a cross-modal semantic bridge.

[0100] Among them, the audio encoder Based on 1D-ResNet, input the denoised audio spectrum and output a 256-dimensional vector ;

[0101] Text Encoder Based on the BERT-Base model, input text fragments and output 256-dimensional vectors ;

[0102] In this embodiment, the similarity between chapter titles and audio paragraphs is calculated based on the coarse-grained aligned chapter-audio paragraph pairs;

[0103] An improved dynamic time warping algorithm is used to align sentence fragments, combined with a semantic cost function to compensate for local timing jitter.

[0104] The semantic cost function is:

[0105]

[0106] Where, Represents the semantic cost function used to measure the chapter title With audio clips The smaller the semantic distance between them, the more similar their semantics are. Indicates the Chapter titles; Indicates the audio segments; Indicates that the chapter title is converted through a text encoder (such as BERT) Feature vectors mapped to high-dimensional semantic space; Indicates that the audio segment is converted through an audio encoder (such as 1D-ResNet) Feature vectors mapped to high-dimensional semantic space; Represents a transpose operation;

[0107] Use CRF sequence tagging to identify professional terms and label the transition matrix Dynamically update and generate three-level alignment results, providing structured multi-granularity alignment guidance for feature fusion, where: Represents the total number of label categories in the CRF (conditional random field) sequence labeling model;

[0108] Dynamically calculate weights based on real-time audio feature vectors and text feature vectors ;

[0109] Dynamic weight calculation:

[0110] ;

[0111] in, is the audio feature vector, the original feature extracted from the audio signal; is the text feature vector, the original feature extracted from the text data; is the training parameter matrix; Represents the Sigmoid function; represents the weight matrix;

[0112] Through the audio projection matrix and the text projection matrix Unify audio / text features to 512 dimensions, eliminate dimensionality differences, achieve additive fusion, and generate fused features :

[0113]

[0114] Where, Represents the audio projection matrix; Represents the text projection matrix; Represents dynamic weights, and the audio projection matrix and the text projection matrix They are learned during the training process through backpropagation and optimization algorithms (such as gradient descent), and dynamically adjusted according to the input data and task objectives.

[0115] Generating robust fusion features , providing noise-immune cross-modal representations for downstream summary generation;

[0116] In response to input stream delays and sudden interference, when the input stream delay is greater than 200ms, the lightweight optimizer is activated to update parameters to prevent cumulative errors from causing system drift:

[0117]

[0118] Learning rate ;

[0119] Where, Represents the model parameters, at time step The neural network weights of represents the updated model parameters; represents the dynamic learning rate, the parameter value adjusted by gradient descent; represents the alignment loss function Parameters gradient; represents the initial learning rate; represents the learning rate adjustment function; Represents the current batch of data, at time step The input data batch is used to calculate the loss and gradient; SNR stands for signal-to-noise ratio. The lower the SNR, the smaller the learning rate to prevent unstable updates under noise interference;

[0120] Through the equation of state When the predicted feature changes and the residual exceeds the limit (when the residual ) Start parameter fine-tuning to resolve feature distortion caused by sudden interference;

[0121] Where, Indicates that at time step The predicted feature vector of , that is, the current state predicted by the model; represents the state transition matrix; Indicates that at time step The actual eigenvector of ; represents the system noise term; Indicates the preset residual threshold; Represents the time step The actual observed feature vector of

[0122] Ensure the system's continuous and stable operation in complex conference scenarios (such as network fluctuations and sudden noise).

[0123] S2. Deploy a hybrid CTC / Attention end-to-end ASR model and insert domain-specific attention heads into the attention layer of the pre-trained Transformer model. In addition, insert K “dialect-adaptive attention heads” in parallel to build a domain-enhanced speech recognition model for converting audio into text sequences with speaker labels. ;

[0124] The specific steps involved in building a domain-enhanced speech recognition model are:

[0125] Deploy a hybrid CTC / Attention end-to-end ASR model, using CTC loss to align frame-level timing and Attention to capture long-range dependencies;

[0126] Furthermore, a dialect classification subnetwork is inserted into the front end of the ASR model encoder to perform real-time dialect type discrimination on the pronunciation style in the input audio stream. The dialect classification subnetwork includes a multi-layer causal convolutional network and a lightweight self-attention layer, and outputs the dialect embedding vector corresponding to the audio segment. , which is used to solve the acoustic model mismatch caused by regional dialects or Mandarin pronunciation differences, thereby reducing the recognition error rate of terms and common words. In each attention layer of the pre-trained Transformer model, in addition to inserting domain-specific attention heads, K "dialect-adaptive attention heads" are also inserted in parallel. Its query / key / value matrix is ​​composed of the dialect embedding vector Linear mapping generation: During multi-head attention calculations, the outputs of the dialect adaptation head and the general attention head are weighted and merged through gated fusion to achieve adaptive bias for pronunciations from different regions. Furthermore, a dialect relevance score is introduced during the decoding stage, enabling the decoding process to flexibly enhance or suppress the language model's prediction confidence for specific phonemes or words based on the current speaker's dialect characteristics, thereby further reducing recognition errors in dialect scenarios.

[0127] The encoder uses a Conformer architecture, which includes a convolutional gating module (capturing local acoustic features), a multi-head self-attention module (modeling global temporal dependencies), and a feedforward network (nonlinear feature transformation).

[0128] The decoder executes synchronously:

[0129] CTC path: directly based on encoder output Compute frame-level posterior distribution ;

[0130] Attention path: dynamically aggregate context through attention weights, calculate ,in, Indicates based on encoder output and the generated Tags , the sequence-level posterior probability distribution of the Attention path output, indicating the generation of the Tags probability;

[0131] Combining the temporal alignment capability of CTC with the long-distance dependency capture of Attention improves the robustness of the ASR model;

[0132] Build a neural network with an encoder-decoder structure, where the encoder maps the acoustic feature sequence to a high-level representation, and the decoder performs the following operations simultaneously:

[0133] Calculate the first loss term based on the frame-level alignment path of Connectionist Temporal Classification (CTC) :

[0134]

[0135] in, Indicates the The target label of the frame, Indicates the encoder The hidden state of the moment;

[0136] Calculate the second loss term based on the sequence dependency modeling path of the attention mechanism :

[0137]

[0138] in, Indicates the target sequence length; Represents the global context representation of the encoder output; Indicates the tags;

[0139] Through learnable dynamic weight coefficients Dynamically fuse the first loss term and the second loss term to form a joint optimization objective :

[0140]

[0141] in, The value of is dynamically adjusted according to the signal-to-noise ratio of the current input speech: when the signal-to-noise ratio is lower than the preset threshold, the value of is increased. To strengthen the timing constraints of the CTC path; when long-distance semantic dependencies are detected, reduce To enhance the context modeling of the Attention path; represents CTC loss;

[0142] Dynamic weight coefficient Compute via a gating network:

[0143]

[0144] in, Represents the sigmoid function; Indicates the encoder The hidden state of the moment; Represents the Attention context vector; represents the learnable weight matrix; represents the learnable bias term;

[0145] In this embodiment, the language model weight is dynamically adjusted during the decoding phase through a context-adaptive mechanism. The contribution weights of the acoustic and language models are dynamically adjusted based on the real-time speech content. When the acoustic feature confidence is high (such as regular words with clear pronunciation), the acoustic model is given priority. When domain terms appear, the language model weight is increased, and pre-trained knowledge is used to complete possible ambiguous parts of the pronunciation.

[0146] Build a gated network containing an acoustic confidence assessment module to receive the posterior probability distribution output by the acoustic model in real time ;

[0147] Analyze the vocabulary distribution of the current speech segment through the pre-trained domain term detector to generate a domain relevance score , reduce the interference of common vocabulary on professional scenarios;

[0148] The fusion features As the input of the gating network, the acoustic confidence feature is introduced at the same time and domain relevance scores , calculate the dynamic language model weight coefficient :

[0149]

[0150] Where, Represents the Sigmoid activation function; represents the bias term;

[0151] Specifically, acoustic confidence features The specific calculation method is:

[0152] Extract the entropy value of the ASR model output probability distribution;

[0153] Calculate the KL divergence between the current frame and its 5 adjacent frames;

[0154] Combine the above features into ;

[0155] When high domain relevance is detected (such as the appearance of professional terms), the language model weight is increased and pre-trained knowledge is used to complete the ambiguous part of the pronunciation;

[0156] In regular conversation scenarios, prioritize trusting the acoustic model to avoid semantic bias caused by over-reliance on the language model.

[0157] Perform context-aware model fusion to generate final decoding probabilities :

[0158]

[0159] in, ,Will As auxiliary input, dynamically adjust the fusion weight of the acoustic model and the language model, Represents the mapping function implemented by the gating network, and the input is the cross-modal fusion feature ;

[0160] Where, Represents the input speech signal, such as audio waveform, spectrogram, or multimodal fusion features; represents the target output sequence, i.e., the text labels to be generated (such as characters, words, or phonemes); Represents generating text sequences based on general language knowledge probability; Indicates that the input voice signal Generate text sequence probability; Represents the dynamic language model weight coefficient, with a value range of [0,1], which is used to balance the contributions of the language model and the acoustic model;

[0161] In common vocabulary scenarios with clear pronunciation (high acoustic confidence), acoustic model output is preferred to ensure transcription efficiency;

[0162] In scenarios with specialized terminology or noise interference (low acoustic confidence), the language model is relied upon to complete semantics and ensure term accuracy and contextual coherence.

[0163] In this embodiment, the domain term detector is implemented as follows:

[0164] Build a domain keyword library And generate the corresponding word embedding matrix ;

[0165] Calculate the semantic similarity between the current speech segment and the domain keyword library through the sliding window, that is, the domain relevance score ;

[0166] Among them, the domain relevance score Specifically:

[0167]

[0168] Where, Represents a collection of professional terms or keywords related to a specific field, for example, in the medical field, Contains words such as "diagnosis", "antibiotics", and "CT scan"; Represents the word embedding matrix; Represents the speech segment in the sliding window, indicating the time point The speech segment is centered and extended by 5 time units before and after (the total window length is 11); Represents a bidirectional long short-term memory network, which is used to process time series data (such as speech fragments) and extract the semantic features of the current window through bidirectional context; represents the cosine similarity function; Represents domain keywords and traverses keyword sets Each of , calculate its semantic similarity with the current speech segment;

[0169] In this embodiment, the language model weight is dynamically adjusted :

[0170]

[0171] Where, Represents the Sigmoid function; represents the weight matrix; represents the acoustic eigenvector; represents the bias term;

[0172] Language model weight coefficient , through the real-time calculation of the gating network, the gating function Adopt sigmoid linear unit (SiLU);

[0173] Furthermore, K domain-specific attention heads are inserted into the attention layer of the pre-trained Transformer model (the Transformer model is the acoustic encoding core of the ASR system. Through the collaborative design of the domain-specific attention mechanism and the mask matrix, the modeling accuracy of professional terms is significantly improved while maintaining the basic speech recognition ability) to form a hybrid attention structure. In the conference scenario, attention heads related to professional terms are injected (3 domain-specific heads are inserted into the standard 12-head Transformer) to enhance the sensitivity to professional vocabulary and retain the basic modeling ability of common speech features (such as intonation and stress). Only the newly added domain-related parameter layer (such as the Query matrix of the topic node attention head) is optimized. The standard attention head maintains the original parameters unchanged and continues to capture common speech features. The domain-specific attention head adopts a restricted parameter update strategy and only fine-tunes its query matrix and key matrix to construct a learnable domain mask matrix. , and uses a multi-task transfer learning strategy to optimize model parameters and transcribe audio features into text sequences; used for preliminary transcription of audio into high-quality text;

[0174] Through a learnable domain mask matrix Strengthen professional terminology recognition, enhance the model's sensitivity to domain keywords, and reduce the interference of general vocabulary in professional scenarios;

[0175] Specifically, the computation process of the domain-specific attention head is defined as:

[0176]

[0177] Where, To query the projection matrix; is the key projection matrix; represents the query matrix; Represents the value projection matrix; represents the bond matrix; represents the value matrix; Represents the domain mask matrix, the attention bias matrix generated by embedding professional terms, which is used to enhance the attention to the relevant positions of the terms; represents the dimension of the attention head; Indicates the The output of each domain-specific attention head is used to enhance the modeling of professional terminology;

[0178]

[0179] Where, A word embedding matrix representing a dictionary of specialized terms; is the trainable projection weight; represents the rectified linear unit activation function;

[0180] Use multi-task transfer learning strategy to optimize model parameters:

[0181]

[0182] Where, represents the domain classification loss based on attention weights; Represents the weight coefficient of the domain classification loss, initially set to 0.5, and adjusted according to the validation set effect (if the domain classification task interferes with the main task, then reduce value; Represents domain classification loss, an auxiliary task loss function, which classifies the domain of input data (such as medical, financial, and technical conferences) based on attention weights, enhancing the model's adaptability to professional scenarios; Represents the main task loss function, a joint loss of hybrid CTC and attention mechanism, used to optimize the speech recognition task (audio to text transcription); represents the gradient operator; represents the learning rate; represents the basic model parameters; Represents the updated model parameters.

[0183] In this embodiment, the specific steps involved in converting the audio into the initial text with speaker tags are:

[0184] Parallel extraction of denoised audio signals MFCC features and FBANK features , the bimodal features are concatenated into ;in, , ;

[0185] Will As the input of the GMM-UBM model (Gaussian mixture model), the speaker embedding vector is obtained by maximum a posteriori probability estimation ; Use Gaussian mixture model (GMM-UBM) to distinguish the fundamental frequency and formant characteristics of different speakers;

[0186] Speaker Embedding Vector

[0187] Where, represents the general background model parameters; Represents the adaptive GMM parameters; Indicates the expectation of all time frames; Indicates the Joint feature vector of frames; Indicated in the model Next, Features The logarithmic probability of

[0188] The training steps of the GMM-UBM model are:

[0189] The UBM is constructed using 512 Gaussian components and the diagonal covariance matrix is ​​initialized;

[0190] Iterative optimization is performed using the Expectation Maximization (EM) algorithm, where only the mean supervector is updated during the adaptation phase;

[0191] Speaker clustering is performed based on Bhattacharyya distance, and the merging threshold is set to ;

[0192] Based on t-SNE dimensionality reduction and DBSCAN clustering algorithm, the voiceprint feature vector is collected Divided into Speaker clusters, output voiceprint cluster labels , cluster the voiceprints to achieve speaker separation, where, express A set of voiceprint feature vectors, where It is The feature vector of the voiceprint sample; and All represent index variables;

[0193] In this embodiment, the text of each voiceprint cluster paragraph and fusion features , is the sequence length; the Bi-LSTM+CRF model is used to calculate the semantic role probability distribution , perform semantic role annotation (such as "questioner" and "decision maker"), and cross-validate the semantic role with the voiceprint clustering results, and finally annotate the corresponding speaker identity for each text segment to generate the final speaker label ; Responsible for accurately labeling the text with the speaker;

[0194] Semantic Role Labeling:

[0195] Split the text into word / character sequences and encode them into vectors;

[0196] Bi-LSTM generates a hidden state for each word that incorporates bidirectional context ;

[0197] Will and hidden state Splicing, as the input feature of the CRF layer, generates enhanced comprehensive features As CRF input to enhance the accuracy of semantic role classification;

[0198] CRF is based on comprehensive features Calculate the global optimal solution of the label sequence and calculate the distribution of semantic role labels:

[0199]

[0200] Where, represents the CRF state transition matrix; represents the comprehensive feature vector; Indicates the Features of words Emission score for each tag; represents the emission matrix; represents any possible sequence of labels;

[0201] Assign semantic role labels to each word (e.g., “B-questioner,” “I-decision maker”);

[0202] Calculating voiceprint confidence score :

[0203]

[0204] Evaluating semantic consistency scores :

[0205]

[0206] Where, Represents the target voiceprint feature vector, extracted by GMM-UBM or deep neural network The feature vector of each voiceprint sample represents the speaker's acoustic characteristics (such as fundamental frequency and formant); Indicates the The center of a speaker cluster is generated by a clustering algorithm (such as DBSCAN), which reflects the typical acoustic characteristics of the cluster. represents the Bhattacharyya distance, which is used to measure the similarity between two probability distributions (here, the voiceprint feature vector and the cluster center); represents the total number of speaker clusters; represents the indicator function; represents the weight coefficient; The PPT semantic vector representing the current chapter is extracted from the chapter nodes in the structured graph;

[0207] Generate final speaker labels through gating mechanism :

[0208]

[0209] Where, Represents a learnable fusion weight, which is used to balance the importance of voiceprint and semantic information. represents the Sigmoid function, represents the trainable weight matrix, Represents a trainable bias term; Indicates the conflict threshold ( =0.7);

[0210] Furthermore, semantic similarity constraints and fusion features are introduced in Dynamic Time Warping (DTW) The temporal features of the speaker are used as constraints to solve the semantic offset problem of pure acoustic alignment, optimize the semantic cost function, and label the speaker. Align with PPT chapter information to generate text sequences with speaker labels ;

[0211] Among them, the text sequence with speaker labels is:

[0212]

[0213] Where, Represents text content; Indicates a timestamp; Speaker labeling that integrates voiceprint and semantics;

[0214] Introducing semantic similarity constraints and fusion features in Dynamic Time Warping (DTW) The temporal characteristics of are used as constraints to improve the DTW cost function. The matching degree between the text content and the PPT chapter topic nodes is integrated into the alignment decision to ensure the time synchronization accuracy between the text paragraph and the PPT chapter. The mathematical expression of the improved DTW cost function is:

[0215]

[0216] Where, The weight representing the alignment of acoustic features; The weight representing the temporal consistency of the fused features; represents the weight of the semantics matching the PPT chapter, and ; represents the cosine similarity function, which measures the consistency between features; Indicates the first Characteristics of each time point; Indicates the first Characteristics of each time point; Indicates the Cross-modal fusion features of audio streams at different time points; Indicates the Cross-modal fusion features of PPT chapters at different time points; Indicates the Acoustic characteristics at each time point; Indicates the Text embedding of each PPT chapter; Indicates the The text semantic role labeling (SRL) results at each time point; Indicates the The topic node embedding of each PPT chapter is generated by a structured graph, including term distribution and hierarchical relationships;

[0217] Specifically, during a conference, when the input audio signal-to-noise ratio (SNR) is detected to be lower than the threshold, the system automatically increases the and The weight of the fusion feature and semantic matching is prioritized to correct the alignment path to ensure the temporal consistency between chapters and speech paragraphs;

[0218] The audio segments and PPT chapters on the forced alignment path have high similarity in the fusion feature space to avoid alignment deviation caused by noise from a single modality (such as pure audio or pure text);

[0219] When the acoustic features of a certain speech are distorted due to environmental noise, but the fusion features After text mode compensation, it can still remain stable, and this constraint can correct the alignment result.

[0220] Dynamic Time Warping (DTW) minimizes Find the minimum mapping path between the audio timeline and the PPT chapter;

[0221] Map audio time segments to PPT chapters based on the DTW path , then the alignment mapping relationship is: , where Indicates the temporal mapping relationship between audio and PPT chapters, generated by the DTW algorithm; Indicates the first time points; Indicates the first chapter in the PPT chapter sequence Chapter index; represents the total length of the alignment path, i.e. the number of point pairs, which is automatically determined by the DTW algorithm;

[0222] For each PPT chapter , aggregate the speaker labels corresponding to all the audio segments mapped by it;

[0223] Using confidence weighting (combined with and ) Determine the final speaker label;

[0224] After alignment, each PPT chapter is associated with a set of audio time periods, and the audio data within the time period is extracted. Label;

[0225] Finally, text with speaker tags is generated;

[0226] The resulting formatted record is as follows:

[0227] [00:12:30]<Speaker A>: Revenue increased *** this quarter, primarily driven by expansion into new markets.

[0228] [00:14:20]<Speaker B>: It is recommended to increase the R&D budget to *** million yuan.

[0229] S3, based on fusion features , text sequences with speaker labels and PPT chapter structure By analyzing the semantic roles and temporal relationships of the text, a heterogeneous graph neural network is used to construct a structured topic evolution map. ;

[0230] The specific steps involved in constructing a structured issue evolution map are:

[0231] The fusion features , text sequences with speaker labels and PPT chapter structure As input; each chapter Contains title and key content text;

[0232] For text sequences with speaker labels Perform multimodal feature extraction to extract semantic, acoustic and temporal features, and fuse semantic role embedding with time interval information to generate a semantic representation vector for the text ;

[0233] The text sequence with speaker labels As input, the BERT model is used to generate a basic text vector, and the semantic role embedding vector is fused to generate a semantic representation vector of the text. :

[0234]

[0235] The time interval is transformed by a linear layer Mapped to time series feature vector :

[0236] ;

[0237] ;

[0238] Where, Indicates the The embedding vector of the semantic role corresponding to the text; A trainable projection matrix representing the semantic role embedding vector into the semantic space; Indicates the Semantic representation vector of a text; Indicates the The content of the text snippet (string); Indicates the Timestamp of the text snippet; Indicates the Article text and The difference between the end time of the text, Indicates the End time of the article; Indicates the End time of the article; Indicates time interval The time series feature vector obtained by linear transformation (Linear); Indicates that the Text snippets The text is passed into the pre-trained language model BERT as input, and the semantic representation of the text is obtained through the model; Indicates the time interval between text segments Perform a linear transformation, that is, convert the original time difference into a corresponding vector representation through a linear layer (such as a fully connected layer);

[0239] The fusion features Through linear projection and semantic representation vector Splicing to generate joint features :

[0240]

[0241] based on Constructing term distribution of topic nodes , used to enhance the modeling of cross-modal semantics (such as PPT keywords and voice content);

[0242] Where, Represents the feature concatenation operation, which represents the semantic vector and fusion features Splicing in feature dimension; represents a trainable projection matrix;

[0243] Furthermore, the nested Dirichlet process is used to model the topic nodes of the conference content, mining the parent-child topic hierarchy in the conference text and generating topic nodes. ;

[0244]

[0245] Where, Indicates the topic nodes; Indicates the total number of topic nodes;

[0246] The Nested Dirichlet Process (NDP) is used to generate the parent-child issue node structure and obtain the issue level set. ;

[0247] Establish topic node-term probability distribution To characterize the semantic features of each topic node:

[0248]

[0249] Where, Indicates the The topic node generates The probability of a term; Indicates that in the corpus, Term under topic node The number of occurrences of Indicates the total number of terms in the vocabulary; Indicates that the parameters of a given topic node Under the condition of The probability distribution of

[0250] In this embodiment, from the text sequence with speaker labels Extract Speakers, forming a speaker node , and record the speaker's corresponding speaking frequency and voiceprint confidence;

[0251] in:

[0252]

[0253] Where, Indicates the total number of speakers; Indicates the Speaker nodes;

[0254] According to the input PPT chapter structure , extract chapter nodes ;

[0255]

[0256] Where, Indicates the total number of PPT chapters; Indicates the PPT chapters;

[0257] The complete node set ;

[0258] Based on the semantic role embedding vector and the topic node embedding vector, a role-topic association scoring model is constructed to quantify the association strength between different semantic roles and specific topics, and dynamically generate contribution relationship edges based on the speaking frequency and topic popularity. ;

[0259] The role-issue association scoring model is as follows:

[0260]

[0261] Combine the frequency of speaking and the popularity of the topic to dynamically adjust the contribution relationship Weight:

[0262]

[0263] Where, Represents the semantic role obtained by cosine similarity calculation and topic nodes 's association score; For the The vector representation of each topic node in the embedding space; Representing semantic roles The corresponding embedding vector; Representing semantic roles Topic Node The final contribution weight of Represents an issue node The popularity index is calculated based on the number of times the topic node is discussed or the cumulative relevance; Representing semantic roles Frequency of speaking;

[0264] According to the above weights Combine all speaker-issue pairs Organized as edge sets:

[0265]

[0266] Where, Indicates that from the speaker node To the topic node ; Represents an element in the speaker node set, Represents a specific speaker node, belonging to the set ; Represents an element in the topic node set, Represents a specific topic node, belonging to the set ;

[0267] and As the weight of each edge, To quantify the speaker On the topic degree of contribution (e.g., frequency of speech, semantic relevance);

[0268] By constructing a temporal adjacency matrix , quantitative issues and topics The transfer relationship between them is combined with the Time Graph Attention Network (TGAT) to dynamically generate evolutionary relationship edges. ; Realize dynamic updating of topic characteristics in the time dimension and modeling of evolution paths;

[0269] Evolution relationship edge The specific steps for building are:

[0270] Constructing a temporal adjacency matrix :

[0271]

[0272] Dynamically update the timing of the node:

[0273]

[0274] Where, Indicates the preset time window; Indicates the topic Total number of transfers; Indicates that in the time window Internal, topic Shift to the topic the number of times; Represents the time step Current topic nodes The eigenvector of Represents the time step Current topic nodes The eigenvector of Represents the time step Current topic nodes The updated feature vector of ;

[0275] In this embodiment, the edge set is defined as:

[0276]

[0277] Where, Indicates the topic and topics There is a temporal evolutionary relationship; Represents the elements of the temporal adjacency matrix; Represents an element in the topic node set;

[0278] For each chapter node With each topic node , calculate chapter nodes and topic nodes The semantic similarity of the relationship is used as the inclusion relationship edge. The weight of the inclusion relationship edge ;

[0279] The complete edge set ;

[0280] Based on heterogeneous graph neural network for complete node set and the complete edge set Perform multi-layer message delivery and node updates, introduce timestamps, and generate a structured topic evolution graph .

[0281] In this embodiment, the structured topic evolution map for:

[0282]

[0283] Where, Represents the set of all nodes in the graph, including topic nodes, speaker nodes, and chapter nodes; Represents the set of all edges in the graph, including contribution relationship edges, evolution relationship edges, and inclusion relationship edges; Represents a timestamp set, used to characterize the timing attributes in the graph;

[0284] Node Collection The following types of nodes are included:

[0285] Topic Node : Indicates the specific discussion content. Each topic is represented by a vector, which is composed of term distribution, heat index and hierarchical coding;

[0286] Speaker Node : Identify the people participating in the discussion. Each node integrates semantic role embedding, voiceprint information, and speaking frequency.

[0287] Chapter Node : Represents chapter information in conference materials or PPTs. Its features include semantic vectors and timestamps for the start and end of the chapters.

[0288] Edge Set The following types of edges are included:

[0289] Contribution relationship edge : It connects the speaker and the topic, reflecting the contribution of a speaker to a specific topic;

[0290] Evolution relationship edge : Used to characterize the temporal evolution and transfer relationship between topics;

[0291] Contains relationship edge : Indicates the affiliation and inclusion relationship between chapters and topics;

[0292] Timestamp Collection Used to divide the meeting timeline into discrete time slices, providing a timing basis for subsequent dynamic modeling;

[0293] Heterogeneous graph neural networks are used for cross-type message passing, and a joint loss function is used to constrain structural and temporal consistency:

[0294] In this embodiment, during the graph construction process, the heterogeneous graph neural network jointly models each node through the transmission and fusion of neighborhood information:

[0295] According to different types of edges, the messages of contribution edges, evolution edges and inclusion edges are calculated separately:

[0296] Contributed side messages (by speakers To the topic ):

[0297]

[0298] Where, Indicates the In the layer, the speaker On the topic Contribution information is used to update the characteristics of the topic node; Multilayer perceptron dedicated to contribution edge; Indicates the Speaker node in the layer The hidden state of Indicates the Topic nodes in the layer The hidden state of Indicates the speaker On the topic Contribution weight;

[0299] Evolution of side messages (by topic To the topic ):

[0300]

[0301] Where, Indicates the Layer, topic Topic The evolution relationship information is used to model the temporal dependencies between topics; represents a multilayer perceptron dedicated to evolving edges; Indicates the Topic nodes in the layer The hidden state of Represents the elements of the temporal adjacency matrix, representing the topic Move to the topic within the time window probability;

[0302] Contains side messages (by section To the topic ):

[0303]

[0304] Among them, the symbol represents vector concatenation, and each edge type corresponds to an independent multi-layer perceptron (MLP) to parameterize the message generation process;

[0305] Where, For the Layer, chapter node To the topic node Message, indicating the chapter On the topic Affiliation information is used to enhance the semantic consistency of topics; represents an edge-specific multilayer perceptron; Indicates the Chapter nodes in layers The hidden state of Indicates chapter On the topic semantic similarity.

[0306] In this embodiment, the message transmission and node update process is specifically as follows:

[0307] According to the edge type (contribution, evolution, inclusion), the corresponding MLP is generated 、 and ;

[0308] Node Update:

[0309] After receiving the messages from neighbors, each node aggregates all its neighbor messages and updates its hidden state based on its own characteristics:

[0310]

[0311] Where, Representation and Node A set of connected neighbors, with different edge types corresponding to different parameter matrices ; The projection matrix representing the node's own features; Indicates edge type Parameter matrix (contribution edge, evolution edge, inclusion edge); Indicates the From neighbor nodes in the layer Pass to node news; Representation node In the The updated hidden state of the layer; Representation node In the The hidden state of the layer;

[0312] At the same time, the node's own characteristics are also added through The transformed information is finally activated by ReLU to form the updated node representation;

[0313] After several layers of message passing, the constructed heterogeneous graph generates the final node embedding. In order to make the node embedding faithfully reflect the information in the original data (such as the term distribution of the topic) and the time evolution characteristics, a joint loss function is constructed. :

[0314] Reconstruction loss , so that the vector of topic nodes restores the original topic distribution:

[0315]

[0316] Where, Topic node, indicating the topic discussed in the meeting; Indicates the The embedding vector of the topic node; Indicates the The actual distribution of topic nodes; Indicates the total number of topic nodes; Represents the topic node index variable;

[0317] Timing consistency loss Ensure that the node feature changes in the graph in adjacent time slices are smooth and consistent with time logic:

[0318]

[0319] Where, Indicates the total number of time slices; Represents the time slice index variable; Indicates time slice The feature matrix of all nodes when ; Indicates time slice The feature matrix of all nodes at the time, including the features of topics, speakers, and chapter nodes; represents the mask matrix, which is used to retain nodes that persist in time; represents the Frobenius norm, which is used to measure the matrix difference; Represents element-wise multiplication;

[0320] Combining the above two losses, the hyperparameters After balancing, define the joint loss function :

[0321]

[0322] Where, represents a hyperparameter used to balance the weight of reconstruction loss and temporal consistency loss;

[0323] By continuously optimizing this joint loss, the model not only obtains an embedding that can accurately reconstruct the attributes of each node, but also ensures good consistency and continuity of the graph in temporal dynamics.

[0324] S4. Extracting structured topic evolution graphs based on reinforcement learning strategies The key decision nodes in the meeting are used to generate the final meeting summary document.

[0325] The specific steps involved in generating the final meeting summary document are:

[0326] After completing the structured issue evolution map After the construction of the graph and multi-layer message passing, each topic node in the graph and its semantic features, temporal features and speaker contributions are used as state input;

[0327] Use deep Q-learning to select the topic node that contributes most to the summary from the graph ;

[0328] State-Action Space:

[0329] State: semantics, timing, and speaker characteristics of the topic node;

[0330] Action: Select the topic node to add to the summary;

[0331] Bonus: Optimize the Q-network based on the quality of the generated summary (ROUGE score);

[0332] For each key decision node, the topic text slot value is determined by the term distribution model, the speaker slot value is determined by the maximum contribution edge weight, the decision content slot value is generated by the SRL model, and the chapter nodes are aligned by cosine similarity to output the time interval slot value and chapter slot value;

[0333] For each selected key decision node , filling the following slots:

[0334] Topic text generation:

[0335] Term distribution method: from the term distribution of the topic Select the top 3 words with the highest probability:

[0336]

[0337] Where, Expressive words Pre-trained embeddings of Represents the topic node The term distribution vector of ; Expressive words Traverse all graph node sets (In this formula, the word Traverse the set of vocabulary nodes in the graph and calculate the relevance of each word to the topic node);

[0338] Use the Seq2Seq model to generate natural language descriptions to ensure consistency between the summary text and the original speech / PPT content:

[0339]

[0340] in, Represents the decoder using the Transformer architecture (the decoding side of the Seq2Seq model), which outputs the natural language sentence used for the final summary; Represents the topic node The term distribution vector of ;

[0341] The spokesperson confirmed:

[0342]

[0343] Where, Representing the spokesperson To the topic node The edge weight is used to quantify the speaker's contribution to the topic (such as the number of speeches and decision-making influence);

[0344] Decision-making content generation:

[0345] Extracting decision actions using the Semantic Role Labeling (SRL) model:

[0346]

[0347] Time intervals are aligned with chapters:

[0348] Time interval: take the timestamp of the chapter node ,in:

[0349]

[0350] Chapter title: corresponding PPT chapter title ; Represents a chapter node The start timestamp of Represents a chapter node End timestamp of

[0351] Concatenate the above slot values ​​in the order of the preset template to form the final meeting summary document:

[0352] Template splicing:

[0353]

[0354] Where, Indicates the Text descriptions corresponding to key topic nodes (generated by term distribution method and Seq2Seq); Indicates the topic node The speaker who made the greatest contribution; Indicates that the SRL model is used to extract the topic node The decision content structure extracted from the associated text; Representation and chapter nodes The corresponding time interval slot, that is, the start and end timestamps of the chapter; Represents a chapter node Corresponding PPT / document chapter title; Indicates the An index of key topic nodes selected by deep Q-learning; Represents a chapter node Semantic vector of Represents the topic node The term distribution vector of ; Represents the semantic role labeling analysis of topic node text; Represents the decision-action triplet output by SRL.

[0355] Example 2:

[0356] This embodiment provides a meeting minutes processing system using AI, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement any one of the above-mentioned meeting minutes processing methods using AI.

[0357] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for processing meeting minutes using AI, characterized in that: The following steps are involved: S1. Use multi-source sensors to collect conference audio streams and associated text data. Perform real-time noise suppression on the collected conference audio streams and associated text data using a noise suppression algorithm. Then use a cross-modal alignment algorithm to perform cross-modal alignment on the noise-reduced data to obtain fusion features. S2. Deploy an end-to-end ASR model with a hybrid CTC / Attention architecture. Insert domain-specific attention heads into the attention layer of the pre-trained Transformer model to build a domain-enhanced speech recognition model for converting audio into text sequences with speaker labels. S3: Based on the fusion features, speaker-labeled text sequences, and PPT chapter structure, and by analyzing the semantic roles and temporal relationships of the text, a heterogeneous graph neural network is used to construct a structured topic evolution graph; S4. Extract key decision nodes from the structured topic evolution graph based on reinforcement learning strategy and generate the final meeting summary document.

2. The method for processing meeting minutes using AI according to claim 1, characterized in that: In said S1, the multi-source sensor includes an audio sensor, a visual sensor and a text input source; The audio sensor captures the speaker's audio directionally through a beamforming algorithm; The text input source is connected to the PPT / PDF document parser in real time to extract text content and page turning event timestamps.

3. The method for processing meeting minutes using AI according to claim 2, characterized in that: In S1, the specific steps involved in performing real-time noise suppression on the collected data using the noise suppression algorithm are: The original audio signal is denoised by spectral subtraction to obtain the denoised amplitude spectrum; The amplitude spectrum and phase spectrum after spectral subtraction are input into the deep neural network for refinement of noise reduction to obtain the denoised audio signal.

4. The method for processing meeting minutes using AI according to claim 3, characterized in that: In S1, the cross-modal alignment algorithm performs cross-modal alignment on the denoised data, and the specific steps involved are: Use SlideTimer algorithm to parse PPT page turning log and extract chapter time set; Detect silent segments using Silero-VAD and segment them into a set of paragraph boundaries; The chapter time set and paragraph boundary set are constructed into a candidate pairing set. The Hungarian algorithm is used to filter the legal timestamp pairing sets in the candidate pairing set to achieve coarse-grained time alignment. The output is coarse-grained aligned chapter-audio paragraph pairs; Train the audio encoder and text encoder through a contrastive learning framework and construct a unified feature space; Based on the coarse-grained aligned chapter-audio paragraph pairs, the similarity between the chapter title and the audio paragraph is calculated; An improved dynamic time warping algorithm is used to align sentence fragments, combined with a semantic cost function to compensate for local timing jitter. Use CRF sequence tagging to identify professional terms, and dynamically update the tag transfer matrix; Dynamically calculate weights based on real-time audio feature vectors and text feature vectors; The audio / text features are unified to 512 dimensions through the audio projection matrix and the text projection matrix to generate fused features.

5. The method for processing meeting minutes using AI according to claim 1, characterized in that: The specific steps involved in building the domain-enhanced speech recognition model are: Deploy a hybrid CTC / Attention end-to-end ASR model; Dynamically adjust the language model weights during the decoding phase through a context-adaptive mechanism; Build a gating network that includes acoustic confidence assessment and receives the posterior probability distribution output by the acoustic model in real time; Analyze the vocabulary distribution of the current speech segment through the pre-trained domain term detector and generate a domain relevance score; The fused features are used as the input of the gating network, and the acoustic confidence features and domain relevance scores are introduced to calculate the weight coefficients of the dynamic language model; Perform context-aware model fusion to generate final decoding probabilities; At the same time, K domain-specific attention heads are inserted into the attention layer of the pre-trained Transformer model to form a hybrid attention structure, construct a learnable domain mask matrix, and adopt a multi-task transfer learning strategy to optimize the model parameters and transcribe audio features into text sequences.

6. The method for processing meeting minutes using AI according to claim 5, characterized in that: The specific steps involved in converting the audio into the initial text with speaker tags are: Extract the MFCC features and FBANK features of the denoised audio signal in parallel, and concatenate the dual modal features into ; Will As the input of the GMM-UBM model, the speaker embedding vector is obtained through maximum a posteriori probability estimation; Based on t-SNE dimensionality reduction and DBSCAN clustering algorithm, the voiceprint feature vector set is divided into Speaker clusters, output voiceprint cluster labels; Based on the text and fusion features of each voiceprint clustering paragraph, the Bi-LSTM+CRF model is used to calculate the probability distribution of semantic roles. The semantic roles are then cross-validated with the voiceprint clustering results. Finally, the corresponding speaker identity is annotated for each paragraph of text to generate the final speaker label. In dynamic time warping, semantic similarity constraints and fused temporal features are introduced as constraints, the semantic cost function is optimized, speaker labels are aligned with PPT chapter information, and text sequences with speaker labels are generated. Among them, the text sequence with speaker labels includes text content, timestamp and speaker labels that integrate voiceprint and semantics.

7. The method for processing meeting minutes using AI according to claim 1, characterized in that: In S3, the specific steps involved in constructing a structured topic evolution graph are: Take the fusion features, text sequence with speaker labels and PPT chapter structure as input; Perform multimodal feature extraction on speaker-labeled text sequences, extracting semantic, acoustic, and temporal features. Then, the semantic role embedding is integrated with the time interval information to generate a semantic representation vector for the text. The fusion feature is concatenated with the semantic representation vector through linear projection to generate a joint feature; The nested Dirichlet process is used to model the topic nodes of the meeting content, mining the parent-child topic hierarchy in the meeting text to generate topic nodes; Extract from text sequences with speaker labels Speakers are formed into speaker nodes, and the corresponding speaking frequency and voiceprint confidence of the speaker are recorded; Extract chapter nodes based on the input PPT chapter structure; Combine the topic nodes, speaker nodes and chapter nodes into a complete node set; Based on the semantic role embedding vector and the topic node embedding vector, a role-topic association scoring model is constructed to quantify the association strength between different semantic roles and specific topics. The contribution relationship edges are dynamically generated by combining the speaking frequency and topic popularity. Quantify the issues by constructing a time-series adjacency matrix and topics The transfer relationship between them is combined with the time graph attention network to dynamically generate evolutionary relationship edges; For each chapter node and each topic node, the semantic similarity between the chapter node and the topic node is calculated, and the semantic similarity value is used as the weight of the inclusion relationship edge to obtain the inclusion relationship edge; The contribution relationship edge, evolution relationship edge and inclusion relationship edge are combined into a complete edge set; Based on heterogeneous graph neural networks, multi-layer message passing and node updates are performed on the complete node set and the complete edge set, and timestamps are introduced to generate a structured issue evolution graph.

8. The method for processing meeting minutes using AI according to claim 1, characterized in that: In S4, the specific steps involved in generating the final meeting summary document are: Each topic node in the graph and its semantic features, temporal features and speaker contributions are used as state input; Use deep Q-learning to select the topic nodes that contribute most to the summary from the graph; For each key decision node, the topic text slot value is determined by the term distribution model, the speaker slot value is determined by the maximum contribution edge weight, the decision content slot value is generated by the SRL model, and the chapter nodes are aligned by cosine similarity to output the time interval slot value and chapter slot value; The above slot values ​​are spliced ​​in the order of the preset template to form the final meeting summary document.

9. A meeting minutes processing system using AI, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes the computer program to implement the method for processing meeting minutes using AI as described in any one of claims 1 to 8.

Images