Conference summary processing method and system using AI

Abstract

The invention relates to the technical field of intelligent conference processing, and relates to a conference summary processing method and system using AI, and the method comprises the steps: carrying out the real-time noise suppression of a collected conference audio stream and associated text data through a noise suppression algorithm, and carrying out the cross-modal alignment of the denoised data through a cross-modal alignment algorithm; a domain-specific attention head is inserted into an attention layer of the pre-trained Transform model, a domain-enhanced speech recognition model is constructed, and audio is converted into a text sequence with a speaker tag; adopting a heterogeneous graph neural network to construct a structured topic evolution graph; key decision nodes in the structured topic evolution graph are extracted based on a reinforcement learning strategy, and a final conference summary document is generated. In the decoding stage, the fusion proportion of the acoustic model and the language model is dynamically adjusted based on the real-time acoustic confidence coefficient, the recognition rate of the vocabularies in the professional field is increased, and the problems of frequent term transcription errors and poor semantic coherence in the professional conference are effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent conference processing, and specifically, to a method and system for processing meeting minutes using AI. Background Art

[0002] Meetings are often held within an enterprise. After a meeting is held, meeting minutes information is recorded so that those who did not attend the meeting can read the meeting minutes to understand the meeting content, and those who attended the meeting can read it again to review the meeting content. Meeting minutes are used for the structured recording of key information such as the content of the meeting discussion, decision-making matters, and task assignments. However, the content of the meeting minutes recorded in text is relatively large. With the development of AI intelligence, AI can achieve real-time transcription and automatically generate complete meeting minutes, realizing the real-time recording and summary of the meeting content.

[0003] However, due to the lack of targeted optimization for industry terms, the misrecognition rate of domain vocabulary is high, term transcription errors frequently occur in professional meetings, and the language model weights cannot be dynamically adjusted according to the context, resulting in poor semantic coherence. Therefore, a method and system for processing meeting minutes using AI are provided. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and system for processing meeting minutes using AI to solve the problems of high misrecognition rate of domain vocabulary due to the lack of targeted optimization for industry terms, frequent term transcription errors in professional meetings, and poor semantic coherence caused by the inability to dynamically adjust the language model weights according to the context.

[0005] To achieve the above purpose, the present invention aims to provide a method for processing meeting minutes using AI, including the following steps: S1. Collect meeting audio streams and associated text data using multi-source sensors, perform real-time noise suppression on the collected meeting audio streams and associated text data through a noise suppression algorithm, and perform cross-modal alignment on the denoised data using a cross-modal alignment algorithm to obtain fusion features ; S2. Deploy an end-to-end ASR model with a hybrid CTC / Attention, and insert domain-specific attention heads into the attention layer of the pre-trained Transformer model to construct a domain-enhanced speech recognition model for converting audio into a text sequence with speaker labels ; S3. Based on the fusion features , the text sequence with speaker labels and the PPT chapter structure and by parsing the semantic roles and temporal relationships of the text, construct a structured topic evolution graph using a heterogeneous graph neural network ; S4. Extract the structured topic evolution graph based on the reinforcement learning strategy, and generate the final meeting summary document by extracting the key decision nodes in it. In the middle, generate the final meeting summary document.

[0006] As a further improvement of this technical solution, in S1, the multi-source sensors include audio sensors, visual sensors, and text input sources; The audio sensor directionally captures the speaker's audio through a beamforming algorithm; The text input source is connected to the PPT / PDF document parser in real time to extract text content and the timestamp of page turning events.

[0007] As a further improvement of this technical solution, in S1, the specific steps involved in real-time noise suppression of the collected data through a noise suppression algorithm are as follows: The original audio signal is denoised by spectral subtraction to obtain the denoised amplitude spectrum; The amplitude spectrum and phase spectrum after spectral subtraction denoising are input into a deep neural network for refined denoising to obtain the denoised audio signal .

[0008] As a further improvement of this technical solution, in S1, the specific steps involved in cross-modal alignment of the denoised data by a cross-modal alignment algorithm are as follows: Use the SlideTimer algorithm to parse the PPT page turning log and extract the chapter time set ; Detect the silent segments through Silero-VAD and segment them into a set of paragraph boundaries ; Construct the chapter time set and the paragraph boundary set into a candidate pairing set, and screen the legal timestamp pairing set in the candidate pairing set through the Hungarian algorithm to achieve coarse-grained time alignment; Output the coarsely aligned chapter-audio paragraph pairs; Train the audio encoder and the text encoder through a contrastive learning framework, and construct a unified feature space: ; In the formula, The loss value used to measure the cross-modal alignment of audio and text; Represents the audio encoder; Represents the text encoder; Represents the audio segment The feature vector extracted by the audio encoder; Represents the text segment The feature vector extracted by the text encoder; Denotes negative sample text, referring to the text segment that does not match the audio segment ; Denotes a preset threshold; Among them, the associated audio segment and the text segment satisfy in the embedding space: , and ; In the formula, Denotes the encoding result of the audio encoder for the audio segment ; Denotes the encoding result of the text encoder for the text segment ; Denotes and Cosine similarity; Denotes a preset boundary margin, used to enhance the distinguishability between positive and negative samples; Denotes that for all text segments that do not match all satisfy ; Based on the chapter-audio paragraph pairs of coarse-grained alignment, calculate the similarity between the chapter title and the audio paragraph; Adopt an improved dynamic time warping algorithm to align the sentence segments, and combine a semantic cost function to compensate for local temporal jitter; Use CRF sequence labeling to identify professional terms, and the labeling transition matrix is dynamically updated; According to the real-time audio feature vector and text feature vector, dynamically calculate the weight ; Through the audio projection matrix and the text projection matrix Unify the audio / text features to 512 dimensions to generate a fused feature .

[0009] As a further improvement of this technical solution, the specific steps involved in constructing the domain-enhanced speech recognition model are: Deploy an end-to-end ASR model with hybrid CTC / Attention; Through a context adaptation mechanism, dynamically adjust the language model weight during the decoding stage; Construct a gated network containing acoustic confidence evaluation, and receive the posterior probability distribution output by the acoustic model in real time ; Analyze the vocabulary distribution of the current speech segment through a pre-trained domain term detector to generate a domain relevance score ; Take the fused feature as the input of the gating network, and simultaneously introduce the acoustic confidence feature and the domain relevance score to calculate the dynamic language model weight coefficient ; Perform context-aware model fusion to generate the final decoding probability ; Meanwhile, insert K domain-specific attention heads into the attention layer of the pre-trained Transformer model to form a hybrid attention structure, construct a learnable domain mask matrix , and adopt a multi-task transfer learning strategy to optimize the model parameters and transcribe the audio features into a text sequence.

[0010] As a further improvement of this technical solution, the specific steps involved in converting the audio into the initial text with speaker tags are as follows: Parallelly extract the MFCC features of the denoised audio signal and the FBANK features , and concatenate the bimodal features into ; Take as the input of the GMM-UBM model, and obtain the speaker embedding vector through maximum a posteriori probability estimation; Based on the t-SNE dimensionality reduction and DBSCAN clustering algorithm, divide the set of voiceprint feature vectors into speaker clusters and output the voiceprint clustering labels ; Based on the text and the fused feature of each voiceprint clustering paragraph, use the Bi-LSTM+CRF model to calculate the semantic role probability distribution , and cross-validate the semantic role with the voiceprint clustering result. Finally, label the corresponding speaker identity for each paragraph of text to generate the final speaker label ; Introduce the semantic similarity constraint and the temporal sequence feature of the fused feature as a constraint term in dynamic time warping, optimize the semantic cost function, and align the speaker label with the PPT chapter information to generate a text sequence with speaker tags ; Among them, the text sequence with speaker tags is: represents the text content; Indicates the timestamp; Is the speaker label that fuses voiceprint and semantics.

[0011] As a further improvement of this technical solution, in step S3, the specific steps involved in constructing the structured topic evolution graph are as follows: Take the fused features , the text sequence with speaker labels and the PPT chapter structure as inputs, where the PPT chapter structure is extracted by a PPT / PDF document parser; Perform multi-modal feature extraction on the text sequence with speaker labels , extract semantic, acoustic, and temporal features, and fuse semantic role embedding and time interval information to generate the semantic representation vector of the text ; Concatenate the fused features with the semantic representation vector through linear projection to generate the joint feature ; Use the nested Dirichlet process to model the topic nodes of the meeting content, mine the parent-child topic hierarchy in the meeting text, and generate topic nodes ; Extract speakers from the text sequence with speaker labels to form speaker nodes , and record the speech frequency and voiceprint confidence corresponding to the speakers; Extract chapter nodes according to the input PPT chapter structure ; Then the complete node set ; Based on the semantic role embedding vector and the topic node embedding vector, construct a role-topic association scoring model to quantify the association strength between different semantic roles and specific topics, and dynamically generate contribution relationship edges in combination with the speech frequency and topic popularity ; By constructing a temporal adjacency matrix , quantify the transition relationship between topic and topic , and dynamically generate evolution relationship edges in combination with the temporal graph attention network ; For each chapter node and each topic node , calculate the chapter node and the topic node of semantic similarity, and use the value of semantic similarity as the weight of the inclusion relationship edge to obtain the inclusion relationship edge ; Then the complete edge set ; Based on the heterogeneous graph neural network, perform multi-layer message passing and node update on the complete node set and the complete edge set , and introduce timestamps to generate a structured topic evolution graph .

[0012] As a further improvement of this technical solution, in S4, the specific steps involved in generating the final meeting summary document are as follows: Take each topic node in the graph and its semantic features, temporal features, and speaker contributions as the state input; Use deep Q-learning to select the topic node with the greatest contribution to the summary from the graph ; For each key decision node, determine the topic text slot value through the term distribution model, determine the speaker slot value through the maximum contribution edge weight, generate the decision content slot value through the SRL model, and align the chapter node output time interval slot value and chapter slot value through cosine similarity; Concatenate the above slot values in the order of the preset template to form the final meeting summary document.

[0013] On the other hand, the present invention provides a meeting minutes processing system using AI, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the method for processing meeting minutes using AI according to any one of the above.

[0014] Compared with the prior art, the beneficial effects of the present invention are: 1. In the method and system for processing meeting minutes using AI, a hybrid CTC / Attention end-to-end ASR model is adopted, combined with a domain-specific attention head and a dynamic language model weight adjustment mechanism, to optimize industry terms specifically. And in the decoding stage, based on the real-time acoustic confidence, dynamically adjust the fusion ratio of the acoustic model and the language model to improve the recognition rate of professional field vocabulary, effectively reducing the problems of frequent term transcription errors and poor semantic coherence in professional meetings.

[0015] 2. In the method and system for processing meeting minutes using AI, a structured topic evolution graph is constructed through a heterogeneous graph neural network, integrating topic nodes, speaker contributions, and PPT chapter information, dynamically quantifying the association strength between semantic roles and topics, realizing multi-dimensional structured representation of meeting content, and facilitating the extraction and traceability of key information; Meanwhile, based on the reinforcement learning strategy, key decision nodes are extracted from the graph, and combined with the term distribution model, semantic role annotation, and temporal alignment, a meeting summary containing key slots such as topics, speakers, decision content, time intervals, etc. is automatically generated to ensure accurate content and compliance with the actual discussion logic. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the overall method flowchart of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0018] Embodiment 1: Please refer to Figure 1 as shown. This embodiment provides a meeting minutes processing method using AI, including the following steps: S1. Use multi-source sensors to collect meeting audio streams and associated text data, perform real-time noise suppression on the collected meeting audio streams and associated text data through a noise suppression algorithm, and perform cross-modal alignment on the denoised data through a cross-modal alignment algorithm to obtain fusion features ; In this embodiment, the multi-source sensors include audio sensors, visual sensors, and text input sources; The audio sensor captures the speaker's audio directionally through a beamforming algorithm; The text input source is connected to a PPT / PDF document parser in real time for extracting text content and page-turning event timestamps.

[0019] Specifically, the specific steps involved in performing real-time noise suppression on the collected data through a noise suppression algorithm are: The original audio signal is denoised by spectral subtraction to obtain the denoised amplitude spectrum; The amplitude spectrum and phase spectrum after spectral subtraction denoising are input into a deep neural network for refined denoising to obtain the denoised audio signal ; The noise suppression algorithm combining spectral subtraction and a deep neural network realizes real-time denoising, and its noise suppression coefficient α(t) is calculated through a dynamic prediction model composed of a 5-layer convolutional layer and a GRU layer.

[0020] In this embodiment, the specific steps involved in the cross-modal alignment algorithm performing cross-modal alignment on the denoised data are: Parse the PPT page-turning log using the SlideTimer algorithm to extract the chapter time set ; used to obtain the structured time markers of the text modality for subsequent global alignment; Detect silent segments through Silero-VAD and segment them into a set of paragraph boundaries ; used to divide the continuous audio stream into semantically complete paragraphs for easy alignment with text chapters; Construct the chapter time set and the paragraph boundary set into a candidate pairing set, and filter the legal timestamp pairing set in the candidate pairing set through the Hungarian algorithm to achieve coarse-grained time alignment: Execute the Hungarian algorithm for matching Yes, and satisfy ; ( = 5 seconds), to achieve coarse-grained time alignment, which is used to eliminate the global timing misalignment between the PPT explanation and the speech; Output the coarsely aligned chapter-audio paragraph pairs to provide spatio-temporal associated positive samples for encoder training; Furthermore, train the audio encoder through the contrastive learning framework and the text encoder and construct a unified feature space to generate an encoder that can measure cross-modal similarity, providing a similarity calculation basis for hierarchical alignment: ; In the formula, is the loss value used to measure the cross-modal alignment between audio and text; represents the audio encoder; represents the text encoder; represents the audio segment The 256-dimensional feature vector extracted by the audio encoder; represents the text segment The 256-dimensional feature vector extracted by the text encoder (such as BERT-Base); represents the negative sample text, referring to the text segment that does not match the audio segment ; represents a preset threshold (usually 0.2 - 0.3), requiring that the similarity of the positive sample pair is at least higher than that of the negative sample pair , otherwise the loss is triggered; Among them, the associated audio segment and the text segment satisfy in the embedding space: , and ; In the formula, represents the audio encoder for the audio segment The encoding result; Indicates the encoding result of the text encoder for the text segment ; Indicates the cosine similarity between and (the value range is ); Indicates the preset boundary margin, which is used to enhance the distinguishability between positive and negative samples, and the value is 0.2 - 0.3; Indicates that all text segments that do not match all satisfy ; Positive samples , from the same chapter - paragraph pair; negative samples , randomly sampled or hard negative mining in the same Batch; Map audio features (such as MFCC, FBANK) and visual features (such as face key points, optical flow) to the same feature space through a unified encoder; The audio - text pairs associated by the loss function are close in the embedding space, and the unrelated pairs are far away, forming a cross - modal semantic bridge; Among them, the audio encoder is based on 1D - ResNet, takes the denoised audio spectrogram as input, and outputs a 256 - dimensional vector ; The text encoder is based on the BERT - Base model, takes the text segment as input, and outputs a 256 - dimensional vector ; In this embodiment, based on the chapter - audio paragraph pairs with coarse - grained alignment, calculate the similarity between the chapter title and the audio paragraph; Adopt the improved dynamic time warping algorithm to align the sentence segments, and combine the semantic cost function to compensate for local temporal jitter; The semantic cost function is: In the formula, represents the semantic cost function, which is used to measure the semantic distance between the chapter title and the audio paragraph . The smaller the value, the more similar the semantics of the two; represents the th chapter title; represents the th audio paragraph; represents the feature vector obtained by mapping the chapter title to the high - dimensional semantic space through the text encoder (such as BERT); Denote the feature vector obtained by mapping an audio passage through an audio encoder (such as 1D-ResNet) into a high-dimensional semantic space; to the high-dimensional semantic space; Denote the transpose operation; Use CRF sequence labeling to identify domain-specific terms, with the labeling transition matrix dynamically updated to generate a three-level alignment result, providing a structured multi-granularity alignment guidance for feature fusion, where denotes the total number of label categories in the CRF (Conditional Random Field) sequence labeling model; Dynamically calculate the weights based on real-time audio feature vectors and text feature vectors ; Dynamic weight calculation: ; where is the audio feature vector, the original feature extracted from the audio signal; is the text feature vector, the original feature extracted from the text data; is the training parameter matrix; Denote the Sigmoid function; Denote the weight matrix; Unify the audio / text features to 512 dimensions through the audio projection matrix and the text projection matrix to eliminate the dimensionality difference and achieve additive fusion, generating the fused feature : In the formula, denotes the audio projection matrix; denotes the text projection matrix; denotes the dynamic weight, and the audio projection matrix and the text projection matrix are respectively learned during the training process through backpropagation and optimization algorithms (such as gradient descent), and are dynamically adjusted according to the input data and task objectives.

[0021] Generate a robust fused feature to provide a noise-immune cross-modal representation for downstream summary generation; In the case of input stream delay and burst interference, etc., when the input stream delay > 200ms, activate the lightweight optimizer to update the parameters to prevent cumulative errors from causing system drift: Learning rate ; In the formula, denotes the model parameter, the neural network weight at time step ; Represents the updated model parameters; Represents the dynamic learning rate, the parameter value adjusted by gradient descent; Represents the alignment loss function For the parameter Gradient; Represents the initial learning rate; Represents the learning rate adjustment function; Represents the current batch of data, the data batch input at time step The input data batch is used to calculate the loss and gradient; SNR represents the signal-to-noise ratio. The lower the SNR, the smaller the learning rate to prevent unstable updates under noise interference; Predict the feature change through the state equation When the residual exceeds the limit (when the residual ), start parameter fine-tuning to solve the feature distortion caused by sudden interference; In the formula, Represents at time step The predicted feature vector, that is, the current state predicted by the model; Represents the state transition matrix; Represents at time step The actual feature vector; Represents the system noise term; Represents the preset residual threshold; Represents time step The actual observed feature vector; Ensure the continuous and stable operation of the system in complex conference scenarios (such as network fluctuations, sudden noise).

[0022] S2. Deploy an end-to-end ASR model with hybrid CTC / Attention, insert domain-specific attention heads in the attention layer of the pre-trained Transformer model, and insert K "dialect adaptation attention heads" in parallel to construct a domain-enhanced speech recognition model for converting audio into a text sequence with speaker labels ; The specific steps involved in constructing a domain-enhanced speech recognition model are as follows: Deploy an end-to-end ASR model with hybrid CTC / Attention, use the CTC loss to align the frame-level timing, and use Attention to capture long-range dependencies; Furthermore, insert a dialect classification sub-network at the front end of the encoder of the ASR model to perform real-time dialect type discrimination on the pronunciation style in the input audio stream. The dialect classification sub-network includes a multi-layer causal convolutional network and a lightweight self-attention layer, and outputs the dialect embedding vector corresponding to the audio segment , which is used to solve the acoustic model mismatch caused by regional dialects or differences in Mandarin pronunciation, thereby reducing the recognition error rate of terms and common words. In each attention layer of the pre-trained Transformer model, in addition to inserting domain-specific attention heads, K "dialect-adaptive attention heads" are inserted in parallel. The query / key / value matrices of these heads are linearly mapped from dialect embedding vectors . During the multi-head attention calculation, the outputs of the dialect-adaptive heads and the general attention heads are weighted and merged through gated fusion to achieve an adaptive bias for pronunciations in different regions. Further, a dialect correlation score is introduced in the decoding stage, enabling the decoding process to flexibly enhance or suppress the prediction confidence of the language model for specific phonemes or words according to the dialect features of the current speaker, thereby further reducing the recognition error in dialect scenarios; The encoder adopts a Conformer architecture, which includes a convolutional gating module (capturing local acoustic features), a multi-head self-attention module (modeling global temporal dependencies), and a feed-forward network (non-linear feature transformation); The decoder executes synchronously: CTC path: Directly based on the encoder output Calculate the frame-level posterior distribution ; Attention path: Dynamically aggregate context through attention weights and calculate , where represents based on the encoder output and the previously generated labels . The sequence-level posterior probability distribution output by the Attention path represents the probability of generating the th label ; Fuse the temporal alignment ability of CTC and the long-distance dependence capture of Attention to improve the robustness of the ASR model; Construct a neural network with an encoder-decoder structure, where the encoder maps the acoustic feature sequence to a high-level representation, and the decoder executes synchronously: Based on the frame-level alignment path of connectionist temporal classification (CTC), calculate the first loss term : where represents the target label of the th frame, represents the hidden state of the encoder at the th moment; Based on the sequence dependence modeling path of the attention mechanism (Attention), calculate the second loss term : Among them, represents the length of the target sequence; represents the global context representation output by the encoder; represents the th label; Dynamically fuse the first loss term and the second loss term through a learnable dynamic weight coefficient to form a joint optimization objective : Among them, The value of is dynamically adjusted according to the signal-to-noise ratio of the current input speech: when the signal-to-noise ratio is lower than the preset threshold, increase to strengthen the temporal constraint of the CTC path; when long-distance semantic dependencies are detected, decrease to enhance the context modeling of the Attention path; represents the CTC loss; The dynamic weight coefficient is calculated through a gating network: Among them, represents the sigmoid function; represents the hidden state of the encoder at the th moment; represents the Attention context vector; represents a learnable weight matrix; represents a learnable bias term; In this embodiment, through a context adaptation mechanism, the language model weight is dynamically adjusted during the decoding stage; the contribution weights of the acoustic model and the language model are dynamically adjusted according to the real-time speech content. When the acoustic feature confidence is high (such as regular vocabulary with clear pronunciation), the acoustic model is trusted preferentially; when domain terms appear, the language model weight is increased to use the pre-trained knowledge to complement the possible pronunciation ambiguity part; Construct a gating network including an acoustic confidence evaluation module to receive the posterior probability distribution output by the acoustic model in real time ; Analyze the vocabulary distribution of the current speech segment through a pre-trained domain term detector to generate a domain relevance score , reducing the interference of general vocabulary on professional scenarios; Take the fused feature as the input of the gating network, and at the same time introduce the acoustic confidence feature and the domain relevance score to calculate the dynamic language model weight coefficient : In the formula, represents the Sigmoid activation function; represents the bias term; Specifically, the acoustic confidence feature is calculated as follows: Extract the entropy value of the output probability distribution of the ASR model; Calculate the KL divergence between the current frame and its adjacent 5 frames; Concatenate the above features into ; When high domain relevance is detected (such as the appearance of technical terms), increase the weight of the language model and use pre-trained knowledge to complete the blurred pronunciation part; In a regular conversation scenario, give priority to trusting the acoustic model to avoid semantic deviation caused by over-relying on the language model; Perform context-aware model fusion to generate the final decoding probability : Among them, , take as the auxiliary input, dynamically adjust the fusion weight of the acoustic model and the language model, represents the mapping function implemented by the gating network, and the input is the cross-modal fusion feature ; In the formula, represents the input speech signal, such as audio waveform, spectrogram or multi-modal fusion feature; represents the target output sequence, that is, the text label to be generated (such as characters, words or phonemes); represents the probability of generating a text sequence under general language knowledge ; represents the probability of generating a text sequence under the input speech signal ; represents the dynamic language model weight coefficient, with a value range of [0,1], used to balance the contributions of the language model and the acoustic model; In a regular vocabulary scenario with clear pronunciation (high acoustic confidence), give priority to using the output of the acoustic model to ensure the transcription efficiency; In a technical term or noise interference scenario (low acoustic confidence), rely on the language model to complete the semantics to ensure the accuracy of the terms and the coherence of the context.

[0023] In this embodiment, the implementation method of the domain term detector is: Construct a domain keyword library and generate the corresponding word embedding matrix ; Calculate the semantic similarity between the current speech segment and the domain keyword library through a sliding window, that is, the domain relevance score ; Among them, the domain relevance score Specifically: In the formula, represents a set containing professional terms or keywords related to a specific domain. For example, in the medical field, contains words such as "diagnosis", "antibiotic", "CT scan", etc.; represents the word embedding matrix; represents the speech segment within the sliding window, which means taking the time point as the center and expanding 5 time units forward and backward for each speech segment (the total window length is 11); represents a bidirectional long short-term memory network, which is used to process time series data (such as speech segments), and extracts the semantic features of the current window through bidirectional context; represents the cosine similarity function; represents the domain keyword, traversing each in the keyword set , and calculating its semantic similarity with the current speech segment; In this embodiment, the weights of the language model are dynamically adjusted : In the formula, represents the Sigmoid function; represents the weight matrix; represents the acoustic feature vector; represents the bias term; The weight coefficient of the language model , which is calculated in real time through a gating network, and the gating function adopts the sigmoid linear unit (SiLU); Furthermore, insert K domain-specific attention heads into the attention layer of the pre-trained Transformer model (the Transformer model is the acoustic encoding core of the ASR system. Through the co-design of the domain-specific attention mechanism and the mask matrix, while maintaining the basic speech recognition ability, it significantly improves the modeling accuracy of professional terms), forming a hybrid attention structure. In the meeting scenario, inject attention heads related to professional terms (insert 3 domain-specific heads into the standard 12-head Transformer), enhance the sensitivity to professional vocabulary, retain the modeling ability of the basic model for general speech features (such as intonation, stress), and only focus on optimizing the newly added domain-related parameter layer (such as the Query matrix of the topic node attention head). The standard attention heads maintain their original parameters unchanged and continue to capture general speech features. The domain-specific attention heads adopt a restricted parameter update strategy and only fine-tune their query matrix and key matrix, and construct a learnable domain mask matrix , and adopt a multi-task transfer learning strategy to optimize the model parameters, and transcribe the audio features into text sequences; for the preliminary transcribing of audio to high-quality text; Through the learnable domain mask matrix Strengthen the recognition of professional terms, enhance the sensitivity of the model to domain keywords, and reduce the interference of general vocabulary on the professional scenario; Specifically, the calculation process of the domain-specific attention head is defined as: In the formula, is the query projection matrix; is the key projection matrix; represents the query matrix; represents the value projection matrix; represents the key matrix; represents the value matrix; represents the domain mask matrix, an attention bias matrix generated by the professional term embedding, used to enhance the attention to the term-related positions; represents the dimension of the attention head; represents the output of the nth domain-specific attention head, used to enhance the modeling of professional terms; In the formula, represents the word embedding matrix of the professional term dictionary; is the trainable projection weight; represents the rectified linear unit activation function; Adopt a multi-task transfer learning strategy to optimize the model parameters: In the formula, Denotes the domain classification loss based on attention weights; Denotes the weight coefficient of the domain classification loss, initially set to 0.5, and adjusted according to the validation set effect (if the domain classification task interferes with the main task, then reduce the value; Denotes the domain classification loss, an auxiliary task loss function, which classifies the domain of the input data (such as medical, financial, technical conferences) based on attention weights to enhance the model's adaptability to professional scenarios; Denotes the main task loss function, a joint loss that combines CTC and attention mechanisms, used to optimize the speech recognition task (audio-to-text transcription); Denotes the gradient operator; Denotes the learning rate; Denotes the basic model parameters; Denotes the updated model parameters.

[0024] In this embodiment, the specific steps involved in converting the audio into the initial text with speaker tags are as follows: Parallelly extract the MFCC features of the denoised audio signal and the FBANK features and splice the bimodal features into ; where , , ; Take as the input of the GMM-UBM model (Gaussian mixture model), and obtain the speaker embedding vector through maximum a posteriori probability estimation; use the Gaussian mixture model (GMM-UBM) to distinguish the fundamental frequency and formant features of different speakers; Speaker embedding vector In the formula, Denotes the universal background model parameters; Denotes the adaptive GMM parameters; Denotes taking the expectation over all time frames; Denotes the th frame of the joint feature vector; Denotes the logarithm probability of the feature under the model ; The training steps of the GMM-UBM model are as follows: Construct a UBM with 512 Gaussian components and initialize the diagonal covariance matrix; Iteratively optimize through the expectation maximization (EM) algorithm, and only update the mean supervector in the adaptive stage; Speaker clustering is performed based on the Bhattacharyya distance, and the merging threshold is set to ; Based on t-SNE dimensionality reduction and the DBSCAN clustering algorithm, the set of voiceprint feature vectors is divided into speaker clusters, and the voiceprint clustering labels are output to cluster the voiceprints and separate the speakers. In the formula, represents a set composed of voiceprint feature vectors, where is the feature vector of the and both represent index variables; In this embodiment, based on the text and the fusion features of each voiceprint clustering paragraph, is the sequence length; the Bi-LSTM+CRF model is used to calculate the semantic role probability distribution for semantic role annotation (such as "questioner", "decision maker"), and the semantic roles are cross-validated with the voiceprint clustering results. Finally, the corresponding speaker identity is annotated for each paragraph of text to generate the final speaker label ; responsible for accurately labeling the text with the speaker label; Semantic role annotation: The text is segmented into word / character sequences and encoded as vectors; Bi-LSTM generates hidden states that fuse bidirectional contexts for each word ; The is concatenated with the hidden state as the input feature of the CRF layer to generate enhanced comprehensive features as the CRF input to enhance the accuracy of semantic role classification; CRF calculates the global optimal solution of the label sequence based on the comprehensive features to calculate the semantic role label distribution: In the formula, represents the CRF state transition matrix; represents the comprehensive feature vector; represents the feature of the represents the emission matrix; represents any possible label sequence; Assign semantic role tags to each word (such as "B-Questioner", "I-Decision Maker"); Calculate the voiceprint confidence score : Evaluate the semantic consistency score : In the formula, represents the target voiceprint feature vector, the feature vector of the th voiceprint sample extracted by GMM-UBM or deep neural network, characterizing the acoustic characteristics of the speaker (such as fundamental frequency, formant); represents the center of the th speaker cluster, the centroid of the speaker cluster generated by the clustering algorithm (such as DBSCAN), reflecting the typical acoustic characteristics of the cluster; represents the Bhattacharyya distance, which is used to measure the similarity between two probability distributions (here the voiceprint feature vector and the cluster center); represents the total number of speaker clusters; represents the indicator function; represents the weight coefficient; represents the PPT semantic vector of the current chapter, extracted from the chapter nodes in the structured graph; Generate the final speaker label through the gating mechanism : In the formula, represents the learnable fusion weight, which is used to balance the importance of voiceprint and semantic information, represents the Sigmoid function, represents the trainable weight matrix, represents the trainable bias term; represents the conflict threshold ( =0.7); Furthermore, introduce semantic similarity constraints and the temporal characteristics of the fusion features in dynamic time warping (DTW) as constraint terms to solve the semantic shift problem of pure acoustic alignment, optimize the semantic cost function, and align the speaker label with the PPT chapter information to generate a text sequence with speaker labels ; Among them, the text sequence with speaker labels is: In the formula, represents the text content; represents the timestamp; Speaker label that fuses voiceprint and semantics; Introduce semantic similarity constraints and fusion features in Dynamic Time Warping (DTW) The temporal features are used as constraint terms to improve the DTW cost function, incorporate the matching degree between the text content and the PPT chapter topic nodes into the alignment decision, and ensure the time synchronization accuracy between the text paragraphs and the PPT chapters. Then the mathematical expression for improving the DTW cost function is: In the formula, represents the weight of acoustic feature alignment; represents the weight of the temporal consistency of the fusion features; represents the weight of the semantic matching with the PPT chapter, and ; represents the cosine similarity function, which measures the consistency between features; represents the feature at the th time point in the audio stream; represents the feature at the th time point in the PPT chapter; represents the cross-modal fusion feature of the audio stream at the th time point; represents the cross-modal fusion feature of the PPT chapter at the th time point; represents the acoustic feature at the th time point; represents the text embedding of the th PPT chapter; represents the result of semantic role labeling (SRL) at the th time point; represents the topic node embedding of the th PPT chapter, generated from the structured graph, including term distribution and hierarchical relationship; Specifically, during the meeting, when it is detected that the signal-to-noise ratio (SNR) of the input audio is lower than the threshold, the system automatically increases the weights of and , and preferentially relies on the fusion features and semantic matching to correct the alignment path to ensure the temporal consistency between the chapter and the speech paragraph; Force the audio segments on the alignment path to have high similarity in the fusion feature space with the PPT chapters, and avoid alignment deviations caused by noise in a single modality (such as pure audio or pure text); When the acoustic features of a certain speech segment are distorted due to environmental noise, but the fusion feature remains stable after compensation through the text modality, this constraint term can correct the alignment result.

[0025] Dynamic Time Warping (DTW) minimizes to find the minimum mapping path between the audio timeline and the PPT sections; According to the DTW path, map the audio time segments to the PPT sections , then the alignment mapping relationship is: , where represents the temporal mapping relationship between the audio and the PPT sections, generated by the DTW algorithm; represents the th time point on the audio timeline; represents the th section index in the PPT section sequence; represents the total length of the alignment path, i.e., the number of point pairs, automatically determined by the DTW algorithm; For each PPT section , aggregate the speaker labels corresponding to all the mapped audio segments; Use confidence weighting (combining and ) to determine the final speaker label; After alignment, each PPT section is associated with a set of audio time segments, and extract the labels within this time segment; Finally, generate the text with speaker tags; Finally, generate a formatted record, for example: [00:12:30]<Speaker A>: The revenue growth this quarter is ***, mainly due to the expansion of new markets.

[0026] [00:14:20]<Speaker B>: It is recommended to increase the R & D budget to *** million yuan.

[0027] S3. Based on the fused features , the text sequence with speaker tags and the PPT section structure and by analyzing the semantic roles and temporal relationships of the text, construct a structured topic evolution graph using a heterogeneous graph neural network ; Among them, the specific steps involved in constructing the structured topic evolution graph are: Take the fused features , the text sequence with speaker tags and the PPT section structure as the input; among them, each section contains a title and key content text; For the text sequence with speaker tags Perform multi-modal feature extraction, extract semantic, acoustic and temporal features, and fuse semantic role embeddings with time interval information to generate a semantic representation vector of the text ; Take the text sequence with speaker labels as input, use the BERT model to generate a basic text vector, and fuse the semantic role embedding vector to generate a semantic representation vector of the text : Map the time interval to a temporal feature vector through a linear layer : ; ; where represents the embedding vector of the semantic role corresponding to the th text; represents the trainable projection matrix from the semantic role embedding vector to the semantic space; represents the semantic representation vector of the th text; represents the content (string) of the th text segment; represents the timestamp of the th text segment; represents the difference between the end time of the th text and the end time of the th text, represents the end time of the th text; represents the end time of the th text; represents the temporal feature vector obtained by linearly transforming the time interval ; represents passing the th text segment as input into the pre-trained language model BERT to obtain the semantic representation of this text through this model; represents linearly transforming the time interval between text segments, that is, converting the original time difference into a corresponding vector representation through a linear layer (such as a fully connected layer); Concatenate the fused features with the semantic representation vector to generate a joint feature : Based on Term distribution for constructing topic nodes , which is used to enhance the modeling of cross-modal semantics (such as PPT keywords and speech content); In the formula, denotes the feature concatenation operation, which concatenates the semantic representation vector with the fused feature in the feature dimension; denotes the trainable projection matrix; Furthermore, the nested Dirichlet process is adopted to model the topic nodes of the meeting content, mine the parent-child topic hierarchy in the meeting text, and generate topic nodes ; In the formula, denotes the th topic node; denotes the total number of topic nodes; The nested Dirichlet process (NDP) is adopted to generate the parent-child topic node structure, and the topic hierarchy set is obtained; The topic node-term probability distribution is established to characterize the semantic features of each topic node: In the formula, denotes the probability that the th topic node generates the th term; denotes the number of occurrences of the term under the th topic node in the corpus; denotes the total number of terms in the vocabulary; denotes the probability distribution of generating the term set under the condition of the given topic node parameter ; In this embodiment, from the text sequence with speaker labels, speakers are extracted to form the speaker nodes , and the speech frequency and voiceprint confidence corresponding to the speakers are recorded; Among them: In the formula, denotes the total number of speakers; denotes the th speaker node; According to the input PPT chapter structure , the chapter nodes are extracted; In the formula, represents the total number of PPT chapters; represents the th PPT chapter; Then the complete node set ; Based on the semantic role embedding vector and the topic node embedding vector, construct a role-topic association scoring model to quantify the association strength between different semantic roles and specific topics, and combine the speech frequency and topic popularity to dynamically generate contribution relationship edges ; The role-topic association scoring model is specifically: Combine the speech frequency and topic popularity to dynamically adjust the weight of the contribution relationship edge : In the formula, represents the association score between the semantic role and the topic node calculated by cosine similarity; is the vector representation of the th topic node in the embedding space; represents the embedding vector corresponding to the semantic role ; represents the final contribution weight of the semantic role to the topic node ; represents the popularity index of the topic node , which is calculated cumulatively according to the number of times the topic node is discussed or the relevance; represents the speech frequency of the semantic role ; According to the above weight Organize all speaker-topic pairs into an edge set: In the formula, represents a directed edge from the speaker node to the topic node ; represents an element in the set of speaker nodes, represents a specific speaker node, belonging to the set ; represents an element in the set of topic nodes, represents a specific topic node, belonging to the set ; and use as the weight of each edge to quantify the contribution degree of the speaker to the topic such as speech frequency, semantic relevance); By constructing a temporal adjacency matrix , quantify the transfer relationship between topics and topics and combine the Temporal Graph Attention Network (TGAT) to dynamically generate evolution relationship edges ; realize the dynamic update of topic features in the time dimension and the modeling of the evolution path; The specific steps for constructing the evolution relationship edges are as follows: Construct a temporal adjacency matrix : Dynamically update the time sequence of nodes: In the formula, represents the preset time window; represents the total number of transfers of the topic ; represents the number of times the topic transfers to the topic within the time window ; represents the feature vector of the topic node at the time step ; represents the feature vector of the topic node at the time step ; represents the updated feature vector of the topic node at the time step ; In this embodiment, the edge set is defined as: In the formula, represents that there is an evolutionary relationship in time sequence between the topic and the topic ; represents the element of the temporal adjacency matrix; represents the element in the set of topic nodes; For each chapter node and each topic node , calculate the semantic similarity between the chapter node and the topic node , and use the value of the semantic similarity as the inclusion relationship edge The weights are obtained to get the inclusion relationship edges ; Then the complete edge set ; Based on the heterogeneous graph neural network, multi-layer message passing and node update are performed on the complete node set and the complete edge set , and a timestamp is introduced to generate a structured topic evolution graph .

[0028] In this embodiment, the structured topic evolution graph is as follows: In the formula, represents all node sets in the graph, including topic nodes, speaker nodes, and chapter nodes; represents all edge sets in the graph, including contribution relationship edges, evolution relationship edges, and inclusion relationship edges; represents the timestamp set, which is used to characterize the temporal attributes in the graph; The node set includes the following types of nodes: Topic nodes : Represent the specific discussion content. Each topic is represented by a vector, which is composed of a term distribution, a popularity index, and a hierarchical encoding; Speaker nodes : Identify the people participating in the discussion. Each node integrates semantic role embedding, voiceprint information, and speech frequency; Chapter nodes : Represent the chapter information in the meeting materials or PPT. Its features include a semantic vector and the start and end timestamps of the chapter; Edge set includes the following types of edges: Contribution relationship edges : Represent the connection between the speaker and the topic, reflecting the contribution of a certain speaker to a specific topic; Evolution relationship edges : Used to characterize the temporal evolution and transfer relationship between topics; Inclusion relationship edges : Represent the subordination and inclusion relationship between the chapter and the topic; Timestamp set is used to divide the time axis of the meeting into discrete time slices, providing a temporal basis for subsequent dynamic modeling; The heterogeneous graph neural network is used for cross-type message passing, and the joint loss function is used to constrain the structural and temporal consistency: In this embodiment, during the construction of the graph spectrum, the heterogeneous graph neural network jointly models each node through the transmission and fusion of neighborhood information: According to different types of edges, calculate the messages of contributing edges, evolving edges, and inclusion edges respectively: Contributing edge message (from the speaker to the topic ): In the formula, represents the contribution information of the speaker to the topic in the layer, which is used to update the features of the topic node; represents the multi-layer perceptron dedicated to the contributing edge; represents the hidden state of the speaker node in the layer; represents the hidden state of the topic node in the layer; represents the contribution weight of the speaker to the topic ; Evolving edge message (from the topic to the topic ): In the formula, represents the evolving relationship information of the topic to the topic in the layer, which is used to model the temporal dependence between topics; represents the multi-layer perceptron dedicated to the evolving edge; represents the hidden state of the topic node in the layer; represents the element of the temporal adjacency matrix, indicating the probability that the topic transfers to the topic within the time window; Inclusion edge message (from the section to the topic ): Among them, the symbol represents vector concatenation, and each edge type corresponds to an independent multi-layer perceptron (MLP) to parameterize the message generation process; In the formula, is the layer, the section node To the issue node The message of Regarding the issue Subordination relationship information, which is used to enhance the semantic consistency of the issue; Indicates a multi-layer perceptron dedicated to the inclusion edge; Indicates the Hidden state of the chapter node In the layer; Indicates the chapter Regarding the issue Semantic similarity;

[0029] In this embodiment, the message passing and node update process is specifically as follows: According to the edge types (contribution, evolution, inclusion), the corresponding MLPs are respectively used to generate , And ; Node update: After receiving the messages passed by the neighbors, each node aggregates all the messages from its neighbors and updates the hidden state in combination with its own features: In the formula, Indicates the set of neighbors connected to the node , and different edge types correspond to different parameter matrices ; Indicates the projection matrix of the node's own features; Indicates the parameter matrix of the edge type (contribution edge, evolution edge, inclusion edge); Indicates the Layer, the message passed from the neighbor node To the node ; Indicates the node In the Updated hidden state after the layer; Indicates the node In the Hidden state of the layer; At the same time, the information obtained by transforming the node's own features through Is also added, and finally, after ReLU activation, the updated node representation is formed; After several layers of message passing in the constructed heterogeneous graph, the final node embedding is generated. In order to make the node embedding faithfully reflect the information in the original data (such as the term distribution of the issue) and the time evolution characteristics, a joint loss function Is constructed: Reconstruction loss , which restores the vector of the issue node to the original issue distribution: In the formula, is the topic node, representing the topics discussed in the meeting; represents the th embedding vector of the topic node; represents the th true distribution of the topic node; represents the total number of topic nodes; represents the topic node index variable; Temporal consistency loss ensures that the node features in the spectrograms of adjacent time slices change smoothly and conform to the time logic: In the formula, represents the total number of time slices; represents the time slice index variable; represents the th feature matrix of all nodes at time slice represents the th feature matrix of all nodes at time slice including the features of topics, speakers, and chapter nodes; represents the mask matrix used to retain the nodes that persist over time; represents the Frobenius norm used to measure the matrix difference; represents element-wise multiplication; Combining the above two parts of losses, the joint loss function is defined after being balanced by the hyperparameter : In the formula, represents the hyperparameter used to balance the weights of the reconstruction loss and the temporal consistency loss; By continuously optimizing this joint loss, the model not only obtains the embeddings that can accurately reconstruct the attributes of each node, but also ensures good consistency and continuity in the temporal dynamics of the spectrogram.

[0030] S4. Extracting the key decision nodes from the structured topic evolution spectrogram based on the reinforcement learning strategy to generate the final meeting summary document. in to generate the final meeting summary document.

[0031] The specific steps involved in generating the final meeting summary document are as follows: After completing the construction of the structured topic evolution spectrogram and the multi-layer message passing, each topic node in the spectrogram and its semantic features, temporal features, and speaker contributions are used as the state input; Use deep Q-learning to select the topic nodes in the graph that contribute the most to the abstract ; State-action space: State: semantic, temporal, and speaker features of the topic nodes; Action: Select topic nodes to add to the abstract; Reward: Optimize the Q-network according to the quality (ROUGE score) of the generated abstract; For each key decision node, determine the topic text slot value through the term distribution model, determine the speaker slot value through the maximum contribution edge weight, generate the decision content slot value through the SRL model, and align the chapter node output time interval slot value and chapter slot value through cosine similarity; For each selected key decision node , fill in the following slots: Topic text generation: Term distribution method: Select the top 3 words with the highest probability from the term distribution of the topic : In the formula, represents the pre-trained embedding of the word ; represents the term distribution vector of the topic node ; represents the word Traverse all graph node sets (In this formula, the word traverses the set of vocabulary nodes in the graph, and calculates the relevance of each word to the topic node); Use the Seq2Seq model to generate a natural language description to ensure the consistency of the abstract text with the original speech / PPT content: Among them, represents the decoder using the Transformer architecture (the decoding end of the Seq2Seq model), and outputs the natural language sentence for the final abstract; represents the term distribution vector of the topic node ; Speaker determination: In the formula, represents the edge weight from the speaker to the topic node , which is used to quantify the contribution of the speaker to the topic (such as the number of speeches and decision-making influence); Decision content generation: Extract decision-making actions using a semantic role labeling (SRL) model: Align time intervals with sections: Time interval: Obtain the timestamp of the section node , where: Section title: The corresponding PPT section title ; Indicates the start timestamp of the section node ; Indicates the start timestamp of the section node ; Concatenate the values of the above slots in the order of the preset template to form the final meeting summary document: Template concatenation: In the formula, Indicates the text description corresponding to the th key issue node (generated by the term distribution method and Seq2Seq); Indicates the speaker who contributes the most to the issue node ; Indicates the decision content structure extracted from the associated text of the issue node by the SRL model; Indicates the time interval slot corresponding to the representative and the section node , that is, the start and end timestamps of this section; Indicates the PPT / document section title corresponding to the section node ; Indicates the th key issue node index selected by deep Q-learning; Indicates the semantic vector of the section node ; Indicates the term distribution vector of the issue node ; Indicates the semantic role labeling analysis of the issue node text; Indicates the decision-making action triple output by SRL.

[0032] Example 2: This embodiment provides a meeting minutes processing system using AI, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the method for processing meeting minutes using AI according to any one of the above.

[0033] The basic principles, main features and advantages of the present invention have been shown and described above. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification are only preferred examples of the present invention and are not used to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. A method for processing meeting minutes using AI, characterized in that, It includes the following steps: S1. Use multi-source sensors to collect conference audio streams and associated text data, perform real-time noise suppression on the collected conference audio streams and associated text data through a noise suppression algorithm, and perform cross-modal alignment on the denoised data through a cross-modal alignment algorithm to obtain fusion features; S2. Deploy an end-to-end ASR model with hybrid CTC / Attention, and insert domain-specific attention heads into the attention layer of the pre-trained Transformer model to construct a domain-enhanced speech recognition model for converting audio into a text sequence with speaker labels; S3. Based on the fusion features, the text sequence with speaker labels, and the PPT chapter structure, and by analyzing the semantic roles and temporal relationships of the text, use a heterogeneous graph neural network to construct a structured topic evolution graph; S4. Extract key decision nodes from the structured topic evolution graph based on a reinforcement learning strategy to generate a final conference summary document.

2. The method for processing meeting minutes using AI according to claim 1, characterized in that: In S1, the multi-source sensors include audio sensors, visual sensors, and text input sources; The audio sensor directionally captures the speaker's audio through a beamforming algorithm; The text input source is connected to a PPT / PDF document parser in real time for extracting text content and page-turning event timestamps.

3. The method for processing meeting minutes using AI according to claim 2, wherein: In S1, the specific steps involved in performing real-time noise suppression on the collected data through a noise suppression algorithm are: The original audio signal is denoised by spectral subtraction to obtain a denoised amplitude spectrum; The amplitude spectrum and phase spectrum after spectral subtraction denoising are input into a deep neural network for refined denoising to obtain a denoised audio signal.

4. The method for processing meeting minutes using AI according to claim 3, characterized in that: In S1, the specific steps involved in the cross-modal alignment algorithm for performing cross-modal alignment on the denoised data are: Use the SlideTimer algorithm to parse the PPT page-turning log and extract the chapter time set; Detect silent segments through Silero-VAD and segment them into a paragraph boundary set; Construct a candidate pairing set from the chapter time set and the paragraph boundary set, and use the Hungarian algorithm to screen the legal timestamp pairing set in the candidate pairing set to achieve coarse-grained time alignment; Output the coarsely aligned chapter-audio paragraph pairs; Train the audio encoder and text encoder through a contrastive learning framework and construct a unified feature space; Based on the coarsely aligned chapter-audio paragraph pairs, calculate the similarity between the chapter title and the audio paragraph; Use an improved dynamic time warping algorithm to align sentence segments and compensate for local temporal jitter by combining a semantic cost function; Use CRF sequence labeling to identify professional terms, and the labeling transition matrix is dynamically updated; Dynamically calculate weights based on real-time audio feature vectors and text feature vectors; Unify the audio / text features to 512 dimensions through an audio projection matrix and a text projection matrix to generate fusion features.

5. The method for processing meeting minutes using AI according to claim 1, characterized in that: The specific steps involved in constructing the domain-enhanced speech recognition model are: Deploy an end-to-end ASR model with hybrid CTC / Attention; Through a context adaptation mechanism, dynamically adjust the language model weights during the decoding stage; Construct a gating network including acoustic confidence evaluation to receive the posterior probability distribution output by the acoustic model in real time; Analyze the lexical distribution of the current speech segment through a pre-trained domain term detector to generate a domain relevance score; Use the fused features as the input of the gating network, and introduce the acoustic confidence feature and the domain relevance score at the same time to calculate the dynamic language model weight coefficient; Perform context-aware model fusion to generate the final decoding probability; At the same time, insert K domain-specific attention heads into the attention layer of the pre-trained Transformer model to form a hybrid attention structure, construct a learnable domain mask matrix, and adopt a multi-task transfer learning strategy to optimize the model parameters, and transcribe the audio features into a text sequence.

6. The method for processing meeting minutes using AI according to claim 5, wherein: The specific steps involved in converting the audio into the initial text with speaker tags are as follows: Parallelly extract the MFCC features and FBANK features of the denoised audio signal, and splice the features of the two modalities into ; Taking as the input of the GMM-UBM model, the speaker embedding vector is obtained through maximum a posteriori probability estimation; Based on the t-SNE dimensionality reduction and DBSCAN clustering algorithms, the voiceprint feature vector set is divided into speaker clusters, and the voiceprint clustering labels are output. Based on the text and fused features of each voiceprint clustering paragraph, use the Bi-LSTM+CRF model to calculate the semantic role probability distribution, and cross-validate the semantic role with the voiceprint clustering result, and finally label the corresponding speaker identity for each paragraph of text to generate the final speaker label; Introduce semantic similarity constraints and the temporal features of the fused features in dynamic time warping as constraint terms to optimize the semantic cost function, align the speaker label with the PPT chapter information, and generate a text sequence with speaker tags; Among them, the text sequence with speaker tags includes text content, timestamps, and speaker tags that fuse voiceprint and semantics.

7. The method for processing meeting minutes using AI according to claim 1, characterized in that: In step S3, the specific steps involved in constructing a structured topic evolution graph are as follows: Use the fused features, the text sequence with speaker tags, and the PPT chapter structure as inputs; Perform multi-modal feature extraction on the text sequence with speaker tags, extract semantic, acoustic, and temporal features, and fuse the semantic role embedding and the time interval information to generate a semantic representation vector of the text; Concatenate the fused features with the semantic representation vector through linear projection to generate joint features; Use the nested Dirichlet process to model the topic nodes of the meeting content, mine the parent-child topic hierarchy in the meeting text, and generate topic nodes; Extract from the text sequence with speaker tags speakers to form speaker nodes, and record the speech frequency and voiceprint confidence corresponding to each speaker; Extract chapter nodes according to the input PPT chapter structure; Union the topic nodes, speaker nodes, and chapter nodes into a complete node set; Based on the semantic role embedding vector and the topic node embedding vector, construct a role-topic association scoring model to quantify the association strength between different semantic roles and specific topics, and combine the speech frequency and topic popularity to dynamically generate contribution relationship edges; Quantify the issue by constructing a temporal adjacency matrix and the issue between the transfer relations, and combine with the time graph attention network to dynamically generate the evolution relation edges; For each chapter node and each topic node, calculate the semantic similarity between the chapter node and the topic node, and use the value of the semantic similarity as the weight of the inclusion relationship edge to obtain the inclusion relationship edge; Union the contribution relationship edges, evolution relationship edges, and inclusion relationship edges into a complete edge set; Based on the heterogeneous graph neural network, perform multi-layer message passing and node update on the complete node set and the complete edge set, and introduce timestamps to generate a structured topic evolution graph.

8. The method for processing meeting minutes using AI according to claim 1, characterized in that: In step S4, the specific steps involved in generating the final meeting summary document are as follows: Use the state of each topic node in the graph and its semantic features, temporal features, and speaker contributions as inputs; Use deep Q-learning to select the topic node that makes the greatest contribution to the summary from the graph; For each key decision node, determine the topic text slot value through the term distribution model, determine the speaker slot value through the maximum contribution edge weight, generate the decision content slot value through the SRL model, and align the output time interval slot value and chapter slot value of the chapter node through cosine similarity; Concatenate the above slot values in the order of the preset template to form the final meeting summary document.

9. A meeting minutes processing system using AI, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes a computer program to implement the meeting minutes processing method using AI according to any one of claims 1-8.

Images