A Method and System for Intelligent Conference Speech Transcription and Minute Generation Based on Voiceprint Recognition and Hotword Optimization in the Power Industry

Abstract

This invention discloses an intelligent meeting speech transcription and minutes generation method and system for the power industry, featuring voiceprint recognition and hot word optimization. The method includes: pre-collecting voice samples to construct an encrypted voiceprint database; mining professional terms from multi-source data in the power industry and dynamically assigning weights to establish a hot word library; performing unified decoding and adaptive preprocessing on the input audio; extracting voice features and using deep embedding clustering to achieve speaker separation and recognition; integrating the hot word library and using a Transformer-based domain adaptive model to generate transcribed text with speaker tags; utilizing a pre-trained model in the power industry for multi-granular semantic understanding and hierarchical summary generation, automatically outputting structured meeting minutes. This invention effectively solves the problems of low accuracy in recognizing professional terms in the power industry, difficulty in separating multiple speakers, and reliance on manual minute generation, significantly improving the efficiency and intelligence level of meeting recording.

Description

Technical Field

[0001] This invention relates to the fields of speech recognition and natural language processing, and in particular to an intelligent meeting speech transcription and minutes generation method and system for voiceprint recognition and hot word optimization for the power industry. Background Technology

[0002] In the daily operation and management of the power industry, various work meetings, dispatch meetings, and technical seminars are frequently held. These meetings typically involve a large number of technical terms, technical parameters, and complex decision-making processes. Accurately and efficiently recording the meeting content and generating standardized meeting minutes is of great significance for ensuring the safe and stable operation of the power grid and promoting the smooth implementation of projects.

[0003] Currently, traditional meeting minutes primarily rely on manual recording and organization, which suffers from inefficiency, a tendency to miss key information, and strong subjectivity. While some general-purpose speech-to-text systems have been tested in meeting scenarios with the development of speech recognition technology, they still have many limitations: First, the power industry has a large number of technical terms that are rapidly updated, resulting in low accuracy for general speech recognition models in identifying specialized vocabulary; second, in meeting scenarios, multiple speakers take turns speaking, and existing systems are ineffective at speaker separation and recognition, making it difficult to accurately distinguish the content of different speakers; third, the generation of meeting minutes still requires significant manual intervention, lacking the ability to automatically generate structured minutes from transcribed text.

[0004] Furthermore, existing voiceprint recognition technologies are mostly applied in security or finance, with limited research on their application in power industry conference scenarios, and a lack of voiceprint model optimization schemes tailored to the characteristics of the power industry. Meanwhile, traditional hot word optimization methods often use static word lists, making it difficult to adapt to the rapid updates and changes in power industry policies and technical terminology.

[0005] Therefore, there is an urgent need for an intelligent meeting speech transcription and minutes generation method and system that can deeply integrate the characteristics of the power industry and integrate voiceprint recognition, hot word optimization and multi-granular semantic understanding technologies to solve the above-mentioned technical problems and improve the intelligence level and efficiency of meeting management in the power industry. Summary of the Invention

[0006] This invention aims to solve the above-mentioned problems and provides an intelligent meeting speech transcription and minutes generation method and system for voiceprint recognition and hot word optimization in the power industry. Through voiceprint pre-registration and power hot word library optimization, the transcription accuracy and meeting management efficiency are improved.

[0007] In a first aspect, embodiments of this application provide an intelligent meeting speech transcription and minutes generation method for the power industry, featuring voiceprint recognition and hot word optimization, the method comprising:

[0008] S1: Collect voice samples from meeting participants in advance, extract voiceprint feature vectors based on deep neural networks, generate standard voiceprint templates through average pooling, and establish an encrypted voiceprint database.

[0009] S2: Collect professional terms from multi-source data in the power industry through text mining, dynamically allocate word weights based on word frequency, inverse document frequency and timeliness factors, and establish a dynamic update mechanism for the hot word library;

[0010] S3: Receives multi-format audio input, performs unified decoding and adaptive preprocessing, and uses a speech activity detection model based on a one-dimensional convolutional neural network for endpoint detection and silence removal;

[0011] S4: Extract the high-dimensional acoustic feature vector of the preprocessed speech segment, perform unsupervised clustering using a deep embedding clustering algorithm, achieve speaker separation by optimizing the KL divergence loss between the soft assignment probability and the target distribution, and match and identify the clustering results with the voiceprint database.

[0012] S5: It adopts a domain-adaptive speech recognition model based on the Transformer structure, integrates a weighted hot word library during the decoding process, and generates transcribed text with speaker tags through a context-aware dynamic weighting algorithm and a hot word conflict resolution mechanism.

[0013] S6: Utilize a pre-trained natural language processing model in the power industry for multi-granular semantic understanding and hierarchical summary generation, automatically generating structured meeting minutes that include full-text summaries, chapter overviews, and to-do lists;

[0014] S7: Provides an editing interface and encrypts and stores the final minutes, supporting local and cloud-based multimodal output formats.

[0015] Optionally, in one implementation of the first aspect of the present invention, the extraction of voiceprint features and the establishment of a voiceprint database in step S1 includes:

[0016] Several segments of registered speech from the target speaker are collected through a dedicated registration channel. The registered speech is text with known content that covers different phonemes and high-frequency words in the power field.

[0017] Each registered speech segment is preprocessed, including noise reduction, silence removal, and volume normalization.

[0018] Extracting voiceprint feature vectors based on deep neural networks from preprocessed speech;

[0019] Average pooling is performed on all voiceprint feature vectors of the same speaker to generate the standard voiceprint template of that speaker.

[0020] The standard voiceprint template is associated with the speaker's identity information and encrypted and stored in the voiceprint database;

[0021] A voiceprint verification interface is provided to re-verify registered voiceprint templates to ensure their validity and accuracy.

[0022] Optionally, in one implementation of the first aspect of the present invention, the construction of the hot word library in S2 includes:

[0023] We collected professional terms and organization names from multi-source data in the power industry to form a basic hot word set;

[0024] Each word in the basic hot word set is assigned an initial weight, and the weight is assigned based on the following factors: a) the word frequency and inverse document frequency in the source document; b) the timeliness of the word in recent industry policies; c) the importance level of the word in a specific conference context.

[0025] The weighted hot word set is input into the hot word optimization module of the speech recognition model. During the ASR decoding process, the probability of words in the hot word library is increased, and the increase is dynamically controlled by the weight.

[0026] Based on the new words discovered in the conference transcription results and user feedback, the hot word database is incrementally learned and its weights are adjusted to achieve dynamic updates.

[0027] Optionally, in one implementation of the first aspect of the present invention, the preprocessing and speech activity detection in S3 include:

[0028] It receives audio input in multiple formats and decodes them uniformly into pulse code modulation waveform data with a preset sampling rate and bit depth;

[0029] The waveform data is subjected to adaptive preprocessing, including: a) using a noise suppression algorithm based on spectral subtraction to suppress environmental steady-state noise and sudden interference noise; b) using an automatic gain control algorithm to normalize the waveform amplitude to the target dynamic range;

[0030] The preprocessed audio data is divided into frames and input into a speech activity detection model based on a one-dimensional convolutional neural network. The speech / non-speech probability is calculated frame by frame to obtain a probability sequence.

[0031] The start and end points of the speech segment are determined based on the probability sequence, and silent segments exceeding a preset duration are cut off to output continuous speech segments.

[0032] Optionally, in one implementation of the first aspect of the present invention, S4 includes:

[0033] Extract high-dimensional acoustic feature vectors from the preprocessed speech segments;

[0034] Unsupervised clustering is performed on the feature vectors, and the soft assignment probability q is optimized. ij With the target distribution p ij Speaker separation is achieved using the KL divergence loss between the two, where:

[0035] a) Calculate the eigenvector z using the Student's t-distribution. i With cluster center μ j The soft assignment probability q between ij :

[0036]

[0037] Among them, z i μ represents the high-dimensional acoustic feature vector of the i-th speech segment. j Let j represent the j-th cluster center, where j represents the traversal of all cluster centers;

[0038] b) Construct the target distribution p by reinforcing the high-confidence assignment. ij :

[0039]

[0040] Where f j =∑ i q ij , representing the sum of soft assignment frequencies of the j-th cluster center;

[0041] c) Synchronously update the cluster centers μ by minimizing the KL divergence loss function L. j And the parameters θ of the deep neural network:

[0042]

[0043] Where P represents the target distribution, Q represents the soft assignment distribution, and L is the KL divergence loss value;

[0044] The clustering results are matched with the standard voiceprint templates in the voiceprint database. If the maximum similarity exceeds the preset threshold, the cluster is mapped to the corresponding speaker identity; otherwise, it is marked as an unknown speaker.

[0045] Optionally, in one implementation of the first aspect of the present invention, the unsupervised clustering process employs an adaptive optimization strategy for the number of clusters, including:

[0046] Automatically estimate the potential number K of speaker roles in conference audio;

[0047] In the process of deep embedding clustering iterative optimization, the stability of clustering is evaluated by calculating the normalized mutual information of the clustering assignment results;

[0048] For each cluster formed after stable clustering, calculate the average silhouette coefficient of its internal members, and perform sub-cluster splitting operation on abnormal clusters whose silhouette coefficient is lower than a preset threshold.

[0049] Calculate the cosine similarity between different cluster centers, and automatically merge cluster pairs that have similarity exceeding the merging threshold and have similar acoustic features.

[0050] By iteratively performing sub-cluster splitting and cluster pair merging operations, the number of clusters is dynamically and adaptively adjusted to obtain the optimal clustering result that matches the actual number of speakers.

[0051] Optionally, in one implementation of the first aspect of the present invention, step S5 includes:

[0052] Constructing a domain-adaptive speech recognition model: An end-to-end speech recognition model based on the Transformer architecture is adopted, and its language model components are pre-trained for domain adaptability using a power industry text corpus;

[0053] During ASR decoding, a weighted hot word library is integrated and a dynamic weighting algorithm is adopted. The bias weight β is dynamically calculated based on the confidence of the hot word in the current acoustic context and the global frequency, thereby increasing the probability of the hot word.

[0054] When there are similar-sounding hot words, the context-based semantic conflict resolution module is activated, and the domain language model is used to select the word that best matches the current semantic context.

[0055] The optimized recognized text is aligned and fused with speaker identification tags and timestamps to generate a structured transcribed text sequence with speaker tags.

[0056] Optionally, in one implementation of the first aspect of the present invention, the dynamic weighting algorithm further includes a hot word enhancement strategy based on an attention mechanism:

[0057] The current speech frame and its context acoustic features are input into a network based on a multi-head self-attention mechanism to generate context-aware acoustic representation vectors.

[0058] Calculate the attention weights between the word embeddings in the hot word library and the current acoustic context representation vector, and generate a hot word-context relevance score;

[0059] The relevance score, global frequency statistics of hot words, and preset basic weights are fused in multiple dimensions, and the final dynamic bias weight β of hot words is generated through a gating mechanism.

[0060] The dynamic bias weight β is smoothed using a sliding window over the time dimension;

[0061] An adversarial training mechanism is introduced during the hot word weighting process to enhance the model's robustness against malicious hot word interference attacks.

[0062] Optionally, in one implementation of the first aspect of the present invention, the minute generation in S6 includes:

[0063] Transcribed text with speaker tags is input into a pre-trained natural language processing model in the power industry for multi-granular semantic understanding, including: semantic error correction and standardization of power industry terms, identification and annotation of meeting elements, and extraction and standardization of technical parameters and indicators.

[0064] A sequence-to-sequence model based on pointer generation network is used to generate hierarchical conference summaries, including: full text summary, chapter overview, and action item extraction;

[0065] According to the standard format of power industry meeting minutes, the summary generation results are assembled into a structured document, including a basic meeting information area, a topic discussion area, and an action item list area;

[0066] It supports generating structured minutes output in various formats, including editable document format, machine-readable data format, and visual dashboard format.

[0067] Secondly, embodiments of this application provide an intelligent conference speech transcription and minutes generation system for the power industry based on voiceprint recognition and hot word optimization, applied to the intelligent conference speech transcription and minutes generation method for the power industry based on voiceprint recognition and hot word optimization as described in the first aspect. The system includes:

[0068] The voiceprint pre-registration module is used to collect voice samples from meeting participants in advance, extract voiceprint feature vectors based on deep neural networks, generate standard voiceprint templates through average pooling, and establish an encrypted voiceprint database.

[0069] The hot term library construction module is used to collect professional terms from multi-source data in the power industry through text mining, dynamically allocate word weights based on word frequency, inverse document frequency and timeliness factors, and establish a dynamic update mechanism for the hot term library.

[0070] The speech processing module is used to receive multi-format audio input, perform unified decoding and adaptive preprocessing, and adopt a speech activity detection model based on a one-dimensional convolutional neural network for endpoint detection and silence removal.

[0071] The speaker separation and recognition module is used to extract high-dimensional acoustic feature vectors of preprocessed speech segments, perform unsupervised clustering using a deep embedding clustering algorithm, achieve speaker separation by optimizing the KL divergence loss between the soft assignment probability and the target distribution, and match and recognize the clustering results with the voiceprint database.

[0072] The speech recognition module employs a domain-adaptive speech recognition model based on the Transformer architecture. During the decoding process, it integrates a weighted hot word library and generates transcribed text with speaker tags through a context-aware dynamic weighting algorithm and a hot word conflict resolution mechanism.

[0073] The minutes generation module is used to perform multi-granular semantic understanding and hierarchical summary generation using a pre-trained natural language processing model in the power field, and automatically generate structured meeting minutes that include a full text summary, chapter overview and to-do list;

[0074] The output storage module provides an editing interface and encrypts the storage of the final minutes, supporting local and cloud-based multimodal output formats.

[0075] Thirdly, embodiments of this application provide an electronic device, including:

[0076] processor;

[0077] Memory used to store processor-executable instructions;

[0078] The processor is configured to implement, when executing the instructions, the intelligent conference speech transcription and minutes generation method for voiceprint recognition and hot word optimization for the power industry as described in the first aspect.

[0079] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program that instructs a device to execute the intelligent conference speech transcription and minutes generation method for voiceprint recognition and hot word optimization for the power industry as described in the first aspect.

[0080] This invention discloses an intelligent meeting speech transcription and minutes generation method and system for the power industry, featuring voiceprint recognition and hot word optimization. The method includes: pre-collecting voice samples to construct an encrypted voiceprint database; mining professional terms from multi-source data in the power industry and dynamically assigning weights to establish a hot word library; performing unified decoding and adaptive preprocessing on the input audio; extracting voice features and using deep embedding clustering to achieve speaker separation and recognition; integrating the hot word library and using a Transformer-based domain adaptive model to generate transcribed text with speaker tags; utilizing a pre-trained model in the power industry for multi-granular semantic understanding and hierarchical summary generation, automatically outputting structured meeting minutes. This invention effectively solves the problems of low accuracy in recognizing professional terms in the power industry, difficulty in separating multiple speakers, and reliance on manual minute generation, significantly improving the efficiency and intelligence level of meeting recording.

[0081] Beneficial effects:

[0082] 1. Significantly improves the accuracy of power industry terminology recognition: By dynamically mining and constructing a professional hot word database from multi-source data in the power industry, and dynamically allocating weights based on word frequency, inverse document frequency, and timeliness factors, the low recognition rate of general speech recognition models for power industry professional terms is effectively solved. Combined with context-aware dynamic weighting and decoding optimization, the recognition accuracy of professional terms, technical parameters, and organization names is significantly improved.

[0083] 2. Achieve high-precision automatic speaker separation and recognition in multi-person conferences: Employing an unsupervised clustering algorithm based on deep embedding clustering, and optimizing the KL divergence between the soft assignment probability and the target distribution, it can effectively distinguish different speakers. Combined with a pre-established voiceprint database for matching, it achieves automatic speaker identification, solving the problem of speech separation and identity recognition in multi-person alternating speaking scenarios.

[0084] 3. Enhanced system adaptability to industry changes and unknown scenarios: The hot word library has a dynamic update mechanism, enabling incremental learning and weight adjustment based on new words in meeting transcription results and user feedback; the voiceprint recognition module supports re-verification of registered templates through a verification interface, ensuring the validity of voiceprint templates. This self-evolutionary mechanism allows the system to continuously adapt to the development and changes in power industry policies and technologies.

[0085] 4. Achieve end-to-end intelligent generation from audio to structured minutes: Utilizing a pre-trained natural language processing model in the power industry for multi-granular semantic understanding, it can automatically complete technical terminology correction, meeting element identification, and action item extraction. Furthermore, employing hierarchical summary generation technology, it automatically produces structured meeting minutes containing a full-text summary, chapter overview, and to-do list, significantly reducing the workload of manual compilation and writing, and improving meeting efficiency.

[0086] 5. Enhanced system usability and security: Supports multi-format audio input and preprocessing, improving the system's adaptability to different meeting recording environments. Provides an editing interface and encrypts the storage of final minutes, supporting multimodal output formats (such as Word, PDF, JSON, and visual dashboards) both locally and in the cloud. This satisfies the needs for flexible editing and convenient viewing while ensuring data security of meeting content, and is easy to integrate with existing enterprise project management systems. Attached Figure Description

[0087] Figure 1 This is a schematic flowchart of an intelligent conference speech transcription and minutes generation method for voiceprint recognition and hot word optimization in the power industry, provided as an embodiment of this application.

[0088] Figure 2 This application provides an architecture diagram for an intelligent conference speech transcription and minutes generation system based on voiceprint recognition and hot word optimization for the power industry.

[0089] Figure 3 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0090] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0091] It should be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0092] It should be noted that in the embodiments of this application, the terms "first," "second," etc., are used only for descriptive purposes and should not be construed as indicating or implying relative importance, nor as indicating or implying order. Features defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, terms such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiments or designs described as exemplary or for example in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present related concepts in a specific manner.

[0093] Based on the embodiments described in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0094] Example 1

[0095] Figure 1 This is a schematic flowchart illustrating a method for intelligent conference speech transcription and minutes generation based on voiceprint recognition and hot word optimization for the power industry, provided as an embodiment of this application. Figure 1 As shown, a method for intelligent conference speech transcription and minutes generation based on voiceprint recognition and hot word optimization for the power industry includes:

[0096] S1: Voice samples from meeting participants are pre-collected, and voiceprint feature vectors based on deep neural networks are extracted. Standard voiceprint templates are generated through average pooling, and an encrypted voiceprint database is established. Through voiceprint registration and database construction, biometric data collection and modeling of speaker identity are achieved. Speaker voiceprint features are extracted using deep neural networks, and standard templates are generated through average pooling. An encrypted database is then established to provide a comparison benchmark for subsequent speaker identification.

[0097] Specifically, in this embodiment, the extraction of voiceprint features and the establishment of a voiceprint database in step S1 includes:

[0098] Several segments of registered speech from the target speaker are collected through a dedicated registration channel. The registered speech is text with known content, covering different phonemes and high-frequency vocabulary from the power industry. Collecting these segments through a dedicated channel ensures the quality and coverage of the registered speech. The dedicated channel allows for control of the recording environment, reducing noise interference. The requirement that the speech content covers different phonemes and high-frequency vocabulary from the power industry aims to obtain comprehensive and representative acoustic features of the speaker, laying the foundation for generating a high-precision voiceprint template.

[0099] Each registered speech segment undergoes preprocessing, including noise reduction, silence removal, and volume normalization. Preprocessing standardizes the speech signal and improves the robustness of feature extraction. Noise reduction, silence removal, and volume normalization eliminate environmental interference and differences in recording conditions, ensuring the purity and consistency of subsequently extracted voiceprint features and reducing errors in later recognition stages.

[0100] Voiceprint feature vectors are extracted from preprocessed speech. These feature vectors are x-vectors extracted based on deep neural networks. Extracting voiceprint feature vectors from preprocessed speech captures the unique identity characteristics of the speaker. Using techniques such as x-vectors based on deep neural networks, high-dimensional, abstract representation vectors are extracted from the speech signal. These vectors can effectively distinguish different speakers and are the core of voiceprint recognition. An x-vector is a fixed-length speaker embedding vector. It can be understood as a digital "fingerprint" or "DNA sequence." It compresses a variable-length speech segment (possibly a few seconds or minutes) into a fixed-dimensional numerical vector (e.g., 512 digits). Regardless of whether the input speech is 5 seconds or 5 minutes, the output x-vector dimension remains fixed. This greatly simplifies subsequent comparison, classification, and clustering operations. This vector can preserve the speaker's identity information to the greatest extent possible while ignoring other information in the speech, such as speech content, environmental noise, channel differences (e.g., mobile phone, microphone), and emotion.

[0101] The x-vector model is essentially a deep neural network with a specially designed structure, typically based on a TDNN (Time Delay Neural Network) or a similar structure. Its training process can be summarized as follows: 1. Training Phase (The model learns "how to extract speaker features"). Input: A large number of speech segments from different speakers (and their corresponding speaker labels). Network Structure: Frame-level Layers: The network first processes short frames of speech (e.g., 25ms per frame), gradually extracting microscopic acoustic features. Statistics Pooling Layer: This is the key innovation of x-vector. This layer receives features from all frames over a period of time and calculates their mean and standard deviation. This step aggregates variable-length frame-level features into a fixed-length global feature vector, thereby capturing the statistical characteristics of the entire speech segment. Segment-level Layers: Further transformations and non-linear processing are applied to the pooled global features, ultimately outputting a fixed-length x-vector. Output Layer: Finally, a classification layer attempts to determine which speaker the speech segment belongs to based on this x-vector. Objective: The goal of training is to enable the network to adjust its parameters so that the x-vectors of different speech from the same speaker are as close as possible in the vector space, while the x-vectors of different speakers are as far apart as possible.

[0102] 2. Inference Phase (Extracting "Voice Fingerprints" Using the Model). After training, we remove the final classification layer. For a new speech segment, we input it into the network and extract the output vector from the end of the segment layer; this vector is the x-vector of that speech segment. This extracted x-vector can be used for: Voiceprint Verification: Comparing the similarity of two x-vectors (e.g., calculating cosine similarity) to determine if they come from the same person. Voiceprint Recognition: Comparing the x-vector to be tested with the x-vectors of all registered templates in the voiceprint database to find the most similar one. Speaker Clustering: As you saw in the claims, directly performing cluster analysis on these x-vectors.

[0103] The x-vector, extracted using deep neural networks, functions as a high-performance speaker feature extractor. It transforms a variable-length speech segment into a fixed-dimensional numerical vector that highly represents the speaker's identity information. This vector serves as the core foundation and input data for subsequent speaker clustering (as in step S4) and matching / identifying with a voiceprint database. It is precisely because of modern deep learning techniques like x-vector that high-precision speaker separation and recognition in subsequent steps are effectively achieved.

[0104] The standard voiceprint template for the same speaker is generated by average pooling all the voiceprint feature vectors in the feature space. This step produces a stable and reliable standard voiceprint template. By average pooling multiple feature vectors of the same speaker, the random variations in a single pronunciation can be smoothed out, resulting in a more representative and robust voiceprint template, thus improving recognition stability.

[0105] Specifically, let's assume we extracted three voiceprint feature vectors (x-vectors) from three different speech segments of Zhang San. Each vector is very simplified, with only four dimensions (in reality, it may have 512 or more dimensions). These four dimensions can be abstractly understood as representing some underlying features of the sound (such as timbre, pitch, formant distribution, etc.). The three extracted original vectors are: vector_1 = [0.8, 1.2, -0.5, 0.3] (from Zhang San's speech when he said "transformer"). vector_2 = [1.0, 0.9, -0.6, 0.5] (from Zhang San's speech when he said "load forecasting"). vector_3 = [0.9, 1.1, -0.4, 0.2] (from Zhang San's speech when he said "inspection"). Average pooling calculation: To generate a standard template, we average the values ​​of each dimension separately. Dimension 1 (D1) average = (0.8 + 1.0 + 0.9) / 3 = 0.9. Dimension 2 (D2) average value = (1.2 + 0.9 + 1.1) / 3 = 1.067. Dimension 3 (D3) average value = (-0.5 + (-0.6) + (-0.4)) / 3 = -0.5. Dimension 4 (D4) average value = (0.3 + 0.5 + 0.2) / 3 = 0.333. The final generated standard voiceprint template: template_zhangsan = [0.9, 1.067, -0.5, 0.333].

[0106] The arithmetic mean of N voiceprint feature vectors from the same speaker is calculated for each dimension, resulting in a new vector with the same dimensions. This new vector is the speaker's "standard voiceprint template," which is more stable and representative, and is used for subsequent recognition and comparison.

[0107] The standard voiceprint template is associated with the speaker's identity information and encrypted and stored in a local voiceprint database. This establishes a recognizable and secure voiceprint database, binding biometric features with identity information to provide a basis for identifying the speaker's identity in speech segments; encrypted storage protects the speaker's biometric privacy and complies with data security regulations.

[0108] A voiceprint verification interface is provided for re-verifying registered voiceprint templates to ensure their validity and accuracy. Voiceprints can drift over time or due to changes in physical condition; this function ensures the system can consistently and accurately identify registered speakers by periodically re-verifying and updating the templates, demonstrating the system's self-maintenance capabilities.

[0109] S2: Technical terms are collected from multi-source data in the power industry through text mining. Lexical weights are dynamically assigned based on word frequency, inverse document frequency, and timeliness, and a dynamic update mechanism for the hot word library is established. Through the construction and dynamic updating of the library, the accuracy of domain-specific terminology recognition is improved. Technical terms are mined from multi-source text in the power industry, and weights are dynamically assigned based on word frequency, inverse document frequency, and timeliness to establish an adaptively updateable hot word library, thus optimizing the speech recognition decoding process.

[0110] Specifically, in this embodiment, the hot word library construction step includes:

[0111] S2.1: Multi-source data collection: From power industry policy documents, technical standards, academic papers, historical meeting minutes, and project documents, professional terms and organization names are collected using text mining and keyword extraction techniques to form a basic hot word set. Through multi-source data collection, a basic professional terminology database is built. From authoritative and diverse data sources in the power industry (policies, standards, papers, minutes, etc.), core professional terms and organization names are collected using automated text mining techniques, ensuring that the hot word database possesses a high degree of industry professionalism and coverage from its inception.

[0112] S2.2: Dynamic Weight Allocation: An initial weight is assigned to each word in the basic hot word set. The weight assignment is based on at least one of the following factors: a) the word's frequency and inverse document frequency in the source document; b) the word's timeliness in recent industry policies; c) the word's importance level in a specific meeting context. Through dynamic weight allocation, the importance of words is quantified and priorities are established. Each word in the hot word library is assigned a dynamic weight that integrates its statistical significance (word frequency and inverse document frequency), timeliness (relevance to recent policies), and contextual importance (meeting context), thereby improving the probability of differentiated and accurate word recognition during speech recognition decoding.

[0113] S2.3: Hotword Optimization Model Integration: The weighted hotword set is input into the hotword optimization module of the speech recognition model. During ASR decoding, the probability of words in the hotword library is increased, and the increase is dynamically controlled by the weight values. Through hotword optimization model integration, hotword knowledge is injected into the speech recognition process. The weighted hotword set is directly integrated into the hotword optimization module of the ASR (Automatic Speech Recognition) model. During decoding, the system dynamically increases the probability of words in the candidate words according to their weight values, thereby significantly improving the recognition accuracy of specialized terms.

[0114] S2.4: Dynamic Update of Hot Word Database: Establish a hot word database update mechanism. Based on new word discoveries in the transcription results of each meeting and user feedback, the hot word database undergoes incremental learning and weight adjustments to achieve self-evolution. Through dynamic updates, the hot word database achieves self-evolution and adaptation. A continuous learning mechanism is established, analyzing the actual transcription results of each meeting (new word discovery) and direct user feedback to incrementally update the hot word database (adding new words) and adjust weights (optimizing the importance of old words), enabling the system to continuously adapt to vocabulary changes brought about by new technologies and policies in the power industry.

[0115] The above steps construct a complete and intelligent hot word management lifecycle. Its functionality begins with basic data collection, quantifies word importance through multi-dimensional weight evaluation, integrates with the core recognition engine for effectiveness, and finally achieves self-iterative optimization through a closed-loop feedback mechanism. These four steps work together to ensure the effective implementation of the aforementioned "dynamic hot word database update mechanism," which is the core guarantee for improving the accuracy of power industry terminology recognition.

[0116] S3: Receives multi-format audio input, performs unified decoding and adaptive preprocessing, and employs a speech activity detection model based on a one-dimensional convolutional neural network for endpoint detection and silence removal. Through audio preprocessing and speech activity detection, it improves audio quality and locates effective speech segments. It supports multi-format audio input, performs decoding, noise reduction, and gain normalization, and utilizes a deep learning model to detect speech endpoints, remove silences, and generate clean speech segments for subsequent processing.

[0117] Specifically, in this embodiment, the preprocessing and speech activity detection steps in S3 include:

[0118] S3.1: Multi-format audio adaptation and decoding: Receives input audio data, including online real-time audio streams or offline audio files; decodes the audio data, converting it into pulse code modulation waveform data with a preset sampling rate and bit depth to adapt to subsequent processing modules. Multi-format audio adaptation and decoding achieves the universality and standardization of the input interface. It receives audio data from different sources (such as online conferencing software, voice recorders, etc.) and in different formats (such as MP3, WAV, AAC, etc.), decodes it uniformly, and converts it into the standardized PCM waveform data format required by subsequent processing modules, ensuring the system can handle various common conference recording inputs.

[0119] S3.2: Adaptive Preprocessing: Preprocessing of the uniformly formatted waveform data includes: a) using a noise suppression algorithm based on spectral subtraction to suppress steady-state environmental noise and sudden interference noise; b) using an automatic gain control algorithm to normalize the waveform amplitude to the target dynamic range, eliminating volume fluctuations caused by changes in the distance between the speaker and the microphone. Adaptive preprocessing improves audio signal quality, providing clean and standardized input for subsequent analysis. This step includes two core processes: a) Noise Suppression: Employing a spectral subtraction algorithm to effectively filter out steady-state noise (such as air conditioner and fan noise) and sudden impulse noise (such as keyboard clicks and coughs) from the environment, preserving a clear speech signal. b) Volume Normalization: Using an automatic gain control (AGC) algorithm to eliminate volume fluctuations caused by changes in the distance between the speaker and the microphone and individual differences in pronunciation habits, adjusting the waveform amplitude to a stable target range to ensure the stability of subsequent processing.

[0120] S3.3: Deep Learning-Based Speech Activity Detection: Preprocessed audio data is segmented into fixed-length frames and input into a pre-trained speech activity detection model. This model is a one-dimensional convolutional neural network-based classifier used to calculate the probability of each frame being speech or non-speech. Through deep learning-based speech activity detection, speech segments are accurately distinguished from non-speech segments (silence or noise). The preprocessed audio is segmented into short frames, and a pre-trained one-dimensional CNN classifier is used to calculate the probability of each frame being speech. This deep learning method is more effective than traditional energy-based VAD algorithms in handling low signal-to-noise ratio environments and non-stationary noise, resulting in higher detection accuracy.

[0121] S3.4: Silence Segment Removal and Speech Segment Endpoint Detection: Based on the probability sequence output by the speech activity detection model, the start and end endpoints of speech segments are determined, and silence segments exceeding a preset duration are removed, generating clean, continuous speech segments for output to the speaker separation and recognition steps. Through silence segment removal and speech segment endpoint detection, valid speech is located and redundant data processing is reduced. Based on the probability sequence output by the VAD model, the start and end endpoints (boundaries) of each speech segment are accurately determined, and silence segments exceeding a preset duration are completely removed. The final output is a continuous, clean speech segment, thereby significantly reducing the computational load and misjudgment risk of subsequent speaker separation and speech recognition modules.

[0122] A robust front-end signal processing pipeline is constructed. It first addresses input compatibility issues, then "cleans" the signal through noise reduction and gain control, intelligently "finds" useful speech using a deep learning model, and finally "extracts" key segments through endpoint detection and silence removal. This series of steps collectively ensures that subsequent core algorithms (speaker separation, ASR) receive high-quality, high signal-to-noise ratio input, providing a crucial foundation for the efficient and accurate operation of the entire system.

[0123] S4: Extract high-dimensional acoustic feature vectors from the preprocessed speech segments, perform unsupervised clustering using a deep embedding clustering algorithm, and achieve speaker separation by optimizing the KL divergence loss between the soft assignment probability and the target distribution. Then, match the clustering results with a speakerprint database for identification. Through speaker separation and recognition, speaker differentiation and identity recognition are achieved in multi-person scenarios. Extract speech acoustic features, perform unsupervised speaker separation using deep embedding clustering, optimize the clustering loss function, and match the results with a speakerprint database to output identified speech segments.

[0124] Specifically, in this embodiment, S4 specifically includes:

[0125] S4.1: Input the preprocessed speech segments into a pre-trained deep neural network model to extract the high-dimensional acoustic feature vector of each speech segment, denoted as: Where n is the number of speech segments and d is the feature dimension. High-dimensional acoustic feature vector extraction converts the speech signal into a machine-processable deep feature representation. Pre-trained deep neural networks (such as TDNN and ResNet) are then used to extract high-dimensional, abstract acoustic feature vectors (such as x-vectors) from the pre-processed speech segments. These features effectively characterize speaker identity information, providing high-quality input for subsequent clustering.

[0126] S4.2: Perform unsupervised clustering on the high-dimensional acoustic feature vectors. This process is achieved by optimizing the following objective function: a) Soft assignment: Calculate the feature vector z using the Student's t-distribution. iWith cluster center μ j The similarity between them is used as the soft assignment probability q ij :

[0127]

[0128] b) Target distribution: Construct a target distribution p by reinforcing the high-confidence assignment. ij :

[0129] Where f j =∑ i q ij ,

[0130] c) Joint optimization: The cluster centers μ are updated synchronously by minimizing the KL divergence loss function L between the soft-assigned distribution Q and the target distribution P. j And the parameters θ of the deep neural network:

[0131]

[0132] Speaker separation is achieved through unsupervised clustering, automatically grouping speech segments according to different speakers even when the speaker's identity is unknown. This step is accomplished by optimizing the KL divergence loss between the soft assignment probability and the target distribution: a) Soft assignment: The similarity between the feature vector and the cluster center is calculated using the Student's t-distribution as the soft assignment probability. This is a "soft" and probabilistic assignment method, more flexible than "hard" assignment, and conducive to optimization. b) Target distribution: A target distribution is constructed by reinforcing high-confidence assignments, guiding the clustering process towards a more deterministic direction. c) Joint optimization: By minimizing the KL divergence, the cluster centers and deep neural network parameters are updated simultaneously, achieving end-to-end optimization. This allows feature extraction and clustering tasks to mutually reinforce each other, improving separation accuracy.

[0133] In deep neural networks, parameters are variables that can be automatically adjusted within the model. They determine how the network transforms and processes input data to obtain the final output. The purpose of the training process is to find a set of "optimal values" for all these parameters so that the model's predictions are most accurate.

[0134] Specifically, the unsupervised clustering process in S4.2 further includes an optimization strategy for adaptive cluster number:

[0135] S4.2.1: A cluster number estimation algorithm based on data distribution density is adopted. By performing spectral analysis on high-dimensional acoustic feature vectors or using methods based on gap statistics, the potential number K of speaker roles in the conference audio is automatically estimated. Automatic cluster number estimation solves the problem of "not knowing how many people are in the meeting." There is no need to pre-set the number of clusters. Through algorithms such as spectral analysis or gap statistics, the distribution structure of high-dimensional acoustic feature vectors is automatically analyzed to estimate the potential number of speaker roles K in the meeting, providing initial parameters for subsequent clustering and avoiding biases caused by manual settings.

[0136] The gap statistic method can be used to find the optimal K value by comparing the clustering error of the actual data with that of a reference distribution (such as a uniform distribution).

[0137] Gap(k) = E * [log(W k )]-log(W k ),

[0138] Among them, W k It is the sum of squared clustering errors of the actual data, E * [log(W k )] is log(W) under the reference distribution k The expectation of Gap(k). Choose the K that maximizes Gap(k) as the estimate.

[0139] S4.2.2: During the DEC iterative optimization process, the normalized mutual information (NMI) of the clustering assignment results in consecutive iterations is calculated. When the evaluation index is higher than the preset stability threshold, the clustering result is determined to be stable. Through clustering stability assessment, the system intelligently determines when the clustering process can be stopped. During the iterative optimization process, the stability of the clustering assignment is evaluated by calculating the normalized mutual information (NMI) of the results in consecutive iterations. When the NMI is higher than the preset threshold, it indicates that the clustering result has stabilized, and the iteration can be terminated to prevent over-iteration or non-convergence.

[0140] The normalized mutual information (NMI) of the clustering assignments between two consecutive iterations (t and t+1) is calculated using the following formula:

[0141]

[0142] Among them, (C) t C t+1 H(C) is mutual information. t ) is the distribution entropy. When NMI exceeds the threshold, the cluster is considered stable.

[0143] S4.2.3: For each cluster formed after stable clustering, calculate the average silhouette coefficient of its internal members. For anomalous clusters with silhouette coefficients below a preset threshold, automatically perform a sub-cluster splitting operation, dividing them into two new cluster centers, and recalculate the soft assignment. The anomalous cluster splitting operation addresses the under-clustering problem of "multiple speakers in one cluster." Calculate the average silhouette coefficient within each cluster to measure the compactness and separation of its members. For anomalous clusters with excessively low silhouette coefficients (indicating a loose internal structure and possible mixing of different speakers), automatically perform a splitting operation, dividing them into two purer sub-clusters, improving the separation granularity.

[0144] Calculate the average silhouette coefficient s(j) for cluster j:

[0145]

[0146] Where a(i) is the average distance from sample i to other points within the cluster, and b(i) is the average distance from sample i to the nearest other cluster point. If s(j) < τ split If so, then perform a K-means (k=2) splitting operation on the cluster.

[0147] S4.2.4: Calculate the cosine similarity between different cluster centers and automatically merge cluster pairs with similar acoustic features and similar similarity exceeding the merging threshold to avoid speaker identity splitting caused by over-clustering. The similarity cluster merging operation solves the over-clustering problem of "the same speaker being split into multiple clusters". It calculates the cosine similarity between different cluster centers and automatically merges cluster pairs with similar acoustic features and similar similarity exceeding the merging threshold. This avoids misidentifying the same speaker as multiple people due to speech changes (such as laughter, coughing) or model fluctuations.

[0148] Calculate the cosine similarity between the centers of clusters j and k:

[0149]

[0150] If Sim(μ) j ,μ k If the average acoustic feature distance between the two clusters is greater than the preset threshold and less than the threshold, then the two clusters are merged.

[0151] S4.2.5: Through iterative operations in steps S4.2.3 and S4.2.4, the number of clusters is dynamically and adaptively adjusted, ultimately obtaining the optimal clustering result that matches the actual number of speakers. Dynamic adaptive adjustment achieves fully automatic optimization of the number of clusters. By iteratively executing splitting (S4.2.3) and merging (S4.2.4) operations, the number of clusters can be dynamically adjusted, ultimately approximating the actual number of speakers and obtaining the optimal clustering result. The entire process requires no manual intervention, demonstrating the system's intelligence.

[0152] By introducing quantitative indicators and criteria such as gap statistics, NMI, silhouette coefficient, and cosine similarity, an adaptive optimization loop for cluster number estimation, stability assessment, splitting and merging is realized, which significantly improves the accuracy and robustness of speaker separation.

[0153] S4.3: After clustering, each resulting cluster is matched with a standard voiceprint template in the voiceprint database for similarity. If the maximum similarity exceeds a preset threshold, the cluster label is mapped to the corresponding speaker identity; otherwise, it is marked as an unknown speaker. By matching the clustering results with the voiceprint database, a specific speaker identity label is assigned to each separated cluster. The central feature (or the average vector of all features within the cluster) of each cluster is matched with a pre-stored standard voiceprint template in the voiceprint database for similarity (e.g., cosine similarity calculation). If the maximum similarity exceeds a preset threshold, the cluster is considered to belong to a known speaker; otherwise, it is marked as an "unknown speaker," thus achieving automatic speaker identification.

[0154] The steps described above detail how deep learning and unsupervised clustering techniques can be used to achieve automatic separation and identification of multiple speakers without prior knowledge of who is speaking. Its core innovation lies in employing a joint optimization strategy (S4.2c), which unifies feature learning and clustering within a single framework. By optimizing the KL divergence loss, the extracted features are made more conducive to clustering, thus significantly improving the accuracy and robustness of speaker separation. Finally, by matching with a speakerprint database, a closed loop from "separation" to "recognition" is completed.

[0155] S5: Employs a domain-adaptive speech recognition model based on the Transformer architecture. During decoding, a weighted hotword library is integrated. Through a context-aware dynamic weighting algorithm and a hotword conflict resolution mechanism, transcribed text with speaker tags is generated. By integrating domain-adaptive speech recognition with hotwords, high-accuracy transcribed text with speaker tags is generated. A Transformer model adapted for the power industry is used, dynamically weighting hotword probabilities during decoding to resolve pronunciation conflicts and output structured transcription results.

[0156] Specifically, in this embodiment, step S5 includes:

[0157] S5.1: Constructing a Domain-Adaptive Speech Recognition Model: An end-to-end speech recognition model based on the Transformer architecture is adopted as the foundational acoustic model. The language model components of this foundational acoustic model are continuously pre-trained using a large-scale power industry text corpus to enhance its ability to model grammatical and semantic patterns within the power industry. This process constructs a domain-adaptive speech recognition model, creating a speech recognition engine capable of understanding the specialized language of the power industry. The end-to-end model based on the Transformer architecture is used as the foundation, effectively modeling the long-term contextual dependencies of speech signals. The language model components are continuously pre-trained using a large-scale power industry text corpus (such as technical standards, operating procedures, and project reports). This step is crucial, as it enables the model to deeply understand and master the specific grammatical structures, terminology collocations, and semantic expression habits of the power industry, thereby significantly improving its overall recognition ability for specialized content.

[0158] S5.2: Context-Aware Dynamic Weighting of Hot Words: During ASR decoding, the aforementioned power industry professional hot word library is integrated, and a dynamic weighting algorithm is introduced. This algorithm dynamically calculates and injects a bias weight β based on the confidence level of the hot word in the current acoustic context and its global frequency of occurrence in historical meetings. Specifically, hot words that appear frequently in recent meetings and have high acoustic confidence are given a higher weight increase, with a weighting probability of P. boosted (w)=P base (w)·e β Context-aware dynamic weighting of hot words intelligently and discriminatively prioritizes important technical terms during the recognition process. The constructed weighted hot word library is integrated into the ASR decoding process. A dynamic weighting algorithm is introduced, which does not simply assign fixed weights to all hot words, but dynamically calculates a bias weight β based on two dimensions: the confidence level of the current acoustic context (the probability of the word being spoken in the current speech environment) and global frequency (the importance of the word in historical conferences). For words that have appeared frequently recently and have clear acoustic evidence (such as words related to hot topics in the current conference), a higher boost (P) is given. boosted (w)=P base (w)·e β ), to ensure that it is accurately identified.

[0159] The context-aware dynamic weighted algorithm for hot words in S5.2 further includes a hot word enhancement strategy based on an attention mechanism:

[0160] S5.2.1: Acoustic Context Encoding: The acoustic features of the current speech frame and its preceding and following context windows are input into a network based on a multi-head self-attention mechanism to encode and generate a context-aware acoustic representation vector; capturing the complete acoustic information of the current speech segment and its surrounding environment. The acoustic features (such as FBank and MFCC) of the current speech frame and its preceding and following windows are input into a network based on a multi-head self-attention mechanism. This network can dynamically weigh the importance of features at different time points and output a high-quality acoustic representation vector that integrates global contextual information.

[0161] S5.2.2: Hot Word-Context Relevance Calculation: Calculate the attention weight between the embedding representation of each word in the hot word library and the current acoustic context representation vector to generate a relevance score between the hot word and the current acoustic context; quantify the relevance of each hot word to the currently spoken content. Calculate the attention weight between the embedding representation of each word in the hot word library and the current acoustic context representation vector generated in S5.2.1. This weight, as a relevance score, reflects the likelihood of mentioning the hot word in the current context.

[0162] S5.2.3: Multi-dimensional Weight Fusion: The hot word-context relevance score, the global frequency statistics of the hot word, and the preset basic weights are fused in multiple dimensions. A learnable gating mechanism is used to generate the final dynamic bias weight β for the hot word. This process integrates various information to generate the final hot word bias weight. The relevance score calculated in S5.2.2, the global frequency statistics (historical importance) of the hot word, and its preset basic weights (derived from S2) are fused. A learnable gating mechanism (e.g., using the Sigmoid function to generate a gating value between 0 and 1) automatically determines the contribution ratio of each dimension, ultimately generating a more accurate and robust dynamic bias weight β.

[0163] S5.2.4: Temporal Awareness Weight Smoothing: A sliding window smoothing process is applied to the dynamic bias weight β in the time dimension to ensure the temporal continuity of hot word weighting and avoid instability in recognition results caused by drastic weight fluctuations; this ensures the stability and continuity of the hot word weighting effect. Since the speech signal is continuous, the contextual information of adjacent frames should be smooth. A sliding window average is applied to the dynamic bias weight β in the time dimension to avoid drastic changes in weight between different frames, thereby preventing discontinuous or flickering recognition results.

[0164] S5.2.5: Adversarial Interference Protection: An adversarial training mechanism is introduced during the hot word weighting process. By adding audio perturbations imperceptible to the human ear, the robustness of the model against maliciously constructed hot word interference attacks is enhanced. This improves system security and prevents malicious attacks. When training the hot word weighting model, an adversarial training mechanism is introduced. This involves intentionally generating subtle audio perturbations that are difficult for the human ear to detect and adding them to the training data, forcing the model to learn to ignore these perturbations and still make correct weight judgments. This improves the system's robustness against maliciously constructed interference attacks designed to trigger incorrect hot words.

[0165] The above steps construct a more refined, intelligent, and secure dynamic weighted system for hot words. It is no longer a simple weighting formula, but a miniature analysis and decision-making network: it understands the current environment by encoding context, evaluates term associations by calculating relevance, makes comprehensive decisions through multi-dimensional fusion, ensures output stability through temporal smoothing, and finally strengthens security through adversarial training. This series of steps collectively ensures the maximization and reliability of hot word optimization results.

[0166] S5.3: Hot Word Conflict Resolution: When similar-sounding words exist in the hot word database, a context-based semantic conflict resolution module is activated. This module utilizes the domain language model trained in step S5.1 to calculate the contextual probability of candidate word sequences and selects the word that best matches the current semantic context as the final recognition result. Hot word conflict resolution addresses the ambiguity between similar-sounding technical terms. When homophones or near-homophones exist in the hot word database (such as "busline" and "bus-line," "circuit breaker" and "circuit breaker"), simple probability boosting may lead to errors. In this case, the context-based semantic conflict resolution module is activated. It uses the domain language model trained in S5.1 to calculate the probability of each candidate word placed in the current sentence context and selects the word with the most fluent semantics and best fit for the power industry context as the final recognition result, thereby achieving accurate differentiation.

[0167] S5.4: Generate Speaker-Tagged Text: Align and fuse the identified text after hot word optimization and conflict resolution with the speaker identity tags and timestamps obtained in step S4 to generate a structured, speaker-tagged transcribed text sequence. Generate speaker-tagged text, fuse the recognition results and identity information to generate structured transcribed text. Align and fuse the clean identified text generated after hot word optimization and conflict resolution with the speaker identity tags and timestamp information output by the speaker separation and recognition module (S4). Finally, output a structured text sequence with speaker tags (e.g., [10:01:25] Zhang San: It is recommended to check whether the oil temperature of the C1 main transformer exceeds the standard.), providing directly usable and information-rich raw materials for subsequent meeting minutes generation.

[0168] The above steps construct a highly customized and intelligent speech recognition core for the power industry. It is no longer a general speech-to-text converter, but rather lays a linguistic foundation through domain-adaptive training, then accurately improves the recognition rate of key terms through dynamic hotword weighting, and is supplemented by a conflict resolution mechanism to handle deep ambiguities, ultimately outputting structured text rich in semantic and identity information. This is the core technological guarantee for achieving high-accuracy transcription.

[0169] S6: Utilizing a pre-trained natural language processing model in the power industry for multi-granular semantic understanding and hierarchical summary generation, automatically generates structured meeting minutes containing a full-text summary, chapter overview, and to-do list. This automatic meeting minutes generation achieves the automatic conversion from transcribed text to structured minutes. Using a pre-trained model in the power industry for semantic understanding, feature extraction, and hierarchical summarization, it automatically generates standardized minutes containing summaries, topics, and to-do lists.

[0170] Specifically, in this embodiment, the minutes generation step in S6 includes:

[0171] S6.1: Multi-granular semantic understanding: Input the transcribed text with speaker tags into a pre-trained natural language processing model in the power industry. This model is based on the Transformer architecture and adopts the following multi-task learning mechanism: a) Perform semantic error correction and standardization of power industry terminology on the input text, and correct homonymous professional terminology errors based on the power knowledge graph; b) Identify and label meeting elements in the text, including topic paragraphs, decision-making viewpoints, responsible parties, and time nodes; c) Extract technical parameters and indicators unique to the power industry and standardize their format.

[0172] S6.2: Hierarchical Summary Generation: Using a sequence-to-sequence model based on pointer generation network, hierarchical meeting summaries are generated: a) First, a full summary is generated to extract the core topics and main conclusions of the meeting; b) Then, a chapter overview is generated, and bullet point-style summaries of key points are generated for each topic paragraph; c) Finally, action items are extracted, to identify the to-do items in the meeting, and the responsible parties, time requirements, and priorities are automatically associated.

[0173] S6.3: Structured Assembly of Meeting Minutes: Based on the standard format for meeting minutes in the power industry, the generated summary results are assembled into a structured document, including: a) Basic Meeting Information Area: Automatically fills in the meeting time, participants, and moderator; b) Topic Discussion Area: Organizes the discussion points and decision results of each topic in chronological order; c) Action Item List Area: Lists all to-do items in tabular form, including task content, responsible person, deadline, and status tracking column.

[0174] S6.4: Multimodal Minutes Output: Supports the generation of structured minutes in multiple formats: a) Editable document formats (such as Word, PDF); b) Machine-readable data formats (JSON, XML) for automatic integration with enterprise project management systems; c) Visual Kanban format, displaying the timeline and progress of action items in Gantt chart form.

[0175] S7: Provides an editing interface and encrypts and stores the final minutes, supporting local and cloud-based multimodal output formats. Through minutes editing and multimodal output, it offers post-processing and secure output of minutes. It supports content editing, encrypted storage, and output in multiple formats (such as Word, PDF, and JSON) to meet viewing, editing, and system integration needs.

[0176] Steps S1-S7 construct an end-to-end intelligent meeting processing flow, from audio input to structured minutes output, and successively solve key problems in the power industry meeting scenario such as voiceprint recognition, professional terminology recognition, multi-person speech separation, domain-adaptive transcription and intelligent summary generation, which significantly improves the accuracy, efficiency and automation of meeting records.

[0177] Example 2

[0178] like Figure 2 As shown, this application provides an architecture diagram of an intelligent conference speech transcription and minutes generation system for the power industry, which uses voiceprint recognition and hot word optimization. It is applied to the intelligent conference speech transcription and minutes generation system for the power industry as described in Embodiment 1, including a voiceprint pre-registration module 11, a hot word library construction module 12, a speech processing module 13, a speaker separation and recognition module 14, a speech recognition module 15, a minutes generation module 16, and an output storage module 17.

[0179] The voiceprint pre-registration module 11 is used to pre-collect voice samples from meeting participants, extract voiceprint feature vectors based on deep neural networks, generate standard voiceprint templates through average pooling, and establish an encrypted voiceprint database.

[0180] The hot term library construction module 12 is used to collect professional terms from multi-source data in the power industry through text mining, dynamically allocate word weights based on word frequency, inverse document frequency and timeliness factors, and establish a dynamic update mechanism for the hot term library.

[0181] The speech processing module 13 is used to receive multi-format audio input, perform unified decoding and adaptive preprocessing, and use a speech activity detection model based on a one-dimensional convolutional neural network for endpoint detection and silence removal.

[0182] The speaker separation and recognition module 14 is used to extract the high-dimensional acoustic feature vector of the preprocessed speech segment, perform unsupervised clustering using a deep embedding clustering algorithm, achieve speaker separation by optimizing the KL divergence loss between the soft assignment probability and the target distribution, and match and recognize the clustering results with the voiceprint database.

[0183] The speech recognition module 15 is used to adopt a domain-adaptive speech recognition model based on the Transformer structure. During the decoding process, a weighted hot word library is integrated. Through a context-aware dynamic weighting algorithm and a hot word conflict resolution mechanism, transcribed text with speaker tags is generated.

[0184] The minutes generation module 16 is used to perform multi-granular semantic understanding and hierarchical summary generation using a pre-trained natural language processing model in the power field, and automatically generate structured meeting minutes that include a full text summary, chapter overview and to-do list.

[0185] The output storage module 17 provides an editing interface and encrypts the storage of the final minutes, supporting local and cloud-based multimodal output formats.

[0186] Figure 3 This is an electronic device provided in one embodiment of this application. For example... Figure 3 As shown, the electronic device includes at least the following components: processor 101 and memory 100, communication interface 103, and bus 102.

[0187] In this embodiment of the application, memory 100 is used to store executable instructions of processor 101, which, when configured to execute instructions, implements the method as described in the first aspect.

[0188] In embodiments of this application, a computer-readable storage medium includes instructions that instruct a device to perform the method as described in the first aspect. For example, the instructions instruct the device to perform... Figure 1 The method is shown in the process steps.

[0189] In one embodiment of this application, the program operating in the electronic device may be a program that controls a central processing unit (CPU) or similar device to achieve the functions of the above-described embodiments of the present invention (a program that enables the computer to function). Information processed by these systems is then temporarily stored in random access memory (RAM) during processing, and subsequently stored in various ROMs such as read-only memory (FlashROM) and hard disk drives (HDDs), and read, corrected, and written by the CPU as needed.

[0190] It should be noted that a portion of the electronic device described above can also be implemented using a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and the program recorded on the recording medium can be read into the computer and executed.

[0191] It should be noted that the computer mentioned here refers to a computer built into an electronic device, employing hardware including an operating system and peripheral devices. Furthermore, computer-readable recording media refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage systems such as hard drives built into the computer.

[0192] Furthermore, computer-readable recording media can include: media that dynamically stores programs for short periods of time, such as communication lines used when transmitting programs via networks like the Internet or communication lines like telephone lines; and media that store programs for fixed periods of time, such as volatile memory inside a computer that serves as a server or client in this case. In addition, the aforementioned program can be a program used to implement the above-mentioned functions, or it can be a program that can implement the above-mentioned functions by combining them with programs already recorded in the computer.

[0193] Furthermore, the electronic device in the above embodiments can also be implemented as an assembly (system group) composed of multiple systems. Each system constituting the system group can possess some or all of the functions or functional blocks of the electronic device in the above embodiments. As a system group, it is sufficient to have all the functions or functional blocks of the electronic device.

[0194] Those skilled in the art should recognize that the above embodiments are only used to illustrate this application and are not intended to limit this application. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of this application fall within the scope of protection claimed in this application.

Claims

1. A method for intelligent conference speech transcription and minutes generation based on voiceprint recognition and hot word optimization in the power industry, characterized in that, Includes the following steps: S1: Collect voice samples from meeting participants in advance, extract voiceprint feature vectors based on deep neural networks, generate standard voiceprint templates through average pooling, and establish an encrypted voiceprint database. S2: Collect professional terms from multi-source data in the power industry through text mining, dynamically allocate word weights based on word frequency, inverse document frequency and timeliness factors, and establish a dynamic update mechanism for the hot word library; S3: Receives multi-format audio input, performs unified decoding and adaptive preprocessing, and uses a speech activity detection model based on a one-dimensional convolutional neural network for endpoint detection and silence removal; S4: Extract the high-dimensional acoustic feature vector of the preprocessed speech segment, perform unsupervised clustering using a deep embedding clustering algorithm, achieve speaker separation by optimizing the KL divergence loss between the soft assignment probability and the target distribution, and match and identify the clustering results with the voiceprint database. S4 includes: Extract high-dimensional acoustic feature vectors from the preprocessed speech segments; Unsupervised clustering is performed on the feature vectors, and the soft assignment probability is optimized. With target distribution Speaker separation is achieved using the KL divergence loss between the two, where: a) Calculate the eigenvectors using the Student's t-distribution. With cluster center Soft allocation probability between : ， in, Indicates the first High-dimensional acoustic feature vectors of each speech segment Indicates the first Cluster centers, This indicates a traversal of all cluster centers; b) Constructing a target distribution by reinforcing high-confidence allocation : ， in , indicating the first The soft assignment frequency of each cluster center; c) Synchronously update cluster centers by minimizing the KL divergence loss function L. and parameters of deep neural networks : ， in, Indicates the target distribution. Indicates a soft-assigned distribution. This represents the KL divergence loss value. The clustering results are matched with the standard voiceprint templates in the voiceprint database. If the maximum similarity exceeds the preset threshold, the cluster is mapped to the corresponding speaker identity; otherwise, it is marked as an unknown speaker. S5: It adopts a domain-adaptive speech recognition model based on the Transformer structure, integrates a weighted hot word library during the decoding process, and generates transcribed text with speaker tags through a context-aware dynamic weighting algorithm and a hot word conflict resolution mechanism. S6: Utilize a pre-trained natural language processing model in the power industry for multi-granular semantic understanding and hierarchical summary generation, automatically generating structured meeting minutes that include full-text summaries, chapter overviews, and to-do lists; S7: Provides an editing interface and encrypts the storage of the final minutes, supporting local and cloud-based multimodal output formats.

2. The intelligent conference speech transcription and minutes generation method based on voiceprint recognition and hot word optimization for the power industry, as described in claim 1, is characterized in that... The extraction of voiceprint features and the establishment of a voiceprint database in S1 include: Several segments of registered speech from the target speaker are collected through a dedicated registration channel. The registered speech is text with known content that covers different phonemes and high-frequency words in the power field. Each registered speech segment is preprocessed, including noise reduction, silence removal, and volume normalization. Extracting voiceprint feature vectors based on deep neural networks from preprocessed speech; Average pooling is performed on all voiceprint feature vectors of the same speaker to generate the standard voiceprint template of that speaker. The standard voiceprint template is associated with the speaker's identity information and encrypted and stored in the voiceprint database; A voiceprint verification interface is provided to re-verify registered voiceprint templates to ensure their validity and accuracy.

3. The intelligent conference speech transcription and minutes generation method based on voiceprint recognition and hot word optimization for the power industry, as described in claim 1, is characterized in that... The construction of the hot word library in S2 includes: We collected professional terms and organization names from multi-source data in the power industry to form a basic hot word set; Each word in the basic hot word set is assigned an initial weight, and the weight is assigned based on the following factors: a) the word frequency and inverse document frequency in the source document; b) the timeliness of the word in recent industry policies; c) the importance level of the word in a specific conference context. The weighted hot word set is input into the hot word optimization module of the speech recognition model. During the ASR decoding process, the probability of words in the hot word library is increased, and the increase is dynamically controlled by the weight. Based on the new words discovered in the conference transcription results and user feedback, the hot word database is incrementally learned and its weights are adjusted to achieve dynamic updates.

4. The intelligent conference speech transcription and minutes generation method based on voiceprint recognition and hot word optimization for the power industry, as described in claim 1, is characterized in that... The preprocessing and speech activity detection in S3 include: It receives audio input in multiple formats and decodes them uniformly into pulse code modulation waveform data with a preset sampling rate and bit depth; The waveform data is subjected to adaptive preprocessing, including: a) using a noise suppression algorithm based on spectral subtraction to suppress environmental steady-state noise and sudden interference noise; b) using an automatic gain control algorithm to normalize the waveform amplitude to the target dynamic range; The preprocessed audio data is divided into frames and input into a speech activity detection model based on a one-dimensional convolutional neural network. The speech / non-speech probability is calculated frame by frame to obtain a probability sequence. The start and end points of the speech segment are determined based on the probability sequence, and silent segments exceeding a preset duration are cut off to output continuous speech segments.

5. The intelligent conference speech transcription and minutes generation method based on voiceprint recognition and hot word optimization for the power industry, as described in claim 4, is characterized in that... The unsupervised clustering process employs an adaptive optimization strategy for the number of clusters, including: Automatically estimate the potential number K of speaker roles in conference audio; In the process of deep embedding clustering iterative optimization, the stability of clustering is evaluated by calculating the normalized mutual information of the clustering assignment results; For each cluster formed after stable clustering, calculate the average silhouette coefficient of its internal members, and perform sub-cluster splitting operation on abnormal clusters whose silhouette coefficient is lower than a preset threshold. Calculate the cosine similarity between different cluster centers, and automatically merge cluster pairs that have similarity exceeding the merging threshold and have similar acoustic features. By iteratively performing sub-cluster splitting and cluster pair merging operations, the number of clusters is dynamically and adaptively adjusted to obtain the optimal clustering result that matches the actual number of speakers.

6. The intelligent conference speech transcription and minutes generation method based on voiceprint recognition and hot word optimization for the power industry, as described in claim 1, is characterized in that... S5 includes: Constructing a domain-adaptive speech recognition model: An end-to-end speech recognition model based on the Transformer architecture is adopted, and its language model components are pre-trained for domain adaptability using a power industry text corpus; During ASR decoding, a weighted hot word library is integrated and a dynamic weighting algorithm is adopted. The bias weight β is dynamically calculated based on the confidence of the hot word in the current acoustic context and the global frequency, thereby increasing the probability of the hot word. When there are similar-sounding hot words, the context-based semantic conflict resolution module is activated, and the domain language model is used to select the word that best matches the current semantic context. The optimized recognized text is aligned and fused with speaker identification tags and timestamps to generate a structured transcribed text sequence with speaker tags.

7. The intelligent conference speech transcription and minutes generation method based on voiceprint recognition and hot word optimization for the power industry, as described in claim 6, is characterized in that... The dynamic weighting algorithm further includes a hot word enhancement strategy based on an attention mechanism: The current speech frame and its context acoustic features are input into a network based on a multi-head self-attention mechanism to generate context-aware acoustic representation vectors. Calculate the attention weights between the word embeddings in the hot word library and the current acoustic context representation vector, and generate a hot word-context relevance score; The relevance score, global frequency statistics of hot words, and preset basic weights are fused in multiple dimensions, and the final dynamic bias weight β of hot words is generated through a gating mechanism. The dynamic bias weight β is smoothed using a sliding window over the time dimension; An adversarial training mechanism is introduced during the hot word weighting process to enhance the model's robustness against malicious hot word interference attacks.

8. The intelligent conference speech transcription and minutes generation method based on voiceprint recognition and hot word optimization for the power industry, as described in claim 1, is characterized in that... The minute generation in S6 includes: Transcribed text with speaker tags is input into a pre-trained natural language processing model in the power industry for multi-granular semantic understanding, including: semantic error correction and standardization of power industry terms, identification and annotation of meeting elements, and extraction and standardization of technical parameters and indicators. A sequence-to-sequence model based on pointer generation network is used to generate hierarchical conference summaries, including: full text summary, chapter overview, and action item extraction; According to the standard format of power industry meeting minutes, the summary generation results are assembled into a structured document, including a basic meeting information area, a topic discussion area, and an action item list area; It supports generating structured minutes output in various formats, including editable document format, machine-readable data format, and visual dashboard format.

9. A smart conference speech transcription and minutes generation system for the power industry, employing voiceprint recognition and hot word optimization, applied to the smart conference speech transcription and minutes generation method for the power industry as described in any one of claims 1 to 8, characterized in that... The system includes: The voiceprint pre-registration module is used to collect voice samples from meeting participants in advance, extract voiceprint feature vectors based on deep neural networks, generate standard voiceprint templates through average pooling, and establish an encrypted voiceprint database. The hot term library construction module is used to collect professional terms from multi-source data in the power industry through text mining, dynamically allocate word weights based on word frequency, inverse document frequency and timeliness factors, and establish a dynamic update mechanism for the hot term library. The speech processing module is used to receive multi-format audio input, perform unified decoding and adaptive preprocessing, and adopt a speech activity detection model based on a one-dimensional convolutional neural network for endpoint detection and silence removal. The speaker separation and recognition module is used to extract high-dimensional acoustic feature vectors of preprocessed speech segments, perform unsupervised clustering using a deep embedding clustering algorithm, achieve speaker separation by optimizing the KL divergence loss between the soft assignment probability and the target distribution, and match and recognize the clustering results with the voiceprint database. The speech recognition module employs a domain-adaptive speech recognition model based on the Transformer architecture. During the decoding process, it integrates a weighted hot word library and generates transcribed text with speaker tags through a context-aware dynamic weighting algorithm and a hot word conflict resolution mechanism. The minutes generation module is used to perform multi-granular semantic understanding and hierarchical summary generation using a pre-trained natural language processing model in the power field, and automatically generate structured meeting minutes that include a full text summary, chapter overview and to-do list; The output storage module provides an editing interface and encrypts the storage of the final minutes, supporting local and cloud-based multimodal output formats.

Images