A conference speech recognition and speaker matching method, device and computer equipment

Abstract

This application relates to the field of speech recognition technology, and provides a method, apparatus, and computer device for conference speech recognition and speaker matching. The method includes receiving a project code carried in a conference creation request or audio submission request; verifying whether the target project is available based on the project code; if the target project is available, binding the project context of the target project; constructing enhanced hot words for the project based on the project code indexing current project hot words; constructing a candidate set of project member voiceprints by matching the project code with the current project member voiceprints; and submitting an automatic speech recognition task to the algorithm side, the automatic speech recognition task including the target conference audio, enhanced project hot words, the candidate set of project member voiceprints, and task parameters. This method improves the speaker identity matching rate and speech recognition accuracy across projects.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a method, apparatus and computer equipment for conference speech recognition and speaker matching. Background Technology

[0002] With the widespread adoption of remote work and digital collaboration, audio and video conferencing has become a core communication scenario for businesses. Automatically converting meeting recordings into transcripts with speaker annotations—a process known as meeting transcription—can significantly reduce manual processing costs and improve information flow efficiency.

[0003] Currently, meeting transcription systems typically use general speech recognition models to submit meeting audio to ASR (Automatic Speech Recognition) services, converting it into text, extracting acoustic features from each speech segment, and matching the speaker's identity against a fully registered pool of available voiceprints. However, in real-world cross-project meeting scenarios, when the meeting involves content from a specific vertical domain, the model often produces recognition biases such as substitution of similar-sounding characters and incorrect segmentation of proper nouns. Furthermore, the same person may participate in multiple projects in different roles, and the participants from different projects often overlap significantly, affecting the readability of the transcribed text. This can even further impact the downstream speaker matching process, as the decreased text quality weakens the auxiliary discrimination signal based on language content, leading to higher matching errors.

[0004] Inventors in this field have discovered that existing solutions for speech recognition in conferences suffer from high mismatch rates of speaker identities across projects and low speech recognition accuracy. Summary of the Invention

[0005] This application provides a method, apparatus, and computer device for conference speech recognition and speaker matching, which can achieve higher-precision speech recognition and more accurate speaker matching between different projects.

[0006] In a first aspect, this application provides a method for conference speech recognition and speaker matching, comprising the following steps:

[0007] Receive the project code carried in the meeting creation request or audio submission request;

[0008] Based on the project code, verify whether the target project is in an available state;

[0009] If the target project is in an available state, then the project context of the target project is bound, wherein the project context is an external structured constraint predetermined by the target project.

[0010] Based on the project coding index of current project hot words, construct enhanced project hot words;

[0011] Based on the project code, match the voiceprints of the current project members to construct a candidate set of project member voiceprints;

[0012] Submit an automatic speech recognition task to the algorithm side. The automatic speech recognition task includes the target meeting audio, the enhanced hot words of the project, the candidate set of voiceprints of the project members, and task parameters.

[0013] The algorithm performs the following steps:

[0014] Perform speech activity detection on the target conference audio to obtain a set of speech segments;

[0015] Speaker segmentation is performed on the target conference audio to obtain a clustered speaker set;

[0016] Based on the enhanced hot words of the project, automatic speech recognition is performed on the set of speech segments to output subtitle segments;

[0017] Extract the clustered speaker voiceprint embeddings from the speech segment set, calculate the similarity between any clustered speaker voiceprint embedding and each candidate voiceprint embedding in the project member voiceprint candidate set, traverse the similarity between the clustered speaker voiceprint embeddings of the clustered speaker set and each corresponding candidate voiceprint embedding, and construct a similarity matrix.

[0018] A one-to-one constraint is established, whereby each cluster speaker is assigned at most one candidate voiceprint, and each candidate voiceprint is assigned at most one cluster speaker. Combined with a preset global optimal algorithm, the cluster speaker identity corresponding to the cluster speaker set is determined from the similarity matrix.

[0019] Optionally, the step of constructing enhanced hot words for the project based on the project coding index of current project hot words includes:

[0020] Based on the project code, obtain hot word records that are associated with the project and are in an available state;

[0021] The system detects whether the line tail weight corresponding to the hot word record meets the preset compliance conditions, wherein the line tail weight is the product of the weight corresponding to the hot word record and the line tail position attenuation coefficient;

[0022] If the line ending weight corresponding to the hot word record meets the preset compliance conditions, then the corresponding hot word record is parsed to obtain the hot word text;

[0023] Save the main text of the hot words and their corresponding line ending weights to form the hot words for the current project;

[0024] System hot words are obtained from a globally maintained system hot word data source. These system hot words are applicable to general product names, high-frequency words within an organization, and industry terms for different projects.

[0025] By concatenating the current project hot words and the system hot words, the enhanced project hot words are obtained.

[0026] Optionally, the step of constructing a candidate set of project member voiceprints based on the project code and matching the voiceprints of current project members includes:

[0027] Detect the existence of a collection of voiceprints of project members;

[0028] If the set of project member voiceprints exists, match the set of project member voiceprints as the current project member voiceprint, and construct a candidate set of project member voiceprints.

[0029] If the project member voiceprint set does not exist, the project member voiceprint candidate set is empty, and the algorithm performs speech activity detection and speaker separation.

[0030] Optionally, the task parameters include the callback address after the algorithm service completes processing. After the step of submitting the automatic speech recognition task to the algorithm side, the method further includes:

[0031] Obtain the automatic speech recognition results from the algorithm side callback and output meeting minutes containing subtitles and speaker identification information.

[0032] Optionally, the step of determining the identity of the clustered speaker corresponding to the clustered speaker set from the similarity matrix, in conjunction with a preset global optimal algorithm, includes:

[0033] From the similarity matrix, select the candidate voiceprint embedding with the highest similarity among the similarity combinations of the clustered speaker voiceprint embedding and the corresponding candidate voiceprint embeddings, which satisfy the one-to-one constraint.

[0034] The similarity of the candidate voiceprint embeddings is compared with a preset voiceprint matching threshold;

[0035] When the similarity of the candidate voiceprint embedding is greater than or equal to the voiceprint matching threshold, the candidate voiceprint embedding is determined as the corresponding clustered speaker identity.

[0036] When the similarity of the candidate voiceprint embedding is less than the voiceprint matching threshold, the speaker in the cluster is retained as an unknown or temporary speaker, and the candidate voiceprint embedding with the highest similarity remains unmatched.

[0037] Add virtual candidate nodes with unknown speaker identities to the speakers in this cluster.

[0038] Optionally, the following steps are also included:

[0039] Calculate the similarity difference between the candidate voiceprint embedding with the highest similarity and the candidate voiceprint embedding with the second highest similarity among the similarity combinations of the clustered speaker voiceprint embedding and each corresponding candidate voiceprint embedding.

[0040] The similarity difference is compared with a preset confidence threshold.

[0041] When the similarity difference is less than the confidence threshold, the candidate voiceprint embedding with the highest similarity is marked as the speaker identity of the low-confidence cluster and a verification is prompted.

[0042] Optionally, the following steps are also included:

[0043] Obtain the set of speech segments and the set of clustered speakers;

[0044] For each element in the speech segment set, calculate its overlap duration with each element in the clustered speaker set, and retain only one clustered speaker segment for the overlap replacement interval;

[0045] Calculate the percentage of speakers in the de-overlapping clustered speaker set relative to the corresponding element in the speech segment set, and compare it with a preset sentence speaker percentage threshold.

[0046] If the speaker percentage is greater than or equal to the speaker percentage threshold of the sentence, then the element in the speech segment set belongs to the corresponding element in the clustered speaker set;

[0047] If the speaker percentage is less than the sentence speaker percentage threshold and the sentence is long, the long sentence is split into clauses, and the speaker percentage for each clause is recalculated and compared with the sentence speaker percentage threshold.

[0048] Optionally, before the step of obtaining the automatic speech recognition result from the algorithm-side callback and outputting meeting minutes containing subtitles and speaker identification information, the method further includes:

[0049] Query meeting audio recordings;

[0050] If the meeting audio recording is canceled, deleted, or in a final state, discard the callback result and record the callback event;

[0051] Query callback status;

[0052] If the callback status is successful, the voiceprints of the identified project members will be mapped to speaker identity information, sentence-level subtitles with timestamps will be saved, and the minutes will be generated.

[0053] If the callback status is failure, save the failure reason and algorithm warning, and do not trigger the generation of minutes.

[0054] Secondly, this application provides a conference speech recognition and speaker matching device, which has the function of implementing the aforementioned conference speech recognition and speaker matching method. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned function, and the modules can be software and / or hardware.

[0055] In one possible design, the conference speech recognition and speaker matching device includes:

[0056] The project coding module is used to receive the project code carried in the meeting creation request or audio submission request;

[0057] The project verification module is used to verify whether the target project is in an available state based on the project code.

[0058] The project context module is used to bind the project context of the target project when the target project is in an available state. The project context is determined by the external structured constraints of the target project in advance.

[0059] The project-enhanced hot word module is used to construct project-enhanced hot words based on the project code index of the current project hot words;

[0060] The project member voiceprint module is used to construct a candidate set of project member voiceprints by matching the current project member's voiceprint based on the project code.

[0061] The task module is used to submit automatic speech recognition tasks to the algorithm side. The automatic speech recognition task includes target meeting audio, project-enhanced hot words, a candidate set of project member voiceprints, and task parameters. The algorithm side performs the following: performing speech activity detection on the target meeting audio to obtain a set of speech segments; performing speaker separation on the target meeting audio to obtain a clustered speaker set; performing automatic speech recognition on the speech segment set based on the project-enhanced hot words to output subtitle segments; extracting clustered speaker voiceprint embeddings from the speech segment set, calculating the similarity between any clustered speaker voiceprint embedding and each candidate voiceprint embedding in the candidate set of project member voiceprints, traversing the similarity between the clustered speaker voiceprint embeddings and the corresponding candidate voiceprint embeddings in the clustered speaker set to construct a similarity matrix; establishing a one-to-one constraint, each clustered speaker is assigned at most one candidate voiceprint, and each candidate voiceprint is assigned at most one clustered speaker; and combining a preset global optimal algorithm, determining the clustered speaker identity corresponding to the clustered speaker set from the similarity matrix.

[0062] Thirdly, this application provides a computer device including at least one connected processor, memory, and transceiver, wherein the memory is used to store program code, and the processor is used to invoke the program code in the memory to execute the methods described in the above aspects.

[0063] Compared to existing technologies, the solution provided in this application involves receiving a project code carried in a meeting creation request or audio submission request; verifying whether the target project is available based on the project code; if the target project is available, binding the project context of the target project; obtaining the external structured constraints of the project context from the project management data based on the business project bound to the meeting before the ASR task starts; then indexing the current project hot words based on the project code to construct enhanced project hot words; matching the current project member voiceprints based on the project code to construct a candidate set of project member voiceprints; and submitting the automatic speech recognition task to the algorithm side. This achieves the acquisition of currently valid project hot words and project member voiceprints using the project code, simultaneously converging two candidate spaces using the same project context, and performing global analysis based on the current project hot words. The system's hot words are optimized, and enhanced hot words for projects are constructed for text correction. Furthermore, a project member voiceprint matching query is performed based on the current project members, constructing a project member voiceprint candidate set. This converges all registered voiceprint candidates into project member voiceprint candidates, narrowing the target voiceprint candidate range. This allows the system to provide ASR with proprietary terminology constraints related to the current project before speech recognition begins, and excludes voiceprint candidates unrelated to the current project during identity matching. This improves the speaker identity matching rate and speech recognition accuracy across projects, achieving higher-precision speech recognition and more accurate speaker matching between different projects. Moreover, since project hot word constraints and project member voiceprint constraints originate from the same project boundary, noise interference caused by inconsistencies between the business scope targeted by the hot words and the personnel scope targeted by the voiceprint candidates can be avoided, further improving recognition accuracy. Attached Figure Description

[0064] Figure 1 This is a flowchart illustrating the main process of a conference speech recognition and speaker matching method according to an embodiment of this application.

[0065] Figure 2 This is a schematic diagram of the structure of a conference speech recognition and speaker matching device according to an embodiment of this application;

[0066] Figure 3 This is a schematic diagram of the structure of a computer device according to an embodiment of this application;

[0067] Figure 4 This is a schematic diagram of the structure of a computer storage medium according to an embodiment of this application;

[0068] Figure 5This is a schematic diagram of the structure of a computer program product according to an embodiment of this application. Detailed Implementation

[0069] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The division of modules in the embodiments of this application is merely a logical division; in actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not performed. Additionally, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interface, and the indirect coupling or communication connection between modules may be electrical or other similar forms, none of which are limited in the embodiments of this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed among multiple circuit modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of this application.

[0070] In existing solutions, enterprise meeting speech recognition faces the following interrelated problems:

[0071] First, meeting content is highly dependent on the project context. Different projects may use abbreviations, customer names, module names, and technical terms with the same or similar pronunciations. Traditional ASR is based only on a general acoustic model and global hot word recognition. The global hot word table cannot reflect the differences in customer names, interface names, and business terms between different projects. It is easy to identify project-specific words as common words, homophones, or terms from other projects. The recognition of specific terms is unstable, which leads to the distortion of key information in subtitles and minutes, resulting in low language recognition accuracy.

[0072] Secondly, if voiceprint matching is applied to all registered users, the probability of mismatch increases with the number of users and voiceprints. An individual within a company may participate in multiple projects, but the actual attendance at a particular meeting is usually significantly smaller than the entire company. Since the boundary for distinguishing different speakers in a general voiceprint model is globally fixed, if the algorithm uses all active voiceprints in every meeting, irrelevant personnel from different projects will also enter the candidate set, causing their voiceprints to compete for similarity. When the number of candidate voiceprints is large and some speakers have high acoustic feature similarity, the system is highly prone to misattributing speech segments to other individuals with the same name or similar acoustics in the voiceprint database. This results in a high risk of mismatches across all voiceprint candidates, leading to incorrect and chaotic speaker labeling. Furthermore, general voiceprint matching lacks constraints on the current meeting session context. For example, it fails to utilize the actual attendees of the current meeting to narrow the matching space, causing historical voiceprints from different projects to interfere, further amplifying the probability of mismatches.

[0073] Third, hot words and voiceprints lack a unified context, and the system cannot simultaneously constrain the candidate range of hot words and voiceprints;

[0074] Fourth, speaker matching is prone to getting stuck in local optima. Selecting the highest similarity can easily lead to multiple clustered voiceprints competing for the same candidate user's voiceprint, or the overall matching result being suboptimal.

[0075] Fifth, project anomalies affect the main process. When project member services are unavailable, hot keyword searches fail, or project members have no voiceprints, the meeting processing flow will be interrupted, affecting the output of recognition results.

[0076] Sixth, the subtitle results lack a stable structured database. The ASR, voiceprint, and speaker separation results need to be uniformly mapped to subtitle segments, speakers in the meeting, and participants; otherwise, it will affect the generation of subsequent minutes.

[0077] Therefore, referring to Figure 1 This application provides a method for conference speech recognition and speaker matching, including the following steps:

[0078] Receive the project code carried in the meeting creation request or audio submission request;

[0079] Based on the project code, verify whether the target project is in an available state;

[0080] If the target project is in an available state, then the project context of the target project is bound, wherein the project context is an external structured constraint predetermined by the target project.

[0081] Based on the project coding index of current project hot words, construct enhanced project hot words;

[0082] Based on the project code, match the voiceprints of the current project members to construct a candidate set of project member voiceprints;

[0083] Submit an automatic speech recognition task to the algorithm side. The automatic speech recognition task includes the target meeting audio, the enhanced hot words of the project, the candidate set of voiceprints of the project members, and task parameters. The task parameters include the callback address after the algorithm service completes the processing.

[0084] Obtain the automatic speech recognition results from the algorithm side callback and output meeting minutes containing subtitles and speaker identification information.

[0085] During implementation, users upload or record meeting audio; the backend submits the meeting audio to the ASR service; the speech recognition module performs speech activity detection, speech recognition, punctuation restoration, and sentence-level timestamp generation; the voiceprint or speaker separation module extracts, clusters, and matches the identities of the audio embeddings; the backend receives the ASR callback and stores the segments as meeting subtitles; the subsequent minutes generation system uses the subtitle text to generate summaries, decisions, and to-do lists.

[0086] Specifically, during the meeting creation, audio upload, or audio processing task submission stages, the meeting backend receives optional project codes and saves a stable snapshot of the project code and project name during the meeting creation stage. The project code is used for subsequent hot word enhancement, voiceprint candidate convergence, meeting processing task tracking, and result auditing, but it is not directly equivalent to meeting content viewing permissions.

[0087] When the meeting audio enters the processing flow, the backend constructs a set of enhanced hot words for the project and a set of candidate voiceprints for project members based on the project context bound to the meeting. Subsequently, the backend submits the target meeting audio, hot word strings, candidate voiceprint sets, voiceprint matching threshold, preset number of speakers, and webhook callback address to the algorithm service.

[0088] The algorithm performs the following steps:

[0089] Perform speech activity detection on the target conference audio to obtain a set of speech segments;

[0090] Speaker segmentation is performed on the target conference audio to obtain a clustered speaker set;

[0091] Based on the enhanced hot words of the project, automatic speech recognition is performed on the set of speech segments to output subtitle segments;

[0092] Extract the clustered speaker voiceprint embeddings from the speech segment set, calculate the similarity between any clustered speaker voiceprint embedding and each candidate voiceprint embedding in the project member voiceprint candidate set, traverse the similarity between the clustered speaker voiceprint embeddings of the clustered speaker set and each corresponding candidate voiceprint embedding, and construct a similarity matrix.

[0093] A one-to-one constraint is established, whereby each cluster speaker is assigned at most one candidate voiceprint, and each candidate voiceprint is assigned at most one cluster speaker. Combined with a preset global optimal algorithm, the cluster speaker identity corresponding to the cluster speaker set is determined from the similarity matrix.

[0094] After receiving the task, the algorithm service performs voice activity detection, automatic speech recognition, speaker segmentation, speaker embedding vector extraction, and candidate voiceprint matching on the conference audio. The algorithm service outputs a subtitle segment with timestamps, text, clustered speaker identifiers, and candidate voiceprint identifiers. Upon receiving the callback, the backend maps the voiceprint_id from the algorithm side to the speaker_id within the conference, and further synchronizes the identified participants based on the voiceprint_id. Finally, it saves the original subtitles, displays the subtitles and speaker information, and triggers the subsequent meeting minutes generation process.

[0095] Through the above technical solution, without changing the main interface form of ASR and voiceprint recognition, a controlled processing link is established that runs through meeting creation, project context snapshot, hot word enhancement, voiceprint candidate constraint, algorithm task submission, algorithm result callback, and speaker database entry within the meeting. Before submitting ASR, the project context is obtained. Using the project to which the meeting belongs, project members, and project hot words, the hot words and candidate voiceprints of the meeting speech recognition are simultaneously constrained by context, thereby improving the recognition quality of meeting proper terminology, reducing the cross-project voiceprint mismatch rate, and the speech recognition, speaker separation, and identity matching are performed separately, which is conducive to the stable and structured database entry of text recognition results.

[0096] The following is a detailed explanation.

[0097] The user inputs a meeting creation request or audio submission request with a project code: Req={meeting_id,user_id,audio_id,project_code,project_name,meeting_title,created_at}, where: meeting_id represents the meeting identifier; user_id represents the current user; audio_id represents the meeting audio identifier; project_code represents the optional project code; project_name represents the project name; meeting_title represents the meeting title; and created_at represents the creation time.

[0098] The meeting backend first verifies whether the project is available based on preset conditions, and verifies whether the current user has the membership relationship or project selection permission for the project.

[0099] After successful verification, the system saves a snapshot of the project context: ProjectSnapshot={project_code,project_name_snapshot,bind_time}, where project_name_snapshot=project_name atbind_time, meaning the project context snapshot is a copy of the project name at the time the project was bound. Even if the project name changes subsequently, it will not affect the project context used at the time of the meeting.

[0100] The project context is a pre-determined external structured constraint of the business project bound to the meeting. The project context comes from project data outside the meeting, including project identifiers, project member sets, and project terminology data, which can be determined by project_code before automatic speech recognition.

[0101] In this embodiment, before the ASR task is submitted, the project_code stored in the same meeting record is used as a unified association key. The active hot words of the current project are queried using the project code and the enhanced hot word constraints of the project are constructed. The active members of the current project are queried using the same project code, and the project member voiceprint candidates are selected from the active voiceprint records accordingly. The voiceprints of the project members are then submitted to the same speech processing task along with the meeting audio.

[0102] The output is the meeting minutes bound to the project context: MeetingContext={meeting_id,project_code,project_name_snapshot,bind_time}.

[0103] Project member relationships are only used for intelligent processing such as project selection, project hot keywords, voiceprint candidates, and project context enhancement, and do not automatically grant access to view meeting content. Access to view meeting content is still determined by independent permission mechanisms such as meeting creator, participants, sharing relationships, and meeting privacy rules.

[0104] In one embodiment, to improve security, the meeting creation request or the audio submission request also carries the current operating user ID;

[0105] Based on the project code, verify whether the current user ID has the user permissions (project selection permission) for the target project.

[0106] If the target project is available and the current user ID has the user permissions for the target project, bind the project context of the target project.

[0107] In one embodiment, if no project is bound, the meeting can still be created normally, but subsequent project hot word enhancement and voiceprint candidate convergence will enter a no-project mode to reduce the impact on the main link when a project is abnormal.

[0108] By simultaneously converging the text recognition candidate space and the speaker identity candidate space before recognition: project hot words can reduce recognition errors of project names, customer names, version numbers and business terms; project member voiceprint candidate constraints can exclude registered voiceprints that are not related to the current project, reducing mismatches caused by irrelevant voiceprints participating in similarity competition; the two types of constraints originate from the same project boundary, which can avoid noise interference caused by the inconsistency between the business scope targeted by hot words and the personnel scope targeted by voiceprint candidates, further improving recognition accuracy.

[0109] Optionally, to ensure the reproducibility of long-running or retrying tasks, the query time, version number, or summary value of the project hotspot set and project member set can be recorded in the task audit information; this optional information is used for consistency verification and auditing, and does not mean that the complete content of members or hotspots must be saved in ProjectSnapshot.

[0110] Optionally, the step of constructing enhanced hot words for the project based on the project coding index of current project hot words includes:

[0111] Based on the project code, obtain hot word records that are associated with the project and are in an available state;

[0112] The system detects whether the line tail weight corresponding to the hot word record meets the preset compliance conditions, wherein the line tail weight is the product of the weight corresponding to the hot word record and the line tail position attenuation coefficient;

[0113] If the line ending weight corresponding to the hot word record meets the preset compliance conditions, then the corresponding hot word record is parsed to obtain the hot word text;

[0114] Save the main text of the hot words and their corresponding line ending weights to form the hot words for the current project;

[0115] System hot words are obtained from a globally maintained system hot word data source. These system hot words are applicable to general product names, high-frequency words within an organization, and industry terms for different projects.

[0116] By concatenating the current project hot words and the system hot words, the enhanced project hot words are obtained.

[0117] In one embodiment, the steps for constructing enhanced hot words for a project include:

[0118] Read the system's hot word set as the global base hot word;

[0119] If the meeting is linked to a project, read the active keywords for that project;

[0120] Merge keywords in order of system hot words first, followed by project hot words;

[0121] Remove duplicates from the normalized whole line, retaining the order of first appearance;

[0122] Ignore blank lines, comment lines, and lines containing illegal hot words.

[0123] Generate the final hotword_str and submit it to the ASR algorithm service.

[0124] Specifically, during the audio processing stage, system hot words are read from the globally maintained system hot word data source. This data source is maintained by the system management terminal, system configuration, or an equivalent global hot word storage, which stores general product names, technical terms, high-frequency words within the organization, and industry terms applicable to different projects. When a hot word record has a status field, only records in the active state are retained, forming H_sys.

[0125] If the meeting is associated with an active project, the system uses the `project_code` from the meeting log to call the project hotword API, retrieving only hotword records related to that project and in an active state, forming `H_proj`. Deactivated, deleted, null, or non-project-related hotwords are not included in this set. Project hotwords can include project name, customer name, project codename, business terminology, version number, deployment terminology, API name, and product module name. When the meeting is not associated with a project, `H_proj` is empty, and only system hotwords are used.

[0126] Each hot word to be processed must include at least the hot word text and source identifier; when the corresponding data source is provided, it may also include status, line end weight, original order, usage frequency or other attributes for pruning.

[0127] The system can perform normalization processing on hot words:

[0128] h'=NormalizeHotword(h), where NormalizeHotword() parses hot word records one by one: unifies line break boundaries, removes leading and trailing whitespace from hot word text, and ignores blank lines, comment lines, and records that cannot form valid hot words.

[0129] When a record contains a valid line ending weight, the weight is retained after parsing the hot words in the text; mixed Chinese and English words and letter abbreviations separated by spaces retain their original word order and their pronunciation is not changed due to standardization.

[0130] Then, the normalized hot word text and the retained legal weight expression are used as the deduplication key for the entire row; for records with the same deduplication key, only the record that appears for the first time in the concatenation sequence of system hot words and project hot words is retained, so as to maintain the stability of the merge order while eliminating duplicate submissions.

[0131] The project-enhanced keyword list includes a keyword set and a project keyword set:

[0132] H_sys={h_1,h_2,...,h_m};

[0133] H_proj={p_1,p_2,...,p_n};

[0134] Wherein: H_sys represents system-level hot words, such as general product names, common terms, and industry terms; H_proj represents project hot words for projects that are currently active, such as customer names, project names, system names, version numbers, deployment terms, and proper nouns.

[0135] The system merges hot keywords in the following order: system hot keywords first, project hot keywords second:

[0136] H_merge=Dedup(H_sys'||H_proj'), where: || means concatenating in order; Dedup() means removing duplicates from the normalized hot words in the whole line.

[0137] Optionally, if the total number of lines or the string length of the merged hot words exceeds the budget B allowed by the algorithm service, the system will perform budget pruning on the hot words to reduce processing costs.

[0138] One possible implementation is as follows: First, select records according to the priority of "project hot words related to the meeting title, project names or customer names, other project hot words, and system hot words"; within the same priority, prioritize records with higher explicit weight or higher usage frequency, and maintain the original order when attributes are the same; then, accumulate the record length according to priority. If adding the next record would cause the accumulated length to exceed B, skip that record, until no more records can be added. After selection, the selected records can be output in the original merging order to stabilize the arrangement of hotword_str. The system can simultaneously record the number of lines before and after pruning, the string length, and whether pruning occurred, but the complete project hot word content is not output in the normal log.

[0139] The final generated algorithm service can use the following hot word strings:

[0140] hotword_str=JoinLines(H_merge).

[0141] Optionally, the system log only records the following statistics:

[0142] Meeting ID; Project Code; Number of System Hot Word Records; Number of Project Hot Word Records; Number of Lines of Hot Words After Merging; Length of String After Merging; Whether Pruning Occurred; Reason for Downgrading.

[0143] The system does not output complete customer hot keywords in the regular logs to avoid leaking project names, customer names, or sensitive business terms, thereby improving security and privacy.

[0144] Optionally, the step of constructing a candidate set of project member voiceprints based on the project code and matching the voiceprints of current project members includes:

[0145] Detect the existence of a collection of voiceprints of project members;

[0146] If the set of project member voiceprints exists, match the set of project member voiceprints as the current project member voiceprint, and construct a candidate set of project member voiceprints.

[0147] If the project member voiceprint set does not exist, the project member voiceprint candidate set is empty, and the algorithm performs speech activity detection and speaker separation.

[0148] Specifically, the input consists of the meeting project context and the voiceprint library:

[0149] ProjectSnapshot={project_code,project_name_snapshot};

[0150] V_all={v_1,v_2,...,v_k};

[0151] Here, V_all represents all active voiceprint records in the system.

[0152] Each voiceprint record may include:

[0153] voiceprint_id: Voiceprint identifier;

[0154] user_id: User identifier;

[0155] embedding_near: Near-field acoustic vector;

[0156] embedding_far: far-field acoustic vector;

[0157] status: Voiceprint status;

[0158] updated_at: Update time.

[0159] When a meeting is linked to an active project and the project members are successfully queried, the system first queries the project member set:

[0160] U_proj=Members(project_code), then only the active voiceprints of project members are selected:

[0161] V_candidate={v∈V_all|v.user_id∈U_proj∧v.status=active}.

[0162] The project_code used and the project code used when querying project hot words come from the same meeting record to generate two consistent input constraints based on the same project context before task submission.

[0163] Let the total number of active voiceprints be k, and the number of candidate active voiceprints for project members be b, then b ≤ k. The convergence of project candidates reduces the number of candidate columns in the subsequent similarity matrix from the total range k to the project range b. This eliminates irrelevant voiceprints outside the project before similarity calculation, reducing the participation of irrelevant candidates in similarity competition and global assignment calculation.

[0164] Users whose active voiceprints are not among the project members will not be added to the candidate set. That is, if the project members exist but there are no active voiceprints among them, i.e. there is no project member voiceprint set, the system will not submit voiceprints, and the system will not fall back to the full voiceprint database. The algorithm will continue to perform voice activity detection and speaker separation, and the clustered speaker corresponding to the user can continue to be retained as an unknown speaker or a temporary speaker.

[0165] The output of the candidate voiceprint set for project members is:

[0166] V_candidate={voiceprint_id,user_id,embeddings}.

[0167] In one embodiment, if a project member query fails, or the management backend times out or the project service becomes unavailable, the system can handle the situation according to a preset deployment strategy:

[0168] Safety-first strategy: Do not roll back the full voiceprint count; if the voiceprint candidate set is empty, only perform ASR and speaker separation.

[0169] Availability-first strategy: roll back all active voiceprints and record structured warnings;

[0170] Blocking strategy: Block voiceprint matching or block the entire audio processing task, and wait for the project members to resume service.

[0171] Preferably, in this embodiment, a security-first strategy is adopted when deploying in a private or high-security environment to avoid mismatches caused by misusing the full voiceprint library when the project context is abnormal.

[0172] A method for conference speech recognition and speaker matching also includes the following steps:

[0173] The meeting backend submits the ASR task to the algorithm side. The input includes the meeting audio, hot word strings, a set of candidate voiceprints, and task parameters. The task parameters include the callback address after the algorithm service completes the processing and the preset voiceprint matching threshold, etc. The input can be:

[0174] TaskReq={audio_ref,hotword_str,V_candidate,webhook_url,threshold,preset_speaker_count}

[0175] Where: audio_ref represents the audio input reference for the conference; hotword_str represents the hotword string; V_candidate represents the set of voiceprint candidates; webhook_url represents the callback address after the algorithm service has finished processing; threshold represents the voiceprint matching threshold; preset_speaker_count represents the optional preset number of speakers.

[0176] The algorithm interface remains compatible, meaning that existing callers that only transmit audio can still work; when a project context exists, hotword_str and voiceprint candidate set are additionally carried without changing the form of the main interface for ASR and voiceprint recognition.

[0177] The task request may include:

[0178] Plain Text{

[0179] "task_id": "task identifier",

[0180] "meeting_id": "Meeting ID",

[0181] "audio_ref": "audio reference",

[0182] "hotword_str": "hotword string",

[0183] "voiceprints": {

[0184] "voiceprint_id_1": ["embedding1", "embedding2"],

[0185] "voiceprint_id_2": ["embedding1"]},

[0186] "voiceprint_threshold": 0.75,

[0187] "preset_speaker_count": 4,

[0188] "webhook_url": "callback address"

[0189] }

[0190] Voiceprints can be organized by voiceprint_id, and each voiceprint_id can carry one or more embeddings, such as near-field embedding and far-field embedding.

[0191] The output is the algorithm task:

[0192] ASRTask={task_id,meeting_id,audio_ref,hotword_str,voiceprints,webhook_url}.

[0193] The algorithm performs the following steps:

[0194] Perform speech activity detection on the target conference audio to obtain a set of speech segments;

[0195] Based on the enhanced hot words of the project, automatic speech recognition is performed on the set of speech segments to output sentence-level subtitle segments with timestamps.

[0196] Specifically, the input is the algorithm task ASRTask. The algorithm service first performs speech activity detection on the audio to obtain a set of speech segments: A={a_1,a_2,...,a_r}, where each speech segment a_i includes the start time and end time.

[0197] Automatic speech recognition uses hotword_str as the input for model-side hotword enhancement, performs automatic speech recognition on a set of speech segments, and outputs sentence-level caption segments:

[0198] S={s_1,s_2,...,s_n}

[0199] Each subtitle segment s_i includes at least:

[0200] start_i: Start time;

[0201] end_i: End time;

[0202] text_i: Recognizes text;

[0203] seq_i: Segment number.

[0204] After automatic speech recognition, the algorithm service can perform hot word post-processing. Hot word post-processing only allows modification of the sentence's text field; it does not allow modification of the following:

[0205] start;

[0206] end

[0207] speaker_id;

[0208] voiceprint_id;

[0209] Segment order.

[0210] Hot word post-processing only corrects the text without modifying the timestamp and speaker fields, in order to minimize the impact of human subjective error.

[0211] Hot word post-processing occurs after the ASR sentence text, timestamp, and speaker attribution are generated but before the callback, ensuring consistency between the algorithm results, callback results, and backend display. This avoids inconsistencies between the algorithm results, callback results, and backend display caused by performing hot word correction after the backend subtitles are stored in the database.

[0212] If the post-processing of hot words fails, the system will record a warning and retain the original ASR text to prevent the task from failing.

[0213] The automatic speech recognition output is a collection of ASR subtitle fragments with timestamps.

[0214] The algorithm also performs the following steps:

[0215] Perform speech activity detection on the target conference audio to obtain a set of speech segments;

[0216] Speaker segmentation is performed on the target conference audio to obtain a clustered speaker set;

[0217] Extract the clustered speaker voiceprint embeddings from the speech segment set, calculate the similarity between any clustered speaker voiceprint embedding and each candidate voiceprint embedding in the project member voiceprint candidate set, traverse the similarity between the clustered speaker voiceprint embeddings of the clustered speaker set and each corresponding candidate voiceprint embedding, and construct a similarity matrix.

[0218] A one-to-one constraint is established, whereby each cluster speaker is assigned at most one candidate voiceprint, and each candidate voiceprint is assigned at most one cluster speaker. Combined with a preset global optimal algorithm, the cluster speaker identity corresponding to the cluster speaker set is determined from the similarity matrix.

[0219] Specifically, the algorithm service obtains the speaker separation results, the candidate voiceprint set, and the voiceprint matching threshold.

[0220] Speaker separation yields a clustered speaker set: D = {d_1, d_2, ..., d_i}; the voiceprint candidate set is: V_candidate = {v_1, v_2, ..., v_j}; where d_i represents the i-th clustered speaker, and v_j represents the j-th candidate voiceprint.

[0221] Based on the set of speech segments, the embedding is extracted for each cluster of speakers: Emb(d_i) = e_i;

[0222] If a candidate user has multiple voiceprint embeddings, such as near-field and far-field embeddings, it is denoted as: Emb(v_j)={u_j1,u_j2,...,u_jm}.

[0223] Automatic speech recognition and speaker voiceprint extraction embedding both use a set of speech segments to share speech activity detection results, avoid repeated audio segmentation, and improve detection efficiency.

[0224] Let the clustered speaker set obtained during the speaker separation stage be D = {d_1, d_2, ..., d_a}, and the candidate voiceprint set of project members be V_candidate = {v_1, v_2, ..., v_b}. Here, a represents the number of speakers in each cluster obtained from the meeting audio, and b represents the number of candidate identities after project context constraints.

[0225] The algorithm service extracts the clustering embedding from the audio segment corresponding to the speaker in the i-th cluster, denoted as Emb(d_i) = e_i. For the j-th candidate identity, if it has multiple registered voiceprint embeddings under near-field, far-field, or other acquisition conditions, it is denoted as Emb(v_j) = {u_j1, u_j2, ..., u_jm}.

[0226] The system calculates the cosine similarity between the cluster embedding and each registered voiceprint embedding of the candidate identity, and takes the maximum value as the similarity between the speaker of that cluster and the candidate identity:

[0227] Sim(d_i,v_j)=max_k Cosine(e_i,u_jk)

[0228] Using the above method, traverse all clustered speakers and all candidate identities to construct a similarity matrix M with dimension a×b: M[i,j]=Sim(d_i,v_j);

[0229] The i-th row of the matrix represents the similarity between the i-th cluster speaker and all candidate identities, and the j-th column represents the similarity between the j-th candidate identity and all cluster speakers. The matrix also retains the highest similarity (Top1_i) and the second highest similarity (Top2_i) in each row for subsequent matching confidence assessment.

[0230] Read the similarity matrix M, take the clustered speakers as the objects to be assigned, and take the candidate identities of project members' voiceprints as the candidate targets, and establish a one-to-one constraint: each clustered speaker can be assigned to at most one candidate identity, and each candidate identity can be assigned to at most one clustered speaker, thereby avoiding multiple clustered speakers occupying the same voiceprint_id at the same time.

[0231] Using a pre-defined global optimal algorithm, the identities of the clustered speakers corresponding to the clustered speaker sets are determined from the similarity matrix. The global optimal algorithm can be represented by the binary assignment variable x_ij as maximizing Σ_i Σ_j M[i,j]x_ij, where x_ij∈{0,1}, Σ_j x_ij≤1, and Σ_i x_ij≤1. When the matrix is ​​not square, the rectangular assignment problem can be solved directly, or it can be solved by adding virtual rows and columns to convert it into a square matrix.

[0232] Optionally, the step of determining the identity of the clustered speaker corresponding to the clustered speaker set from the similarity matrix, in conjunction with a preset global optimal algorithm, includes:

[0233] From the similarity matrix, select the candidate voiceprint embedding with the highest similarity among the similarity combinations of the clustered speaker voiceprint embedding and the corresponding candidate voiceprint embeddings, which satisfy the one-to-one constraint.

[0234] The similarity of the candidate voiceprint embeddings is compared with a preset voiceprint matching threshold;

[0235] When the similarity of the candidate voiceprint embedding is greater than or equal to the voiceprint matching threshold, the candidate voiceprint embedding is determined as the corresponding clustered speaker identity.

[0236] Specifically, the Hungarian algorithm or an equivalent global optimal assignment method is used to find the combination with the highest total similarity among the combinations that satisfy the one-to-one constraint; Assign*=argmax_Assign Σ M[i,Assign(i)].

[0237] After obtaining the globally optimal combination, the similarity of each assigned edge is checked. The matching acceptance condition is:

[0238] M[i,Assign(i)]≥τ_v, where τ_v is the voiceprint matching threshold. When the threshold is met, the corresponding voiceprint_id is output.

[0239] Optionally, the following steps are also included:

[0240] When the similarity of the candidate voiceprint embedding is less than the voiceprint matching threshold, the speaker in the cluster is retained as an unknown or temporary speaker, and the candidate voiceprint embedding with the highest similarity remains unmatched.

[0241] Add virtual candidate nodes with unknown speaker identities to the speakers in this cluster.

[0242] Specifically, when the similarity of the candidate voiceprint embedding is lower than the voiceprint matching threshold, the candidate set is empty and no effective assignment is obtained. At this time, the speaker of the cluster is not forcibly mapped to the project member, but is retained as an unknown speaker or temporary speaker. The candidate identity is not selected and remains in an unmatched state.

[0243] During implementation, candidate edges below τ_v can be masked before solving, or virtual candidate nodes representing "unknown speakers" can be added to the clustered speakers. This allows the algorithm to actively select unmatched states, reduce erroneous real-name labeling, and avoid low-scoring candidates being forced to accept assignments, which could lead to an expanded candidate range and increased risk of cross-item mismatches.

[0244] Optionally, if a project member exists but has no valid voiceprint, the entire pool of voiceprint candidates will not be rolled back. Prioritizing the reduction of incorrect real-name labeling, this avoids increasing the risk of cross-project mismatches by expanding the candidate range to achieve surface hits.

[0245] Optionally, the following steps are also included:

[0246] Calculate the similarity difference between the candidate voiceprint embedding with the highest similarity and the candidate voiceprint embedding with the second highest similarity among the similarity combinations of the clustered speaker voiceprint embedding and each corresponding candidate voiceprint embedding.

[0247] The similarity difference is compared with a preset confidence threshold.

[0248] When the similarity difference is less than the confidence threshold, the candidate voiceprint embedding with the highest similarity is marked as the speaker identity of the low-confidence cluster and a verification is prompted.

[0249] Specifically, the system calculates the matching discrimination score based on the highest and second-highest scores in the i-th row of the similarity matrix:

[0250] Margin_i = Top1_i - Top2_i;

[0251] If Margin_i < τ_margin, where τ_margin is the confidence threshold, then the speaker in this cluster is considered to have a close similarity to two or more candidate identities. The system records a low-confidence warning for auditing or manual review, which helps to further improve the accuracy of speaker identity matching. This judgment does not change the one-to-one constraint.

[0252] The speaker separation stage outputs the matching results between clustered speakers and candidate voiceprints:

[0253] SpeakerMatch={cluster_speaker_id,voiceprint_id,score,margin,confidence_flag,

[0254] The function `match_status` is used to distinguish between matched, low-confidence matched, and unknown speaker; the `voiceprint_id` of an unknown speaker is empty.

[0255] Optionally, to further improve speaker matching accuracy, a conference speech recognition and speaker matching method further includes the following steps:

[0256] Obtain the set of speech segments and the set of clustered speakers;

[0257] For each element in the speech segment set, calculate its overlap duration with each element in the clustered speaker set, and retain only one clustered speaker segment for the overlap replacement interval;

[0258] Calculate the percentage of speakers in the de-overlapping clustered speaker set relative to the corresponding element in the speech segment set, and compare it with a preset sentence speaker percentage threshold.

[0259] If the speaker percentage is greater than or equal to the speaker percentage threshold of the sentence, then the element in the speech segment set belongs to the corresponding element in the clustered speaker set;

[0260] If the speaker percentage is less than the sentence speaker percentage threshold and the sentence is long, the long sentence is split into clauses, and the speaker percentage for each clause is recalculated and compared with the sentence speaker percentage threshold.

[0261] Specifically, obtain the ASR caption fragment set S (i.e., the speech fragment set) and the speaker separation fragment set DiarSegs (i.e., the clustered speaker set).

[0262] Each speaker's segmentation includes:

[0263] speaker_cluster_id;

[0264] start;

[0265] end.

[0266] For each ASR sentence segment s_i=[start_i,end_i], the system calculates the overlap duration with the segment separated by each speaker, and retains only one candidate for the overlapping replacement interval:

[0267] Overlap(s_i,d_j)=max(0,min(end_i,end_j)-max(start_i,start_j));

[0268] The system calculates the percentage of a particular speaker's contribution to the sentence:

[0269] Ratio(s_i,speaker_k)=Σ Overlap(s_i,d_j where d_j.speaker=speaker_k) /

[0270] Duration(s_i);

[0271] Among them: Duration(s_i)=end_i-start_i.

[0272] If there exists a speaker_k that satisfies: Ratio(s_i,speaker_k)≥τ_s, then the sentence belongs to speaker_k, where τ_s is the threshold for the percentage of speakers in the sentence.

[0273] For long sentences with insufficient speaker proportion, the system can split the long sentence into clauses according to punctuation, pauses or ASR sub-fragments: SubSentences(s_i)={sub_i1,sub_i2,...,sub_it}; then recalculate the speaker proportion for each clause and assign speakers.

[0274] Update the corresponding voiceprint_id.

[0275] After clause allocation, the system merges adjacent clauses from the same speaker and filters out extremely short, meaningless clauses using a preset clause threshold.

[0276] AlignedSegments={start,end,text,cluster_speaker_id,voiceprint_id}.

[0277] In the event of an anomaly, the system retains the original ASR text and continues the task.

[0278] Finally, the callback is written to the database and triggers the generation of the minutes, with the input being the callback result of the algorithm service webhook:

[0279] Callback = {task_id, status, segments, warnings, algorithm_meta}, where segments must include at least: start; end; text; cluster_speaker_id; voiceprint_id; score.

[0280] The backend retrieves the automatic speech recognition results from the algorithm's callback and queries the meeting audio records based on the task_id.

[0281] If the meeting has been canceled, deleted, or is in a final state, the backend idempotently discards the late callback and records the callback event.

[0282] If the callback status is failure, the system saves the failure reason and algorithm warning, and does not trigger the generation of minutes.

[0283] When the callback succeeds, the backend performs data validation:

[0284] Is the segment list valid?

[0285] Does each segment contain a valid timestamp?

[0286] start<=end;

[0287] Is the order of the segments sortable?

[0288] Is the text empty?

[0289] Whether voiceprint_id belongs to the candidate set for this task or is an unknown value allowed by the algorithm.

[0290] After the data validation is successful, the voiceprint_id will be mapped to the speaker_id.

[0291] The backend extracts the voiceprint_id collection from the callback:

[0292] VP={voiceprint_id_1,voiceprint_id_2,...,voiceprint_id_q};

[0293] The system queries the mapping from voiceprint_id to user_id:

[0294] user_id=MapVoiceprintToUser(voiceprint_id);

[0295] For each voiceprint_id, the system creates or reuses a speaker entity within the current session:

[0296] speaker_id=GetOrCreateMeetingSpeaker(meeting_id,voiceprint_id,user_id);

[0297] Then map the voiceprint_id in segments to the speaker_id within the meeting.

[0298] For unknown speakers or low-confidence matches, the system creates a temporary speaker_id, such as "unknown speaker 1" or "unknown speaker 2".

[0299] For voiceprint_id successfully mapped to user_id, the system can synchronize the user_id as a meeting participant or identify the participant for subsequent use in meeting minutes generation, speech statistics, and front-end display.

[0300] Save the subtitles:

[0301] original_segments: Original segments of the algorithm;

[0302] display_segments: Front-end display segments;

[0303] speaker_mapping: The mapping from voiceprint_id to speaker_id;

[0304] Warnings: Hot words, voiceprints, alignment, etc.

[0305] subtitle_status: Subtitle status.

[0306] After the subtitle results are saved, the system triggers the subsequent meeting minutes generation process. The subsequent meeting minutes generation uses the enhanced subtitle text, speaker_id, participant information, and project context as input, and outputs the subtitles already stored in the database, the speakers in the meeting, and the subsequent minutes generation task.

[0307] Finally, the meeting minutes, including captions and speaker identification information, are generated through the subsequent minutes generation system.

[0308] In one example, a user creates a meeting and selects Project A. Project A has 8 members, 5 of whom have registered active voiceprints. Project A maintains 60 lines of project hotwords, including customer names, product modules, and interface abbreviations. The backend saves `project_code` and `project_name_snapshot`. Before submitting the ASR (Automatic Speech Recognition), the backend reads the system hotwords and the active hotwords of Project A, merges and removes duplicates to generate `hotword_str`; simultaneously, it queries the `user_id` of Project A members and only reads the active voiceprints of these 5 members. After the algorithm completes automatic speech recognition, speaker separation, and voiceprint matching, project-specific terms in the subtitle fragment are corrected to the standard spelling in the project hotwords, and non-project members are not incorrectly labeled as speakers.

[0309] In one example, a meeting is bound to Project B, but none of the members of Project B have registered their voiceprints. After the backend successfully obtains the list of project members, the query for active voiceprints returns empty. The backend submits an ASR task to the algorithm but does not send voiceprints. The algorithm still performs ASR and speaker separation, and the subtitles display different unknown speakers or temporary speakers. Users can subsequently register voiceprints from audio clips or manually correct the speakers. This solution does not interrupt meeting transcription due to missing voiceprints and avoids mismatches caused by reverting to a full set of voiceprints.

[0310] In one example, a meeting is bound to project C, but the management backend reports a timeout when submitting an ASR (Active Voice Request). The backend captures the project member query exception and records a structured warning, including meeting_id, project_code, fallback_reason, and candidate_count. If the deployment strategy allows rollback, the backend reads the full activevoiceprints and continues executing the ASR; if the project hotword query also fails, only system hotwords are used. This ensures the main ASR process continues, and the warning can be used later to assess whether there is a risk of mismatch in the voiceprint candidates.

[0311] In one example, speakers S1 and S2 in a conference cluster have similarities to registered users U1 and U2. By constructing a complete similarity matrix, the Hungarian algorithm is used to find the pair with the highest overall similarity. Pairs with similarity below a threshold are not accepted, and a low-confidence warning is recorded when the difference between the best and second-best is too small, for subsequent manual review. However, if a greedy strategy using existing techniques is employed, S1 might occupy U1's slot first, leading to S2 being incorrectly assigned.

[0312] Compared to existing technologies, a new conference speech recognition and speaker matching method incorporates a project context mechanism. Before algorithm execution, the conference backend persists project_code, project_name_snapshot, and bind_time during the conference binding phase to obtain the external structured project context determined by the business project bound to the conference. When audio enters the processing flow, the system queries the current project status based on the project_code and obtains the current active project member set and active project hot word set through the corresponding interface. The obtained data participates in the construction of enhanced project hot words and voiceprint candidate screening for this task. Based on the same project context, enhanced project hot word constraints and project member voiceprint candidate constraints are generated simultaneously, jointly converging the text recognition candidate space and speaker identity candidate space. This improves the quality of hot words and speaker matching from the source, enhances the accuracy of cross-project conference speech recognition, and reduces the probability of irrelevant personnel participating in the matching.

[0313] The project context originates from the management backend. The meeting backend does not directly use project information as meeting viewing permissions. Project member relationships are only used for project selection, project hot words, voiceprint candidate range, project default templates, and other intelligent processing convergence. They are not automatically granted meeting content reading permissions to avoid mistaking "project member relationships" for "meeting content reading authorization".

[0314] The project_name_snapshot can additionally record the collection version number, summary value, or query time. When processing audio, it can query the current project members and project hot words based on the project code, thereby achieving the purpose of saving a complete copy of the project members or project hot words to support consistency verification and result traceability.

[0315] A conference speech recognition and speaker matching method separates speech activity detection and speaker separation on the algorithm side, and maximizes the overall matching score by constructing a similarity matrix and global optimization, thereby further improving the accuracy of conference speech transcription.

[0316] A meeting speech recognition and speaker matching method rejects low-confidence identity mapping, retains unknown identities for clustered speakers below the voiceprint threshold, those not effectively assigned, or those with empty candidate sets, does not force assignment, and marks low-confidence matches that are difficult to distinguish with the difference between the largest and second largest similarity. By reducing erroneous real-name annotations, it avoids expanding the candidate range to obtain superficial hits and increasing the risk of cross-project mismatches, thereby further improving the accuracy of speaker identity matching.

[0317] A conference speech recognition and speaker matching method determines the attribution of a sentence to a speaker based on the overlap ratio between the sentence's time range and the speaker's separated segments when aligning sentences with speakers. If the overlap ratio is insufficient, clauses can be split and redistributed. This method enables sentence-level subtitles, speaker separation results, and voiceprint identity matching to be unified into structured subtitle segments.

[0318] A conference speech recognition and speaker matching method constructs a similarity matrix between all clustered speakers and all candidate members, and uses the Hungarian algorithm or an equivalent method to solve for the global one-to-one combination with the highest overall similarity. Compared with the greedy method of selecting the highest-scoring candidate from each cluster, this method can avoid multiple clustered speakers occupying the same registered identity, reduce the impact of the matching order on the overall result, and avoid getting trapped in local optima.

[0319] An abnormal degradation mechanism is designed for a meeting speech recognition and speaker matching method. When the meeting is not bound to a project, the system can use system hot words and all active voiceprints. When the project hot word query fails, the system can use system hot words. When the project member query fails, the system records a warning and rolls back to all voiceprints or blocks according to the strategy. When the project member has no voiceprint, the system does not roll back to all voiceprints, continues ASR and outputs unknown speaker. When hot word parsing is abnormal, the system ignores the abnormality. When hot word error correction is abnormal, the system records a warning and returns the original ASR text. When the ASR returns empty segments, the system marks the subtitle as failed and advances the meeting to a failed state. When the callback arrives and the meeting has been canceled or deleted, and the system discards it idempotently, no subtitles are created and no minutes are triggered. When the voiceprint similarity is lower than the threshold, registered users are not matched and unknown speakers are retained. When a project member exists but has no valid voiceprint, the system does not roll back to all voiceprint candidates. This maintains configurable processing capacity, ensures that the ASR main link is not easily blocked when the project context enhancement fails, and retains a basis for subsequent quality assessment and problem investigation.

[0320] In summary, a meeting speech recognition and speaker matching method obtains external structured constraints of the project context from project management data based on the business projects bound to the meeting before the ASR task starts; constructs project-enhanced hot words and project member voiceprint candidate sets; submits the automatic speech recognition task to the algorithm side; achieves the acquisition of currently valid project hot words and project member voiceprints based on project codes; simultaneously converges two candidate spaces using the same project context; optimizes the globally maintained system hot words based on the current project hot words; constructs project-enhanced hot words for text correction; and queries and matches project member voiceprints within the scope of the current project members to construct project member voiceprints. The voiceprint candidate set converges all registered voiceprint candidates into project member voiceprint candidates, narrowing the target voiceprint candidate range. This allows the ASR to provide proprietary terminology constraints related to the current project before speech recognition begins, and excludes voiceprint candidates unrelated to the current project during identity matching. This improves the speaker identity matching rate and speech recognition accuracy across projects, achieving higher-precision speech recognition and more accurate speaker matching between different projects. Furthermore, since project hot word constraints and project member voiceprint constraints originate from the same project boundary, they can avoid noise interference caused by the inconsistency between the business scope targeted by the hot words and the personnel scope targeted by the voiceprint candidates, further improving recognition accuracy.

[0321] The project context constraints do not depend on whether the current meeting has generated usable transcribed text. For project members with registered voiceprints, even if the transcribed text does not contain identity information such as self-identification, other-identification, position, or role responsibilities, the system can still perform identity matching based on the similarity between the clustered speaker voiceprint embedding and the candidate voiceprint embedding. It does not require the appearance of names, titles, or role words in the speech as a necessary condition, making it more applicable.

[0322] It should be understood that although the steps in the flowcharts of the embodiments of this application are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0323] The embodiments disclosed in this application describe in detail a method for conference speech recognition and speaker matching. The method disclosed in this application can be implemented using various types of devices. Therefore, this application also discloses a conference speech recognition and speaker matching device corresponding to the above method. Specific embodiments are given below for detailed description.

[0324] Reference Figure 2 This application also provides a conference speech recognition and speaker matching device, including:

[0325] The project coding module is used to receive the project code carried in the meeting creation request or audio submission request;

[0326] The project verification module is used to verify whether the target project is in an available state based on the project code.

[0327] The project context module is used to bind the project context of the target project when the target project is in an available state. The project context is determined by the external structured constraints of the target project in advance.

[0328] The project-enhanced hot word module is used to construct project-enhanced hot words based on the project code index of the current project hot words;

[0329] The project member voiceprint module is used to construct a candidate set of project member voiceprints by matching the current project member's voiceprint based on the project code.

[0330] The task module is used to submit automatic speech recognition tasks to the algorithm side. The automatic speech recognition task includes target meeting audio, project-enhanced hot words, a candidate set of project member voiceprints, and task parameters. The task parameters include a callback address after the algorithm service completes processing. The algorithm side performs the following: performs speech activity detection on the target meeting audio to obtain a set of speech segments; performs speaker separation on the target meeting audio to obtain a clustered speaker set; performs automatic speech recognition on the speech segment set based on the project-enhanced hot words to output subtitle segments; extracts the clustered speaker voiceprint embeddings from the speech segment set, calculates the similarity between any clustered speaker voiceprint embedding and each candidate voiceprint embedding in the candidate set of project member voiceprints, iterates through the similarity between the clustered speaker voiceprint embeddings and the corresponding candidate voiceprint embeddings of the clustered speaker set, and constructs a similarity matrix; establishes a one-to-one constraint, where each clustered speaker is assigned at most one candidate voiceprint, and each candidate voiceprint is assigned at most one clustered speaker; and, combined with a preset global optimal algorithm, determines the clustered speaker identity corresponding to the clustered speaker set from the similarity matrix.

[0331] The output module is used to obtain the automatic speech recognition results from the algorithm side callback and output meeting minutes containing subtitles and speaker identification information.

[0332] For specific limitations regarding the conference speech recognition and speaker matching device, please refer to the method limitations above, which will not be repeated here. Each module in the above device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the server's processor, or stored in software in the server's memory, so that the processor can call and execute the corresponding operations of each module.

[0333] A conference speech recognition and speaker matching device achieves the effect of any of the above-mentioned conference speech recognition and speaker matching methods, which will not be elaborated here.

[0334] Reference Figure 3 This application also provides a computer device, including at least one processor, a memory, and a transceiver; wherein the memory is used to store program code, and the processor is used to call the program code stored in the memory to execute any of the above-described conference speech recognition and speaker matching methods.

[0335] A processor may include one or more processing cores. The processor connects to various parts of the computer device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, and by accessing data stored in memory.

[0336] Optionally, the processor can be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor can integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem can also be implemented separately as a communication chip, without being integrated into the processor.

[0337] The memory may include random access memory (RAM) or read-only memory (ROM). The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created during the use of the computer device.

[0338] Those skilled in the art will understand that the structure shown in the figure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0339] Reference Figure 4 This application also provides a computer storage medium including instructions that, when run on a computer, cause the computer to execute any of the above-described conference speech recognition and speaker matching methods.

[0340] Computer storage media can be computer-readable storage media. Computer-readable storage media can be any usable medium that a computer can store, or a data storage device such as a server or data center that integrates one or more usable media. The usable medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)). Computer-readable storage media can be electronic storage devices such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disks, or ROM.

[0341] Optionally, computer-readable storage media include non-transitory computer-readable storage media.

[0342] The computer-readable storage medium has storage space for instructions to perform any of the method steps described above. These instructions can be read from or written to one or more computer program products. The instructions can be stored in the computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0343] Reference Figure 5 This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described file enhancement processing methods.

[0344] The computer program product includes one or more computer programs. When the computer program is loaded and executed on a computer, it produces, in whole or in part, the processes or functions described in the embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program code may be compressed in an appropriate form.

[0345] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0346] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0347] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, apparatuses, or modules, and may be electrical, mechanical, or other forms.

[0348] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0349] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0350] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0351] The technical solutions provided in the embodiments of this application have been described in detail above. Specific examples have been used in the embodiments of this application to illustrate the principles and implementation methods of the embodiments of this application. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments of this application. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this application.

Claims

1. A method for conference speech recognition and speaker matching, characterized in that, Includes the following steps: Receive the project code carried in the meeting creation request or audio submission request; Based on the project code, verify whether the target project is in an available state; If the target project is in an available state, then the project context of the target project is bound, wherein the project context is an external structured constraint predetermined by the target project. Based on the project coding index of current project hot words, construct enhanced project hot words; Based on the project code, match the voiceprints of the current project members to construct a candidate set of project member voiceprints; Submit an automatic speech recognition task to the algorithm side. The automatic speech recognition task includes the target meeting audio, the enhanced hot words of the project, the candidate set of voiceprints of the project members, and task parameters. The algorithm performs the following steps: Perform speech activity detection on the target conference audio to obtain a set of speech segments; Speaker segmentation is performed on the target conference audio to obtain a clustered speaker set; Based on the enhanced hot words of the project, automatic speech recognition is performed on the set of speech segments to output subtitle segments; Extract the clustered speaker voiceprint embeddings from the speech segment set, calculate the similarity between any clustered speaker voiceprint embedding and each candidate voiceprint embedding in the project member voiceprint candidate set, traverse the similarity between the clustered speaker voiceprint embeddings of the clustered speaker set and each corresponding candidate voiceprint embedding, and construct a similarity matrix. A one-to-one constraint is established, whereby each cluster speaker is assigned at most one candidate voiceprint, and each candidate voiceprint is assigned at most one cluster speaker. Combined with a preset global optimal algorithm, the cluster speaker identity corresponding to the cluster speaker set is determined from the similarity matrix.

2. The method according to claim 1, characterized in that, The step of constructing enhanced hot words for the project based on the project coding index of the current project hot words includes: Based on the project code, obtain hot word records that are associated with the project and are in an available state; The system detects whether the line tail weight corresponding to the hot word record meets the preset compliance conditions, wherein the line tail weight is the product of the weight corresponding to the hot word record and the line tail position attenuation coefficient; If the line ending weight corresponding to the hot word record meets the preset compliance conditions, then the corresponding hot word record is parsed to obtain the hot word text; Save the main text of the hot words and their corresponding line ending weights to form the hot words for the current project; System hot words are obtained from a globally maintained system hot word data source. These system hot words are applicable to general product names, high-frequency words within an organization, and industry terms for different projects. By concatenating the current project hot words and the system hot words, the enhanced project hot words are obtained.

3. The method according to claim 1, characterized in that, The step of constructing a candidate set of project member voiceprints based on the project code and matching the voiceprints of current project members includes: Detect the existence of a collection of voiceprints of project members; If the set of project member voiceprints exists, match the set of project member voiceprints as the current project member voiceprint, and construct a candidate set of project member voiceprints. If the project member voiceprint set does not exist, the project member voiceprint candidate set is empty, and the algorithm performs speech activity detection and speaker separation.

4. The method according to claim 1, characterized in that, The task parameters include the callback address after the algorithm service completes processing. After submitting the automatic speech recognition task to the algorithm side, the following steps are also included: Obtain the automatic speech recognition results from the algorithm side callback and output meeting minutes containing subtitles and speaker identification information.

5. The method according to claim 1, characterized in that, The steps for determining the identity of the clustered speaker corresponding to the clustered speaker set from the similarity matrix, based on a preset global optimal algorithm, include: From the similarity matrix, select the candidate voiceprint embedding with the highest similarity among the similarity combinations of the clustered speaker voiceprint embedding and the corresponding candidate voiceprint embeddings, which satisfy the one-to-one constraint. The similarity of the candidate voiceprint embeddings is compared with a preset voiceprint matching threshold; When the similarity of the candidate voiceprint embedding is greater than or equal to the voiceprint matching threshold, the candidate voiceprint embedding is determined as the corresponding clustered speaker identity. When the similarity of the candidate voiceprint embedding is less than the voiceprint matching threshold, the speaker in the cluster is retained as an unknown or temporary speaker, and the candidate voiceprint embedding with the highest similarity remains unmatched. Add virtual candidate nodes with unknown speaker identities to the speakers in this cluster.

6. The method according to claim 5, characterized in that, It also includes the following steps: Calculate the similarity difference between the candidate voiceprint embedding with the highest similarity and the candidate voiceprint embedding with the second highest similarity among the similarity combinations of the clustered speaker voiceprint embedding and each corresponding candidate voiceprint embedding. The similarity difference is compared with a preset confidence threshold. When the similarity difference is less than the confidence threshold, the candidate voiceprint embedding with the highest similarity is marked as the speaker identity of the low-confidence cluster and a verification is prompted.

7. The method according to claim 1, characterized in that, It also includes the following steps: Obtain the set of speech segments and the set of clustered speakers; For each element in the speech segment set, calculate its overlap duration with each element in the clustered speaker set, and retain only one clustered speaker segment for the overlap replacement interval; Calculate the percentage of speakers in the de-overlapping clustered speaker set relative to the corresponding element in the speech segment set, and compare it with a preset sentence speaker percentage threshold. If the speaker percentage is greater than or equal to the speaker percentage threshold of the sentence, then the element in the speech segment set belongs to the corresponding element in the clustered speaker set; If the speaker percentage is less than the sentence speaker percentage threshold and the sentence is long, the long sentence is split into clauses, and the speaker percentage for each clause is recalculated and compared with the sentence speaker percentage threshold.

8. The method according to claim 4, characterized in that, Before the step of obtaining the automatic speech recognition result from the algorithm-side callback and outputting meeting minutes containing subtitles and speaker identification information, the following steps are also included: Query meeting audio recordings; If the meeting audio recording is canceled, deleted, or in a final state, discard the callback result and record the callback event; Query callback status; If the callback status is successful, the voiceprints of the identified project members will be mapped to speaker identity information, sentence-level subtitles with timestamps will be saved, and the minutes will be generated. If the callback status is failure, save the failure reason and algorithm warning, and do not trigger the generation of minutes.

9. A conference speech recognition and speaker matching device, characterized in that, include: The project coding module is used to receive the project code carried in the meeting creation request or audio submission request; The project verification module is used to verify whether the target project is in an available state based on the project code. The project context module is used to bind the project context of the target project when the target project is in an available state. The project context is determined by the external structured constraints of the target project in advance. The project-enhanced hot word module is used to construct project-enhanced hot words based on the project code index of the current project hot words; The project member voiceprint module is used to construct a candidate set of project member voiceprints by matching the current project member's voiceprint based on the project code. The task module is used to submit automatic speech recognition tasks to the algorithm side. The automatic speech recognition task includes target meeting audio, project-enhanced hot words, a candidate set of project member voiceprints, and task parameters. The algorithm side performs the following: performing speech activity detection on the target meeting audio to obtain a set of speech segments; performing speaker separation on the target meeting audio to obtain a clustered speaker set; performing automatic speech recognition on the speech segment set based on the project-enhanced hot words to output subtitle segments; extracting clustered speaker voiceprint embeddings from the speech segment set, calculating the similarity between any clustered speaker voiceprint embedding and each candidate voiceprint embedding in the candidate set of project member voiceprints, traversing the similarity between the clustered speaker voiceprint embeddings and the corresponding candidate voiceprint embeddings in the clustered speaker set to construct a similarity matrix; establishing a one-to-one constraint, each clustered speaker is assigned at most one candidate voiceprint, and each candidate voiceprint is assigned at most one clustered speaker; and combining a preset global optimal algorithm, determining the clustered speaker identity corresponding to the clustered speaker set from the similarity matrix.

10. A computer device, characterized in that, include: At least one processor, memory, and transceiver; The memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the method as described in any one of claims 1-8.

Images