Speech recognition method and system

Abstract

The invention provides a voice recognition method and system, and the method comprises the steps: carrying out the voice activity detection of an audio stream received in real time, and extracting at least one voice paragraph from the audio stream; for each voice paragraph, extracting a target voiceprint feature vector of the voice paragraph; determining the similarity between the target voiceprint feature vector of the voice paragraph and each voiceprint feature vector in a registered voiceprint feature library, registering a new second speaker in the voiceprint feature library according to the similarity, and determining the new second speaker as a target speaker corresponding to the voice paragraph; performing voice recognition on the voice paragraph to generate a text corresponding to the voice paragraph; and generating a voice recognition result of the voice paragraph according to the identifier of the target speaker corresponding to the voice paragraph and the text. Therefore, automatic separation, labeling and transcription of multiple speakers can be realized in a real-time dialogue environment, and the availability and the intelligent level of a voice interaction system are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a speech recognition method and system. Background Technology

[0002] With the rapid development of information and network technologies, people have an increasing demand for speech recognition technology, and the requirements for its security and reliability are becoming more stringent. In the field of speech processing, existing speech recognition (ASR) systems mainly focus on converting speech into text, but they cannot accurately distinguish the speech of different speakers in multi-speaker environments. On the other hand, existing speaker verification / identification usually requires pre-registration of speaker voiceprints, and cannot achieve real-time annotation for temporarily accessing users, resulting in insufficient support for real-time performance and failing to meet the needs of large-scale real-time multi-user voice interaction.

[0003] In intelligent voice applications, due to multiple speakers taking turns in a conversation, existing recognition methods based on voiceprint models are prone to misjudgments and high misjudgment rates, failing to accurately identify the speaker's identity and the context. Furthermore, during meetings, the voices of multiple people can interfere with the accuracy of the voiceprint system; incomplete voiceprint registration can lead to model bias; and the latency of speech recognition increases the system's workload and hardware costs.

[0004] Therefore, how to achieve efficient speech separation, identity recognition, and speech transcription in real-time multi-speaker speech recognition processes, while taking into account computational resources, real-time performance, and accuracy, has become a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a speech recognition method and system to solve the problems of insufficient recognition accuracy and poor real-time performance of existing speech recognition systems in multi-speaker environments.

[0006] This application provides a speech recognition method, the method comprising: Speech activity detection is performed on the real-time received audio stream, and at least one speech segment is extracted from the audio stream; For each speech segment, extract the target voiceprint feature vector for that speech segment; The similarity between the target voiceprint feature vector of the speech segment and each voiceprint feature vector in the registered voiceprint feature library is determined. Based on the similarity, a new second speaker is registered in the voiceprint feature library and determined as the target speaker corresponding to the speech segment. The voiceprint feature library includes the original first speaker and the corresponding voiceprint feature vector. Perform speech recognition on the speech segment and generate the corresponding text. Based on the identifier of the target speaker corresponding to the speech segment and the text, the speech recognition result of the speech segment is generated.

[0007] This application also provides a speech recognition system, the system comprising: The voice activity detection module is used to detect voice activity in the real-time received audio stream and extract at least one voice segment from the audio stream. The voiceprint feature extraction module is used to extract the target voiceprint feature vector for each speech segment. The voiceprint management module is used to determine the similarity between the target voiceprint feature vector of the speech segment and each voiceprint feature vector in the registered voiceprint feature library, and register a new second speaker in the voiceprint feature library according to the similarity and determine it as the target speaker corresponding to the speech segment; wherein, the voiceprint feature library includes the original first speaker and the corresponding voiceprint feature vector. The speech recognition module is used to perform speech recognition on the speech segment and generate the corresponding text for the speech segment. The output module generates the speech recognition result for the speech segment based on the identifier of the target speaker and the text corresponding to the speech segment.

[0008] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the speech recognition method described above are performed.

[0009] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the speech recognition method described above.

[0010] This application provides a speech recognition method and system that integrates speech activity detection, voiceprint feature extraction, voiceprint management, speech recognition, and result output to process multi-speaker scenarios holistically. It enables real-time automatic separation, annotation, and transcription of multiple speakers, avoiding the inefficiency caused by independent processing of each functional module and improving the system's real-time performance and processing efficiency. Employing a dynamic speaker management mechanism, it eliminates the need for pre-registration of speakers. Upon detecting a new speaker's speech segment, it dynamically registers the new speaker based on voiceprint feature vectors, overcoming the traditional limitations of voiceprint recognition on new user registration. This significantly improves the system's adaptability in dynamic environments and the accuracy of speech recognition in multi-speaker scenarios. Furthermore, the registration and recognition processes are completed within the same processing flow, further enhancing real-time performance and processing efficiency. Real-time speech activity detection effectively filters out invalid audio, improving speech recognition accuracy, especially in noisy environments, maintaining high recognition precision and ensuring the reliability of voice interaction.

[0011] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This document shows one of the flowcharts of a speech recognition method provided in an embodiment of this application; Figure 2 A second flowchart of a speech recognition method provided in an embodiment of this application is shown; Figure 3 This paper shows a schematic diagram of the structure of a speech recognition system provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0015] Research has revealed the following drawbacks in existing speech recognition technologies: 1. Existing speech recognition (ASR) systems mainly focus on converting speech into text, but they cannot accurately distinguish the speech of different speakers in multi-speaker environments, making it difficult to achieve accurate speech recognition and transcription in complex voice interaction scenarios.

[0016] 2. Existing voiceprint recognition systems typically require pre-registration of the speaker's voiceprint, and cannot perform real-time annotation for temporarily connected users, which limits the system's adaptability and real-time performance in dynamic environments.

[0017] 3. In noisy environments, invalid audio can interfere with recognition, affecting its accuracy and efficiency.

[0018] 4. Current technical solutions handle VAD, ASR and voiceprint recognition separately, which is insufficient for real-time support and cannot meet the needs of large-scale real-time multi-user voice interaction, making it difficult to achieve efficient processing of multi-speaker scenarios.

[0019] 5. In intelligent voice applications, existing technical solutions are insufficient in terms of segmentation accuracy, making it difficult to achieve fine segmentation of audio streams and role recognition, which affects the accuracy and intelligence level of meeting recordings.

[0020] Based on this, the present application provides a speech recognition method to solve the problems of insufficient recognition accuracy and poor real-time performance of existing speech recognition systems in multi-speaker environments. It aims to achieve automatic separation, annotation and transcription of multiple speakers in real-time dialogue environments, significantly improve the usability and intelligence level of the speech interaction system, and enhance recognition accuracy and real-time performance.

[0021] The speech recognition method provided in this application embodiment can be implemented based on a speech recognition system. The speech recognition system integrates a speech activity detection module, a voiceprint feature extraction module, a voiceprint management module, a speech recognition module, and a result output module. It can process multi-speaker scenarios as a whole, realize real-time automatic separation, annotation, and transcription of multiple speakers, avoid the inefficiency caused by independent processing of each functional module, and improve the real-time performance and processing efficiency of the system.

[0022] Please see Figure 1 , Figure 1 This is one of the flowcharts for a speech recognition method provided in an embodiment of this application. For example... Figure 1 As shown in the embodiments of this application, the speech recognition method includes: S101. Perform voice activity detection on the real-time received audio stream and extract at least one voice segment from the audio stream.

[0023] In this step, a pre-trained speech activity detection model can be used to detect speech activity in the real-time received audio stream, achieving speech / silence segmentation, and then extracting at least one valid speech segment from the audio stream. For example, the Silo-VAD model can be used as the speech activity detection model, and the accuracy of detection can be improved by setting a smaller speech activity detection threshold. For instance, the parameters can be set to a minimum silence duration of 0.25s and a minimum speech duration of 0.25s, with a 30ms frame window for real-time detection. Through short frame windows and parameter optimization, short noise triggers can be effectively avoided, ensuring real-time detection.

[0024] S102. For each speech segment, extract the target voiceprint feature vector of that speech segment.

[0025] In this step, a pre-trained voiceprint feature extraction model can be used to extract the target voiceprint feature vector for each separated speech segment. For example, a speaker embedding model based on ResNet34 can be used, which can extract effective voiceprint feature vectors of fixed dimensions (such as 256 dimensions) by setting appropriate convolution kernel size and number of channels, and has strong discriminativeness and robustness.

[0026] S103. Determine the similarity between the target voiceprint feature vector of the speech segment and each voiceprint feature vector in the registered voiceprint feature library, and register a new second speaker in the voiceprint feature library according to the similarity and determine it as the target speaker corresponding to the speech segment.

[0027] The voiceprint feature database includes the original first speaker and the corresponding voiceprint feature vector.

[0028] In this step, the similarity between the target voiceprint feature vector of the speech segment and each voiceprint feature vector in the registered voiceprint feature library is determined. For example, cosine similarity can be used. If the similarity (or the highest similarity) between the target voiceprint feature vector and each of the original first speakers is lower than a preset threshold, a new second speaker needs to be dynamically registered, the target voiceprint feature vector is recorded, and a unique speaker identifier is assigned.

[0029] In addition, if the similarity between the target voiceprint feature vector and any existing first speaker is higher than a preset threshold, the speech segment can be directly identified as belonging to the existing first speaker, and the identifier of the first speaker can be returned directly without a new registration process.

[0030] Here, the preset threshold can be set according to system needs, such as 0.5, or it can be dynamically adjusted. For example, the initial value of the preset threshold is 0.5. To improve the adaptability of the system, a caching mechanism is set up so that when the same voiceprint feature vector is detected multiple times, the corresponding similarity threshold is dynamically adjusted to a higher value, such as 0.7.

[0031] It should be noted that, unlike existing schemes that register voiceprints separately, the voiceprint registration and speech recognition processes in this embodiment are completed in the same stream. This design of registering while recognizing avoids the phased operations of independent voiceprint registration, voiceprint database updates, and then recognition based on the updated voiceprint database in existing technologies. In this embodiment, when a new speaker's audio segment is detected, the system instantly generates a voiceprint vector and dynamically registers it with a latency of less than 50ms, thereby improving real-time performance and processing efficiency. In practical applications, this is more suitable for multi-user real-time interaction scenarios, such as online meetings, customer service, and collaborative office work.

[0032] S104. Perform speech recognition on the speech segment and generate the corresponding text.

[0033] In this step, a pre-trained end-to-end speech recognition model can be used to transcribe the same speech segment. For example, the SenseVoice model can be used as the speech recognition model, and the recognition accuracy can be optimized by setting an appropriate learning rate and batch size. During the recognition process, the system can also use the speech endpoint detection results from S101 as additional speech features to further improve recognition accuracy.

[0034] It should be noted that the execution order between S103 and S104 is not restricted in the embodiments of this application. That is, S103 can be executed first and then S104, or S104 can be executed first and then S103, or S103 and S104 can be executed simultaneously.

[0035] S105. Based on the identifier of the target speaker corresponding to the speech segment and the text, generate the speech recognition result of the speech segment.

[0036] In this step, the speech recognition result of the speech segment can be generated based on the target speaker's identifier and text. For example, the speech recognition result can be packaged into a JSON format, consisting of "target speaker ID + text + timestamp".

[0037] Subsequently, the speech recognition results can be returned to the client in real time via WebSocket by the speech recognition system. Simultaneously, the speech recognition system can also provide an HTTP static service interface, supporting direct access by the front-end browser. This dual interface approach of WebSocket and HTTP provides a flexible service interface, simplifying the system integration process, reducing system integration costs, and improving the system's scalability and usability. Furthermore, to improve the system's real-time performance and concurrency, a multi-process architecture is adopted, dynamically adjusting the number of worker processes based on system resource availability. For example, the system can support 200 concurrent connections, each with millisecond-level latency processing. Based on an asyncio + websockets architecture, compared to traditional end-to-end voiceprint recognition solutions, this embodiment is more suitable for scenarios requiring large-scale simultaneous online operation.

[0038] Thus, the speech recognition method provided in this application integrates speech activity detection, voiceprint feature extraction, voiceprint management, speech recognition, and result output to process multi-speaker scenarios holistically. This enables real-time automatic separation, annotation, and transcription of multiple speakers, avoiding the inefficiency caused by independent processing of each functional module and improving the system's real-time performance and processing efficiency. Employing a dynamic speaker management mechanism, it eliminates the need for pre-registration of speakers. Upon detecting a new speaker's speech segment, it dynamically registers the new speaker based on voiceprint feature vectors, overcoming the traditional limitations of voiceprint recognition on new user registration. This significantly improves the system's adaptability in dynamic environments and the accuracy of speech recognition in multi-speaker environments. Furthermore, the registration and recognition processes are completed in the same stream, further enhancing real-time performance and processing efficiency. Real-time speech activity detection effectively filters out invalid audio, improving speech recognition accuracy, especially in noisy environments, maintaining high recognition precision and ensuring the reliability of voice interaction.

[0039] In summary, the above methods can achieve real-time automatic separation, annotation, and transcription of multiple speakers, effectively solving the problems existing in the prior art and improving the usability and intelligence level of the voice interaction system.

[0040] In one possible implementation, step S103, registering a new second speaker in the voiceprint feature database based on the similarity and determining it as the target speaker corresponding to the speech segment, may include: If the highest similarity between the target voiceprint feature vector of the speech segment and the voiceprint feature vector corresponding to each first speaker in the voiceprint feature library is lower than the second registration threshold, then a new second speaker and its corresponding voiceprint feature vector are registered in the voiceprint feature library according to the speech segment, and the target speaker corresponding to the speech segment is determined as the second speaker.

[0041] In this step, if the highest similarity is lower than the second registration threshold... If it is determined that the speaker corresponding to the speech segment is likely not the first speaker already existing in the voiceprint feature database, then the dynamic registration process begins. Based on the speech segment, a new second speaker *m* and its corresponding voiceprint feature vector are registered in the voiceprint feature database. Accordingly, the target speaker corresponding to the speech segment becomes this second speaker *m*. It can be understood that once this second speaker *m* is registered in the voiceprint feature database, in subsequent voiceprint recognition of other speech segments, the current second speaker *m* is equivalent to the original first speaker in the voiceprint feature database, thus achieving real-time dynamic registration of the voiceprint database.

[0042] Furthermore, the identification method also includes: If the highest similarity between the target voiceprint feature vector of the speech segment and the voiceprint feature vector corresponding to the first speaker in the voiceprint feature database is greater than the first recognition threshold, then the target speaker corresponding to the speech segment is determined as the first speaker corresponding to the highest similarity.

[0043] In this step, the target voiceprint feature vector of the speech segment is compared with the voiceprint feature vector corresponding to each first speaker in the voiceprint feature database to determine the similarity between the target voiceprint feature vector and the voiceprint feature vector corresponding to each first speaker, and then the highest similarity and the corresponding first speaker are determined. If the highest similarity score is greater than the first recognition threshold Then the target speaker can be identified as the first speaker corresponding to the highest similarity. .

[0044] It should be noted that the first recognition threshold With the second registration threshold They can be the same or different. Preferably, the first recognition threshold... It should be slightly higher than the second registration threshold. .For example , The first recognition threshold is the threshold value for determining the speaker. A higher setting can reduce false positives and prevent crosstalk.

[0045] Optionally, when the highest similarity is at the first recognition threshold With the second registration threshold During this period, one can continuously observe subsequent speech segments of the same speaker and determine the speaker's identity based on the overall similarity of the subsequent speech segments.

[0046] In addition, it can be tailored to each speaker. Set a personalized first recognition threshold. With the second registration threshold To adapt to the speech features of different speakers in the voiceprint feature database and achieve more accurate voiceprint recognition.

[0047] In specific implementation, step S103 involves registering a second speaker and its corresponding voiceprint feature vector in the voiceprint feature database based on the speech segment, and identifying the target speaker corresponding to the speech segment as the second speaker, including: Step a1: Create a temporary speaker for the audio segment and start the observation period.

[0048] In this step, when a new speaker is detected, this embodiment of the application will not immediately register the speaker as a long-term persistent speaker, but will first treat it as a temporary speaker and place it under observation for a period of time.

[0049] Step a2: If multiple speech segments of the temporary speaker that meet the preset quality requirements are collected during the observation period, then the temporary speaker is registered as the second speaker in the voiceprint feature database.

[0050] In this step, voice segments of the temporary speaker are continuously collected during the observation period. If M voice segments (M is a positive integer) that meet the preset quality requirements can be collected, the temporary speaker can be registered as the second speaker in the voiceprint feature database, that is, become a permanent speaker.

[0051] For example, preset quality requirements may include at least one of the following: minimum duration ≥ 0.5 s, in which case speech segments shorter than 0.5 s are directly discarded; speech proportion ≥ 80%, which can be estimated by combining the preceding and following edges of VAD; SNR ≥ 10 dB, which can be approximated by the ratio of energy to silence energy; embedding norm: L2 normalized and checked for numerical stability, not NaN / Inf.

[0052] In this way, by applying a quality threshold to each sample speech segment, a lightweight quality assessment is introduced to ensure that the speaker feature vector is composed of high-quality segments, thus avoiding registering noisy / overlapping speech as new speakers.

[0053] Step a3: Aggregate the voiceprint feature vectors corresponding to multiple speech segments that meet the preset quality requirements and the target voiceprint feature vector of the speech segment to obtain the voiceprint feature vector corresponding to the second speaker.

[0054] In this step, the voiceprint feature vector {e_k} corresponding to the M speech segments collected during the observation period of the temporary speaker and the target voiceprint feature vector c extracted in S102 can be robustly aggregated to generate the voiceprint feature vector (i.e., speaker prototype, centroid) corresponding to the second speaker.

[0055] In one example, the weight of each speech segment is determined based on its speech quality. The second speaker's voiceprint feature vector is then obtained by weighted aggregation using the voiceprint feature vectors and weights corresponding to multiple speech segments, along with the target voiceprint feature vector and weight for that speech segment. The formula is expressed as EMA aggregation: c ← (1-α)·c + α·e_k, where α represents the weight, which is adaptively adjusted according to the sample speech quality (duration, SNR). In this weighted averaging method, higher-quality speech segments (e.g., longer, clearer) have a larger weight.

[0056] or, The speech samples are grouped based on the voiceprint feature vectors corresponding to multiple speech segments and the target voiceprint feature vector of that speech segment. The mean of each voiceprint feature vector in each group is calculated to obtain the average voiceprint feature vector for each group. The median vector among the average voiceprint feature vectors of each group is then determined as the voiceprint feature vector corresponding to the second speaker. This median-of-means aggregation method, which calculates the mean of grouped samples before taking the median, can avoid biasing the results due to certain noisy segments.

[0057] Finally, the aggregated voiceprint feature vector c is L2 normalized and stored in the voiceprint feature library. At this point, the temporary speaker is transformed into a persistent speaker in the voiceprint feature library.

[0058] By setting the observation period and the aggregation method during the observation period, we can avoid misjudging the appearance of new users due to a very short or noisy voice segment; and ensure that each new user registered in the voiceprint feature database is based on sufficiently stable voiceprint information.

[0059] In addition, if a temporary speaker has been converted into a persistent speaker in the voiceprint feature database, but subsequent speech segments show that the persistent speaker is very similar to another speaker and often appears in the same conversation, speaker merging can be triggered, and the relevant dialogue history recognition results can be relabeled (optional offline batch processing).

[0060] Furthermore, the speech recognition method also includes: If, during the observation period, the similarity between the voiceprint feature vector of the temporary speaker's speech segment and the voiceprint feature vector corresponding to any first speaker in the voiceprint feature database is consistently greater than the third retention threshold, then the temporary speaker is modified to become the first speaker.

[0061] Here, speech segments of the temporary speaker can be continuously collected during the observation period, and the target voiceprint feature vector of the speaker can be updated based on the collected speech segments to obtain the voiceprint feature vector of the temporary speaker's speech segments; if this voiceprint feature vector during the observation period matches an existing first speaker in the voiceprint feature database... If the similarity between the corresponding voiceprint feature vectors is consistently greater than the third retention threshold, then the temporary speaker can be merged into the first speaker. To avoid registering new speakers incorrectly.

[0062] Among them, the third retention threshold Less than the first recognition threshold Similarly, this can be done for each speaker. Set the corresponding third retention threshold. ,For example, Third retention threshold This means that once the speaker of a certain speech segment is identified... Subsequent speech segments only need to meet this lower third retention threshold to maintain speaker identification. This prevents frequent shifts in the speaker's identity within the same conversation.

[0063] It is worth noting that, in order to adapt to the speech features of different speakers in the voiceprint feature database and achieve more accurate voiceprint recognition, each first speaker in the voiceprint feature database of this application embodiment... Each threshold is configured with a personalized multi-level threshold, and the threshold can be adaptively adjusted based on the recognition results.

[0064] For each known speaker in the voiceprint feature database The maintainable parameters are: The number of valid speech samples accumulated by the speaker; The speaker's current voiceprint feature vector (prototype); :and The within-class variance estimate can be continuously updated using a sliding update method; for The sufficient distribution parameters of the 'em' can be determined statistically based on the highest similarity of the current 'em' to the 'em' of the 'em' of other speakers' 'em' (prototypes), including the mean of the set of highest similarity. and the standard deviation of the set of highest similarity .

[0065] Among them, the first speaker The corresponding second registration threshold It can be set based on experience.

[0066] Based on the parameters maintained above, the method for determining the first recognition threshold and the third retention threshold for each first speaker in the voiceprint feature database includes: Step b1: For any first speaker, calculate at least one of the following parameters in real time: the number of accumulated valid speech samples. The intra-class variance estimation parameters between effective speech samples and the voiceprint feature vector of the first speaker. And the probability distribution parameter of the false positive for the first speaker; the probability distribution parameter of the false positive refers to the probability distribution parameter of the similarity between the voiceprint feature vectors when other first speakers are mistakenly identified as the first speaker, which can quantify the speaker. The ease with which another speaker can successfully impersonate the speaker.

[0067] Step b2: Determine the first identification threshold based on at least one parameter of the first speaker.

[0068] In practical implementation, the first correction term can be determined based on the accumulated number of valid speech samples and the first correction coefficient, expressed by the formula as follows: .

[0069] Based on the intra-class variance estimation parameters and the second correction coefficient, a second correction term is determined; wherein the intra-class variance estimation parameters are determined using a sliding update method. The formula is expressed as follows: .

[0070] Based on the sufficient distribution parameter of the bluff and the third correction coefficient, a third correction term is determined; wherein, the sufficient distribution parameter of the bluff includes the mean of the set of highest similarity between the voiceprint feature vectors of other first speakers and the voiceprint feature vector of the first speaker. and standard deviation The formula is expressed as follows: .

[0071] The sum of a preset base value and at least one of the first, second, and third correction terms is determined, and the first identification threshold is determined based on the sum, a preset upper limit value, and a preset lower limit value. When all three terms are included, the formula is as follows: .

[0072] in, This represents the preset base value, which is optional. ; Indicates the preset upper limit value; This indicates a preset lower limit value. In one example, , , , , , .

[0073] Step b3: Determine the third retention threshold based on the first identification threshold and the preset hysteresis interval of the first speaker.

[0074] The formula is expressed as: Exemplary .

[0075] It is worth noting that existing technologies often use a fixed threshold for all speakers for identification, such as determining that a similarity greater than 0.6 indicates the same speaker. However, a fixed threshold approach can lead to several problems: when a new user first appears, the sample size is small and the voiceprint information is not yet stable, so if the threshold is too high, it is difficult for them to register in the system; when an experienced user has spoken a lot, their voiceprint information is very stable, so if the threshold is too low, they are easily impersonated.

[0076] The embodiments of this application adaptively adjust the threshold using the above formula: the more samples, the higher the threshold; the more compact the samples within a class, the higher the threshold; the closer the distribution of the caps, the higher the threshold. Therefore, for new users: when the number of samples collected by the system is very small ( When the threshold is relatively low (e.g., around 0.5), it is easier for new users to register in the voiceprint feature database. For existing users: as their sample size increases, the system gradually raises the threshold (e.g., to 0.7 or even higher), making it difficult for others to impersonate them. Simultaneously, the system monitors the risk of impersonation and dynamically adjusts it. If a speaker's voiceprint is found to be very similar to others, that speaker's threshold will automatically increase to prevent confusion. Therefore, the dynamic threshold adjustment method designed to be tailored to each individual in this embodiment ensures that new users can smoothly enter the system while providing more accurate identification of existing users and reducing the likelihood of misjudgment.

[0077] Furthermore, the speech recognition method also includes: If it is detected that the similarity between the voiceprint feature vectors of multiple first speakers in the voiceprint feature library is greater than the merging threshold, and they appear alternately multiple times in the same audio stream, then the multiple first speakers will be merged into the same first speaker.

[0078] If it is detected that the intra-class variance estimation parameter of any first speaker in the voiceprint feature library is greater than the splitting threshold, and the effective speech samples of the first speaker are split into multiple sub-clusters, then the first speaker is split according to the number of sub-clusters.

[0079] As the system operates, the speaker's voiceprint feature database becomes increasingly complex, necessitating merging and splitting mechanisms to maintain data clarity. For merging mechanisms, if the voiceprint features of multiple speakers are very similar—for example, if the similarity between the voiceprint feature vectors of two speakers consistently exceeds the splitting threshold—further compatibility issues arise. If a speaker has a similarity greater than 0.85 and appears repeatedly in the same audio stream with frequent alternation, or even if such alternation occurs in multiple audio streams, the system can automatically merge these multiple speakers into one speaker to prevent one person from being registered repeatedly.

[0080] Regarding the segmentation mechanism, if the internal variation of a speaker's speech samples is too large (e.g., a person's tone is sometimes very high and sometimes very low), the system will detect that the speaker is not consistent enough, and the estimated intra-class variance parameter of the first speaker is greater than the segmentation threshold. If the effective speech samples of the first speaker can be stably split into multiple sub-clusters, for example, they can be divided into two classes through a clustering algorithm, then the first speaker can be automatically segmented into multiple different speakers and assigned IDs to them.

[0081] In this way, the splitting and merging mechanism can ensure that the same speaker is not registered with multiple different IDs, nor can two speakers with very different characteristics be mixed up as one, thus keeping the voiceprint feature database clear and accurate.

[0082] Furthermore, to ensure the stability of recognition results within a session, this speech recognition method can also smooth continuous decisions within the same session connection. Specific methods may include: determining the speaker by voting within the most recent W segments; switching speakers when the cumulative score reaches a threshold; setting a cooling-off period after a new speaker ID is registered, during which splitting / merging is not allowed unless there is strong evidence.

[0083] Please see Figure 2 , Figure 2 This is a second flowchart illustrating a speech recognition method provided in an embodiment of this application. Figure 2 As shown in the embodiments of this application, the speech recognition method includes: The speech recognition system receives and caches the audio stream uploaded by the client in real time; the speech activity detection module extracts at least one speech segment from the audio stream; for each speech segment, the voiceprint feature extraction module extracts features to obtain the target voiceprint feature vector; the voiceprint management module matches the target voiceprint feature vector of the speech segment with the registered voiceprint feature database; if it matches an existing speaker... The highest similarity between them is greater than or equal to that between the speakers. If the first recognition threshold is reached, then an existing speaker ID is associated with it, thus identifying the target speaker corresponding to the speech segment as the speaker. If the highest similarity with each existing speaker is less than the first recognition threshold of each speaker, a new speaker ID is dynamically registered in the voiceprint feature library, such as registering a new second speaker m; the speech recognition module recognizes the text corresponding to the speech segment; the result output module packages the target speaker ID, text and timestamp into a speech recognition result and returns it to the client. The speech recognition method provided in this application has the following beneficial effects: 1. By integrating voice activity detection, voiceprint feature extraction, voiceprint management, speech recognition, and result output, the system performs overall processing on multi-speaker scenarios, enabling real-time automatic separation, annotation, and transcription of multiple speakers. This avoids the inefficiency caused by independent processing of each functional module, thereby improving the system's real-time performance and processing efficiency.

[0084] 2. The system adopts a dynamic speaker management mechanism, which eliminates the need for pre-registration of speakers. When a new speaker's speech segment is detected, the system dynamically registers the new speaker based on the voiceprint feature vector. This breaks through the traditional limitations of voiceprint recognition on new user registration, significantly improving the system's adaptability in dynamic environments and the accuracy of speech recognition in multi-speaker environments.

[0085] 3. The registration and recognition processes are completed within the same processing flow, further improving real-time performance and processing efficiency. Upon detecting a new speaker's audio segment, the system instantly generates a voiceprint vector and dynamically registers it, with a latency of less than 50ms, thereby improving real-time performance and processing efficiency. In practical applications, it is more suitable for multi-user real-time interactive scenarios, such as online meetings, customer service, and collaborative office work.

[0086] 4. Real-time voice activity detection can effectively filter out invalid audio, improve the accuracy of speech recognition, especially in noisy environments, and maintain high recognition accuracy to ensure the reliability of voice interaction.

[0087] 5. Based on lightweight models such as the ResNet34-based speaker embedding model, efficient extraction of speech features and similarity calculation are achieved, reducing the computational complexity of the system, making it suitable for deployment on edge devices, and effectively controlling the consumption of hardware resources.

[0088] 6. By using both WebSocket and HTTP interfaces, a flexible service interface is provided, which simplifies the system integration process, reduces system integration costs, and improves the system's scalability and usability.

[0089] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a speech recognition system provided in an embodiment of this application. Figure 3 As shown, the speech recognition system 300 includes: The voice activity detection module 310 is used to detect voice activity in a real-time received audio stream and extract at least one voice segment from the audio stream. The voiceprint feature extraction module 320 is used to extract the target voiceprint feature vector for each speech segment. The voiceprint management module 330 is used to determine the similarity between the target voiceprint feature vector of the speech segment and each voiceprint feature vector in the registered voiceprint feature library, and register a new second speaker in the voiceprint feature library according to the similarity and determine it as the target speaker corresponding to the speech segment; wherein, the voiceprint feature library includes the original first speaker and the corresponding voiceprint feature vector. The speech recognition module 340 is used to perform speech recognition on the speech segment and generate the text corresponding to the speech segment. The output module 350 generates the speech recognition result of the speech segment based on the identifier of the target speaker and the text corresponding to the speech segment.

[0090] Furthermore, when the voiceprint management module 330 registers a new second speaker in the voiceprint feature database based on the similarity and determines it as the target speaker corresponding to the speech segment, the voiceprint management module 330 is used to: If the highest similarity between the target voiceprint feature vector of the speech segment and the voiceprint feature vector corresponding to each first speaker in the voiceprint feature library is lower than the second registration threshold, then a new second speaker and its corresponding voiceprint feature vector are registered in the voiceprint feature library according to the speech segment, and the target speaker corresponding to the speech segment is determined as the second speaker.

[0091] Furthermore, when the voiceprint management module 330 registers a second speaker and its corresponding voiceprint feature vector in the voiceprint feature database based on the speech segment, and identifies the target speaker corresponding to the speech segment as the second speaker, the voiceprint management module 330 is used to: Create a temporary speaker for this audio segment and start an observation period; If multiple speech segments of the temporary speaker that meet the preset quality requirements are collected during the observation period, the temporary speaker will be registered as the second speaker in the voiceprint feature database. The voiceprint feature vectors corresponding to multiple speech segments that meet the preset quality requirements and the target voiceprint feature vector of the speech segment are aggregated to obtain the voiceprint feature vector corresponding to the second speaker.

[0092] Furthermore, the voiceprint management module 330 is also used for: If the highest similarity between the target voiceprint feature vector of the speech segment and the voiceprint feature vector corresponding to the first speaker in the voiceprint feature database is greater than the first recognition threshold, then the target speaker corresponding to the speech segment is determined as the first speaker corresponding to the highest similarity.

[0093] Furthermore, the voiceprint management module 330 is also used for: If, during the observation period, the similarity between the voiceprint feature vector of the temporary speaker's speech segment and the voiceprint feature vector corresponding to any first speaker in the voiceprint feature database is consistently greater than a third retention threshold, then the temporary speaker is modified to become the first speaker; wherein, the third retention threshold is less than the first recognition threshold.

[0094] Furthermore, when the voiceprint management module 330 aggregates the voiceprint feature vectors corresponding to multiple speech segments that meet preset quality requirements and the target voiceprint feature vector of the speech segment to obtain the voiceprint feature vector corresponding to the second speaker, the voiceprint management module 330 is used to: Based on the speech quality of each speech segment, the weight of each speech segment is determined; according to the voiceprint feature vectors and weights corresponding to multiple speech segments, and the target voiceprint feature vector and weight of that speech segment, a weighted aggregation is performed to obtain the voiceprint feature vector corresponding to the second speaker; or, The speech segments are grouped according to their corresponding voiceprint feature vectors and the target voiceprint feature vector of the speech segment. The mean of each voiceprint feature vector in each group is calculated to obtain the average voiceprint feature vector of each group. The median vector among the average voiceprint feature vectors of each group is determined as the voiceprint feature vector of the second speaker.

[0095] Furthermore, the voiceprint management module 330 is also used to determine a first recognition threshold and a third retention threshold corresponding to each first speaker in the voiceprint feature database; specifically, the voiceprint management module 330 is used to: For any first speaker, at least one of the following parameters is calculated in real time: the number of accumulated valid speech samples, the intra-class variance estimation parameter between the valid speech samples and the speaker's voiceprint feature vector, and the false full number distribution parameter for the first speaker; the false full number distribution parameter refers to the probability distribution parameter of the similarity between voiceprint feature vectors when other first speakers are misidentified as the first speaker. The first identification threshold is determined based on at least one parameter of the first speaker; The third retention threshold is determined based on the first identification threshold and the preset hysteresis interval.

[0096] Furthermore, when the voiceprint management module 330 determines the first recognition threshold based on at least one parameter of the first speaker, the voiceprint management module 330 is used to: Based on the accumulated number of valid speech samples and the first correction coefficient, the first correction term is determined; Based on the intra-class variance estimation parameters and the second correction coefficient, a second correction term is determined; wherein the intra-class variance estimation parameters are determined using a sliding update method. Based on the sufficient distribution parameters of the 'sauger' and the third correction coefficient, a third correction term is determined; wherein, the sufficient distribution parameters of the 'sauger' include the mean and standard deviation of the set of highest similarity between the voiceprint feature vectors of other first speakers and the voiceprint feature vector of the first speaker; The sum of a preset base value and at least one of the first correction term, the second correction term, and the third correction term is determined, and the first identification threshold is determined based on the sum, the preset upper limit value, and the preset lower limit value.

[0097] Furthermore, the voiceprint management module 330 is also used for: If it is detected that the similarity between the voiceprint feature vectors of multiple first speakers in the voiceprint feature library is greater than the merging threshold, and they appear alternately multiple times in the same audio stream, then the multiple first speakers will be merged into the same first speaker. If it is detected that the intra-class variance estimation parameter of any first speaker in the voiceprint feature library is greater than the splitting threshold, and the effective speech samples of the first speaker are split into multiple sub-clusters, then the first speaker is split according to the number of sub-clusters.

[0098] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0099] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1 as well as Figure 2 The steps of the speech recognition method in the illustrated method embodiment can be found in the method embodiment for specific implementation methods, which will not be repeated here.

[0100] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 as well as Figure 2 The steps of the speech recognition method in the illustrated method embodiment can be found in the method embodiment for specific implementation methods, which will not be repeated here.

[0101] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0105] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A speech recognition method, characterized in that, The method includes: Speech activity detection is performed on the real-time received audio stream, and at least one speech segment is extracted from the audio stream; For each speech segment, extract the target voiceprint feature vector for that speech segment; The similarity between the target voiceprint feature vector of the speech segment and each voiceprint feature vector in the registered voiceprint feature library is determined. Based on the similarity, a new second speaker is registered in the voiceprint feature library and determined as the target speaker corresponding to the speech segment. The voiceprint feature library includes the original first speaker and the corresponding voiceprint feature vector. Perform speech recognition on the speech segment and generate the corresponding text. Based on the identifier of the target speaker corresponding to the speech segment and the text, the speech recognition result of the speech segment is generated.

2. The method according to claim 1, characterized in that, Based on the similarity, a new second speaker is registered in the voiceprint feature database and identified as the target speaker corresponding to the speech segment, including: If the highest similarity between the target voiceprint feature vector of the speech segment and the voiceprint feature vector corresponding to each first speaker in the voiceprint feature library is lower than the second registration threshold, then a new second speaker and its corresponding voiceprint feature vector are registered in the voiceprint feature library according to the speech segment, and the target speaker corresponding to the speech segment is determined as the second speaker.

3. The method according to claim 2, characterized in that, Based on the speech segment, a second speaker and its corresponding voiceprint feature vector are newly registered in the voiceprint feature database, and the target speaker corresponding to the speech segment is identified as the second speaker, including: Create a temporary speaker for this audio segment and start an observation period; If multiple speech segments of the temporary speaker that meet the preset quality requirements are collected during the observation period, the temporary speaker will be registered as the second speaker in the voiceprint feature database. The voiceprint feature vectors corresponding to multiple speech segments that meet the preset quality requirements and the target voiceprint feature vector of the speech segment are aggregated to obtain the voiceprint feature vector corresponding to the second speaker.

4. The method according to claim 3, characterized in that, The method further includes: If the highest similarity between the target voiceprint feature vector of the speech segment and the voiceprint feature vector corresponding to the first speaker in the voiceprint feature database is greater than the first recognition threshold, then the target speaker corresponding to the speech segment is determined as the first speaker corresponding to the highest similarity.

5. The method according to claim 4, characterized in that, The method further includes: If, during the observation period, the similarity between the voiceprint feature vector of the temporary speaker's speech segment and the voiceprint feature vector corresponding to any first speaker in the voiceprint feature database is consistently greater than a third retention threshold, then the temporary speaker is modified to become the first speaker; wherein, the third retention threshold is less than the first recognition threshold.

6. The method according to claim 3, characterized in that, The voiceprint feature vectors corresponding to multiple speech segments that meet the preset quality requirements and the target voiceprint feature vector of the speech segment are aggregated to obtain the voiceprint feature vector corresponding to the second speaker, including: Based on the speech quality of each speech segment, the weight of each speech segment is determined; according to the voiceprint feature vectors and weights corresponding to multiple speech segments, and the target voiceprint feature vector and weight of that speech segment, a weighted aggregation is performed to obtain the voiceprint feature vector corresponding to the second speaker; or, The speech segments are grouped according to their corresponding voiceprint feature vectors and the target voiceprint feature vector of the speech segment. The mean of each voiceprint feature vector in each group is calculated to obtain the average voiceprint feature vector of each group. The median vector among the average voiceprint feature vectors of each group is determined as the voiceprint feature vector of the second speaker.

7. The method according to claim 5, characterized in that, The methods for determining the first recognition threshold and the third retention threshold for each first speaker in the voiceprint feature database include: For any first speaker, at least one of the following parameters is calculated in real time: the number of accumulated valid speech samples, the intra-class variance estimation parameter between the valid speech samples and the speaker's voiceprint feature vector, and the false full number distribution parameter for the first speaker; the false full number distribution parameter refers to the probability distribution parameter of the similarity between voiceprint feature vectors when other first speakers are misidentified as the first speaker. The first identification threshold is determined based on at least one parameter of the first speaker; The third retention threshold is determined based on the first identification threshold and the preset hysteresis interval.

8. The method according to claim 7, characterized in that, Determining the first identification threshold based on at least one parameter of the first speaker includes: Based on the accumulated number of valid speech samples and the first correction coefficient, the first correction term is determined; Based on the intra-class variance estimation parameters and the second correction coefficient, a second correction term is determined; wherein the intra-class variance estimation parameters are determined using a sliding update method. Based on the sufficient distribution parameters of the 'sauger' and the third correction coefficient, a third correction term is determined; wherein, the sufficient distribution parameters of the 'sauger' include the mean and standard deviation of the set of highest similarity between the voiceprint feature vectors of other first speakers and the voiceprint feature vector of the first speaker; The sum of a preset base value and at least one of the first correction term, the second correction term, and the third correction term is determined, and the first identification threshold is determined based on the sum, the preset upper limit value, and the preset lower limit value.

9. The method according to claim 1, characterized in that, The method further includes: If it is detected that the similarity between the voiceprint feature vectors of multiple first speakers in the voiceprint feature library is greater than the merging threshold, and they appear alternately multiple times in the same audio stream, then the multiple first speakers will be merged into the same first speaker. If it is detected that the intra-class variance estimation parameter of any first speaker in the voiceprint feature library is greater than the splitting threshold, and the effective speech samples of the first speaker are split into multiple sub-clusters, then the first speaker is split according to the number of sub-clusters.

10. A speech recognition system, characterized in that, The system includes: The voice activity detection module is used to detect voice activity in the real-time received audio stream and extract at least one voice segment from the audio stream. The voiceprint feature extraction module is used to extract the target voiceprint feature vector for each speech segment. The voiceprint management module is used to determine the similarity between the target voiceprint feature vector of the speech segment and each voiceprint feature vector in the registered voiceprint feature library, and register a new second speaker in the voiceprint feature library according to the similarity and determine it as the target speaker corresponding to the speech segment; wherein, the voiceprint feature library includes the original first speaker and the corresponding voiceprint feature vector. The speech recognition module is used to perform speech recognition on the speech segment and generate the corresponding text for the speech segment. The output module generates the speech recognition result for the speech segment based on the identifier of the target speaker and the text corresponding to the speech segment.

Images