Speech recognition method, device and electronic equipment for multi-speaker environment

Abstract

The application provides a speech recognition method, device and electronic equipment for a multi-speaker environment. The method comprises: obtaining audio data; using a voice activity detection technology to mark the start and end times of each voice in the audio data to obtain an audio marking result, and then using an automatic speech recognition technology to transcribe the audio marking result to obtain a transcription text set corresponding to the audio data, the transcription text set comprising a plurality of audio text segments, and the audio text segments being marked with start and end times; using a speaker separation technology to perform clustering analysis on the audio data to obtain a segment data set grouped by speakers, wherein the segment data set comprises a plurality of segment data recording the start and end times of the segments and the speaker numbers; and using a time alignment algorithm to fuse the transcription text set and the segment data set to obtain a final recognition result of the audio data. The application solves the problem of low recognition accuracy of existing speech recognition technologies in a multi-person simultaneous speaking scenario.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology for multi-speaker environments, and more specifically, to a speech recognition method, apparatus, computer-readable storage medium, and electronic device for multi-speaker environments. Background Technology

[0002] With the rapid development of speech recognition technology, speech transcription systems have been widely used in scenarios such as real-time conferencing, remote communication, and high-precision post-meeting transcription. In these applications, multiple speakers often take turns speaking, and accurately identifying the content and duration of each speaker's speech has become a key challenge for speech recognition systems. However, in multi-speaker environments, traditional ASR methods often exhibit limitations in handling speaker alternation, short pauses, and background noise, potentially leading to recognition errors and difficulty in accurately locating the speaker for each segment of speech.

[0003] To address this issue, speaker diarization (SD) technology has been gradually introduced into speech recognition systems in recent years, aiming to improve recognition accuracy in multi-speaker scenarios. Despite the powerful capabilities of SD technology, speech recognition using only SD technology still suffers from inaccurate recognition in scenarios where multiple people speak simultaneously. Summary of the Invention

[0004] The main objective of this application is to provide a speech recognition method, apparatus, computer-readable storage medium, and electronic device for multi-speaker environments, so as to at least solve the problem of low recognition accuracy of existing speech recognition technologies in scenarios where multiple people speak at the same time.

[0005] To achieve the above objectives, according to one aspect of this application, a speech recognition method for multi-speaker environments is provided, comprising: an acquisition step of acquiring audio data; a first processing step of using speech activity detection technology to calibrate the start and end times of each speech segment in the audio data to obtain an audio calibration result, and then using automatic speech recognition technology to transcribe the audio calibration result to obtain a transcribed text set corresponding to the audio data, wherein the transcribed text set includes multiple audio text segments, and the audio text segments are marked with start and end times; a second processing step of using speaker separation technology to perform cluster analysis on the audio data to obtain a segment dataset grouped by speaker, wherein the segment dataset includes multiple segment data recording the start and end times of the segments and speaker numbers; and a third processing step of using a time alignment algorithm to fuse the transcribed text set and the segment dataset to obtain the final recognition result of the audio data.

[0006] Optionally, a time alignment algorithm is used to fuse the transcribed text set and the fragment dataset to obtain the final recognition result of the audio data, including: a traversal step: obtaining the audio text fragments in the transcribed text set, traversing the fragment dataset, and determining the fragment data corresponding to the audio text fragments based on the start and end times of the audio text fragments; a numbering step: numbering the audio text fragments according to the speaker numbers of the fragment data to obtain speech recognition fragments; and executing the traversal step and the numbering step multiple times until all the audio text fragments in the transcribed text set have been traversed to obtain the final recognition result.

[0007] Optionally, determining the segment data corresponding to the audio text segment includes: determining at least one segment data that overlaps with the time of the audio text segment based on the start and end time of the audio text segment and the start and end time of the segment data; determining whether the number of segment data that overlaps with the time of the audio text segment is 1; and if the number of segment data is 1, determining that this segment data corresponds to the audio text segment.

[0008] Optionally, determining the segment data corresponding to the audio text segment further includes: if the number of segment data that overlaps with the audio text segment in time is greater than 1, calculating the weight of each segment data that overlaps with the audio text segment in time based on the overlap duration between the segment data and each audio text segment; and taking the segment data with the largest weight as the segment data corresponding to the audio text segment.

[0009] Optionally, the weight of each segment data that overlaps with the audio text segment is calculated based on the overlap duration between the segment data and the audio text segment, including: according to the formula: Determine the weights of each of the data segments, where ω i The weight of the segment data, slice_duration is the overlap duration between the segment data and the audio text segment, and seg_duration is the weight of the segment data. m The total duration of the audio text segment.

[0010] Optionally, after obtaining the fragment dataset grouped by speaker, the method further includes: determining whether the start and end times of the fragment dataset cover the start and end times of the transcribed text set; if the start and end times of the transcribed text set are not covered, continuing to use the speaker separation technique to obtain fragment data.

[0011] Optionally, the first processing step and the second processing step are executed in parallel.

[0012] According to another aspect of this application, a speech recognition device for multi-speaker environments is provided, comprising: a first acquisition unit for performing an acquisition step to acquire audio data; a first processing unit for performing a first processing step, using speech activity detection technology to calibrate the start and end times of each speech in the audio data to obtain an audio calibration result, and then using automatic speech recognition technology to transcribe the audio calibration result to obtain a transcribed text set corresponding to the audio data, wherein the transcribed text set includes multiple audio text segments, and the audio text segments are marked with start and end times; a second processing unit for performing a second processing step, using speaker separation technology to perform cluster analysis on the audio data to obtain a segment dataset grouped by speaker, wherein the segment dataset includes multiple segment data recording the start and end times of segments and speaker numbers; and a third processing unit for performing a third processing step, using a time alignment algorithm to fuse the transcribed text set and the segment dataset to obtain the final recognition result of the audio data.

[0013] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform any of the aforementioned speech recognition methods for multi-speaker environments.

[0014] According to another aspect of this application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for performing any of the described speech recognition methods for multi-speaker environments.

[0015] The technical solution of this application involves the following steps: First, acquiring audio data; second, using speech activity detection technology to calibrate the start and end times of each speech segment in the audio data, obtaining audio calibration results; third, using automatic speech recognition technology to transcribe the audio calibration results, obtaining a transcribed text set corresponding to the audio data, wherein the transcribed text set includes multiple audio text segments, and each audio text segment is marked with its start and end times; fourth, using speaker separation technology to perform cluster analysis on the audio data, obtaining a segment dataset grouped by speaker, wherein the segment dataset includes multiple segment data recording the start and end times of segments and speaker numbers; fifth, using a time alignment algorithm to fuse the transcribed text set and the segment dataset, obtaining the final recognition result of the audio data. The combined use of speech activity detection, automatic speech recognition, speaker separation, and time alignment algorithms significantly improves the accuracy and efficiency of recognition. Speech activity detection technology ensures accurate labeling of speech segments, automatic speech recognition technology provides rapid text transcription, speaker separation technology enables the separation of speech from different speakers, and time alignment algorithm ensures the accuracy of the correspondence between transcribed text and speaker. This method is particularly suitable for scenarios such as meeting minutes, court records, and large-scale multi-person discussions, and can effectively distinguish the speeches of multiple speakers, thereby improving the practical value of speech recognition and user experience. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 A hardware block diagram of a mobile terminal for performing a speech recognition method for a multi-speaker environment is shown in an embodiment of this application.

[0018] Figure 2 A schematic flowchart of a speech recognition method for multi-speaker environments provided according to an embodiment of this application is shown.

[0019] Figure 3 A flowchart illustrating a specific speech recognition method for multi-speaker environments provided according to an embodiment of this application is shown.

[0020] Figure 4 A schematic diagram of a specific speech recognition system for multi-speaker environments provided according to embodiments of this application is shown;

[0021] Figure 5 A structural block diagram of a speech recognition device for multi-speaker environments provided according to an embodiment of this application is shown.

[0022] The above figures include the following reference numerals:

[0023] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0028] Automatic Speech Recognition (ASR) is a technology that converts speech signals into text, aiming to enable computers to understand and process human spoken language. ASR systems typically rely on complex models and algorithms to analyze speech signals, extract features, and match them with pre-trained speech models to generate corresponding text. Common ASR applications include intelligent assistants, speech-to-text systems, and voice-controlled devices.

[0029] Speaker Diarization (SD) is a technique used in multi-speaker environments to mark and differentiate speech segments from different speakers in an audio file.

[0030] Voice Activity Detection (VAD) is a technique used to detect the presence of speech activity in an audio signal, distinguishing between speech segments and non-speech segments (such as silence, noise, etc.).

[0031] As described in the background section, existing speech recognition technologies have low accuracy in scenarios where multiple people speak simultaneously. To address this issue, embodiments of this application provide a speech recognition method, apparatus, computer-readable storage medium, and electronic device for multi-speaker environments.

[0032] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0033] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a speech recognition method in a multi-speaker environment according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0034] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the speech recognition method for multi-speaker environments in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0035] This embodiment provides a speech recognition method for multi-speaker environments that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0036] Figure 2 This is a flowchart of a speech recognition method for multi-speaker environments according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0037] Step S201, Acquisition Step, Acquire Audio Data;

[0038] Step S202, the first processing step, uses speech activity detection technology to calibrate the start and end times of each speech in the above audio data to obtain audio calibration results, and then uses automatic speech recognition technology to transcribe the above audio calibration results to obtain the transcribed text set corresponding to the above audio data, wherein the transcribed text set includes multiple audio text segments, and the above audio text segments are marked with start and end times.

[0039] Specifically, Voice Activity Detection (VAD) technology is mainly used to identify the speech and non-speech components in audio. By analyzing features of the audio signal, such as energy and zero-crossing rate, it determines the start and end times of the speech. Automatic Speech Recognition (ASR) technology then converts these speech segments into text.

[0040] Step S203, the second processing step, uses speaker separation technology to perform cluster analysis on the above audio data to obtain a segment dataset grouped by speaker, wherein the above segment dataset includes multiple segment data that record the start and end times of the segments and speaker numbers;

[0041] Specifically, speaker separation techniques, such as speaker recognition models in deep learning, can distinguish the voices of different speakers and effectively separate them even when they are speaking in overlapping sentences.

[0042] Step S204, the third processing step, uses a time alignment algorithm to fuse the above transcribed text set and the above fragment dataset to obtain the final recognition result of the above audio data.

[0043] Specifically, time alignment algorithms, such as Dynamic Time Warping (DTW) or Hidden Markov Models (HMMs), are used to match and align the timestamps of transcribed text with speaker segments to ensure the accuracy of the recognition results.

[0044] This embodiment acquires audio data through the following steps: First, speech activity detection technology is used to calibrate the start and end times of each speech segment in the audio data, obtaining an audio calibration result. Then, automatic speech recognition technology is used to transcribe the audio calibration result, obtaining a transcribed text set corresponding to the audio data. The transcribed text set includes multiple audio text segments, each labeled with its start and end times. Second, speaker separation technology is used to perform cluster analysis on the audio data, obtaining a segment dataset grouped by speaker. This segment dataset includes multiple segments recording their start and end times and speaker numbers. Third, a time alignment algorithm is used to fuse the transcribed text set and the segment dataset, obtaining the final recognition result of the audio data. The combined use of speech activity detection, automatic speech recognition, speaker separation, and time alignment algorithms significantly improves the accuracy and efficiency of recognition. Speech activity detection technology ensures accurate calibration of speech segments, automatic speech recognition technology provides rapid text transcription, speaker separation technology separates the speech of different speakers, and the time alignment algorithm ensures the accuracy of the correspondence between the transcribed text and the speaker. This method is particularly suitable for scenarios such as meeting minutes, court records, and large-scale discussions with multiple people. It can effectively distinguish the speech of multiple speakers, thereby improving the practical value of speech recognition and user experience.

[0045] This method is not limited to the combination of the above-mentioned technical features. It can also further integrate preprocessing techniques such as noise suppression and echo cancellation, as well as post-processing techniques such as text correction and semantic understanding, to enhance the adaptability to complex environments and the usability of the recognition results.

[0046] In the specific implementation process, a time alignment algorithm is used to fuse the above-mentioned transcribed text set and the above-mentioned fragment dataset to obtain the final recognition result of the above-mentioned audio data. The process includes: a traversal step: obtaining the above-mentioned audio text fragments in the above-mentioned transcribed text set, traversing the above-mentioned fragment dataset, and determining the fragment data corresponding to the above-mentioned audio text fragments based on the start and end times of the above-mentioned audio text fragments; a numbering process step: numbering the above-mentioned audio text fragments according to the speaker numbers of the above-mentioned fragment data to obtain speech recognition fragments; and executing the above-mentioned traversal step and the above-mentioned numbering process step multiple times until all the above-mentioned audio text fragments in the above-mentioned transcribed text set have been traversed to obtain the above-mentioned final recognition result.

[0047] The time alignment algorithm in this method is crucial for ensuring correct matching between transcribed text and speaker segments. During the traversal, the algorithm compares the start and end times of the audio text segment with each segment in the segment dataset, finding the most temporally matching speaker segment. By mapping speaker numbers, the identified text segment can be categorized under the correct speaker, thus constructing a complete recognition result. This method effectively handles time alignment problems in multi-speaker environments, maintaining high recognition accuracy even with frequent speaker alternation and speech overlap.

[0048] The implementation of time alignment algorithms can include, but is not limited to, Dynamic Time Warping (DTW), Hidden Markov Models (HMM), and deep learning-based time series alignment models. These algorithms can flexibly adjust the alignment strategy according to the needs of different scenarios, thereby further improving the accuracy and robustness of recognition.

[0049] Specifically, determining the segment data corresponding to the aforementioned audio text segment includes: determining at least one segment data that overlaps with the time of the aforementioned audio text segment based on the start and end time of the aforementioned audio text segment and the start and end time of the aforementioned segment data; determining whether the number of the aforementioned segment data that overlaps with the time of the aforementioned audio text segment is 1; and determining that this segment data corresponds to the aforementioned audio text segment if the number of the aforementioned segment data is 1.

[0050] This method, when determining the correspondence between fragment data and audio text fragments, first filters possible matching fragments by temporal overlap. If only one fragment data point overlaps temporally with an audio text fragment, then that fragment data point is the correct match. This method is simple and efficient, quickly locating the correct speaker fragment, reducing computational load, and improving processing speed. Furthermore, fragment data can also be determined by calculating the similarity between the fragment data and the audio text fragment, such as through deep learning-based similarity calculation, or by combining contextual information for intelligent matching, thereby improving recognition accuracy in complex environments.

[0051] More specifically, determining the segment data corresponding to the aforementioned audio text segment further includes: when the number of segment data that overlaps with the time of the aforementioned audio text segment is greater than 1, calculating the weight of each segment data that overlaps with the time of the aforementioned audio text segment based on the overlap duration between the aforementioned segment data and each of the aforementioned audio text segments; and taking the segment data with the largest weight as the segment data corresponding to the aforementioned audio text segment.

[0052] This method determines the best-matching segment when multiple data segments overlap with audio-text segments in time. It calculates the weight of the overlap duration, effectively handling situations where speakers alternate or overlap. By quantifying the degree of time overlap, it ensures the accuracy of the recognition results. The weight calculation is based on the ratio of the overlap duration between the data segment and the audio-text segment to the total duration of the audio-text segment. This considers not only the degree of time matching but also the integrity of the audio-text segment, thus improving the reliability of the recognition. Furthermore, in addition to weight calculation based on time overlap, it can combine audio features (such as frequency and pitch) and semantic analysis of the text content to further optimize the segment matching process. This makes it applicable to a wider range of scenarios, such as meeting minutes in noisy environments and discussions in multilingual environments.

[0053] Furthermore, based on the overlap duration between the aforementioned segment data and the aforementioned audio text segment, the weight of each of the aforementioned segment data that overlaps with the aforementioned audio text segment in time is calculated, including: according to the formula: Determine the weights of each of the above data segments, where ω i The weights of the above segment data are defined as follows: slice_duration is the overlap duration between the above segment data and the above audio text segment, and seg_duration is the weight of the above segment data. m This represents the total duration of the aforementioned audio text segment.

[0054] This method calculates the weights of segment data using the formula described above, quantifying the temporal matching degree between the segment data and the audio text segment. A higher weight value indicates a higher degree of matching between the segment data and the audio text segment. This method is simple and intuitive, enabling rapid evaluation of the matching of multiple segment data, providing a reliable basis for the final recognition result. Furthermore, the weight calculation formula can be further optimized; for example, by incorporating speaker feature similarity calculations or considering the contextual information of the audio text segment, to adapt to more complex and varied recognition environments and improve recognition accuracy and efficiency.

[0055] Furthermore, after obtaining the fragment dataset grouped by speaker, the method further includes: determining whether the start and end times of the fragment dataset cover the start and end times of the transcribed text set; if the start and end times of the transcribed text set are not covered, continuing to use the speaker separation technique to obtain fragment data.

[0056] After obtaining an ASR segment, the system checks whether the current SD (Speaker Segmentation) results are sufficient to cover that segment. If there are insufficient SD segments, the system retrieves more SD segmentation results using a function that retrieves SD results. If not enough SD results are obtained, the system considers the speaker segmentation task problematic and terminates processing. To address the issue of insufficient SD segments, the system calls a function that retrieves SD results. This typically involves interacting with the SD service to request more SD processing results to cover the time range of ASR text segments for which speakers have not yet been assigned. For example, if the ASR identifies a 4-second segment of text, but the SD service currently only has 2 seconds of speaker information, the system will request the SD service to continue processing the remaining 2 seconds of audio until enough SD results are returned to cover the entire time range of the ASR text segment.

[0057] Additionally, if the system attempts to acquire more SD results but fails, or if the SD service cannot provide a sufficient timeframe to cover all ASR segments, the system will consider the speaker segmentation task problematic. This problem may stem from delays in SD processing, limitations of the SD algorithm itself, or poor input audio quality, causing the system to fail to accurately or completely identify the speaker. In such cases, to avoid outputting inaccurate or incomplete results, the system will terminate the current speaker ID assignment process. This may mean temporarily halting the fusion of ASR and SD results until the problem is resolved or other remedial measures are taken.

[0058] By checking the integrity of the segment dataset, this method ensures that all audio text segments have corresponding speaker segments, avoiding recognition omissions caused by incomplete segment datasets. This approach effectively improves the completeness of the recognition results, guaranteeing accurate identification and recording of all speakers' statements even in long recordings. Integrity checks can include, but are not limited to, time coverage checks and speaker feature checks to ensure the comprehensiveness and accuracy of the recognition process. It is suitable for scenarios with long recordings and frequent speaker alternation, such as long meeting transcripts and court trial records.

[0059] Specifically, the first processing step and the second processing step described above are executed in parallel.

[0060] This method significantly improves the efficiency of speech processing in real-time conferencing scenarios through parallel ASR and SD result acquisition and ASR and SD result merging processing mechanisms. It is particularly suitable for large-scale real-time audio data processing and supports high-concurrency real-time conferencing transcription applications.

[0061] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the speech recognition method for multi-speaker environments of this application will be described in detail below with reference to specific embodiments.

[0062] This embodiment relates to a specific speech recognition method for multi-speaker environments, aiming to achieve efficient and accurate multi-speaker speech recognition. First, it utilizes VAD technology to determine the start and end times of speech in the audio in real time, and then applies speech recognition technology to obtain the transcribed text of the speech. Simultaneously, it uses SD technology to separate segments from different speakers. Finally, the method in this system is used to tag each ASR recognition result segment with a specific speaker, achieving fusion of ASR and SD results to obtain clear and accurate recognition results for each speaker. This method can be applied to real-time meeting transcription or high-precision post-meeting transcription scenarios, and is particularly suitable for multi-speaker meeting scenarios. Specifically, it includes the following:

[0063] In this embodiment, the merging module is responsible for merging the outputs of the speech recognition module (ASR module) and the speaker segmentation module (SD module). The specific logical flow is as follows: The entire technical solution flow is as follows: Figure 3 As shown:

[0064] Step 1, Streaming Speech Recognition (ASR) Request: The service receives the audio sent by the client into the VAD and ASR services, and obtains the recognition result of streaming ASR. The result includes the text, the start time and end time of the current speech segment, and puts it into the ASR result queue.

[0065] Step 2: Obtain the Automatic Speech Recognition (ASR) segment: The system retrieves the recognition result of a speech segment from the ASR result queue. The result includes the recognized text, the start time of the segment, the end time, and other metadata. Simultaneously, the system determines whether this segment is the last segment (is_asr_eof). If it is the last segment, the speech recognition task is considered complete.

[0066] Step 3, Streaming Speaker Recognition (SD) Request: While performing Step 1, send the audio to the SD service. The result contains the speaker ID, the start time and end time of the speaker's speech in the SD audio segment, and put the SD service result into the SD result queue.

[0067] Step 4: Check Speaker Segmentation (SD) Fragments: After obtaining the ASR fragment, the system checks whether the current SD results have sufficient time to cover the ASR fragment. If there are insufficient SD fragments, more SD segmentation results are obtained using the function that retrieves SD results. If not enough SD results can be obtained, the speaker segmentation task is considered to have a problem, and processing is terminated.

[0068] Step 5. Determining speaker ID: On the premise of ensuring sufficient SD results, the system calculates the speaker ID corresponding to the start time and end time of the ASR segment through a time alignment algorithm. In this implementation, the system uses a "speaker ID discrimination" function to accurately determine the speaker identity in multi-speaker scenarios. The key steps of this function include the following aspects:

[0069] 1. Input the start time and end time of the current ASR recognition segment, and the start time and end time of the SD segment;

[0070] 2. Calculate segment overlap:

[0071] 1) The code first traverses each speaker segment obtained by speaker diarization according to the time range of the current ASR segment to find overlapping time segments.

[0072] 2) For the time segment (sd_begin_time, sd_end_time) of each speaker i and the m-th segment (begin_time, end_time) of ASR, calculate the start time and end time of the overlapping part:

[0073] slice_begin_time=max(sd_begin_time,begin_time);

[0074] slice_end_time=min(sd_end_time,end_time);

[0075] When slice_begin_time < slice_end_time, the duration of the overlapping segment is:

[0076] slice_duration=slice_end_time-slice_begin_time;

[0077] 3) Through the time segments of SD, find the speaker time period overlapping with the ASR segment, and calculate the time proportion of speaker i in this ASR segment: seg_duration m is the total duration of the current ASR segment.

[0078] 3. Record weights:

[0079] For each overlapping speaker segment, its weight is accumulated into a hash speaker_weight_map. This hash table records each speaker ID and its corresponding weight value.

[0080] Select the speaker with the largest weight:

[0081] 1) After traversing all speaker segments, the system selects the speaker with the highest weight as the speaker for the current ASR segment by comparing weight values.

[0082] 2) Find the element with the largest weight in the speaker_weight_map and obtain the corresponding speaker ID.

[0083] Establish speaker mapping:

[0084] 1) The speaker's ID will be mapped to a unique identification ID. estimated The mapping method is the speaker ID returned by SD. SD Each key and its corresponding ID in the system is mapped one-to-one for future use. If the speaker's ID has not appeared before, a new ID is assigned to it. estimated Numbering starts from 0.

[0085] 2) This ID estimated It will be stored in session→speaker_id_map so that the same ID can be used to identify the same speaker later.

[0086] 4. Result Return: Finally, ID estimated It will be output as the speaker ID of the ASR segment, and the ID, text, start time and end time of the segment will be returned to the requesting client as the result.

[0087] To improve speech recognition and speaker identification in real-time conferencing systems, the streaming VAD and speaker segmentation and merging system proposed in this embodiment can be used to encapsulate a service based on the gRPC standard protocol. This system's service connects to the streaming ASR service with VAD and the streaming SD service via the gRPC protocol in two separate threads. It acquires ASR and SD result requests in real time, queues the results separately, and merges the results in the other thread.

[0088] Example of an ASR results queue:

[0089] {"text":"Welcome everyone to today's meeting","start_time":0,"end_time":4000,"is_asr_eof":false};

[0090] {"text":"Let's first discuss this quarter's sales performance","start_time":4500,"end_time":10000,"is_asr_eof":false};

[0091] {"text":"The following is the progress report from the R&D department","start_time":10500,"end_time":14000,"is_asr_eof":false};

[0092] {"text":"Finally, we summarize today's decisions","start_time":14500,"end_time":18000,"is_asr_eof":true};

[0093] Example of an SD result queue:

[0094] {"speaker_id":"0","start_time":0,"end_time":3000};

[0095] {"speaker_id":"2","start_time":3000,"end_time":8000};

[0096] {"speaker_id":"0","start_time":8000,"end_time":11000};

[0097] {"speaker_id":"1","start_time":11000,"end_time":16000};

[0098] {"speaker_id":"0","start_time":16000,"end_time":18000};

[0099] Example of processing and merging results:

[0100] 1) For ASR segment 1 ("Welcome everyone to today's meeting"), find overlapping SD results within the time range of 0 to 4000:

[0101] Overlapping SD clips:

[0102] {"speaker_id":"0","start_time":0,"end_time":3000}: Overlap time 3000 milliseconds, weight 0.75;

[0103] {"speaker_id":"2","start_time":3000,"end_time":4000}: Overlap time 1000 milliseconds, weight 0.25;

[0104] Speaker_id: 0 is selected as the speaker because 0 has a longer overlap time and a greater weight. The speaker ID in this system is obtained. Since there is no speaker yet, the result ID_sd = 0 returned by SD is also mapped to the speaker ID of this system as 0, ID = 0.

[0105] 2) For ASR segment 2 ("Let's first discuss this quarter's sales performance"), with a time range of 4500 to 10000, compare with the SD results:

[0106] Overlapping SD clips:

[0107] {"speaker_id":"2","start_time":4500,"end_time":8000}: Overlap time is 3500 milliseconds, weight is approximately 0.64;

[0108] {"speaker_id":"0","start_time":8000,"end_time":10000}: Overlap time 2000 milliseconds, weight approximately 0.37;

[0109] The speaker_id is chosen as 2 because it has a longer overlap time and a greater weight. This gives the speaker ID within the system. The maximum system ID is 0, and ID_sd=2 cannot be found in the system map. Therefore, the result returned by SD, ID_sd=2, maps to a speaker ID of 1 in this system, resulting in ID=1.

[0110] 3) For ASR segment 3 ("Next is the R&D department's progress report"), with a time range of 10500 to 14000, compare with the SD results:

[0111] Overlapping SD clips:

[0112] {"speaker_id":"1","start_time":11000,"end_time":14000}: Overlap time 3000 milliseconds;

[0113] Select speaker_id:1, the corresponding ID in this system is 2.

[0114] 4) For ASR segment 4 ("Finally, we summarize today's decisions"), with a time range of 14500 to 18000, compare with the SD results:

[0115] Overlapping SD clips:

[0116] {"speaker_id":"2","start_time":14500,"end_time":16000}: Overlap time 1500 milliseconds;

[0117] {"speaker_id":"0","start_time":16000,"end_time":18000}: Overlap time 2000 milliseconds;

[0118] Select speaker_id:0.

[0119] 5) The final result of the merger:

[0120] Based on the above analysis, the merged result is as follows:

[0121] {"speaker_id":"0","text":"Welcome everyone to today's meeting","start_time":0,"end_time":4000};

[0122] {"speaker_id":"1","text":"Let's first discuss this quarter's sales performance","start_time":4500,"end_time":10000};

[0123] {"speaker_id":"2","text":"Next is the R&D department's progress report","start_time":10500,"end_time":14000};

[0124] {"speaker_id":"0","text":"Finally, we summarize today's decisions","start_time":14500,"end_time":18000};

[0125] This process combines ASR and SD information to ensure that text fragments are correctly matched with their corresponding speakers, and the speaker ID of each ASR fragment is immediately returned to the client to achieve low latency. The entire implementation process is as follows: Figure 4 As shown.

[0126] This embodiment has the following advantages:

[0127] 1. High-efficiency parallel processing: Through the parallel ASR and SD result acquisition and ASR and SD result merging processing mechanism, this system can significantly improve the efficiency of speech processing in real-time conferencing scenarios. It is particularly suitable for large-scale real-time audio data processing and supports high-concurrency real-time conferencing transcription applications.

[0128] 2. Intelligent merging of ASR and SD fragments: By merging the ASR and SD results, the problem of fragment inconsistency is avoided, ensuring the integrity of the ASR recognition results.

[0129] 3. Time Alignment and Speaker Prediction: By employing a combined algorithm of time alignment and speaker ID prediction, and through weighting, the system can ensure the accuracy of the identified text and speaker information, reduce false matches, and enable speakers to be accurately distinguished to the greatest extent possible in application scenarios, greatly improving the user experience.

[0130] 4. Speaker ID Mapping: By remapping speaker IDs, the continuity of speaker IDs in the output results is ensured, improving the user experience in multi-speaker recognition scenarios.

[0131] 5. Precise flow termination mechanism: Through reasonable flow termination conditions, the system avoids the risk of task termination or result loss in the middle, ensuring a complete processing flow.

[0132] This application also provides a speech recognition device for multi-speaker environments. It should be noted that the speech recognition device for multi-speaker environments provided in this application can be used to execute the speech recognition method for multi-speaker environments provided in this application. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0133] The following describes the speech recognition device for multi-speaker environments provided in the embodiments of this application.

[0134] Figure 5 This is a schematic diagram of a speech recognition device for multi-speaker environments according to an embodiment of this application. Figure 5 As shown, the device includes:

[0135] The first acquisition unit 51 is used to perform the acquisition steps and acquire audio data;

[0136] The first processing unit 52 is used to perform a first processing step, which uses speech activity detection technology to calibrate the start and end times of each speech in the audio data to obtain an audio calibration result, and then uses automatic speech recognition technology to transcribe the audio calibration result to obtain a transcribed text set corresponding to the audio data. The transcribed text set includes multiple audio text segments, and the audio text segments are marked with start and end times.

[0137] The second processing unit 53 is used to perform the second processing step, which uses speaker separation technology to perform cluster analysis on the above audio data to obtain a segment dataset grouped by speaker, wherein the above segment dataset includes multiple segment data that record the start and end times of the segments and speaker numbers;

[0138] The third processing unit 54 is used to perform the third processing step, which uses a time alignment algorithm to fuse the above-mentioned transcribed text set and the above-mentioned fragment dataset to obtain the final recognition result of the above-mentioned audio data.

[0139] In this embodiment, a first acquisition unit is used to perform an acquisition step to acquire audio data; a first processing unit is used to perform a first processing step, using speech activity detection technology to calibrate the start and end times of each speech segment in the audio data to obtain an audio calibration result, and then using automatic speech recognition technology to transcribe the audio calibration result to obtain a transcribed text set corresponding to the audio data, wherein the transcribed text set includes multiple audio text segments, and the audio text segments are marked with start and end times; a second processing unit is used to perform a second processing step, using speaker separation technology to perform cluster analysis on the audio data to obtain a segment dataset grouped by speaker, wherein the segment dataset includes multiple segment data recording the start and end times of segments and speaker numbers; a third processing unit is used to perform a third processing step, using a time alignment algorithm to fuse the transcribed text set and the segment dataset to obtain the final recognition result of the audio data. By combining speech activity detection, automatic speech recognition, speaker separation, and time alignment algorithms, the accuracy and efficiency of recognition are significantly improved. Speech activity detection technology ensures accurate labeling of speech segments, automatic speech recognition technology provides rapid text transcription, speaker separation technology enables the separation of speech from different speakers, and time alignment algorithms ensure the accuracy of the correspondence between transcribed text and speakers. This approach is particularly suitable for scenarios such as meeting minutes, court transcripts, and large-scale multi-person discussions, effectively distinguishing the speeches of multiple speakers and enhancing the practical value and user experience of speech recognition.

[0140] As an optional approach, the third processing unit includes a traversal module, a numbering module, and an execution module. The traversal module performs the following steps: acquiring the audio text segments from the transcribed text set, traversing the segment dataset, and determining the segment data corresponding to the audio text segments based on their start and end times. The numbering module performs the following steps: numbering the audio text segments according to the speaker numbers of the segment data to obtain speech recognition segments. The execution module performs the traversal and numbering steps multiple times until all audio text segments in the transcribed text set have been traversed, resulting in the final recognition result.

[0141] In one optional scheme, the traversal module includes a first determining submodule, a second determining submodule, and a third determining submodule; the first determining submodule is used to determine at least one segment data that overlaps with the time of the audio text segment based on the start and end time of the audio text segment and the start and end time of the segment data; the second determining submodule is used to determine whether the number of segment data that overlaps with the time of the audio text segment is 1; the third determining submodule is used to determine that if the number of segment data is 1, this segment data corresponds to the audio text segment.

[0142] In one optional scheme, the traversal module further includes a calculation submodule and a fourth determination submodule; the calculation submodule is used to calculate the weight of each of the above-mentioned segment data that overlaps with the time of the above-mentioned audio text segment when the number of the above-mentioned segment data that overlaps with the time of the above-mentioned audio text segment is greater than 1; the fourth determination submodule is used to determine the above-mentioned segment data with the largest weight as the segment data corresponding to the above-mentioned audio text segment.

[0143] In one alternative approach, the calculation submodule includes a fifth determining submodule, used to determine the formula: Determine the weights of each of the above data segments, where ω i The weights of the above segment data are defined as follows: slice_duration is the overlap duration between the above segment data and the above audio text segment, and seg_duration is the weight of the above segment data. m This represents the total duration of the aforementioned audio text segment.

[0144] In one alternative embodiment, the apparatus further includes a determining unit and a second acquiring unit. The determining unit is used to determine, after obtaining the fragment dataset grouped by speaker, whether the start and end times of the fragment dataset cover the start and end times of the transcribed text set. The second acquiring unit is used to continue using the speaker separation technique to acquire fragment data if the start and end times of the transcribed text set are not covered.

[0145] An alternative approach is to execute the first processing step and the second processing step described above in parallel.

[0146] The aforementioned speech recognition device for multi-speaker environments includes a processor and a memory. The first acquisition unit, first processing unit, second processing unit, and third processing unit are all stored as program units in the memory. The processor executes these program units stored in the memory to achieve the corresponding functions. All of the above modules are located in the same processor; alternatively, the modules may be located in different processors in any combination.

[0147] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the issue of low recognition accuracy in scenarios where multiple people speak simultaneously.

[0148] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0149] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device containing the computer-readable storage medium to perform the speech recognition method for multi-speaker environments.

[0150] Specifically, speech recognition methods for multi-speaker environments include:

[0151] Step S201, Acquisition Step, Acquire Audio Data;

[0152] Step S202, the first processing step, uses speech activity detection technology to calibrate the start and end times of each speech in the above audio data to obtain audio calibration results, and then uses automatic speech recognition technology to transcribe the above audio calibration results to obtain the transcribed text set corresponding to the above audio data, wherein the transcribed text set includes multiple audio text segments, and the above audio text segments are marked with start and end times.

[0153] Step S203, the second processing step, uses speaker separation technology to perform cluster analysis on the above audio data to obtain a segment dataset grouped by speaker, wherein the above segment dataset includes multiple segment data that record the start and end times of the segments and speaker numbers;

[0154] Step S204, the third processing step, uses a time alignment algorithm to fuse the above transcribed text set and the above fragment dataset to obtain the final recognition result of the above audio data.

[0155] This invention provides a processor for running a program, wherein the program executes the speech recognition method for multi-speaker environments.

[0156] Specifically, speech recognition methods for multi-speaker environments include:

[0157] Step S201, Acquisition Step, Acquire Audio Data;

[0158] Step S202, the first processing step, uses speech activity detection technology to calibrate the start and end times of each speech in the above audio data to obtain audio calibration results, and then uses automatic speech recognition technology to transcribe the above audio calibration results to obtain the transcribed text set corresponding to the above audio data, wherein the transcribed text set includes multiple audio text segments, and the above audio text segments are marked with start and end times.

[0159] Step S203, the second processing step, uses speaker separation technology to perform cluster analysis on the above audio data to obtain a segment dataset grouped by speaker, wherein the above segment dataset includes multiple segment data that record the start and end times of the segments and speaker numbers;

[0160] Step S204, the third processing step, uses a time alignment algorithm to fuse the above transcribed text set and the above fragment dataset to obtain the final recognition result of the above audio data.

[0161] This invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0162] Step S201, Acquisition Step, Acquire Audio Data;

[0163] Step S202, the first processing step, uses speech activity detection technology to calibrate the start and end times of each speech in the above audio data to obtain audio calibration results, and then uses automatic speech recognition technology to transcribe the above audio calibration results to obtain the transcribed text set corresponding to the above audio data, wherein the transcribed text set includes multiple audio text segments, and the above audio text segments are marked with start and end times.

[0164] Step S203, the second processing step, uses speaker separation technology to perform cluster analysis on the above audio data to obtain a segment dataset grouped by speaker, wherein the above segment dataset includes multiple segment data that record the start and end times of the segments and speaker numbers;

[0165] Step S204, the third processing step, uses a time alignment algorithm to fuse the above transcribed text set and the above fragment dataset to obtain the final recognition result of the above audio data.

[0166] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0167] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0168] Step S201, Acquisition Step, Acquire Audio Data;

[0169] Step S202, the first processing step, uses speech activity detection technology to calibrate the start and end times of each speech in the above audio data to obtain audio calibration results, and then uses automatic speech recognition technology to transcribe the above audio calibration results to obtain the transcribed text set corresponding to the above audio data, wherein the transcribed text set includes multiple audio text segments, and the above audio text segments are marked with start and end times.

[0170] Step S203, the second processing step, uses speaker separation technology to perform cluster analysis on the above audio data to obtain a segment dataset grouped by speaker, wherein the above segment dataset includes multiple segment data that record the start and end times of the segments and speaker numbers;

[0171] Step S204, the third processing step, uses a time alignment algorithm to fuse the above transcribed text set and the above fragment dataset to obtain the final recognition result of the above audio data.

[0172] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0173] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0174] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0175] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0176] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0177] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0178] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0179] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0180] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0181] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0182] 1) This application discloses a speech recognition method for multi-speaker environments, comprising: an acquisition step of acquiring audio data; a first processing step of using speech activity detection technology to calibrate the start and end times of each speech segment in the audio data to obtain an audio calibration result, and then using automatic speech recognition technology to transcribe the audio calibration result to obtain a transcribed text set corresponding to the audio data, wherein the transcribed text set includes multiple audio text segments, and the audio text segments are marked with start and end times; a second processing step of using speaker separation technology to perform cluster analysis on the audio data to obtain a segment dataset grouped by speaker, wherein the segment dataset includes multiple segment data recording the start and end times of the segments and speaker numbers; and a third processing step of using a time alignment algorithm to fuse the transcribed text set and the segment dataset to obtain the final recognition result of the audio data. By combining speech activity detection, automatic speech recognition, speaker separation, and time alignment algorithms, the accuracy and efficiency of recognition are significantly improved. Speech activity detection technology ensures accurate labeling of speech segments, automatic speech recognition technology provides rapid text transcription, speaker separation technology enables the separation of speech from different speakers, and time alignment algorithms ensure the accuracy of the correspondence between transcribed text and speakers. This approach is particularly suitable for scenarios such as meeting minutes, court transcripts, and large-scale multi-person discussions, effectively distinguishing the speeches of multiple speakers and enhancing the practical value and user experience of speech recognition.

[0183] 2) A speech recognition device for multi-speaker environments according to this application includes: a first acquisition unit for performing an acquisition step to acquire audio data; a first processing unit for performing a first processing step, using speech activity detection technology to calibrate the start and end times of each speech in the audio data to obtain an audio calibration result, and then using automatic speech recognition technology to transcribe the audio calibration result to obtain a transcribed text set corresponding to the audio data, wherein the transcribed text set includes multiple audio text segments, and the audio text segments are marked with start and end times; a second processing unit for performing a second processing step, using speaker separation technology to perform cluster analysis on the audio data to obtain a segment dataset grouped by speaker, wherein the segment dataset includes multiple segment data recording the start and end times of segments and speaker numbers; and a third processing unit for performing a third processing step, using a time alignment algorithm to fuse the transcribed text set and the segment dataset to obtain the final recognition result of the audio data. The combined use of speech activity detection, automatic speech recognition, speaker separation, and time alignment algorithms significantly improves the accuracy and efficiency of recognition. Speech activity detection technology ensures accurate labeling of speech segments, automatic speech recognition technology provides rapid text transcription, speaker separation technology enables the separation of speech from different speakers, and time alignment algorithms ensure the accuracy of the correspondence between transcribed text and speakers. This approach is particularly suitable for scenarios such as meeting minutes, court transcripts, and large-scale multi-person discussions, effectively distinguishing the speeches of multiple speakers and enhancing the practical value and user experience of speech recognition.

[0184] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A speech recognition method for multi-speaker environments, characterized in that, include: The steps to obtain audio data are as follows: The first processing step involves using speech activity detection technology to calibrate the start and end times of each speech segment in the audio data to obtain audio calibration results. Then, automatic speech recognition technology is used to transcribe the audio calibration results to obtain a transcribed text set corresponding to the audio data. The transcribed text set includes multiple audio text segments, and the audio text segments are marked with start and end times. The second processing step involves using speaker separation technology to perform cluster analysis on the audio data to obtain a segment dataset grouped by speaker. The segment dataset includes multiple segment data that record the start and end times of the segments and the speaker numbers. The third processing step involves using a time alignment algorithm to fuse the transcribed text set and the fragment dataset to obtain the final recognition result of the audio data. A time alignment algorithm is used to fuse the transcribed text set and the fragment dataset to obtain the final recognition result of the audio data. The process includes: a traversal step: acquiring the audio text fragments in the transcribed text set, traversing the fragment dataset, and determining the fragment data corresponding to the audio text fragments based on their start and end times; a numbering step: numbering the audio text fragments according to the speaker numbers of the fragment data to obtain speech recognition fragments; and repeatedly executing the traversal step and the numbering step until all audio text fragments in the transcribed text set have been traversed to obtain the final recognition result. Determining the segment data corresponding to the audio text segment further includes: when the number of segment data that overlaps with the audio text segment in time is greater than 1, calculating the weight of each segment data that overlaps with the audio text segment in time based on the overlap duration between the segment data and each audio text segment; and determining the segment data with the largest weight as the segment data corresponding to the audio text segment. The step of determining the segment data corresponding to the audio text segment includes: Based on the start and end times of the audio text segment and the start and end times of the segment data, determine at least one segment data that overlaps with the time of the audio text segment; Determine whether the number of data segments that overlap with the time of the audio text segment is 1; When the number of the segment data is 1, it is determined that this segment data corresponds to the audio text segment; The step of calculating the weight of each segment data that overlaps with the audio text segment in time based on the overlap duration between the segment data and each of the audio text segments includes: According to the formula: Determine the weight of each of the data segments, wherein, The weights of the data segments are... The duration of overlap between the segment data and the audio text segment. The total duration of the audio text segment.

2. The method according to claim 1, characterized in that, After obtaining the fragment dataset grouped by speaker, the method further includes: Determine whether the start and end times of the fragment dataset cover the start and end times of the transcribed text set; Even if the start and end times of the transcribed text set are not covered, the speaker separation technique is still used to obtain fragment data.

3. The method according to claim 1, characterized in that, The first processing step and the second processing step are executed in parallel.

4. A speech recognition device for multi-speaker environments, characterized in that, include: The first acquisition unit is used to perform the acquisition steps and acquire audio data; The first processing unit is used to perform a first processing step, which uses speech activity detection technology to calibrate the start and end times of each speech in the audio data to obtain an audio calibration result, and then uses automatic speech recognition technology to transcribe the audio calibration result to obtain a transcribed text set corresponding to the audio data. The transcribed text set includes multiple audio text segments, and the audio text segments are marked with start and end times. The second processing unit is used to perform the second processing step, which uses speaker separation technology to perform cluster analysis on the audio data to obtain a segment dataset grouped by speaker, wherein the segment dataset includes multiple segment data that record the start and end times of the segments and speaker numbers; The third processing unit is used to perform the third processing step, which uses a time alignment algorithm to fuse the transcribed text set and the fragment dataset to obtain the final recognition result of the audio data; The third processing unit includes a traversal module, a numbering module, and an execution module. The traversal module performs the following steps: acquiring the audio text segments from the transcribed text set, traversing the segment dataset, and determining the segment data corresponding to the audio text segments based on their start and end times. The numbering module performs the following step: numbering the audio text segments according to the speaker numbers of the segment data to obtain speech recognition segments. The execution module performs the traversal and numbering steps multiple times until all audio text segments in the transcribed text set have been traversed, resulting in the final recognition result. The traversal module further includes a calculation submodule and a fourth determination submodule; the calculation submodule is used to calculate the weight of each segment data that overlaps with the audio text segment time based on the overlap duration between the segment data and each audio text segment when the number of segment data that overlaps with the time of the audio text segment is greater than 1; the fourth determination submodule is used to determine the segment data with the largest weight as the segment data corresponding to the audio text segment. The traversal module includes: The first determining submodule is used to determine at least one segment data that overlaps with the time of the audio text segment based on the start and end time of the audio text segment and the start and end time of the segment data; The second determining submodule is used to determine whether the number of segment data that overlaps with the time of the audio text segment is 1; The third determining submodule is used to determine that, when the number of fragment data is 1, this fragment data corresponds to the audio text fragment; The calculation submodule includes: a fifth determination submodule, used to determine the formula: Determine the weight of each of the data segments, wherein, The weights of the data segments are... The duration of overlap between the segment data and the audio text segment. The total duration of the audio text segment.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the speech recognition method for multi-speaker environments as described in any one of claims 1 to 3.

6. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a speech recognition method for multi-speaker environments as described in any one of claims 1 to 3.

Images