Multi-channel call recording recognition method and device based on single-channel artificial intelligence model

Abstract

The invention discloses a multichannel call recording recognition method based on a monaural artificial intelligence model. The method comprises the following steps: inputting multichannel call audio data; performing voice activity detection on each sound channel to obtain a voice segment in each sound channel and a corresponding original timestamp during detection; sorting the original timestamps based on the sequence of the original timestamps, constructing a monaural audio, and recording and synthesizing a timestamp; inputting the constructed monaural audio into a pre-trained monaural speech recognition model to generate a recognition text sequence and record an output timestamp; based on an overlapping interval between the synthetic timestamp and the output timestamp, backtracking and matching to the original timestamp; and constructing a triple containing the speaker, the timestamp and the recognition text according to a backtracking matching result. The invention further provides a multichannel call recording recognition device. According to the method provided by the invention, the speaker can be identified and the call logic sequence can be reconstructed while the semantic accuracy is kept.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of speech recognition, and particularly relates to a multi-channel call recording recognition method and device based on a single-channel artificial intelligence model. BACKGROUND

[0002] In recent years, speech recognition technology has made significant progress under the impetus of deep learning. Representative models include the self-supervised pre-training model wav2vec2.0 proposed by Facebook and HuBERT, as well as the end-to-end speech recognition system Whisper model proposed by OpenAI and the SenseVoice model open-sourced by Alibaba. These models have strong robustness and can adapt to multi-lingual and multi-scene language transcription needs, and have been widely used in subtitle generation, voice assistants, conference recording and other scenarios.

[0003] Among them, existing end-to-end speech recognition models (such as Whisper and SenseVoice) only support single-channel input. For multi-channel audio processing, there are often the following shortcomings: (1) the speaker order in the conversation content cannot be accurately restored; (2) direct recognition after multi-channel merging can easily cause information misplacement or time overlap; (3) there is a lack of fine-grained call turn structure division and speaker labeling mechanism. To achieve the goal of recognizing "who said what", speaker separation (Separation) and speaker segmentation (Diarization) modules need to be introduced. Some existing systems attempt to use end-to-end speaker separation models or speaker clustering algorithms to solve this problem, but such methods have high computational complexity, high resource consumption, poor interpretability, and are difficult to deploy in resource-constrained real-world scenarios.

[0004] At the same time, a large amount of voice communication data in reality is collected in a multi-channel manner, such as telephone call recordings and online meetings, each channel corresponding to a speaker channel. Such multi-channel audio often has characteristics such as overlapping speech, irregular start and end, and unequal speaking time. Traditional single-channel speech recognition processes are difficult to directly adapt, which can easily cause text misplacement, speaker recognition errors, and semantic fragmentation, seriously affecting the usability of the recognition results.

[0005] Patent document CN119296522A discloses a multi-channel speech recognition method, device, equipment and medium, which can accurately identify which effective speech segments may have signal mixing problems by judging whether any two channels of effective speech segments satisfy a pre-configured signal mixing condition. The signal mixing condition combines the judgment of content intersection and time intersection. The content intersection ensures the similarity of the speech content, and the time intersection ensures that the similarity occurs within the same time period. Through this signal mixing condition, it is beneficial to accurately identify the two effective speech segments that have signal mixing problems. When it is determined that the effective speech segments of the two channels satisfy the signal mixing condition, the energy size of the content intersection in the two channels can be further compared, and the content intersection of the speech segment with larger energy is selected.

[0006] Patent document CN111883132A discloses a speech recognition method, device, system and storage medium, including: obtaining a speech signal to be recognized, wherein the speech signal to be recognized includes a multi-channel speech signal, and the multi-channel speech signal includes speech signals of at least two channels; performing channel separation on the multi-channel speech signal to obtain at least two first single-channel speech signals; obtaining at least one first speech segment corresponding to the first single-channel speech signal; obtaining a first speech recognition result of the first speech segment; and obtaining a speech recognition result of the first single-channel speech signal according to the first speech recognition result. SUMMARY

[0007] The purpose of the present application is to provide a multi-channel call recording recognition method and device based on a single-channel artificial intelligence model, which can identify the speaker and reconstruct the call logical sequence while maintaining semantic accuracy.

[0008] To achieve the first purpose of the present application, the following technical solution is provided: a multi-channel call recording recognition method based on a single-channel artificial intelligence model, comprising the following steps: inputting multi-channel call audio data, which includes multiple channels and speech data of at least one speaker contained in each channel; performing speech activity detection on each channel to obtain speech segments in each channel and corresponding original time stamps at the time of detection; sorting the original time stamps based on their order, and judging whether the adjacent two speech segments are from the same channel to select different length of silence segments as the connection to construct a single-channel audio and record the synthesized time stamp; inputting the constructed single-channel audio into a pre-trained single-channel speech recognition model to generate an identified text sequence with time annotation and record the output time stamp; based on the overlapping interval between the synthetic timestamp and the output timestamp, and using a nearest neighbor alignment strategy to backtrack to match to the original timestamp; constructing a triple including a speaker, a timestamp, and recognized text according to the result of the backtracking matching.

[0009] The present application accurately pairs original voice data and recognized text by using a voice activity detection (VAD), a silence insertion synthesis strategy, a monophonic speech recognition model, and a timestamp matching algorithm, thereby outputting a speech recognition result with a logical sequence.

[0010] Specifically, the voice activity detection is completed using a short-time energy detection strategy based on time window sliding to filter non-speech passages and effectively separate clear sentence intervals.

[0011] Specifically, the silence segments of different lengths include a first silence segment and a second silence segment. When the adjacent two voice segments are from different sound channels, the first silence segment is used for connection. When the adjacent two voice segments are from the same sound channel, the second silence segment is used for connection. The duration of the first silence segment is greater than the duration of the second silence segment.

[0012] Specifically, the duration of the first silence segment is 1000 milliseconds, and the duration of the second silence segment is 50 milliseconds.

[0013] Specifically, the pre-trained monophonic speech recognition model is constructed using an end-to-end speech recognition model.

[0014] Specifically, data filtering is also required when constructing a triple including a speaker, a timestamp, and recognized text, and the data filtering includes removing abnormal segments caused by short-time noise, time overlap recognition conflicts, or invalid empty segments.

[0015] Specifically, the format of the triple includes CSV, Excel, or JSON format.

[0016] To achieve the second object of the present application, the following technical solution is provided: a multi-channel call recording recognition device for executing the steps of the multi-channel call recording recognition method based on a monophonic artificial intelligence model as described above.

[0017] Compared with the prior art, the present application has the following beneficial effects: Avoid complex modeling of multi-channel data, and only use a monophonic speech recognition model (such as Whisper and SenseVoice) to complete accurate recognition without additional training or fine-tuning, with high compatibility and deployment efficiency; By using the strategy of inserting silent segments of different lengths, speaker discrimination can be effectively enhanced and speech recognition accuracy can be improved. Speaker reconstruction is achieved through timestamp comparison, improving the ability to restore dialogue structure and making it applicable to various scenarios. Attached Figure Description

[0018] Figure 1 A flowchart of the multi-channel call recording recognition method based on a mono artificial intelligence model provided in this embodiment; Figure 2 This is a schematic diagram of the process of synthesizing multi-channel audio into mono audio and matching timestamps provided in this embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0020] like Figure 1 As shown in this embodiment, a multi-channel call recording recognition method based on a mono artificial intelligence model is provided, which includes the following specific steps: Step S101: Input multi-channel call voice data. Each channel records the complete voice content of one speaker. That is, the input multi-channel call recording (such as telephone recording, remote conference recording, etc.) is loaded into the system. Each channel usually corresponds to an independent speaker.

[0021] Step S102, as follows Figure 2 As shown, speech activity detection is performed for each channel to obtain speech segments in each channel and the corresponding original timestamps at the time of detection. Specifically, after multi-channel separation, the VAD algorithm detects speech activity, extracts speech segments, and extracts multiple channels into independent audio tracks. The speech start and end positions are detected based on the speech activity detection algorithm, silent segments and background noise are removed, and valid speech segments are retained. Simultaneously, the original timestamps are marked. Figure 2 The value is represented by T1.

[0022] Step S103, sort the original timestamps based on their chronological order, and determine whether the adjacent two speech segments are from the same channel to select the different length of silence segments as the connection to construct a single-channel audio and record the synthesis timestamp, that is, all extracted speech segments are recombined into a single-channel track according to the time sequence, if the segments before and after come from different channels (i.e. different speakers), a first silence segment of a preset length is inserted between the segments; if the continuous segments come from the same channel (i.e. the same speaker), a second silence segment of a shorter length is inserted to simulate natural pauses, and the synthesis timestamp of the new synthesized audio is recorded, which is represented by T2 in Figure 2 .

[0023] Step S104, based on the overlapping interval between the synthesis timestamp and the output timestamp, and using the nearest neighbor alignment strategy to backtrack to match the original timestamp, a triple containing the speaker, the timestamp and the recognized text is constructed according to the backtracking matching result, that is, the synthesized single-channel audio is input into a speech recognition model (such as Whisper) for speech transcription, and the transcription text is obtained segment by segment, and the output timestamp of the transcription text in the synthesized audio is recorded, which is represented by T3 in Figure 2 .

[0024] Step S105, based on the overlapping interval between the synthesis timestamp (T2) and the single-channel speech recognition model output timestamp (T3) and the nearest neighbor alignment strategy, the original timestamp (T1) is backtracked and matched for comparison, the transcription text is classified into the corresponding channel and speaker, and the short noise or invalid recognition segment in the transcription content that does not conform to the context is removed to form a triple structure of "speaker ID-start and end time-text".

[0025] Step S106, export the structured result. The final recognized speaker label, start and end time, transcription text and other information are exported in a structured format (such as CSV or JSON) for subsequent data analysis, quality inspection or intelligent question and answer application scenarios.

[0026] The triple structure mentioned in the embodiment can be exported in a standard data format (such as CSV, Excel, JSON, etc.) for subsequent system use.

[0027] The embodiment also provides a multi-channel call recording recognition device for executing the steps of the multi-channel call recording recognition method based on the single-channel artificial intelligence model proposed in the above embodiment, which includes an input unit, a data processing unit and an output unit.

[0028] The input unit is used to input multi-channel call audio data.

[0029] The data processing unit is configured to reconstruct the input multi-channel call audio data to generate speech data in a "speaker ID-start and end time-text" triple structure.

[0030] The output unit is configured to output the speech data generated by the data processing unit in a structured manner according to a file format selected by a user.

[0031] Through the above steps, the present application automatically extracts semantic structured text and speaker sequences from original multi-channel recordings, greatly simplifying the process of multi-channel transcription and dialogue structure reconstruction, and is suitable for practical application scenarios such as telephone call recording, remote consultation, judicial investigation recording, online education, etc.

[0032] In order to make the technical solutions, implementation processes and technical effects of the present application clearer and more explicit, the multi-channel call recording recognition method based on a single-channel artificial intelligence model provided by the present embodiment is described in detail with reference to Figure 1 , Figure 2 and the actual application scenario of a certain psychological assistance hotline.

[0033] In this embodiment, the two-channel call recording processing of the hotline is taken as a specific application scenario, and the two channels correspond to the hotline operator and the caller, respectively. The recording format is MP3, the sampling rate is 16 kHz, and the bit depth is 16 bits.

[0034] This embodiment relies on real call data of a certain psychological assistance hotline, and the call recording uses a two-channel collection mode (channel 1 corresponds to the operator and channel 2 corresponds to the caller). It has typical multi-channel call characteristics such as overlapping speech, irregular start and end, and background noise. To verify the effectiveness of the method, 60 (.mp3 format) were randomly selected from 21527 hotline telephone samples from January 1, 2023 to December 31, 2023, and the first 5 minutes of valid audio were extracted from each sample, for a total of 300 minutes. Two professionals with experience in psychological hotlines manually annotated the audio, including speaker identity (operator / caller), start and end time stamp of each speech segment (accurate to milliseconds), and corresponding text content, which served as the benchmark data for performance evaluation.

[0035] The experimental hardware environment is Intel Xeon Gold 6330 CPU, NVIDIA A800 GPU, and the software environment is based on Python 3.9, relying on Librosa library for audio processing, Whisper-large-v3 model (end-to-end single-channel speech recognition model) for text transcription, and timestamp matching algorithm based on NumPy.

[0036] The specific execution process is as follows: Step 1: Multi-channel call audio data input, load 60 double-channel hotline recording samples through the audio input interface, the system automatically parses the audio channel number, sampling rate and other metadata, separates the voice data of channel 1 (operator) and channel 2 (caller) into independent audio streams, each audio stream is associated with a unique channel identifier (ID1 corresponds to the operator, ID2 corresponds to the caller).

[0037] Step 2: Voice activity detection and original timestamp extraction (corresponding to Figure 1 in 102, Figure 2 in T1), for each separated channel audio stream, a short-time energy detection strategy based on time window sliding is used to perform voice activity detection (VAD): set the time window length to 20ms, the sliding step to 10ms, calculate the short-time energy value in each time window, set the energy threshold to 1.5 times the average energy of the background noise of this channel. When the energy values of 3 consecutive time windows exceed the threshold, it is determined that the voice starts; when the energy values of 5 consecutive time windows are lower than the threshold, it is determined that the voice ends.

[0038] Through the above detection, the effective voice segments in each channel are extracted, and the original timestamps T1 (including the start time T1_start and the end time T1_end) of each segment are recorded synchronously. The original timestamp takes the audio start time as the time origin (0ms) and is accurate to the millisecond level. Pure noise segments (continuous segments with energy lower than the threshold) are automatically filtered during detection.

[0039] Step 3: Synthesis channel construction and synthesis timestamp recording (corresponding to Figure 1 in 103, Figure 2 in synthesis channel and T2); Voice segment sorting: sort all effective voice segments extracted from two channels in ascending order of original timestamp T1_start to form a unified segment sequence; Silence segment selection and insertion: traverse the sorted segment sequence, and judge the channel source of the adjacent two segments. If the adjacent segments come from different channels (such as the previous segment is ID1 and the next segment is ID2), insert a first silence segment of 1000ms; if the adjacent segments come from the same channel (such as the two consecutive segments are both ID1), insert a second silence segment of 50ms; Synthesis timestamp generation: concatenate the sorted voice segments and inserted silence segments in sequence to form a single-channel audio stream, and record the synthesis timestamp T2, i.e. the start time T2_start and the end time T2_end of each original voice segment in the synthesized audio. Silence segments are not assigned a speaker identifier and are only used as segment separation markers. The time axis mapping relationship of the synthesis process is as follows: Figure 2As shown, the speech segments of channel 1 and channel 2 are integrated into a single synthesized channel after being connected by different lengths of silence.

[0040] Step 4: Single-channel speech recognition and output timestamp acquisition (corresponding to Figure 1 In 104, Figure 2 In T3), the constructed single-channel audio input is pre-trained into the Whisper-large-v3 end-to-end speech recognition model, with the recognition language set to Chinese. The model outputs a recognized text sequence with time annotations, and simultaneously records the output timestamp T3 (including T3_start and T3_end) of each text segment in the synthesized audio. This model does not require additional fine-tuning for multi-channel scenarios and directly reuses the pre-trained weights, meeting the compatibility design requirements.

[0041] Step 5: Timestamp backtracking matching (corresponding to Figure 1 In 105, Figure 2 In T1-T2-T3 mapping) The timestamp matching is achieved using the nearest neighbor alignment strategy: the overlapping interval of the synthesized timestamp T2 and the output timestamp T3 is calculated, and the overlap rate calculation formula is: Overlap = (min(T2_end, T3_end) - max(T2_start, T3_start)) / (T2_end - T2_start). The matching pairs with an overlap rate ≥ 50% are retained. For each matching pair, the original speech segment corresponding to the synthesized timestamp T2 is traced back to obtain its original timestamp T1 and the speaker ID (speaker ID) belonging to it, completing the association mapping of "recognized text - synthesized timestamp - original timestamp - speaker".

[0042] Step 6: Triplet construction and data filtering (corresponding to Figure 1 In 106), which includes: Data filtering: used to eliminate abnormal segments.

[0043] Short noise: eliminate recognized text segments with a length <100ms.

[0044] Time overlap conflict: when multiple recognized texts correspond to the same original speech segment, the text with the highest ROUGE-L (F1) is retained.

[0045] Invalid empty segment: eliminate text segments that are pure punctuation, emotional words, or semantic blanks.

[0046] Triplet generation: construct the filtered speaker ID, original timestamp (T1_start, T1_end), and recognized text into a structured triplet, which supports export in JSON format, as shown in the following example: ```json [ { "Speaker ID": "ID1", "Start time": "00:00:12.350", "End time": "00:02:36.120", "Recognized text": "Hello, this is the psychological assistance hotline, is there anything I can help you with?" }, { "Speaker ID": "ID2", "Start time": "00:01:05.780", "End time": "00:04:44.210", "Recognized text": "I've been having trouble sleeping lately and feel like life has no meaning..." } ] ```。

[0047] Implementation effect evaluation: Based on the artificial annotation results, the performance of the method of the present application is quantitatively evaluated, and the traditional method of "directly inputting the original multi-channel audio into the Whisper model" is compared. The evaluation indicators and results are shown in Table 1.

[0048] Note: The evaluation data is based on 300 minutes of audio of 60 hotline samples, and all indicators are rounded to four decimal places.

[0049] From Table 1, the method of the present application realizes a speaker recognition accuracy of 96.43% through the silent segment insertion and timestamp matching strategy, solving the core pain point of traditional single-channel models that cannot distinguish speakers. At the same time, the CER is reduced by 29.3% compared to the traditional method, and the ROUGE-L(F1) is increased by 15.9%, proving its significant advantage in text transcription accuracy.

[0050] This embodiment is based on a real multi-channel call scenario of a certain psychological assistance hotline, and fully presents the execution process of the method of the present application. The technical details of each step and the data flow relationship are clarified, and the effectiveness of the method is verified through quantitative experiments. Figure 1 、 Figure 2 In the implementation process, no additional model training or complex hardware dependence is introduced, but only through audio structural reconstruction and timestamp alignment strategy, the precise association of "speaker-time-text" is realized, fully embodying the technical advantages of the present application, such as strong compatibility, low deployment cost, and excellent recognition effect, which can be widely applied to multi-channel speech transcription scenarios such as telephone recording and remote consultation.

[0051] Compared with the traditional scheme relying on manual annotation or high-cost separation model, the application has the following advantages: first, the training of the speaker separation model is avoided, and the calculation burden is avoided; second, based on the insertion of the silent segment and the timestamp matching strategy, the speaker reconstruction of the multi-round dialogue is realized without introducing high complexity algorithm; third, combined with the mainstream single-channel speech recognition model such as Whisper, the efficient and stable text generation capability is realized. In addition, the method has flexible structure, clear module division, and is suitable for deployment in various call recording scenes, and emotion recognition model, text summary model, multilingual transcription module and the like can be extended and accessed in actual use, so as to build a more rich voice understanding system.

[0052] The application not only has high operability in engineering implementation, but also has significant performance advantages in key indicators such as semantic fidelity, speaker tracking and processing efficiency, and has good industrial transformation prospects.

[0053] In addition, the terms "upper", "lower", "inner", "outer", "front", "back" are only for description purposes, and cannot be understood as indicating or implying relative importance. Unless otherwise specified, the relative steps, numerical expressions and numerical values of the components and steps set forth in these embodiments do not limit the scope of the application.

[0054] Of course, the above only describes specific embodiments of the application and does not limit the scope of the application, and equivalent changes or modifications made in accordance with the application are included in the scope of the application.

[0055] Finally, it should be noted that the above-described embodiments are only specific embodiments of the application, and are used to illustrate the technical solutions of the application, but not to limit it, the protection scope of the application is not limited thereto, although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any skilled person in the art can modify or easily think of changes to the technical solutions recorded in the foregoing embodiments within the technical range disclosed by the application, or make equivalent replacement to part of the technical features; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments, and should be covered within the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A method for recognizing multi-channel call recordings based on a monophonic artificial intelligence model, characterized in that, The method comprises the following steps: inputting multi-channel call audio data, which comprises a plurality of channels and speech data of at least one speaker contained in each channel; performing speech activity detection on each channel to obtain a speech segment in each channel and a corresponding original timestamp at the time of detection; sorting the original timestamps based on the order of the original timestamps, and determining whether adjacent two speech segments are from the same channel to select a silence segment with different lengths as a link to construct a single-channel audio and record a synthesized timestamp; inputting the constructed single-channel audio into a pre-trained single-channel speech recognition model to generate a recognized text sequence with time annotation and record an output timestamp; based on the overlapping interval between the synthesized timestamp and the output timestamp, and using a nearest neighbor alignment strategy to backtrack to match the original timestamp; constructing a triple comprising a speaker, a timestamp and a recognized text according to the backtracking result. 2.The method of claim 1, wherein, The speech activity detection is completed by using a short-time energy detection strategy based on time window sliding. 3.The method of claim 1, wherein the mono AI model is a pre-trained model. The silence segment with different lengths comprises a first silence segment and a second silence segment; ​ when the adjacent two speech segments are from different channels, the first silence segment is used for linking; when the adjacent two speech segments are from the same channel, the second silence segment is used for linking; the length of the first silence segment is greater than the length of the second silence segment. 4.The method of claim 3, wherein, The length of the first silence segment is 1000 milliseconds, and the length of the second silence segment is 50 milliseconds. 5.The method of claim 1, wherein, The pre-trained single-channel speech recognition model uses an end-to-end speech recognition model. 6.The method of claim 1, wherein, When constructing the triple comprising a speaker, a timestamp and a recognized text, data filtering is also required, and the data filtering comprises eliminating abnormal segments caused by short-time noise, time overlapping recognition conflict or invalid empty segment. 7.The method of claim 1 or 6, wherein, The format of the triple comprises CSV, Excel or JSON format.

8. A multi-channel talk recording recognition apparatus, characterized by comprising: A device for performing the steps of the multi-channel call recording recognition method based on a single-channel artificial intelligence model according to any one of claims 1-7.

Images