Multi-channel speech recognition method, device, equipment and medium

Abstract

The invention discloses a multi-channel speech recognition method and device, equipment and a medium. By judging whether the effective voice segments of any two sound channels meet the pre-configured signal mixing condition or not, which effective voice segments may have a signal mixing problem can be accurately identified. The signal mixing condition combines judgment in two aspects of content intersection and time intersection. The content intersection ensures the similarity of the voice content, the time intersection ensures that the similarity occurs in the same time period, and through the signal mixing condition, two effective voice segments with the signal mixing problem can be accurately recognized. When it is determined that the effective voice segments of the two sound channels meet the signal mixing condition, the energy of the content intersection in the two sound channels can be further compared, and the content intersection of the voice segment with the large reserved energy is selected. According to the method, wrong recognition caused by signal mixing can be reduced, so that the accuracy of a voice recognition result is improved.

Description

Technical Field

[0001] The present application relates to the field of digital signal processing and deep learning technology, and in particular to a multi-channel speech recognition method, device, equipment and medium. Background Art

[0002] In the vast field of speech recognition technology, multi-channel speech processing occupies a pivotal position. Multi-channel speech, as the name implies, refers to multiple independent audio signals or channels recorded from different speech sources or different angles. These signals usually come from scenes such as microphone arrays in conference rooms and synthetic recordings at telephone exchange ends. For example, a well-designed microphone array can capture speech from different speakers according to different directions to generate a multi-channel speech signal. Figure 1 A schematic diagram of a multi-channel microphone provided in an embodiment of the present application. The multi-channel microphone has a microphone array inside, and can realize recording of 4 channels (90 degrees), 2 channels (180 degrees, i.e., bidirectional), and 1 channel (omnidirectional) through an algorithm.

[0003] Taking dual-channel as an example, in the existing multi-channel speech recognition process, the voices of two speakers can be split into two channels through sound source localization, sound source separation, echo cancellation and other technologies. Then, the speech data of the two channels are recognized by two concurrent speech recognition modules to obtain speech recognition results. At the same time, the start and end time of the valid speech segments in the speech data of the two channels are determined by the speech activity detection module. Then, according to the time sequence and the channel, the order of the speech recognition results can be further determined, and the speech can be presented in the form of a dialogue with correct time sequence.

[0004] However, multi-channel speech processing is not an easy task, and the signal mixing problem remains a major challenge in the field of multi-channel speech processing. When the voices of multiple speakers appear in one channel at the same time, the signals of different sound sources will overlap and interfere, making it difficult for the recognition system to accurately separate and identify these sound sources. Even with the use of advanced blind source separation algorithms or microphone array technology, this signal mixing phenomenon is often not completely eliminated. Moreover, the signal mixing problem is particularly prominent in question-and-answer dialogue scenarios. In this scenario, if the speech data of the two channels are mixed, it may cause ambiguity and errors in the speech recognition results. For example, in a conversation between a doctor and a patient, if the doctor's channel is mixed with the patient's speech data, the patient's answer may be mistakenly added to the doctor's recognition results, causing misunderstandings. This misunderstanding not only reduces the accuracy of speech recognition, but may also have an adverse impact on subsequent tasks such as natural language processing and information extraction.

[0005] Therefore, how to solve the signal mixing problem in multi-channel speech processing and improve the accuracy and stability of speech recognition is an important issue that needs to be urgently addressed in the current field of speech recognition. Summary of the invention

[0006] The present application provides a multi-channel speech recognition method, apparatus, device and medium for solving the problem that signal mixing in existing multi-channel speech processing affects the accuracy and stability of speech recognition.

[0007] In a first aspect, the present application provides a multi-channel speech recognition method, the method comprising:

[0008] Acquire the voice data of each channel in the same time period;

[0009] Determine the valid speech segments respectively included in each of the speech data and the start and end time of each of the valid speech segments;

[0010] Obtaining initial speech recognition results for each of the valid speech segments;

[0011] According to whether the initial speech recognition results of the valid speech segments satisfy a preset signal mixing condition, the initial speech recognition results of the valid speech segments are adjusted to obtain the final speech recognition results of the valid speech segments, including:

[0012] For the valid speech segments of any two channels, determine whether the valid speech segments of the two channels meet the signal mixing condition; wherein, the signal mixing condition includes: there is a content intersection between the initial speech recognition results of the valid speech segments of the two channels and there is a time intersection between the start and end times of the valid speech segments of the two channels; the content intersection means that the contents of the two initial speech recognition results are exactly the same, or the start position or end position of one initial speech recognition result contains all the contents of the other initial speech recognition result; if it is determined that the valid speech segments of the two channels meet the preset signal mixing condition, then compare the energies corresponding to the content intersection in the valid speech segments of the two channels, save the content intersection in the initial speech recognition result corresponding to the valid speech segment with large energy, and delete the content intersection in the initial speech recognition result corresponding to the valid speech segment with small energy.

[0013] In a second aspect, the present application further provides a multi-channel speech recognition device, the device comprising:

[0014] An acquisition unit, used for acquiring the voice data of each channel in the same time period;

[0015] An activity detection unit, used to determine the valid speech segments respectively included in each of the speech data and the start and end time of each of the valid speech segments;

[0016] A speech recognition unit, used to obtain an initial speech recognition result of each of the valid speech segments;

[0017] A signal separation unit, used to adjust the initial speech recognition results of each valid speech segment according to whether the initial speech recognition results of each valid speech segment meet a preset signal mixing condition, so as to obtain a final speech recognition result of each valid speech segment, including:

[0018] For the valid speech segments of any two channels, determine whether the valid speech segments of the two channels meet the signal mixing condition; wherein, the signal mixing condition includes: there is a content intersection between the initial speech recognition results of the valid speech segments of the two channels and there is a time intersection between the start and end times of the valid speech segments of the two channels; the content intersection means that the contents of the two initial speech recognition results are exactly the same, or the start position or end position of one initial speech recognition result contains all the contents of the other initial speech recognition result; if it is determined that the valid speech segments of the two channels meet the preset signal mixing condition, then compare the energies corresponding to the content intersection in the valid speech segments of the two channels, save the content intersection in the initial speech recognition result corresponding to the valid speech segment with large energy, and delete the content intersection in the initial speech recognition result corresponding to the valid speech segment with small energy.

[0019] In a third aspect, the present application provides a computer device, the computer device comprising a processor, the processor being configured to implement the steps of the multi-channel speech recognition method as described above when executing a computer program stored in a memory.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, which implements the steps of the multi-channel speech recognition method as described above when the computer program is executed by a processor.

[0021] The beneficial effects of this application are as follows:

[0022] 1. By judging whether the valid speech segments of any two channels meet the pre-configured signal mixing conditions, it is possible to accurately identify which valid speech segments may have signal mixing problems, which is conducive to the subsequent processing of valid speech segments with signal mixing problems.

[0023] 2. The signal mixing condition combines the judgment of content intersection and time intersection. Content intersection ensures the similarity of speech content, while time intersection ensures that this similarity occurs in the same time period. Through this signal mixing condition, it is helpful to accurately identify two valid speech segments with signal mixing problems.

[0024] 3. When it is determined that the valid speech segments of the two channels meet the signal mixing condition, the energy of the content intersection in the two channels can be further compared, and the content intersection of the speech segment with larger energy can be selected. This method can reduce the misrecognition caused by signal mixing, thereby improving the accuracy of speech recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0026] Figure 1 A schematic diagram of a multi-channel microphone provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of a multi-channel speech recognition process provided by an embodiment of the present application;

[0028] Figure 3 is a schematic diagram of the spectrum of a two-channel speech signal with signal mixing;

[0029] Figure 4 A schematic diagram of a specific multi-channel speech recognition process provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of a specific process for adjusting the initial speech recognition result provided in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of the structure of a multi-channel speech recognition device provided in an embodiment of the present application;

[0032] Figure 7 It is a structural schematic diagram of a computer device provided in an optional embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0034] In order to solve the signal mixing problem in multi-channel speech processing and improve the accuracy and stability of speech recognition, the present application provides a multi-channel speech recognition method, device, equipment and medium.

[0035] Embodiment 1:

[0036] The present application provides a multi-channel speech recognition method. Figure 2 A schematic diagram of a multi-channel speech recognition process provided in an embodiment of the present application includes:

[0037] S101: Acquire the voice data of each channel in the same time period.

[0038] The multi-channel speech recognition method provided in the present application is applied to a computer device, which may be an intelligent terminal, such as a mobile phone, a computer, etc., or a server, such as an application server, a business server, etc.

[0039] In actual application scenarios, multi-channel voice acquisition equipment, such as multi-channel microphones, multi-channel speakers, etc., is pre-installed in the working environment to capture voice information in the working environment in all directions and at multiple angles. The multi-channel voice acquisition equipment can continuously collect voice data (recorded as original voice data) in the working environment. The computer device performing multi-channel voice recognition can determine the voice recognition results of different channels based on the multi-channel voice recognition method provided by the present application through the original voice data collected by the multi-channel voice acquisition equipment.

[0040] After acquiring the original voice data in the working environment, the computer device can split the original voice data into voice data of different channels. For example, the original voice data is split into left channel voice data and right channel voice data using a channel splitting tool (such as ffmpeg, etc.). For another example, the voice data of each channel is extracted from the original voice data using sound source localization and separation technology, such as a blind source separation algorithm, a beamforming algorithm, etc.

[0041] After splitting the original voice data, the computer device will extract the corresponding voice data for each channel in the same time period to ensure that the voice information of each channel is synchronized in subsequent processing, thereby avoiding recognition errors caused by time differences.

[0042] For example, in a hospital outpatient clinic, a doctor is having a face-to-face consultation with a patient. At the same time, a multi-channel voice acquisition device (such as a dual-channel microphone array) is installed in the room. The dual-channel microphone array continuously collects all the original voice data in the environment. The original voice data includes the conversation between the doctor and the patient, environmental noise (such as conversations with other patients, the operating sound of medical equipment), etc. After the computer device obtains the original voice data, it can use tools such as ffmpeg to split the original voice data into left channel (which may be biased towards the doctor's voice) and right channel (which may be biased towards the patient's voice) voice data.

[0043] S102: Determine the valid voice segments respectively included in each voice data and the start and end time of each valid voice segment.

[0044] Taking into account that the voice data of each channel may contain interference from non-valid voice parts such as silence and noise, the accuracy of voice recognition is improved. Based on this, in the present application, after acquiring the voice data of each channel in the same time period, the valid voice segment contained in the voice data of the channel and the start and end time of the valid voice segment can be determined for each channel, that is, the continuous voice signal segment in the voice signal of the channel and the start time and end time of the voice signal segment can be determined. Exemplarily, the valid voice segments respectively contained in each of the voice data and the start and end time of each of the valid voice segments can be determined by a voice activity detection (VAD) algorithm. Among them, the VAD algorithm can be based on energy threshold detection, spectral feature detection, and machine learning-based detection.

[0045] For example, for dual-channel speech data, a complete valid speech segment is obtained from the speech data of the channel through the VAD algorithm. The valid speech segment index of one channel is recorded as m, and the valid speech segment index of one channel is recorded as n. The start and end time of each valid speech segment is recorded, such as the start and end time of valid speech segment m is T1_start (start time) and T1_end (end time), and the start and end time of valid speech segment n is T2_start (start time) and T2_end (end time).

[0046] In a possible implementation manner, after the voice data of each channel is acquired based on the above embodiment, the voice data of each channel may be subjected to denoising, enhancement and other processing to improve the accuracy of subsequent recognition.

[0047] S103: Obtaining initial speech recognition results of each of the valid speech segments.

[0048] After obtaining each valid speech segment, speech recognition can be performed on each valid speech segment in parallel to obtain initial speech recognition results corresponding to each valid speech segment. For example, the speech recognition engine is called in parallel to perform speech recognition on each valid speech segment in parallel to obtain initial speech recognition results corresponding to each valid speech segment.

[0049] In a possible implementation, before performing parallel speech recognition on each valid speech segment, speech features of each valid speech segment can be obtained to facilitate subsequent speech recognition based on the speech features of each valid speech segment and reduce the amount of calculation required for speech recognition. The speech features may include Mel frequency cepstral coefficients, spectrograms, Mel filter energy, short-time Fourier transform coefficients, pitch, timbre, rhythm, etc.

[0050] S104: adjusting the initial speech recognition results of the valid speech segments according to whether the initial speech recognition results of the valid speech segments meet a preset signal mixing condition to obtain a final speech recognition result of the valid speech segments, including:

[0051] For the valid speech segments of any two channels, determine whether the valid speech segments of the two channels meet the signal mixing condition; wherein, the signal mixing condition includes: there is a content intersection between the initial speech recognition results of the valid speech segments of the two channels and there is a time intersection between the start and end times of the valid speech segments of the two channels; the content intersection means that the contents of the two initial speech recognition results are exactly the same, or the start position or end position of one initial speech recognition result contains all the contents of the other initial speech recognition result; if it is determined that the valid speech segments of the two channels meet the preset signal mixing condition, then compare the energies corresponding to the content intersection in the valid speech segments of the two channels, save the content intersection in the initial speech recognition result corresponding to the valid speech segment with large energy, and delete the content intersection in the initial speech recognition result corresponding to the valid speech segment with small energy.

[0052] In the prior art, whether sound source separation is achieved through a mono blind source separation algorithm or a microphone array, in actual use, there is always a situation where the voices of different speakers cannot be completely separated. Figure 3 Figure 1 is a schematic diagram of the spectrum of a dual-channel speech signal with mixed signals. Figure 3 In the video, the doctor and the patient are talking. In theory, the sound source separation technology can clearly separate the voices of the two into two independent channels, such as the patient's channel 1 and the doctor's channel 2. However, in actual operation, it can be found that in certain time periods (such as Figure 3In the position marked by the box), the voice of the patient in channel 1 is not completely eliminated, but mixed into another channel (i.e., channel 2) in a relatively weakened state. Ideally, the part of channel 2 at the time indicated by the box should not contain the voice of the other channel at all, that is, complete sound source separation is achieved. But the reality is not the case. In this case, the effect of the speech recognition result is as follows: channel 1 is the patient's speech recognition result "three times a day.", and channel 2 is the doctor's speech recognition result "How do you take this medicine? Three times a day." It can be seen that since the doctor's channel is mixed with the voice data of the patient's channel, the recognition content of "three times a day" is added to the doctor's speech recognition result, which causes ambiguity. This is a typical signal mixing phenomenon. This situation often occurs in a question-and-answer scenario. Therefore, it is necessary to process the speech segments with signal mixing in the effective speech segments of each channel to reduce the impact of the signal mixing problem on the speech recognition results of each channel and improve the quality of the speech recognition results.

[0053] In order to effectively eliminate the impact of signal mixing problems on speech recognition results, in this application, it is first necessary to determine the speech segments with signal mixing problems from each valid speech segment, and then make special adjustments to the speech segments with signal mixing problems. To this end, a set of signal mixing conditions are pre-configured, and these signal mixing conditions include but are not limited to: the valid speech segments of the two channels overlap in time (i.e., time intersection), and the initial speech recognition results of the two channels intersect in content (i.e., content intersection). Optionally, the signal mixing condition may also include: the voiceprint features of the content intersection in the valid speech segments of the two channels are similar.

[0054] Exemplarily, each signal mixing condition is described below:

[0055] Type 1: Content intersection.

[0056] When the speech signals captured by the two channels are highly consistent in content, it is likely because the two signals come from the same sound source, or because the sounds of the two sound sources are mixed during the propagation process. Therefore, whether the initial speech recognition results of the two channels have content intersection is also an important basis for judging signal mixing. The content intersection means that the content of the two initial speech recognition results is exactly the same, or the beginning or end of one initial speech recognition result contains the entire content of the other initial speech recognition result. This means that the same speech content may be mixed in the speech recognition results of the two channels.

[0057] Exemplarily, for valid speech segments of any two channels, a substring matching method can be used to detect whether there is a content intersection between them. Specifically, the initial speech recognition results of the two channels are first preprocessed with text, including removing irrelevant characters, normalizing the text format, etc., to ensure the accuracy of subsequent comparisons. Next, a substring matching algorithm (such as the KMP algorithm, the Boyer-Moore algorithm, etc.) is used to check whether the initial speech recognition result of one channel contains the initial speech recognition result of another channel as its substring. This includes two situations: complete inclusion and partial inclusion. If it is determined that the initial speech recognition result of one channel is exactly the same as the initial speech recognition result of another channel, or the beginning or end part of the initial speech recognition result of one channel contains the entire content of the initial speech recognition result of another channel, then it is determined that there is a content intersection between the initial speech recognition results of the two channels. If it is determined that the initial speech recognition result of one channel is not completely the same as the initial speech recognition result of another channel, and the beginning or end part of the initial speech recognition result of one channel does not contain all the content of the initial speech recognition result of the other channel, it is determined that there is no content intersection between the initial speech recognition results of the two channels.

[0058] For example, the initial speech recognition result of valid speech segment m is Text1, and the initial speech recognition result of valid speech segment n is Text2. If the following relationship exists, it is determined that there is an intersection of content:

[0059] 1. The content of text1 and text2 is exactly the same. For example, text1 = "three times a day", text2 = "three times a day".

[0060] 2. text1 contains text2 at the beginning or end. For example, text1 = "How do you eat? Three times a day", text2 = "Three times a day".

[0061] 3. text2 contains text1 at the beginning or end. For example, text1 = "Is it effective three times a day?", text2 = "Three times a day".

[0062] Type 2: Time intersection.

[0063] In actual environments, sounds from different sources may be captured by multiple microphones or channels at the same time, resulting in temporal overlap of speech signals. Therefore, an important basis for judging signal mixing is whether the effective speech segments of the two channels overlap in time, that is, temporal intersection. The temporal intersection means that the start and end times of the two effective speech segments overlap. This means that in terms of time, both channels captured speech signals in the same time period. For example, in a question-and-answer scenario, if the doctor's initial speech recognition results accidentally include the patient's answer, it is likely that the patient's voice was captured by the microphone while the doctor was speaking, resulting in content intersection of the initial speech recognition results of the two channels, rather than the patient repeating the doctor's words, resulting in content intersection between the initial speech recognition results of the effective speech segments of the two channels.

[0064] Exemplarily, for two valid speech segments with content intersection, the start and end times of the two valid speech segments can be compared to determine whether there is a time intersection between the start and end times of the two valid speech segments. If it is determined that there is a time intersection between the start and end times of the two valid speech segments, it is determined that there is a signal mixing problem between the two valid speech segments. If it is determined that there is no time intersection between the start and end times of the two valid speech segments, it means that there is a content intersection between the initial speech recognition results of the two valid speech segments and it is not caused by a signal mixing problem, and it is determined that there is no signal mixing problem between the two valid speech segments.

[0065] For example, the initial speech recognition result of valid speech segment m is Text1, the start and end time of valid speech segment m is {T1_start, T1_end}, the initial speech recognition result of valid speech segment n is Text2, the start and end time of valid speech segment n is {T2_start, T2_end}, if there is content intersection between Text1 and Text2, then it is further determined whether there is a time intersection between the start and end time {T1_start, T1_end} of valid speech segment m and the start and end time {T2_start, T2_end} of valid speech segment n. If it is determined that there is a time intersection between the start and end time {T1_start, T1_end} of valid speech segment m and the start and end time {T2_start, T2_end} of valid speech segment n, then it is determined that there is a signal mixing problem between valid speech segment m and valid speech segment n. If it is determined that there is no time intersection between the start and end time {T1_start, T1_end} of the effective speech segment m and the start and end time {T2_start, T2_end} of the effective speech segment n, it is determined that there is no signal mixing problem between the effective speech segment m and the effective speech segment n.

[0066] Type 3: Similar voiceprint features.

[0067] In order to avoid the problem of different speakers uttering the same content at the same time, which may lead to the misidentification of signal mixing between two valid speech segments, in the present application, for the speech segments of two channels with time intersection and content intersection, the speaker to whom the speech signal corresponding to the content intersection belongs can be distinguished.

[0068] Voiceprint features refer to the unique and stable acoustic characteristics of each person's voice signal, which can be used to distinguish different speakers. In the speech recognition system, the similarity of voiceprint features is an important indicator to measure whether two voice signals come from the same speaker. When the effective speech segments of two channels show high similarity in voiceprint features, this may mean that the two voice signals come from the same sound source, or at least two sound sources that are very close in acoustic characteristics. Therefore, the similarity of voiceprint features is also an important basis for judging signal mixing problems.

[0069] When judging the similarity of voiceprint features, a series of voiceprint recognition algorithms and technologies can be used, such as Mel Frequency Cepstral Coefficients (MFCC), Linear Prediction Cepstral Coefficients (LPCC), Gaussian Mixture Model (GMM), Support Vector Machine (SVM) and deep learning model, etc. These algorithms and technologies can extract voiceprint features from speech signals and calculate the similarity or distance between these voiceprint features.

[0070] Exemplarily, for two valid speech segments with content intersection and time intersection, the speech signals corresponding to the content intersection in the valid speech segments of the two channels can be extracted. The voiceprint feature vectors of each speech signal are determined by the voiceprint recognition algorithm. The voiceprint similarity between the two voiceprint feature vectors is determined. For example, this is achieved by calculating indicators such as the Euclidean distance, cosine similarity, and Mahalanobis distance between the voiceprint feature vectors. If the voiceprint feature similarity is greater than a preset voiceprint similarity threshold, it means that the speech signals corresponding to the content intersection are likely to come from the same sound source, and it is determined that the voiceprint features of the content intersection in the valid speech segments of the two channels are similar. If the voiceprint feature similarity is not greater than the preset voiceprint similarity threshold, it means that the speech signals corresponding to the content intersection do not come from the same sound source, and it is determined that the voiceprint features of the content intersection in the valid speech segments of the two channels are not similar.

[0071] It should be noted that the pre-configured signal mixing condition may include only category 1 and category 2, or may include category 1, category 2 and category 3. When determining whether there is a signal mixing problem between any two valid speech segments, it is necessary for the two valid speech segments to simultaneously meet the requirements of all categories included in the signal mixing condition before it is determined that the two valid speech segments meet the pre-configured signal mixing condition. If the two valid speech segments cannot simultaneously meet the requirements of all categories included in the signal mixing condition, it is determined that the two valid speech segments do not meet the pre-configured signal mixing condition.

[0072] Based on the above embodiment, it can be determined whether there is a signal mixing problem between any two valid speech segments. If it is determined that the valid speech segments of the two channels meet the signal mixing condition, the energy (or intensity) corresponding to the content intersection part of the two speech segments can be further compared. Select the initial speech recognition result corresponding to the speech segment with larger energy, and retain the content intersection part therein. In the initial speech recognition result corresponding to the speech segment with smaller energy, delete or ignore the content intersection part. For example, text1 = "How do you eat? Three times a day", text2 = "Three times a day", if the energy of the corresponding segment of text2 in the intersection speech segment of text1 and text2 is large, then text1 is processed as "How do you eat?", otherwise text2 is processed.

[0073] In a possible implementation, when determining the energy corresponding to the content intersection in the valid speech segments of the two channels, the speech signals corresponding to the content intersection in the valid speech segments of the two channels can be extracted. Then, for each speech signal, the energy of the speech signal is determined by summing the squares of the amplitudes of the speech signals. This is expressed by the following formula:

[0074] E=∑(|s[n]| 2 )

[0075] Wherein, s[n] represents the amplitude value of the nth frame speech signal, and E represents the energy of the speech signal.

[0076] In another possible implementation, when determining the energies corresponding to the content intersection in the valid speech segments of the two channels, the speech signals corresponding to the content intersection in the valid speech segments of the two channels may also be extracted. Then, for each speech signal, the absolute value of the amplitude of the speech signal is summed to determine the energy of the speech signal.

[0077] In one example, if it is determined that the valid speech segments of the two channels do not meet the signal mixing conditions, it means that there is no signal mixing problem between the two valid speech segments. The two valid speech segments are independent of each other, and their initial speech recognition results should also be independent of each other. Therefore, there is no need to perform additional signal separation or mixing processing steps based on the initial speech recognition results of the two valid speech segments.

[0078] Based on the above embodiment, the initial speech recognition result of each valid speech segment is adjusted to obtain the final speech recognition result of each valid speech segment.

[0079] In one example, after obtaining the final speech recognition results of each channel, the final speech recognition results of each channel can be displayed to provide a user experience. The display method may include one or more of the following: text display, graphical display, voice playback, etc. The specific display method selected depends on the specific application scenario and user needs. In actual applications, the above-mentioned display methods can be flexibly selected or combined according to actual conditions to provide the best user experience. For example, the final speech recognition result of each channel is displayed on the screen in text form, or a waveform or spectrum diagram is used to display the original voice signal, and the final speech recognition result of each channel is marked on the diagram, or the final speech recognition result is converted into speech and played through a speaker.

[0080] Considering that there may be problems such as signal confusion and voice masking in the effective speech segments of different channels, the final speech recognition results of each channel are displayed so that users can check and process them, thereby avoiding the influence of problems such as signal confusion and voice masking on the speech recognition results. Among them, the signal confusion means that on the basis of signal mixing, if the effective speech segments of the two channels overlap in time, that is, sound source A and sound source B pronounce at the same time, then the inference of the sound source will become more difficult. Voice masking means that if the speech energy of the interference source is greater than the speech energy of the target source, the speech of the target source may be masked, which will seriously affect the accuracy of speech recognition. By displaying the final speech recognition results, users can check the final speech recognition results of different channels through visual and auditory comparison to identify potential errors or inconsistencies. Then, the user inputs a modification operation on any final speech recognition result through the display corresponding to the computer device, such as deleting an erroneous recognition result, adding missing content, or correcting erroneous text. When the computer device receives the user's modification operation on any final speech recognition result, the final speech recognition result can be updated based on the modification content carried in the modification operation.

[0081] The beneficial effects of this application are as follows:

[0082] 1. By judging whether the valid speech segments of any two channels meet the pre-configured signal mixing conditions, it is possible to accurately identify which valid speech segments may have signal mixing problems, which is conducive to the subsequent processing of valid speech segments with signal mixing problems.

[0083] 2. The signal mixing condition combines the judgment of content intersection and time intersection. Content intersection ensures the similarity of speech content, while time intersection ensures that this similarity occurs in the same time period. Through this signal mixing condition, it is helpful to accurately identify two valid speech segments with signal mixing problems.

[0084] 3. When it is determined that the valid speech segments of the two channels meet the signal mixing condition, the energy of the content intersection in the two channels can be further compared, and the content intersection of the speech segment with larger energy can be selected. This method can reduce the misrecognition caused by signal mixing, thereby improving the accuracy of speech recognition results.

[0085] Embodiment 2:

[0086] The multi-channel speech recognition method provided by the present application is described below through specific embodiments. Figure 4 A schematic diagram of a specific multi-channel speech recognition process provided in an embodiment of the present application, the process includes:

[0087] S401: Channel separation: obtaining the voice data of each channel in the same time period.

[0088] The following S402 to S404 are performed in parallel for the voice data of each channel:

[0089] S402: Voice activity detection: determining the valid voice segment contained in the voice data of the channel and the start and end time of the valid voice segment.

[0090] S403: Speech recognition: determining the initial speech recognition result of the valid speech segment.

[0091] S404: Obtain the text of the initial speech recognition result.

[0092] S405: According to whether the initial speech recognition results of each valid speech segment meet the preset signal mixing condition, each text is adjusted to obtain the final text of each valid speech segment.

[0093] Among them, the preset signal mixing conditions include: there is a content intersection between the initial speech recognition results of the effective speech segments of the two channels, there is a time intersection between the start and end times of the effective speech segments of the two channels, and the voiceprint features of the content intersection in the effective speech segments of the two channels are similar.

[0094] Figure 5 A schematic diagram of a specific process for adjusting the initial speech recognition result provided in an embodiment of the present application. The process includes:

[0095] For any valid speech segment of two channels, the following processing is performed:

[0096] S501: Determine whether there is content intersection between the initial speech recognition results of the valid speech segments of the two channels. If so, execute S502; otherwise, execute S510.

[0097] The content intersection means that the contents of the two initial speech recognition results are exactly the same, or the start position or end position of one initial speech recognition result contains the entire content of another initial speech recognition result.

[0098] S502: Determine whether there is a time intersection between the start and end times of the valid speech segments of the two channels. If so, execute S503; otherwise, execute S510.

[0099] S503: Extracting speech signals corresponding to the intersection of the contents of the valid speech segments of the two channels.

[0100] S504: Determine the voiceprint feature vector of each speech signal through a voiceprint recognition algorithm.

[0101] S505: Determine the voiceprint similarity between the voiceprint feature vectors.

[0102] S506: Determine whether the voiceprint similarity is greater than a preset voiceprint similarity threshold, if so, execute S507, otherwise, execute S510.

[0103] S507: Determine that the voiceprint features of the content intersection in the valid speech segments of the two channels are similar.

[0104] S508: For each speech signal, determine the energy of the speech signal by summing the squares of the amplitudes of the speech signal.

[0105] S509: Compare the energies of the two speech signals, save the content intersection in the initial speech recognition result corresponding to the valid speech segment with greater energy, and delete the content intersection in the initial speech recognition result corresponding to the valid speech segment with less energy.

[0106] S510: Obtain the next two valid voice segments.

[0107] Embodiment 3:

[0108] Based on the same inventive concept, the present application also provides a multi-channel speech recognition device, Figure 6 A schematic diagram of the structure of a multi-channel speech recognition device provided in an embodiment of the present application, the device comprising:

[0109] An acquisition unit 61 is used to acquire the voice data of each channel in the same time period;

[0110] An activity detection unit 62, used to determine the valid speech segments respectively included in each of the speech data and the start and end time of each of the valid speech segments;

[0111] A speech recognition unit 63, used to obtain an initial speech recognition result of each of the valid speech segments;

[0112] The signal separation unit 64 is used to adjust the initial speech recognition results of each valid speech segment according to whether the initial speech recognition results of each valid speech segment meet the preset signal mixing condition to obtain the final speech recognition results of each valid speech segment, including:

[0113] For the valid speech segments of any two channels, determine whether the valid speech segments of the two channels meet the signal mixing condition; wherein, the signal mixing condition includes: there is a content intersection between the initial speech recognition results of the valid speech segments of the two channels and there is a time intersection between the start and end times of the valid speech segments of the two channels; the content intersection means that the contents of the two initial speech recognition results are exactly the same, or the start position or end position of one initial speech recognition result contains all the contents of the other initial speech recognition result; if it is determined that the valid speech segments of the two channels meet the preset signal mixing condition, then compare the energies corresponding to the content intersection in the valid speech segments of the two channels, save the content intersection in the initial speech recognition result corresponding to the valid speech segment with large energy, and delete the content intersection in the initial speech recognition result corresponding to the valid speech segment with small energy.

[0114] The multi-channel speech recognition device in this embodiment is presented in the form of a functional module, where the module refers to an application specific integrated circuit (ASIC), a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0115] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0116] Embodiment 4:

[0117] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present application, such as Figure 7As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0118] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0119] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0120] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0121] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0122] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0123] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0124] Embodiment 5:

[0125] On the basis of the above embodiments, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program executable by a processor is stored. When the program runs on the processor, the processor implements the following steps when executing:

[0126] Acquire the voice data of each channel in the same time period;

[0127] Determine the valid speech segments respectively included in each of the speech data and the start and end time of each of the valid speech segments;

[0128] Obtaining initial speech recognition results for each of the valid speech segments;

[0129] According to whether the initial speech recognition results of the valid speech segments satisfy a preset signal mixing condition, the initial speech recognition results of the valid speech segments are adjusted to obtain the final speech recognition results of the valid speech segments, including:

[0130] For the valid speech segments of any two channels, determine whether the valid speech segments of the two channels meet the signal mixing condition; wherein, the signal mixing condition includes: there is a content intersection between the initial speech recognition results of the valid speech segments of the two channels and there is a time intersection between the start and end times of the valid speech segments of the two channels; the content intersection means that the contents of the two initial speech recognition results are exactly the same, or the start position or end position of one initial speech recognition result contains all the contents of the other initial speech recognition result; if it is determined that the valid speech segments of the two channels meet the preset signal mixing condition, then compare the energies corresponding to the content intersection in the valid speech segments of the two channels, save the content intersection in the initial speech recognition result corresponding to the valid speech segment with large energy, and delete the content intersection in the initial speech recognition result corresponding to the valid speech segment with small energy.

[0131] Since the principle of solving the problem by the above computer-readable storage medium is similar to that of the multi-channel speech recognition method, the implementation of the above computer-readable storage medium can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0132] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A multi-channel speech recognition method, characterized in that: The method comprises: Acquire the voice data of each channel in the same time period; Determine the valid speech segments respectively included in each of the speech data and the start and end time of each of the valid speech segments; Obtaining initial speech recognition results for each of the valid speech segments; According to whether the initial speech recognition results of the valid speech segments satisfy a preset signal mixing condition, the initial speech recognition results of the valid speech segments are adjusted to obtain the final speech recognition results of the valid speech segments, including: For the valid speech segments of any two channels, determine whether the valid speech segments of the two channels meet the signal mixing condition; wherein, the signal mixing condition includes: there is a content intersection between the initial speech recognition results of the valid speech segments of the two channels and there is a time intersection between the start and end times of the valid speech segments of the two channels; the content intersection means that the contents of the two initial speech recognition results are exactly the same, or the start position or end position of one initial speech recognition result contains all the contents of the other initial speech recognition result; if it is determined that the valid speech segments of the two channels meet the preset signal mixing condition, then compare the energies corresponding to the content intersection in the valid speech segments of the two channels, save the content intersection in the initial speech recognition result corresponding to the valid speech segment with large energy, and delete the content intersection in the initial speech recognition result corresponding to the valid speech segment with small energy.

2. The method according to claim 1, characterized in that The effective speech segments respectively included in each of the speech data and the start and end time of each of the effective speech segments are determined through a speech activity detection algorithm.

3. The method according to claim 1, characterized in that The signal mixing condition also includes: the voiceprint features of the valid speech segments of the two channels whose contents intersect are similar.

4. The method according to claim 3, characterized in that Determine that the voiceprint features of the valid speech segments of the two channels where the content is intersected are similar, including: Extracting speech signals corresponding to the intersection of the contents of the valid speech segments of the two channels; Determining the voiceprint feature vector of each of the voice signals through a voiceprint recognition algorithm; Determining the voiceprint similarity between the voiceprint feature vectors; If the voiceprint similarity is greater than a preset voiceprint similarity threshold, it is determined that the voiceprint features of the valid speech segments of the two channels whose contents intersect are similar.

5. The method according to claim 1, characterized in that The method further comprises: The final speech recognition results of each of the valid speech segments are displayed.

6. The method according to claim 5, characterized in that The method further comprises: Receive modification operations on any final speech recognition result; The final speech recognition result is updated based on the modification content carried in the modification operation.

7. The method according to claim 1, characterized in that Determine the energy corresponding to the content intersection in the valid speech segments of the two channels, including: Extracting speech signals corresponding to the intersection of the contents of the valid speech segments of the two channels; For each of the speech signals, the energy of the speech signal is determined by summing the squares of the amplitudes of the speech signals.

8. A multi-channel speech recognition device, characterized in that: The device comprises: An acquisition unit, used for acquiring the voice data of each channel in the same time period; An activity detection unit, used to determine the valid speech segments respectively included in each of the speech data and the start and end time of each of the valid speech segments; A speech recognition unit, used to obtain an initial speech recognition result of each of the valid speech segments; A signal separation unit, used to adjust the initial speech recognition results of each valid speech segment according to whether the initial speech recognition results of each valid speech segment meet a preset signal mixing condition, so as to obtain a final speech recognition result of each valid speech segment, including: For the valid speech segments of any two channels, determine whether the valid speech segments of the two channels meet the signal mixing condition; wherein, the signal mixing condition includes: there is a content intersection between the initial speech recognition results of the valid speech segments of the two channels and there is a time intersection between the start and end times of the valid speech segments of the two channels; the content intersection means that the contents of the two initial speech recognition results are exactly the same, or the start position or end position of one initial speech recognition result contains all the contents of the other initial speech recognition result; if it is determined that the valid speech segments of the two channels meet the preset signal mixing condition, then compare the energies corresponding to the content intersection in the valid speech segments of the two channels, save the content intersection in the initial speech recognition result corresponding to the valid speech segment with large energy, and delete the content intersection in the initial speech recognition result corresponding to the valid speech segment with small energy.

9. A computer device, characterized in that: The computer device comprises a processor, and the processor is used to implement the steps of the multi-channel speech recognition method as claimed in any one of claims 1 to 7 when executing a computer program stored in a memory.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the multi-channel speech recognition method as described in any one of claims 1 to 7.

Images