Speech recognition methods, devices and electronic equipment

Abstract

This application provides a speech recognition method, apparatus, and electronic device. The speech recognition method includes: acquiring speech to be recognized, wherein the speech to be recognized is multi-channel audio and is emitted by multiple sound sources; acquiring the voiceprint features of the multiple sound sources, wherein the sound sources and voiceprint features correspond one-to-one; inputting the speech to be recognized and the voiceprint features into a pre-trained speech recognition model; and recognizing the speech based on the spatial information and voiceprint features of the speech to be recognized to obtain the recognized text corresponding to the speech to be recognized. The recognized text includes: a text segment and the corresponding sound source identifier of the text segment. This application can accurately recognize the text corresponding to multi-channel audio and the sound source.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to speech recognition methods, devices and electronic devices. Background Technology

[0002] Multichannel audio refers to audio recorded by multiple recording devices. For example, in a conference setting, there are multiple speakers and multiple microphones placed in different locations, with multiple microphones recording simultaneously to obtain multichannel audio.

[0003] In related technologies, when recognizing multi-channel audio, the multi-channel audio is usually beamformed to obtain single-channel audio, and then the single-channel speaker-related speech recognition model is used to perform speech recognition of the single-channel audio. This method has the problem of low recognition accuracy. Summary of the Invention

[0004] This application provides various aspects of speech recognition methods, apparatus, and electronic devices to improve the recognition accuracy of multi-channel audio.

[0005] The first aspect of this application provides a speech recognition method, comprising: acquiring speech to be recognized, wherein the speech to be recognized is multi-channel audio and is emitted by multiple sound sources; acquiring the voiceprint features of the multiple sound sources, wherein the sound sources and voiceprint features correspond one-to-one; inputting the speech to be recognized and the voiceprint features into a pre-trained speech recognition model; and recognizing the speech based on the spatial information and voiceprint features of the speech to be recognized to obtain the recognized text corresponding to the speech to be recognized, wherein the recognized text includes: a text segment and a corresponding sound source identifier of the text segment.

[0006] The second aspect of this application provides a speech recognition method applied to a terminal device. The speech recognition method includes: acquiring speech to be recognized, wherein the speech to be recognized is multi-channel audio; sending the speech to be recognized to a cloud server to obtain the recognized text corresponding to the speech to be recognized, wherein the recognized text includes a text segment and a sound source identifier corresponding to the text segment, and the cloud server obtains the recognized text corresponding to the speech to be recognized based on the method of the first aspect.

[0007] A third aspect of this application provides a voice recognition device, comprising:

[0008] The first acquisition module is used to acquire the speech to be recognized, which is multi-channel audio and is emitted by multiple sound sources;

[0009] The second acquisition module is used to acquire the voiceprint features of multiple sound sources, with a one-to-one correspondence between the sound sources and the voiceprint features;

[0010] The recognition module is used to input the speech to be recognized and the voiceprint features into a pre-trained speech recognition model. Based on the spatial information and voiceprint features of the speech to be recognized, the module recognizes the speech to be recognized and obtains the recognized text corresponding to the speech to be recognized. The recognized text includes: text fragments and the corresponding sound source identifiers of the text fragments.

[0011] A fourth aspect of this application provides a speech recognition system, including:

[0012] Cloud servers and terminal devices, with speech recognition models deployed on the cloud servers;

[0013] The terminal device is used to acquire the speech to be recognized and send the speech to be recognized to the cloud server. The speech to be recognized is multi-channel audio.

[0014] A cloud server is used to acquire the speech to be recognized, which is multi-channel audio and is emitted by multiple sound sources; the voiceprint features of multiple sound sources are acquired, and there is a one-to-one correspondence between the sound sources and the voiceprint features; the speech to be recognized and the voiceprint features are input into a pre-trained speech recognition model, and the speech to be recognized is recognized based on the spatial information and voiceprint features of the speech to be recognized, so as to obtain the recognized text corresponding to the speech to be recognized. The recognized text includes: text fragments and the corresponding sound source identifiers of the text fragments;

[0015] The terminal device is used to receive the identified text sent by the cloud server. The identified text includes text fragments and the corresponding sound source identifiers of the text fragments.

[0016] A fifth aspect of this application provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the speech recognition method of the first aspect.

[0017] A sixth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the speech recognition method of the first aspect.

[0018] This application embodiment is applied to a speech recognition scenario. It obtains the speech to be recognized, which is multi-channel audio and is emitted by multiple sound sources; it obtains the voiceprint features of multiple sound sources, with a one-to-one correspondence between the sound sources and the voiceprint features; it inputs the speech to be recognized and the voiceprint features into a pre-trained speech recognition model, and recognizes the speech based on the spatial information and voiceprint features of the speech to be recognized, thereby obtaining the recognized text corresponding to the speech to be recognized. The recognized text includes: text fragments and corresponding sound source identifiers of the text fragments, so as to accurately identify the text and sound source of any part of the speech in the multi-channel audio. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 An application scenario diagram provided for an exemplary embodiment of this application;

[0021] Figure 2 A flowchart illustrating the steps of a speech recognition method provided in an exemplary embodiment of this application;

[0022] Figure 3 A schematic diagram of a speech recognition method provided for an exemplary embodiment of this application;

[0023] Figure 4 A schematic diagram of an attention mechanism provided for an exemplary embodiment of this application;

[0024] Figure 5 A schematic diagram of another attention mechanism provided for an exemplary embodiment of this application;

[0025] Figure 6 A schematic diagram of another speech recognition method provided for an exemplary embodiment of this application;

[0026] Figure 7 A schematic diagram of yet another attention mechanism provided for an exemplary embodiment of this application;

[0027] Figure 8 A schematic diagram illustrating another speech recognition method provided as an exemplary embodiment of this application;

[0028] Figure 9 A schematic diagram of a multi-channel speech separation sub-model provided for an exemplary embodiment of this application;

[0029] Figure 10 A structural block diagram of a speech recognition device provided for an exemplary embodiment of this application;

[0030] Figure 11 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] Speaker-related speech recognition in conference settings is a hot research topic and a challenging problem in speech signal processing, also known as the cocktail party problem. Its goal is to identify multiple speakers, potentially with overlapping speech, and assign a corresponding speaker to each identified text segment, i.e., to identify who said what. Because conference settings typically use arrays of multiple microphones to record audio, the different positions of the microphones result in rich spatial information in the multi-channel audio, which is very helpful for multi-speaker speech recognition. However, current technologies for multi-channel audio recognition often rely solely on single-channel audio obtained through beamforming, leading to inaccuracies in the identified text.

[0033] To address the aforementioned issues, this application provides a speech recognition method, comprising: acquiring a speech to be recognized, wherein the speech to be recognized is multi-channel audio and is emitted by multiple sound sources; acquiring the voiceprint features of the multiple sound sources, wherein the sound sources and voiceprint features correspond one-to-one; inputting the speech to be recognized and the voiceprint features into a pre-trained speech recognition model; and recognizing the speech based on the spatial information and voiceprint features of the speech to be recognized to obtain the recognized text corresponding to the speech to be recognized, wherein the recognized text includes: a text segment and a corresponding sound source identifier. This application directly recognizes multi-channel audio, which can make full use of the spatial information in multi-channel audio to improve the recognition efficiency of multi-channel audio.

[0034] In this embodiment, the device executing the speech recognition method is not limited. Optionally, the speech recognition method can be applied to a terminal, or the overall speech recognition method can be implemented using a cloud computing system. For example, the speech recognition method can be applied to a cloud server to leverage the advantages of cloud resources to run various models; instead of applying it to the cloud, the speech recognition method can also be applied to server-side devices such as conventional servers, cloud servers, or server arrays.

[0035] In addition, refer to Figure 1 This is one application scenario of this application. Figure 10 This application scenario Figure 10 It includes multiple sound sources (speaker A, speaker B, speaker C and speaker D) and a recording device 11 (including 8 microphones). The recording device can record the surrounding sounds to obtain multi-channel audio. If there are 8 microphones, it is 8-channel audio. The multi-channel audio includes the speech of at least one speaker. If there are multiple speakers, this application is used to identify which speaker said what in the multi-channel audio.

[0036] Figure 1 This is merely one example of an application scenario exemplified by this application. This application can also be applied to other related scenarios, which are not limited here.

[0037] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0038] Figure 2 A flowchart illustrating the steps of a speech recognition method provided as an exemplary embodiment of this application. Figure 2 The speech recognition method shown includes the following steps:

[0039] S201, Obtain the speech to be recognized.

[0040] The speech to be recognized is multi-channel audio, and the speech to be recognized is emitted by multiple sound sources.

[0041] In this embodiment of the application, the sound source includes the speaker, such as... Figure 1 Speaker A, Speaker B, Speaker C, and Speaker D are the sound sources. Furthermore, the speech to be recognized is multi-channel audio, such as... Figure 1 If the speech to be recognized is obtained from 8 microphones, then the speech to be recognized is 8-channel audio.

[0042] S202, acquire voiceprint features from multiple sound sources.

[0043] Among them, the sound source and voiceprint features correspond one-to-one.

[0044] In this embodiment, the voiceprint features include timbre features and / or prosodic features. Specifically, the timbre feature is a feature vector obtained by a timbre feature extractor after extracting timbre features from the speech, and the prosodic feature is a feature vector obtained by a prosodic feature extractor after extracting prosodic features from the speech. Furthermore, both the timbre feature extractor and the prosodic feature extractor are pre-trained, and this application does not limit the specific extraction process.

[0045] For example, before the meeting begins, a segment of speech can be collected from each speaker, and then voiceprint features can be extracted from that speech segment to obtain corresponding voiceprint features, which contain corresponding sound source identifiers. (Refer to...) Figure 2Speaker A speaks a segment of speech a1 beforehand. Voiceprint features are extracted from this speech, resulting in voiceprint feature a2. Speaker A is then associated with voiceprint feature a2, and speaker A is identified as the sound source. Similarly, speaker B speaks a segment of speech b1. Voiceprint features are extracted from this speech, resulting in voiceprint feature b2. Speaker B is then associated with voiceprint feature b2, and speaker B is identified as the sound source. Speaker C speaks a segment of speech c1. Voiceprint features are extracted from this speech, resulting in voiceprint feature c2. Speaker C is then associated with voiceprint feature c2, and speaker C is identified as the sound source. Speaker D speaks a segment of speech d1. Voiceprint features are extracted from this speech, resulting in voiceprint feature d2. Speaker D is then associated with voiceprint feature d2, and speaker D is identified as the sound source. Therefore, the voiceprint features of multiple sound sources include: (speaker A, voiceprint feature a2), (speaker B, voiceprint feature b2), (speaker C, voiceprint feature c2), and (speaker D, voiceprint feature d2).

[0046] S203, input the speech to be recognized and the voiceprint features into the pre-trained speech recognition model, and recognize the speech based on the spatial information and voiceprint features of the speech to be recognized to obtain the recognized text corresponding to the speech to be recognized.

[0047] In this embodiment, the speaker's voice is transmitted to the microphone via air in the form of sound waves. Different microphones pick up the same speaker's voice, and the voice content is the same. However, because the relative positions of the microphones to the speaker are different, the spatial information of the voice picked up by different microphones is different. The speech recognition model of this application directly inputs the multi-channel voice to be recognized into the speech recognition model. The speech recognition model can use the spatial information of the voice to be recognized to accurately identify the content of the voice. Here, spatial information can be understood as the speaker's position relative to the recording device, and this position information can be reflected in the voice to be recognized.

[0048] Reference Figure 3 The speech recognition model includes a multi-channel speech recognition sub-model and a sound source recognition sub-model. The speech to be recognized and its voiceprint features are input into a pre-trained speech recognition model. Based on the spatial information and voiceprint features of the speech to be recognized, the model performs speech recognition to obtain the corresponding recognized text. This includes: inputting the speech to be recognized into the multi-channel speech recognition sub-model; performing speech recognition based on the spatial information of the speech to be recognized to obtain text segments, with adjacent text segments originating from different sound sources; inputting the speech to be recognized and its voiceprint features into the sound source recognition sub-model for sound source recognition to obtain sound source identifiers for the speech to be recognized at different time periods; and aligning the text segments and sound source identifiers according to the time sequence to determine the correspondence between the text segments and sound source identifiers.

[0049] Among them, the multi-channel speech recognition sub-model and the sound source recognition sub-model are both pre-trained. Figure 3 In this model, the multi-channel speech recognition sub-model recognizes the speech to be recognized and obtains multiple text segments, such as S1 / S2 / ... / Sn, where n is a positive integer. These text segments are arranged in a time sequence, for example, the timestamp corresponding to text segment S1 precedes the timestamp corresponding to text segment S2. Furthermore, adjacent text segments originate from different sound sources (speakers), for example, text segment S1 corresponds to speaker A, text segment S2 to speaker B, text segment S3 to speaker A, text segment S4 to speaker C, and text segment S5 to speaker B.

[0050] Furthermore, the sound source recognition sub-model implements time-frame-level speaker logging, i.e., to identify "who spoke during what time period," refer to... Figure 3 By inputting the voiceprint features of multiple speakers and the audio to be identified into the sound source recognition sub-model, the sound source identifier for each time period can be identified. For example, referring to... Figure 3 The sound source identification sub-model identifies speaker A as the sound source identifier for time period T1, speaker B as the sound source identifier for time period T2, and speaker B as the sound source identifier for time period Tn. Finally, by aligning the multiple text segments identified by the multi-channel speech recognition sub-model with the sound source identifiers identified by the sound source identification sub-model, the recognized text can be obtained. The recognized text indicates "which speaker spoke each text segment". In addition, the recognized text can also include the timestamps corresponding to the text segments, so the recognized text can indicate "which speaker spoke each text segment during what time period".

[0051] Furthermore, referring to Figure 4 The multi-channel speech recognition sub-model employs a multi-frame cross-channel attention (MFCCA) mechanism to recognize the speech to be recognized. MFCCA utilizes the complementarity between frame-level and channel-level information to recognize the speech. Specifically, in... Figure 4In the diagram, 41 represents the feature vector 41 obtained after processing the speech and voiceprint features to be recognized by the feature extraction layer (pre-trained). Feature vector 41 is a C×T×D feature vector, where C represents the number of channels of the speech to be recognized (e.g., 8 channels), T represents the duration of the speech to be recognized, and D represents the dimension (e.g., 256 dimensions). Feature vector 42 is the concatenation of channel information from several frames before and after feature vector 41, enabling MFCCA to focus on the contextual information of adjacent frames, thereby improving the ability to jointly model frame-level and channel-level contextual information. 43, 44, and 45 represent computation units, where computation unit 43 calculates feature vector 41 using pre-trained weight parameters. Computation unit 44 calculates feature vector 42 using pre-trained weight parameters. Computation unit 45 calculates feature vector 42 using pre-trained weight parameters. The result of computation unit 43 is multiplied by the result of computation unit 44 to obtain feature vector 46, and feature vector 45 is multiplied by the result of computation unit 45 to obtain feature vector 47. Feature vector 47 is used to represent text segments.

[0052] Furthermore, the sound source recognition sub-model employs a channel-level cross-channel attention (CLCCA) mechanism to recognize the speech to be identified. CLCCA can leverage spatial diversity and fine-grained modeling of channel information at each time step. The channel-level cross-channel attention mechanism is as follows: Figure 5 As shown, 51 represents the feature vector 51 obtained after the speech and voiceprint features to be recognized are processed by the feature extraction layer (pre-trained). Feature vector 51 is a C×T×D feature vector, where C represents the number of channels of the speech to be recognized (e.g., 8 channels), T represents the duration of the speech to be recognized, and D represents the dimension (e.g., 256 dimensions). 52, 53, and 54 represent computation units, which calculate feature vector 51 using pre-trained weight parameters. The result of computation unit 52 is multiplied by the result of computation unit 53 to obtain feature vector 55. Feature vector 55 is multiplied by the result of computation unit 54 to obtain feature vector 56. Feature vector 56 is used for source identification in each time period.

[0053] Reference Figure 6The speech recognition model includes a multi-channel speech recognition sub-model and a text recognition sub-model. The text segment includes at least one character. The speech to be recognized and the voiceprint features are input into the pre-trained speech recognition model. Based on the spatial information and voiceprint features of the speech to be recognized, the speech to be recognized is recognized to obtain the recognized text corresponding to the speech to be recognized. This includes: inputting the speech to be recognized into the multi-channel speech recognition sub-model, performing speech recognition on the speech to be recognized based on the spatial information of the speech to be recognized to obtain a text segment, with adjacent text segments coming from different sound sources; and inputting the text segment, the speech to be recognized, and the voiceprint features into the text recognition sub-model for recognition to obtain the sound source identifiers corresponding to the characters in the text segment.

[0054] exist Figure 6 In the process, the multi-channel speech recognition sub-model outputs multiple text segments, such as S1 / S2 / ... / Sn. Each text segment includes multiple characters, such as text segment S1 including characters w11, w12...; text segment S2 including characters w21, w22...; and text segment Sn including characters wn1, wn2.... Further, by inputting the text segments, the audio to be recognized, and the voiceprint features into the character recognition sub-model for recognition, the sound source identifier corresponding to each character in the text segment can be obtained, that is, the speaker who uttered each character can be identified. Figure 6 It is designed for character-level recognition, enabling more accurate identification of the text.

[0055] For example, refer to Figure 6 Text segment S1 is spoken by speaker A, and text w21 in text segment S2 is also spoken by speaker A. All other texts in text segment S2 except for w21 are spoken by speaker B.

[0056] Furthermore, referring to Figure 7 The text recognition sub-model implements character-level speaker logging, identifying which speaker uttered the text. This sub-model employs Frame-Level Cross-Channel Attention (FLCCA) to recognize text segments. Figure 7In the diagram, 71 represents the feature vector 71 obtained after processing the speech, voiceprint features, and text fragments to be recognized by the feature extraction layer (pre-trained). Feature vector 71 is a C×T×D feature vector, where C represents the number of channels of the speech to be recognized (e.g., 8 channels), T represents the duration of the speech to be recognized, and D represents the dimension (e.g., 256 dimensions). Averaging feature vector 71 yields feature vector 72, where the averaging process involves averaging all channels of feature vector 71 except for the Cth channel along the time dimension. 73, 74, and 75 represent computation units, where computation unit 73 calculates feature vector 71 using pre-trained weight parameters, computation unit 74 calculates feature vector 72 using pre-trained weight parameters, and computation unit 75 calculates feature vector 72 using pre-trained weight parameters. The result output by the calculation unit 73 is multiplied by the result output by the calculation unit 74 to obtain the feature vector 76. The feature vector is multiplied by the result output by the calculation unit 75 to obtain the feature vector 77. The feature vector 77 is used to characterize the sound source identifier corresponding to each character in the text segment (i.e., which speaker said the character).

[0057] Reference Figure 8 The speech recognition model includes a multi-channel speech separation sub-model and a speech recognition sub-model. The speech to be recognized and the speaker features are input into the pre-trained speech recognition model. Based on the spatial information and speaker features of the speech to be recognized, the speech is recognized to obtain the recognized text corresponding to the speech. This includes: inputting the speech to be recognized and the speaker features into the multi-channel speech separation sub-model for separation to obtain multiple speech segments, each speech segment corresponding to a sound source identifier; and inputting the speech segments from the multiple speech segments into the speech recognition sub-model for recognition to obtain the text segments corresponding to the speech segments.

[0058] Reference Figure 8 The multi-channel speech separation sub-model is used to separate the audio to be recognized into multiple speech segments, each corresponding to a speaker. The speech segments can be ordered according to a time sequence. Figure 8 In the process, speech segments y1, y2, and so on up to yn are identified. Each speech segment can carry a timestamp. Then, a speech recognition sub-model is used to recognize the speech segments corresponding to a single speaker, thus obtaining the recognized text.

[0059] Reference Figure 9The multi-channel speech separation sub-model includes a feature extraction layer, a network processing layer, a multi-channel weighted processing layer, and a decoding layer. The speech to be recognized and its voiceprint features are input into the multi-channel speech separation sub-model for separation, resulting in multiple speech segments. This includes: inputting the speech to be recognized into the feature extraction layer for feature extraction, obtaining a first feature vector representing the speech features; for the voiceprint features among the multiple voiceprint features, inputting the voiceprint features and the first feature vector into the network processing layer for feature matching extraction, obtaining a second feature vector matching the voiceprint features; inputting the first and second feature vectors into the multi-channel weighted processing layer for multi-channel weighted processing, obtaining a single-channel third feature vector; and inputting the third feature vector into the decoding layer for decoding, obtaining the corresponding speech segment.

[0060] Reference Figure 9 The feature extraction layer performs feature extraction on the audio to be recognized, obtaining a first feature vector of C×T×D. This first feature vector represents the audio to be recognized, where C represents the number of channels (e.g., 8 channels), T represents the duration of the speech, and D represents the dimension (e.g., 256 dimensions). The network processing layer, such as BF (a type of neural network), extracts the first feature vector and speaker features, resulting in multiple second feature vectors. Each second feature vector is an 8-channel feature vector, representing a speech segment spoken by the corresponding speaker in the audio. The multi-channel weighted processing layer weights these 8-channel feature vectors to obtain a single-channel third feature vector, which also represents a speech segment spoken by the corresponding speaker. The decoding layer decodes different third feature vectors to obtain the corresponding speech segments.

[0061] This application embodiment is applied to a speech recognition scenario. It obtains the speech to be recognized, which is multi-channel audio and is emitted by multiple sound sources; it obtains the voiceprint features of multiple sound sources, with a one-to-one correspondence between the sound sources and the voiceprint features; it inputs the speech to be recognized and the voiceprint features into a pre-trained speech recognition model, and recognizes the speech based on the spatial information and voiceprint features of the speech to be recognized, thereby obtaining the recognized text corresponding to the speech to be recognized. The recognized text includes: text fragments and corresponding sound source identifiers of the text fragments, so as to accurately identify the text and sound source of any part of the speech in the multi-channel audio.

[0062] This application also provides a speech recognition method applied to a terminal device. The speech recognition method includes: acquiring speech to be recognized, wherein the speech to be recognized is multi-channel audio; sending the speech to be recognized to a cloud server to obtain the recognized text corresponding to the speech to be recognized, wherein the recognized text includes a text segment and a sound source identifier corresponding to the text segment; and the cloud server obtaining the recognized text corresponding to the speech to be recognized based on the above method.

[0063] In addition, this application also provides a speech recognition system, including:

[0064] Cloud servers and terminal devices, with speech recognition models deployed on the cloud servers;

[0065] The terminal device is used to acquire the speech to be recognized and send the speech to be recognized to the cloud server. The speech to be recognized is multi-channel audio.

[0066] A cloud server is used to acquire the speech to be recognized, which is multi-channel audio and is emitted by multiple sound sources; the voiceprint features of multiple sound sources are acquired, and there is a one-to-one correspondence between the sound sources and the voiceprint features; the speech to be recognized and the voiceprint features are input into a pre-trained speech recognition model, and the speech to be recognized is recognized based on the spatial information and voiceprint features of the speech to be recognized, so as to obtain the recognized text corresponding to the speech to be recognized. The recognized text includes: text fragments and the corresponding sound source identifiers of the text fragments;

[0067] The terminal device is used to receive the identified text sent by the cloud server. The identified text includes text fragments and the corresponding sound source identifiers of the text fragments.

[0068] The specific implementation process is described above and will not be repeated here.

[0069] In this application embodiment, in addition to providing a speech recognition method, a speech recognition device is also provided, such as... Figure 10 As shown, the voice recognition device 100 includes:

[0070] The first acquisition module 101 is used to acquire the speech to be recognized, which is multi-channel audio and is emitted by multiple sound sources;

[0071] The second acquisition module 102 is used to acquire the voiceprint features of multiple sound sources, with a one-to-one correspondence between the sound sources and the voiceprint features;

[0072] The recognition module 103 is used to input the speech to be recognized and the voiceprint features into a pre-trained speech recognition model. Based on the spatial information and voiceprint features of the speech to be recognized, the speech to be recognized is recognized to obtain the recognized text corresponding to the speech to be recognized. The recognized text includes: text fragments and the corresponding sound source identifiers of the text fragments.

[0073] In an optional embodiment, the speech recognition model includes a multi-channel speech recognition sub-model and a sound source recognition sub-model. The recognition module 103 is specifically used to: input the speech to be recognized into the multi-channel speech recognition sub-model; perform speech recognition on the speech to be recognized based on the spatial information of the speech to be recognized to obtain text segments, with adjacent text segments originating from different sound sources; input the speech to be recognized and voiceprint features into the sound source recognition sub-model for sound source recognition to obtain sound source identifiers of the speech to be recognized at different time periods; and align the text segments and sound source identifiers according to the time sequence to determine the correspondence between the text segments and sound source identifiers.

[0074] In an optional embodiment, the speech recognition model includes a multi-channel speech recognition sub-model and a text recognition sub-model. The text segment includes at least one character. The recognition module 103 is specifically used to: input the speech to be recognized into the multi-channel speech recognition sub-model; perform speech recognition on the speech to be recognized based on the spatial information of the speech to be recognized to obtain a text segment; adjacent text segments originate from different sound sources; and input the text segment, the speech to be recognized, and the voiceprint features into the text recognition sub-model for recognition to obtain the sound source identifier corresponding to the character in the text segment.

[0075] In an optional embodiment, the speech recognition model includes a multi-channel speech separation sub-model and a speech recognition sub-model. The recognition module 103 is specifically used to input the speech to be recognized and the voiceprint features into the multi-channel speech separation sub-model for separation to obtain multiple speech segments, each speech segment corresponding to a sound source identifier; and for each speech segment among the multiple speech segments, input the speech segment into the speech recognition sub-model for recognition to obtain the text segment corresponding to the speech segment.

[0076] In an optional embodiment, the multi-channel speech separation sub-model includes: a feature extraction layer, a network processing layer, a multi-channel weighted processing layer, and a decoding layer. When the recognition module 103 inputs the speech to be recognized and the voiceprint features into the multi-channel speech separation sub-model for separation to obtain multiple speech segments, it specifically performs the following: inputting the speech to be recognized into the feature extraction layer for feature extraction to obtain a first feature vector representing the speech features to be recognized; for the voiceprint features among the multiple voiceprint features, inputting the voiceprint features and the first feature vector into the network processing layer for feature matching extraction to obtain a second feature vector matching the voiceprint features; inputting the first feature vector and the second feature vector into the multi-channel weighted processing layer for multi-channel weighted processing to obtain a single-channel third feature vector; and inputting the third feature vector into the decoding layer for decoding to obtain the corresponding speech segment.

[0077] The speech recognition device provided in this application embodiment is capable of [the following]. The specific implementation process is the same as described in the above method embodiment, and will not be repeated here.

[0078] In addition, this application also provides a voice recognition device (not shown) applied to a terminal device, comprising:

[0079] The acquisition module is used to acquire the speech to be recognized, which is multi-channel audio.

[0080] The sending module is used to send the speech to be recognized to the cloud server in order to obtain the recognized text corresponding to the speech. The recognized text includes text fragments and the sound source identifiers corresponding to the text fragments. The cloud server obtains the recognized text corresponding to the speech based on the method described above.

[0081] The specific implementation process is the same as described in the above embodiments, and will not be repeated here.

[0082] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0083] Figure 11 This is a schematic diagram of an electronic device provided as an exemplary embodiment of this application. The electronic device 110 is used to run the aforementioned speech recognition method and speech recognition method. For example... Figure 11 As shown, the electronic device includes a memory 114 and a processor 115.

[0084] Memory 114 is used to store computer programs and can be configured to store various other data to support operation on electronic devices. Memory 114 may be object storage (OSS).

[0085] The memory 114 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0086] The processor 115, coupled to the memory 114, is used to execute the computer program in the memory 114 for: acquiring speech to be recognized, wherein the speech to be recognized is multi-channel audio and is emitted by multiple sound sources; acquiring the voiceprint features of the multiple sound sources, wherein the sound sources and voiceprint features correspond one-to-one; inputting the speech to be recognized and the voiceprint features into a pre-trained speech recognition model, and recognizing the speech based on the spatial information and voiceprint features of the speech to be recognized, thereby obtaining the recognized text corresponding to the speech to be recognized, wherein the recognized text includes: text fragments and the corresponding sound source identifiers of the text fragments.

[0087] Optionally, the speech recognition model includes a multi-channel speech recognition sub-model and a sound source recognition sub-model. Specifically, when the processor 115 inputs the speech to be recognized and its voiceprint features into a pre-trained speech recognition model, and performs speech recognition based on the spatial information and voiceprint features of the speech to be recognized to obtain the corresponding recognized text, it inputs the speech to be recognized into the multi-channel speech recognition sub-model, performs speech recognition based on the spatial information of the speech to be recognized to obtain text segments, with adjacent text segments originating from different sound sources; inputs the speech to be recognized and its voiceprint features into the sound source recognition sub-model for sound source recognition, obtaining sound source identifiers for the speech to be recognized at different time periods; and aligns the text segments and sound source identifiers according to the time sequence to determine the correspondence between the text segments and the sound source identifiers.

[0088] Optionally, the speech recognition model includes a multi-channel speech recognition sub-model and a text recognition sub-model. The text segment includes at least one character. When the processor 115 inputs the speech to be recognized and the voiceprint features into the pre-trained speech recognition model, and performs speech recognition based on the spatial information and voiceprint features of the speech to be recognized to obtain the recognized text corresponding to the speech to be recognized, it specifically performs the following: inputting the speech to be recognized into the multi-channel speech recognition sub-model, performing speech recognition based on the spatial information of the speech to be recognized to obtain a text segment, wherein adjacent text segments originate from different sound sources; and inputting the text segment, the speech to be recognized, and the voiceprint features into the text recognition sub-model for recognition to obtain the sound source identifier corresponding to the character in the text segment.

[0089] Optionally, the speech recognition model includes a multi-channel speech separation sub-model and a speech recognition sub-model. When the processor 115 inputs the speech to be recognized and the voiceprint features into the pre-trained speech recognition model, and recognizes the speech based on the spatial information and voiceprint features of the speech to be recognized to obtain the recognized text corresponding to the speech to be recognized, specifically, it inputs the speech to be recognized and the voiceprint features into the multi-channel speech separation sub-model for separation to obtain multiple speech segments, each speech segment corresponding to a sound source identifier; for each speech segment among the multiple speech segments, it inputs the speech segment into the speech recognition sub-model for recognition to obtain the text segment corresponding to the speech segment.

[0090] In one optional embodiment, the multi-channel speech separation sub-model includes: a feature extraction layer, a network processing layer, a multi-channel weighted processing layer, and a decoding layer. When the processor 115 inputs the speech to be recognized and the voiceprint features into the multi-channel speech separation sub-model for separation to obtain multiple speech segments, it specifically performs the following: inputting the speech to be recognized into the feature extraction layer for feature extraction to obtain a first feature vector representing the speech features to be recognized; for the voiceprint features among the multiple voiceprint features, inputting the voiceprint features and the first feature vector into the network processing layer for feature matching extraction to obtain a second feature vector matching the voiceprint features; inputting the first feature vector and the second feature vector into the multi-channel weighted processing layer for multi-channel weighted processing to obtain a single-channel third feature vector; and inputting the third feature vector into the decoding layer for decoding to obtain the corresponding speech segment.

[0091] In one optional embodiment, the processor 115, coupled to the memory 114, is configured to execute a computer program in the memory 114, and is further configured to: acquire speech to be recognized, the speech to be recognized being multi-channel audio; send the speech to be recognized to a cloud server to obtain the recognized text corresponding to the speech to be recognized, the recognized text including text fragments and sound source identifiers corresponding to the text fragments, the cloud server obtaining the recognized text corresponding to the speech to be recognized based on the semantic method as described above.

[0092] Furthermore, such as Figure 11 As shown, the electronic device also includes other components such as a firewall 111, a load balancer 112, a communication component 116, and a power supply component 113. Figure 11 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 11 The components shown.

[0093] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the steps in the method described above.

[0094] Accordingly, embodiments of this application also provide a computer program product, including a computer program / instructions, which, when executed by a processor, cause the processor to implement the steps in the method described above.

[0095] The above Figure 11 The communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related text from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0096] The above Figure 11 The power supply component provides power to the various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0097] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0098] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable text processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable text processing device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0099] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable text processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable text processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0101] In a typical configuration, a computing device includes one or more processors (CPU and / or GPU), input / output interfaces, network interfaces, and memory.

[0102] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0103] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be used to store text by any method or technology. Text can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store text accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0104] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0105] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A speech recognition method, characterized in that, include: Acquire the speech to be recognized, wherein the speech to be recognized is multi-channel audio and is emitted by multiple sound sources; Acquire the voiceprint features of the multiple sound sources, wherein the sound sources and the voiceprint features correspond one-to-one; The speech to be recognized and the voiceprint features are input into a pre-trained speech recognition model. Based on the spatial information of the speech to be recognized and the voiceprint features, the speech to be recognized is recognized to obtain the recognized text corresponding to the speech to be recognized. The recognized text includes: a text segment and a corresponding sound source identifier for the text segment.

2. The speech recognition method according to claim 1, characterized in that, The speech recognition model includes a multi-channel speech recognition sub-model and a sound source recognition sub-model. The process involves inputting the speech to be recognized and the voiceprint features into a pre-trained speech recognition model, and recognizing the speech based on its spatial information and voiceprint features to obtain the corresponding recognized text. The speech to be recognized is input into the multi-channel speech recognition sub-model. Based on the spatial information of the speech to be recognized, speech recognition is performed on the speech to be recognized to obtain text segments. Adjacent text segments come from different sound sources. The speech to be identified and the voiceprint features are input into the sound source identification sub-model to perform sound source identification, thereby obtaining the sound source identifiers of the speech to be identified at different time periods; The text segments and the sound source identifiers are aligned according to the time sequence to determine the correspondence between the text segments and the sound source identifiers.

3. The speech recognition method according to claim 2, characterized in that, The sound source recognition sub-model uses a channel-level cross-channel attention mechanism to recognize the speech to be recognized.

4. The speech recognition method according to claim 1, characterized in that, The speech recognition model includes a multi-channel speech recognition sub-model and a text recognition sub-model. The text segment includes at least one character. The process involves inputting the speech to be recognized and the voiceprint features into a pre-trained speech recognition model, and recognizing the speech based on the spatial information of the speech to be recognized and the voiceprint features to obtain the recognized text corresponding to the speech to be recognized, including: The speech to be recognized is input into the multi-channel speech recognition sub-model. Based on the spatial information of the speech to be recognized, speech recognition is performed on the speech to be recognized to obtain text segments. Adjacent text segments come from different sound sources. The text fragment, the speech to be recognized, and the voiceprint features are input into the text recognition sub-model for recognition, and the sound source identifiers corresponding to the text fragments are obtained.

5. The speech recognition method according to claim 4, characterized in that, The text recognition sub-model uses a frame-level cross-channel attention mechanism to recognize the text fragments.

6. The speech recognition method according to any one of claims 2 to 5, characterized in that, The multi-channel speech recognition sub-model uses a multi-frame cross-channel attention mechanism to recognize the speech to be recognized.

7. The speech recognition method according to claim 1, characterized in that, The speech recognition model includes a multi-channel speech separation sub-model and a speech recognition sub-model. The process involves inputting the speech to be recognized and the voiceprint features into a pre-trained speech recognition model, and recognizing the speech based on its spatial information and voiceprint features to obtain the recognized text corresponding to the speech. The speech to be identified and the voiceprint features are input into the multi-channel speech separation sub-model for separation to obtain multiple speech segments, each of which corresponds to a sound source identifier. For each of the multiple speech segments, the speech segment is input into the speech recognition sub-model for recognition, and the corresponding text segment is obtained.

8. The speech recognition method according to claim 7, characterized in that, The multi-channel speech separation sub-model includes: a feature extraction layer, a network processing layer, a multi-channel weighted processing layer, and a decoding layer. The speech to be recognized and the speaker features are input into the multi-channel speech separation sub-model for separation, resulting in multiple speech segments, including: The speech to be recognized is input into the feature extraction layer for feature extraction to obtain a first feature vector representing the speech features of the speech to be recognized; For the voiceprint feature among multiple voiceprint features, the voiceprint feature and the first feature vector are input into the network processing layer for feature matching and extraction to obtain a second feature vector that matches the voiceprint feature; The first feature vector and the second feature vector are input into the multi-channel weighted processing layer for multi-channel weighted processing to obtain the third feature vector of a single channel; The third feature vector is input into the decoding layer for decoding to obtain the corresponding speech segment.

9. A speech recognition method, characterized in that, The voice recognition method, applied to terminal devices, includes: Acquire the speech to be recognized, wherein the speech to be recognized is multi-channel audio; The speech to be recognized is sent to a cloud server to obtain the recognized text corresponding to the speech to be recognized. The recognized text includes a text segment and a sound source identifier corresponding to the text segment. The cloud server obtains the recognized text corresponding to the speech to be recognized based on the method described in any one of claims 1 to 8.

10. A voice recognition device, characterized in that, include: The first acquisition module is used to acquire the speech to be recognized, wherein the speech to be recognized is multi-channel audio and is emitted by multiple sound sources; The second acquisition module is used to acquire the voiceprint features of the plurality of sound sources, wherein the sound sources and the voiceprint features correspond one-to-one. The recognition module is used to input the speech to be recognized and the voiceprint features into a pre-trained speech recognition model, and to recognize the speech based on the spatial information of the speech to be recognized and the voiceprint features to obtain the recognized text corresponding to the speech to be recognized. The recognized text includes: a text segment and a corresponding sound source identifier of the text segment.

11. A speech recognition system, characterized in that, include: A cloud server and a terminal device, wherein a speech recognition model is deployed on the cloud server; The terminal device is used to acquire the speech to be recognized and send the speech to be recognized to the cloud server, wherein the speech to be recognized is multi-channel audio; The cloud server is used to acquire the speech to be recognized, which is multi-channel audio and is emitted by multiple sound sources; acquire the voiceprint features of the multiple sound sources, with each sound source and voiceprint feature corresponding one-to-one; input the speech to be recognized and the voiceprint features into a pre-trained speech recognition model, and recognize the speech based on the spatial information of the speech to be recognized and the voiceprint features to obtain the recognized text corresponding to the speech to be recognized, wherein the recognized text includes: a text segment and a corresponding sound source identifier for the text segment; The terminal device is used to receive recognition text sent by the cloud server, the recognition text including text fragments and the corresponding sound source identifiers of the text fragments.

12. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the speech recognition method as described in any one of claims 1 to 9.

Images