Speech recognition method, apparatus, device, storage medium and product

Abstract

The present disclosure provides a speech recognition method, device and equipment, and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical field of deep learning and speech. The speech recognition method comprises: obtaining speech data; extracting acoustic feature vectors of the speech data according to at least two frame lengths; clustering the acoustic feature vectors to obtain a speaker label and a time label corresponding to the speaker label; and identifying the speech data according to the speaker label and the time label to generate a speech recognition result with the speaker label and the time label. The speech recognition method can improve the accuracy of distinguishing different speakers in the speech data, improve the accuracy and reliability of the speech recognition result, and can be applied to real-time speech recognition scenarios such as meetings and interviews.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of deep learning and speech technology, and particularly to a speech recognition method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] With the rapid development of technology and society, information exchange and communication between people have become increasingly frequent and important. In scenarios such as seminars, training courses, internal meetings, product demonstrations, surveys, and interviews, the recording and organization of communication content consumes a lot of human resources and is inefficient.

[0003] The basic function of speech-to-text products is to automatically transcribe speech content in scenarios such as meetings into text, greatly reducing the cost of organizing communication content and improving recording efficiency. Summary of the Invention

[0004] This disclosure provides a speech recognition method, apparatus, device, storage medium, and computer program product. The speech recognition method and apparatus can be applied to multi-person speech scenarios such as intelligent meetings and interviews, improving the ability to distinguish speakers.

[0005] According to a first aspect of this disclosure, a speech recognition method is provided, comprising:

[0006] Acquire voice data;

[0007] Extract acoustic feature vectors from speech data according to at least two frame lengths;

[0008] Clustering of acoustic feature vectors yields speaker labels and corresponding time labels;

[0009] Based on speaker tags and time tags, the speech data is identified, and speech recognition results with speaker tags and time tags are generated.

[0010] According to a second aspect of this disclosure, a voice recognition device is provided, comprising:

[0011] The acquisition module is configured to acquire voice data;

[0012] The extraction module is configured to extract acoustic feature vectors from speech data according to at least two frame lengths;

[0013] The clustering module is configured to cluster acoustic feature vectors to obtain speaker labels and time labels corresponding to the speaker labels.

[0014] The recognition module is configured to recognize speech data based on speaker tags and time tags, and generate speech recognition results with speaker tags and time tags.

[0015] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0016] At least one processor; and

[0017] A memory that is communicatively connected to at least one processor; wherein,

[0018] The memory stores instructions that can be executed by at least one processor, such that the instructions are executed by at least one processor to enable the at least one processor to perform the method provided in the first aspect.

[0019] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the method provided in the first aspect.

[0020] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided according to the first aspect.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0023] Figure 1 An exemplary system architecture to which the speech recognition method of this disclosure can be applied is shown;

[0024] Figure 2 A flowchart of one embodiment of the speech recognition method according to the present disclosure is shown;

[0025] Figure 3 A schematic diagram of the short-time Fourier transform process in the speech recognition method according to the present disclosure is shown;

[0026] Figure 4 A graph showing the relationship between time-domain resolution and frequency-domain resolution in short-time Fourier transform is presented.

[0027] Figure 5 A comparison of the recognition process of the speech recognition method in related technologies and the speech recognition method of this disclosure is shown;

[0028] Figure 6 A flowchart of a second embodiment of the speech recognition method according to the present disclosure is shown;

[0029] Figure 7 A schematic diagram of the structure of one embodiment of a speech recognition device according to the present disclosure is shown;

[0030] Figure 8 A block diagram of an electronic device used to implement the speech recognition method of this disclosure is shown. Detailed Implementation

[0031] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] Speech-to-text products can automatically transcribe the content of seminars, training sessions, internal meetings, product demonstrations, surveys, interviews, and other meetings into text, reducing the cost of organizing meeting content and improving meeting recording efficiency. However, a significant drawback of simple speech-to-text products is their inability to automatically distinguish speakers. Because the speeches of different speakers are mixed together, the transcription results need to be manually edited by meeting recorders. More importantly, without distinguishing the speeches of different speakers, it is impossible to analyze the intentions, positions, and referents of different speakers, which greatly limits the application of natural language understanding technology.

[0034] Some speech-to-text products on the market have added the function of automatically distinguishing speakers, but some products are not very effective in distinguishing speakers, while others require specific hardware support. Their high cost and limited usage conditions make it difficult to promote their use in many scenarios.

[0035] Voice technology in meeting or interview scenarios needs to be able to distinguish the speech of different speakers, thereby differentiating different speakers at the text level. This requirement is essentially to annotate different parts of the audio based on the speaker's voiceprint characteristics and to generate timestamps for different speakers' speaking periods to guide speech recognition.

[0036] Voiceprint segmentation and clustering technology is used to distinguish the speech of different speakers. Existing technologies, depending on the application scenario, often employ methods such as speech separation, clustering algorithms, and voiceprint extraction models.

[0037] Speech separation methods typically require a pre-registered speaker audio clip as a reference to separate specific sounds from the original audio. This makes the method unsuitable for scenarios where the number and identities of speakers are unknown.

[0038] Most existing clustering algorithms take segments obtained from complete recordings as input and cannot process real-time audio streams, thus failing to distinguish speakers in real time in scenarios such as meetings and interviews.

[0039] Existing speaker recognition models utilize neural networks to extract audio features. Despite numerous innovations in network structure and training strategies, they still primarily use MFCCs (Mel-Frequency Cepstral Coefficients) and spectrograms as inputs to the neural network. Spectrogram extraction requires segmenting the input audio into frames, with the length of each frame determining the spectrogram's resolution in both the time and frequency domains. While longer frame lengths reduce the minimum frequencies that can be captured, resulting in a more accurate spectrum, they also decrease the number of frames, thus reducing time-domain information. The fixed frame lengths in existing spectrogram technologies prevent the complete preservation of acoustic information, further impacting the feature extraction capabilities of the neural network.

[0040] This disclosure provides a speech recognition method that uses at least two frame lengths to segment speech data, thereby obtaining spectrograms with at least two frequency domain resolutions. These spectrograms are then aggregated to obtain an enhanced spectrogram. The acoustic feature vectors corresponding to the enhanced spectrogram are then clustered to obtain speaker labels and corresponding time labels. Speech data can be labeled using speaker labels and corresponding time labels. Speech data recognition results labeled with speaker labels and corresponding time labels can then be obtained, effectively distinguishing speakers in the speech recognition results, forming an accurate dialogue process, and improving the speech recognition accuracy in real-time dialogue scenarios.

[0041] It should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein all comply with relevant laws and regulations and do not violate public order and good morals. The user information obtained in this embodiment is not targeted at any specific user, nor does it reflect the personal information of any specific user.

[0042] Figure 1 An exemplary system architecture 100 is shown that can be applied to embodiments of the speech recognition methods or speech recognition devices disclosed herein.

[0043] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102, and a server 103. The network 102 is used to provide a communication link between the terminal device 101 and the server 103, and may include various connection types, such as wired communication links, wireless communication links, or fiber optic cables, etc.

[0044] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send information, etc. For example, various client applications can be installed on terminal device 101. Users can send voice recognition requests or voice data to be recognized to server 103 through terminal device 101, create voice recognition tasks on server 103 through terminal device 101, and obtain information such as voice recognition results from server 103 through terminal device 101.

[0045] Terminal device 101 can be hardware or software. When terminal device 101 is hardware, it can be various electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal device 101 is software, it can be installed in the aforementioned electronic devices. It can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are made here.

[0046] Server 103 can be either hardware or software. When server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 103 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0047] The speech recognition method provided in this embodiment is generally executed by server 103, and correspondingly, the speech recognition device is generally installed in server 103.

[0048] It should be noted that, Figure 1 The number of terminal devices 101, network 102, and server 103 shown is merely illustrative. Depending on implementation needs, any number of terminal devices 101, network 102, and server 103 can be included.

[0049] In this embodiment of the disclosure, the speech recognition method is executed by server 103, and the speech recognition result is sent to terminal device 101 with client installed. For example, the speech recognition result is sent to the client or browser page of terminal device 101 in text form or other visual form for display.

[0050] Figure 2A flow 200 of one embodiment of the speech recognition method according to the present disclosure is shown, with reference to Figure 2 As shown, the speech recognition method includes the following steps:

[0051] Step S201: Obtain voice data.

[0052] In this embodiment of the disclosure, the entity executing the speech recognition method is, for example... Figure 1 The server 103 shown acquires the voice data to be recognized. This voice data can be voice data generated or transmitted in real time, or it can be existing complete voice data; no limitation is made here.

[0053] The aforementioned entities can acquire voice data in various ways. For example, they can acquire voice data in real time through one or more voice acquisition devices (such as microphones) at the meeting or interview site; they can also acquire voice data in real time during communication (such as voice or video calls) through smart devices; and they can also acquire voice data sent or stored by terminals through networks.

[0054] In this embodiment of the disclosure, the voice data can be unprocessed voice data after acquisition, or it can be voice data that has undergone preliminary preprocessing during or after acquisition. Preliminary preprocessing may include noise reduction, voice enhancement, etc., to improve the audio quality of the voice data; it may also include filtering out silent segments where no one is speaking.

[0055] In some optional implementations, noise reduction and voice enhancement modules can be set up on the voice acquisition device. For example, voice data collected in scenarios such as shopping malls, restaurants, and train stations often has a lot of noise. This can be achieved using a voice acquisition device with noise reduction or voice enhancement functions. Alternatively, preprocessing can be performed using software or hardware with noise reduction and voice enhancement functions after acquisition and before output. The specific choice between noise reduction and enhancement of the voice data is not limited, but can be selectively made based on the actual voice scenario or the requirements for the recognition results.

[0056] In some alternative implementations, the interfaces in the audio processing (librosa) library can be used to detect audio activity, extract the valid audio portion from the speech data, and filter out silent portions where no one is speaking.

[0057] It should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein all comply with relevant laws and regulations and do not violate public order and good morals. The user information obtained in this embodiment is not targeted at any specific user, nor does it reflect the personal information of any specific user.

[0058] Step S202: Extract acoustic feature vectors from speech data according to at least two frame lengths.

[0059] In this embodiment of the disclosure, the entity executing the speech recognition method is, for example... Figure 1 The server 103 shown extracts the acoustic feature vectors of the speech data obtained in step S201 according to at least two frame lengths.

[0060] In this embodiment, the execution entity segments the speech data according to at least two different frame lengths, forming speech segments of different frame lengths. Then, it extracts the voiceprint features from each speech segment, and finally aggregates the voiceprint features corresponding to the speech segments of different frame lengths to obtain the acoustic feature vector of the speech data. By specifically analyzing the voiceprint features corresponding to speech segments of various frame lengths, both temporal and frequency domain resolutions of the speech segments are considered, effectively improving the accuracy of the acoustic feature vector.

[0061] For example, the acoustic feature vector of speech data includes a two-dimensional feature vector containing time-domain and frequency-domain information, such as a spectrogram.

[0062] In some optional implementations of the embodiments of this disclosure, extracting acoustic feature vectors of speech data according to at least two frame lengths includes: segmenting speech data according to at least two frame lengths to generate at least two frame data sets, wherein the frame data sets correspond to the frame lengths; and extracting acoustic feature vectors of speech data based on the at least two frame data sets.

[0063] In some alternative implementations, short-time Fourier transform can be used to segment the speech data into frames, and the speaker information of each frame can be extracted in units of frame length, and then aggregated into an acoustic feature vector, such as an enhanced spectrogram.

[0064] Figure 3 A schematic diagram of a short-time Fourier transform process 300 is shown, as follows: Figure 3 As shown, the speech data is divided into equal segments of a certain frame length. Then, the Fourier transform of each segmented data is performed to extract the corresponding voiceprint information. Finally, the Fourier transform results are stacked frame by frame to generate the acoustic feature vector of the speech data.

[0065] Then, the execution entity changes the frame length and repeats the above process. After changing the frame length at least once and completing the above process, the execution entity aggregates the acoustic feature vectors extracted based on the various frame lengths.

[0066] Since the voiceprint information of each part of the voice data formed in scenarios such as a multi-speaker conversation is different, if the Fourier transform is performed on the entire voice data, most of the local information will be filtered out; while the short-time Fourier transform can greatly retain the local information in the entire audio. By adjusting the duration parameter in the short-time Fourier transform, the adjustment of the frame length can be achieved. For example, the frame length of the short-time Fourier transform can be controlled at durations such as 12.5 ms, 18.75 ms, and 25 ms respectively.

[0067] Exemplarily, if the voice data is "We are Chinese", different sets of framed data can be generated according to different frame lengths. For example, the set of framed data generated according to the frame length of one character includes "I", "we", "are", "China", "people"; the set of framed data generated according to the frame length of two characters includes "We are", "China people"; the set of framed data generated according to the frame length of three characters includes "We are Chinese"; the set of framed data generated according to the frame length of six characters includes "We are Chinese".

[0068] This solution uses such multiple different frame lengths to segment the voice data respectively, which can not only extract the voiceprint information of each smaller local part, but also extract the voiceprint information of each larger local part, and can also extract the global voiceprint information, that is, the voiceprint information can be extracted from different frequency domain resolutions and different time domain resolutions respectively, thus effectively ensuring the extraction accuracy and accuracy of the voiceprint information, and further ensuring the accuracy of the acoustic feature vector.

[0069] It should be noted that in this solution, when segmenting the voice data according to each frame length, it is an equal-length segmentation. Among them, the insufficient part of the last segmentation length can be filled with spaces or blank content. For example, if "We are Chinese" is segmented according to the frame length of four characters, the generated set of framed data includes "We are Chi" and "na people**", where each "*" represents the space length of one Chinese character.

[0070] As Figure 4 shows the influence relationship diagram between the time domain resolution and the frequency domain resolution in the short-time Fourier transform. As shown in the figure, it can be seen that when segmenting voice data with a single frame length, it is impossible to balance the time domain resolution and the frequency domain resolution of the voice segment, thus affecting the extraction accuracy of the acoustic feature vector. In the embodiments of the present disclosure, at least two frame lengths are used. For example, three different frame lengths are controlled by the short-time Fourier transform at three durations of 12.5 ms, 18.75 ms, and 25 ms respectively, and the voice data is segmented respectively to obtain the set of framed data corresponding to each frame length, so as to effectively balance the frequency domain resolution and the time domain resolution of the voice data.

[0071] After the aforementioned execution entity segments and forms at least two sets of framed data, it extracts voiceprint information from the set of framed data corresponding to at least two different frame lengths to obtain voiceprint information with different temporal resolutions. Then, it aggregates the extracted voiceprint information to form the acoustic feature vector of the speech data.

[0072] In some optional implementations of the embodiments of this disclosure, the acoustic feature vector of the speech data is extracted based on at least two frame data sets, including: extracting the voiceprint information of each frame data in at least two frame data sets respectively; and aggregating the voiceprint information of each frame data to obtain the acoustic feature vector of the speech data.

[0073] One approach is to aggregate the voiceprint information corresponding to each frame according to the frame length, and then aggregate the aggregated results; another approach is to directly aggregate the voiceprint information corresponding to all frames simultaneously.

[0074] Because the speech data is segmented using at least two different frame lengths, the extracted speaker information from different frame data sets has different time-frequency resolutions, meaning it has different lengths in the time dimension. In some optional implementations, to avoid affecting the aggregation results by aggregating speaker information with different time-domain resolutions, the execution entity aggregates the speaker information corresponding to frame data of the same frame length according to the frame data set, and then aggregates the aggregation results corresponding to different frame data sets again to generate the acoustic feature vector of the speech data, thus obtaining the enhanced spectrogram of the speech data. This implementation can balance the fusion accuracy of speaker information in both the time and frequency domains, thereby ensuring the accuracy of the acoustic feature vector.

[0075] Step S203: Cluster the acoustic feature vectors to obtain speaker labels and time labels corresponding to the speaker labels.

[0076] In this embodiment of the disclosure, the entity executing the speech recognition method is, for example... Figure 1 The server 103 shown clusters the acoustic feature vectors extracted in step S202 to obtain speaker labels and time labels corresponding to the speaker labels.

[0077] Among them, the speaker tag is used to distinguish and mark different speakers in the voice data; the time tag corresponding to the speaker tag is used to mark the speaking time of different speakers.

[0078] In this embodiment, the execution entity clusters the acoustic feature vectors extracted in step S202 to distinguish the voiceprint information of different speakers and the speaking time of different speakers, thereby generating different speaker labels and time labels corresponding to each speaker label.

[0079] This embodiment clusters the acoustic feature vectors extracted in step S202 according to at least two frame lengths, taking into account both temporal and frequency domain resolution, which can effectively ensure the accuracy of speaker labels and time labels.

[0080] The acoustic feature vector obtained by segmenting speech data with a single frame length and extracting speakerprint information corresponds to a standard spectrogram, while the acoustic feature vector obtained by segmenting speech data with multiple frame lengths and extracting speakerprint information in this scheme corresponds to an enhanced spectrogram. The two spectrograms can be compared using clustering results. Table 1 below shows a comparison of clustering indices between the standard and enhanced spectrograms in an exemplary embodiment.

[0081] Table 1 Comparison of clustering indices between ordinary and enhanced spectrograms

[0082] Comparison parameters Confusion DER JER Ordinary spectrogram 18.60％ 41.93％ 37.43％ Enhanced spectrogram 15.90％ 39.23％ 36.88％

[0083] Among them, Confusion (the confusion rate) is the percentage of speech data where speaker voiceprint information is not distinguished from other speakers; DER (Diarization Error Rate) is the percentage of speech data where separation errors occur; and JER (Jaccard Error Rate) is the average separation error rate of all speaker voiceprint information in the speech data. It is evident that the smaller the values ​​of these three parameters, the more accurate the clustering results of the spectrogram.

[0084] As can be seen from the comparison in Table 1, in this embodiment of the present disclosure, at least two frame lengths are used to segment, extract acoustic feature vectors and cluster the speech data, which can effectively improve the accuracy of speaker differentiation in the speech data and improve the accuracy of speaker labels and time labels.

[0085] Step S204: Based on the speaker tags and time tags, identify the speech data and generate a speech recognition result with speaker tags and time tags.

[0086] In this embodiment of the disclosure, the entity executing the speech recognition method is, for example... Figure 1 The server 103 shown identifies speech data based on the speaker tags and time tags generated in step S203, and generates a speech recognition result with speaker tags and time tags.

[0087] In the speech recognition results generated in this embodiment, the speech content of different speakers is distinguished by speaker tags in order to clearly distinguish the positions, attitudes, etc. of different speakers; at the same time, the speaking time of different speakers is distinguished by time tags, thereby forming a complete dialogue process and distinguishing and recording it.

[0088] In this embodiment, during the process of recognizing speech data based on speaker tags and time tags, the complete speech data can be recognized directly; alternatively, the speech data can be divided into several parts based on speaker tags or time tags, and then each part can be recognized and marked, and then the recognition results can be combined into one.

[0089] In some optional implementations of the embodiments of this disclosure, the execution entity recognizes the speech data and generates text data corresponding to the speech data; then, it marks the content of the text data according to the speaker tag and time tag to distinguish different speakers and speaking times, thereby forming a speech recognition result with speaker tag and time tag.

[0090] In some optional implementations of the embodiments of this disclosure, recognizing speech data and generating speech recognition results with speaker tags and time tags based on speaker tags and time tags includes: segmenting speech data according to speaker tags and time tags to obtain sentence data corresponding to speaker tags and time tags; recognizing sentence data and generating sentence recognition results; and generating speech recognition results with speaker tags and time tags based on speaker tags, time tags, and sentence recognition results.

[0091] Specifically, segmenting speech data based on speaker tags and time tags can be done in two ways: either by segmenting the speech data based on time tags to generate sentence data with speaker tags and corresponding to different time tags, or by segmenting the speech data based on speaker tags to generate sentence data with time tags and corresponding to different speaker tags.

[0092] Then, the aforementioned execution entity identifies and segments the various sentence data, generates corresponding sentence recognition results, and integrates the various sentence recognition results according to certain rules, such as chronological order, to generate speech recognition results with speaker labels and time labels.

[0093] In this implementation, speech data is segmented using speaker tags and time tags, and then the segmented sentence data is recognized separately, thereby improving the recognition accuracy of speech data and the matching accuracy between speech data and speaker tags and time tags.

[0094] Figure 5The document compares the recognition process of the speech recognition method in the related art with that of the present disclosure, wherein (a) is the speech recognition process in the related art and (b) is the speech recognition process of the present disclosure.

[0095] like Figure 5 As shown, in related technologies, speech data is segmented according to a single frame length, and then acoustic feature vectors are extracted for clustering and recognition. In this disclosure, however, short-time Fourier transform is used to segment the speech data according to multiple different frame lengths, extract speakerprint information, fuse them to generate acoustic feature vectors, and then perform clustering and recognition. Therefore, the speech recognition method of this disclosure generates acoustic feature vectors with higher accuracy and reliability.

[0096] The speech recognition method provided in this disclosure uses at least two frame lengths to segment speech data, extracts acoustic feature vectors, and then clusters these acoustic feature vectors. This effectively improves the accuracy of speaker labels and corresponding time labels, enhancing the differentiation accuracy of different speakers in the speech data. Furthermore, by recognizing speech data based on speaker labels and time labels, the accuracy and reliability of speech recognition results are significantly improved. This speech recognition method can not only accurately identify and differentiate existing speech data by speaker, but also accurately identify and differentiate different speakers and speech content in real-time speech data generated in scenarios such as meetings and interviews, and mark and record them according to speaker labels and time labels.

[0097] The speech recognition method provided in this disclosure can cluster acoustic feature vectors in various ways. For example, the clustering process can be completed through clustering algorithms or clustering models (such as neural network models).

[0098] In some optional embodiments, the acoustic feature vectors are clustered to obtain speaker labels and time labels corresponding to the speaker labels, including: using a pre-trained clustering model to cluster the acoustic feature vectors to obtain speaker labels and time labels corresponding to the speaker labels.

[0099] The clustering model can be a single neural network model or a neural network model composed of multiple sub-models.

[0100] Clustering acoustic feature vectors using a pre-trained clustering model can improve clustering efficiency and ensure the accuracy and reliability of the clustering results.

[0101] Figure 6The flowchart 600 of one embodiment of the speech recognition method according to the present disclosure is shown. In this embodiment, the clustering model includes at least a generation sub-model, an allocation sub-model, and a transformation sub-model. The generation sub-model is used to generate different speaker labels, the allocation sub-model is used to allocate sentences in the speech data according to the speaker labels, and the transformation sub-model is used to determine the transition time between different speakers and generate corresponding time tags.

[0102] Reference Figure 6 As shown, the speech recognition method includes the following steps:

[0103] Step S601: Obtain voice data.

[0104] Step S602: Extract the acoustic feature vector of the speech data according to at least two frame lengths.

[0105] In this embodiment of the disclosure, the entity executing the speech recognition method is, for example... Figure 1 The server 103 shown executes the above steps S601-S602 in sequence. Steps S601-S602 are basically the same as steps S201-S202 in the previous embodiment. For the specific implementation, please refer to the above description of steps S201-S202, which will not be repeated here.

[0106] Step S603: Input the acoustic feature vector into the generator sub-model to obtain the speaker label.

[0107] In this embodiment of the disclosure, the entity executing the speech recognition method is, for example... Figure 1 The server 103 shown inputs the acoustic feature vectors extracted in step S602 into the sub-model to obtain different speaker labels.

[0108] In some optional implementations of the embodiments of this disclosure, a neural network model may be selected as the generating sub-model.

[0109] During the training of the generative sub-model, speech data or voiceprint feature vectors from multiple different speakers and corresponding speaker labels can be used for training. For example, the voiceprint feature vectors from multiple different speakers can be used as input to the generative sub-model, and one speaker label can be used as output in turn. The generative sub-model can be trained multiple times until it can output the corresponding speaker label based on the voiceprint feature vectors from multiple different speakers.

[0110] Step S604: Input the speaker label and speech data into the allocation sub-model to obtain the speech data corresponding to the speaker label.

[0111] In this embodiment of the disclosure, the entity executing the speech recognition method is, for example... Figure 1The server 103 shown inputs the speaker label and acoustic feature vector into the assigner model to obtain the speech data corresponding to the speaker label.

[0112] In this scheme, the allocation sub-model is used to determine which speaker label each voiceprint information in the acoustic feature vector belongs to. That is, the voiceprint information in the acoustic feature vector is matched and labeled with the speaker label, thereby outputting the voiceprint sequence corresponding to different speaker labels.

[0113] For example, the assigning sub-model can be a Bayesian nonparametric model, such as the Chinese restaurant process. The Chinese restaurant process is as follows: Suppose there are an unlimited number of tables in a Chinese restaurant. The first customer who comes to eat sits at the first table. For each subsequent customer, they choose a table to sit at according to the following rules: they choose to sit at an already occupied table according to a first probability, or they choose to sit at an empty table according to a second probability.

[0114] In actual dialogues, speakers who have spoken frequently in the past are more likely to speak frequently in subsequent dialogues. Therefore, the Chinese restaurant process is suitable for allocating voiceprint information in acoustic feature vectors according to different speaker labels.

[0115] In some optional implementations of the embodiments of this disclosure, the allocation sub-model is selected from the Chinese restaurant process model. Each time the allocation sub-model receives an input, such as a sentence in speech data or an acoustic feature vector, it assigns an existing speaker label to the input based on a first probability, and assigns a completely new speaker label to the input based on a second probability.

[0116] For example, the first probability of assigning an input to an existing speaker label in the allocation sub-model is positively correlated with the number of inputs already assigned under that speaker label.

[0117] Step S605: Input the voiceprint sequence corresponding to the speaker label into the conversion sub-model to obtain the time label corresponding to the speaker label.

[0118] In this embodiment of the disclosure, the entity executing the speech recognition method is, for example... Figure 1 The server 103 shown inputs the voiceprint sequence corresponding to the speaker tag obtained in step S604 into the conversion sub-model to obtain the time tag corresponding to the speaker tag.

[0119] In some optional implementations, the transformation sub-model can use a binomial distribution model to realize the time when two adjacent speaker tags in the voiceprint sequence alternate, for example, the time corresponding to when speaker A finishes speaking and speaker B begins speaking, that is, to determine the start and end time tags of the speaker tag transformation in the voiceprint sequence.

[0120] In some alternative implementations, the aforementioned execution entity may also determine only the time stamp of the start time when the speaker label changes in the voiceprint sequence.

[0121] Step S606: Based on the speaker tags and time tags, recognize the speech data and generate a speech recognition result with speaker tags and time tags.

[0122] In this embodiment of the disclosure, the entity executing the speech recognition method is, for example... Figure 1 The server 103 shown executes step S606 as described above. Step S606 is basically the same as step S204 in the previous embodiment. For the specific implementation method, please refer to the description of step S204 above, which will not be repeated here.

[0123] According to the speech recognition method provided in this disclosure, a generation sub-model obtains speaker labels based on acoustic feature vectors, an allocation sub-model obtains a voiceprint sequence corresponding to the speaker labels based on the speaker labels and acoustic feature vectors, and a conversion sub-model obtains a time label corresponding to the speaker labels based on the voiceprint sequence corresponding to the speaker labels. This completes the clustering processing of acoustic feature vectors, resulting in accurate speaker labels and time labels corresponding to the speaker labels, which can be used to accurately label speech data and improve the recognition accuracy of speech data.

[0124] As an implementation of the methods shown in the above figures, Figure 7 An embodiment of a speech recognition device according to the present disclosure is shown. This speech recognition device is related to... Figure 2 Corresponding to the method embodiments shown, this device can be applied to various electronic devices.

[0125] Reference Figure 7 As shown, the speech recognition device 700 provided in this embodiment includes: an acquisition module 701, an extraction module 702, a clustering module 703, and a recognition module 704. The acquisition module 701 is configured to acquire speech data; the extraction module 702 is configured to extract acoustic feature vectors from the speech data according to at least two frame lengths; the clustering module 703 is configured to cluster the acoustic feature vectors to obtain speaker labels and time labels corresponding to the speaker labels; and the recognition module 704 is configured to recognize the speech data based on the speaker labels and time labels, generating a speech recognition result with speaker labels and time labels.

[0126] In the speech recognition device 700 of this embodiment, the specific processing of the acquisition module 701, extraction module 702, clustering module 703, and recognition module 704, and the resulting technical effects, can be found in the following references: Figure 2The relevant descriptions of steps S201-S204 in the corresponding embodiments will not be repeated here.

[0127] In some optional implementations of embodiments of this disclosure, the extraction module 702 includes a segmentation unit and an extraction unit. The segmentation unit is configured to segment the speech data according to at least two frame lengths to generate at least two frame data sets, each frame data set corresponding to a frame length. The extraction unit is configured to extract acoustic feature vectors from the speech data based on the at least two frame data sets.

[0128] In some optional implementations of the embodiments of this disclosure, the extraction unit is configured to: extract the voiceprint information of each frame data in at least two frame data sets respectively; and aggregate the voiceprint information of each frame data to obtain the acoustic feature vector of the speech data.

[0129] In some optional implementations of the embodiments of this disclosure, the clustering module 703 is configured to: cluster the acoustic feature vectors using a pre-trained clustering model to obtain speaker labels and time labels corresponding to the speaker labels.

[0130] In some optional implementations of embodiments of this disclosure, the clustering model includes a generating sub-model, an assigning sub-model, and a transforming sub-model, and the clustering module 703 is configured to: input acoustic feature vectors into the generating sub-model to obtain speaker labels; input speaker labels and acoustic feature vectors into the assigning sub-model to obtain a voiceprint sequence corresponding to the speaker labels; and input the voiceprint sequence corresponding to the speaker labels into the transforming sub-model to obtain a time label corresponding to the speaker labels.

[0131] In the speech recognition device 700 of this embodiment, the specific processing of the clustering module 703 and the resulting technical effects can be found in [reference needed]. Figure 6 The relevant descriptions of steps S603-S605 in the corresponding embodiments will not be repeated here.

[0132] In some optional implementations of the embodiments of this disclosure, the recognition module 704 is configured to: segment speech data according to speaker tags and time tags to obtain sentence data corresponding to speaker tags and time tags; recognize the sentence data and generate a sentence recognition result; and generate a speech recognition result with speaker tags and time tags according to speaker tags, time tags and sentence recognition results.

[0133] According to embodiments of this disclosure, this disclosure also provides an electronic device, a non-transitory computer-readable storage medium storing computer instructions, and a computer program product.

[0134] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described speech recognition method.

[0135] In some embodiments, a non-transitory computer-readable storage medium storing computer instructions is provided, which are used to cause a computer to perform the above-described speech recognition method.

[0136] In some embodiments, a computer program product includes a computer program that, when executed by a processor, implements the above-described speech recognition method.

[0137] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0138] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0139] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0140] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as methods for generating backbone networks or image processing methods. For example, in some embodiments, a speech recognition method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the methods for generating backbone networks or image processing methods described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform a speech recognition method by any other suitable means (e.g., by means of firmware).

[0141] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0143] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0144] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0145] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0146] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0147] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0148] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A speech recognition method, comprising: Acquire voice data; The acoustic feature vector of the speech data is extracted according to at least two frame lengths. Cluster the acoustic feature vectors to obtain speaker labels and time labels corresponding to the speaker labels; Based on the speaker tag and the time tag, the speech data is identified, and a speech recognition result with the speaker tag and the time tag is generated.

2. The speech recognition method according to claim 1, wherein, The step of extracting the acoustic feature vector of the speech data according to at least two frame lengths includes: The speech data is segmented according to the at least two frame lengths to generate at least two frame data sets, wherein the frame data sets correspond to the frame lengths. Based on the at least two sets of segmented data, the acoustic feature vector of the speech data is extracted.

3. The speech recognition method according to claim 2, wherein, The step of extracting the acoustic feature vector of the speech data based on the at least two sets of segmented data includes: Extract the voiceprint information of each frame data from the at least two frame data sets respectively; The acoustic feature vector of the speech data is obtained by aggregating the voiceprint information of each frame of data.

4. The speech recognition method according to any one of claims 1-3, wherein, The process of clustering the acoustic feature vectors to obtain speaker labels and corresponding time labels includes: The acoustic feature vectors are clustered using a pre-trained clustering model to obtain the speaker labels and the time labels corresponding to the speaker labels.

5. The speech recognition method according to claim 4, wherein, The clustering model includes a generating sub-model, an assigning sub-model, and a transformation sub-model, and The step of clustering the acoustic feature vectors using a pre-trained clustering model to obtain the speaker labels and the corresponding time labels includes: The acoustic feature vector is input into the generating sub-model to obtain the speaker label; The speaker label and the acoustic feature vector are input into the allocation sub-model to obtain the voiceprint sequence corresponding to the speaker label; The voiceprint sequence corresponding to the speaker label is input into the transformation sub-model to obtain the time label corresponding to the speaker label.

6. The speech recognition method according to claim 1, wherein, The step of recognizing the speech data based on the speaker tag and the time tag, and generating a speech recognition result with the speaker tag and the time tag, includes: Based on the speaker tag and the time tag, the speech data is segmented to obtain the sentence data corresponding to the speaker tag and the time tag; Identify the statement data and generate statement identification results; Based on the speaker tag, the time tag, and the sentence recognition result, a speech recognition result with the speaker tag and the time tag is generated.

7. A voice recognition device, comprising: The acquisition module is configured to acquire voice data; The extraction module is configured to extract acoustic feature vectors from the speech data according to at least two frame lengths; The clustering module is configured to cluster the acoustic feature vectors to obtain speaker labels and time labels corresponding to the speaker labels; The recognition module is configured to recognize the speech data based on the speaker tag and the time tag, and generate a speech recognition result with the speaker tag and the time tag.

8. The speech recognition device according to claim 7, wherein, The extraction module includes: The segmentation unit is configured to segment the speech data according to the at least two frame lengths respectively, and generate at least two frame data sets, the frame data sets corresponding to the frame lengths; The extraction unit is configured to extract acoustic feature vectors from the speech data based on the at least two sets of segmented data.

9. The speech recognition device according to claim 8, wherein, The extraction unit is configured as follows: Extract the voiceprint information of each frame data from the at least two frame data sets respectively; The acoustic feature vector of the speech data is obtained by aggregating the voiceprint information of each frame of data.

10. The speech recognition device according to any one of claims 7-9, wherein, The clustering module is configured to cluster the acoustic feature vectors using a pre-trained clustering model to obtain the speaker labels and the time labels corresponding to the speaker labels.

11. The speech recognition device according to claim 10, wherein, The clustering model includes a generation sub-model, an allocation sub-model, and a transformation sub-model, and the clustering module is configured as follows: The acoustic feature vector is input into the generating sub-model to obtain the speaker label; The speaker label and the acoustic feature vector are input into the allocation sub-model to obtain the voiceprint sequence corresponding to the speaker label; The voiceprint sequence corresponding to the speaker label is input into the transformation sub-model to obtain the time label corresponding to the speaker label.

12. The speech recognition device according to claim 7, wherein the recognition module is configured as follows: Based on the speaker tag and the time tag, the speech data is segmented to obtain the sentence data corresponding to the speaker tag and the time tag; Identify the statement data and generate statement identification results; Based on the speaker tag, the time tag, and the sentence recognition result, a speech recognition result with the speaker tag and the time tag is generated.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images