Speaker recognition and voice transfer processing method and system based on multi-strategy fusion

Abstract

The invention discloses a multi-strategy fusion speaker recognition and voice transcription processing method and system, and the method comprises the steps: obtaining the real-time voice and / or voice file information of a target speaker, selecting at least one voice working mode, and carrying out the voiceprint analysis of the real-time voice and / or voice file information; performing voiceprint registration on the recognized target speaker according to the current voice working mode, and performing selectable ASR primary transcription on the registered voiceprint of the target speaker to obtain voice text information of the target speaker; performing selectable CTTransform secondary transcription on the voice text information of the target speaker according to the context information of the voice text information of the target speaker, and recovering punctuations of the voice text information of the target speaker; and mapping and storing the voice text information of the target speaker carrying the punctuation after the second-level transcription, the first-level transcription information or non-transcription information according to the registered identity tag of the target speaker.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a multi-strategy fusion speaker recognition and speech transcription processing method and system. Background Technology

[0002] In existing technologies, speaker recognition technology mostly adopts a single-mode voiceprint registration (such as only supporting real-time microphone acquisition or only supporting file upload). It achieves identity recognition by extracting voiceprint features from speech and comparing them with database templates. Moreover, voiceprint databases are mostly stored in conventional formats, with limited retrieval and expansion capabilities. Speech-to-text technology relies on a basic automatic speech recognition (ASR) engine to complete the conversion from speech to text. After transcription, the original text is directly output without optimizing for the readability of the text. Some systems also only support a single input method of either real-time speech or file input. The above-mentioned existing technologies have the following core technical problems: low speaker recognition accuracy, complex voiceprint registration process, lack of logicality in speech-to-text results, and insufficient system adaptability and scalability. Summary of the Invention

[0003] One of the objectives of this invention is to provide a multi-strategy fusion speaker recognition and speech transcription processing method and system. The method and system employ multi-mode voiceprint registration through methods including real-time microphone acquisition and voice file uploading. Furthermore, this invention uses binary SPKDB format voiceprint data to achieve rapid retrieval, horizontal expansion, and incremental updates of voiceprint data, enabling the speaker's speech recognition in this invention to balance recognition accuracy and processing efficiency in large-scale scenarios.

[0004] Another objective of this invention is to provide a multi-strategy fusion speaker recognition and speech transcription processing method and system. The method and system provide a two-level transcription process of the recognized voiceprint information using optional ASR+CTTransformer. First, optional ASR transcription is performed on speakers with specific tags. Then, based on the context of the ASR transcription, it is determined whether to enable secondary punctuation transcription using CTTransformer. Therefore, this invention can improve the contextual logic and readability of the transcribed text through the two-level transcription of ASR+CTTransformer.

[0005] Another objective of this invention is to provide a multi-strategy fusion speaker recognition and speech transcription processing method and system. The method and system provide multiple working modes for voiceprint registration only, separation only, and registration + separation, so that the technical solution of this invention can be adapted to various scenarios. In this invention, unregistered voiceprints are temporarily registered, thereby ensuring the stability and scalability of voiceprint recognition and transcription, and can be adapted to differentiated speech recognition records including only text content or speaker tags combined with timestamps.

[0006] To achieve at least one of the above-mentioned objectives, the present invention further provides a multi-strategy fusion speaker recognition and speech-to-text processing method, the method comprising:

[0007] Acquire real-time speech and / or speech file information of the target speaker, select at least one speech working mode, and perform voiceprint analysis on the real-time speech and / or speech file information according to the speech working mode;

[0008] Based on the current voice working mode, the voiceprint of the identified target speaker is registered, and the registered voiceprint of the target speaker is selectively transcribed using ASR Level 1 to obtain the voice text information of the target speaker.

[0009] Based on the context information of the target speaker's speech-text information, perform selectable CTTransformer secondary transcription on the target speaker's speech-text information to recover the punctuation of the target speaker's speech-text information;

[0010] The speech and text information of the target speaker, including the second-level transcription with punctuation, the first-level transcription information, or the untranscribed information, will be mapped and stored according to the registered target speaker identity tags.

[0011] According to a preferred embodiment of the present invention, the voice working mode includes: a registration-only mode, a separation-only mode, and a registration + separation mode, wherein the registration-only mode only performs new user voiceprint collection and storage, and is only used to register the identity information of the new speaker; the separation-only mode is used to match the input voice in the voiceprint database, and is only used to determine the identity information of the speaker corresponding to the input voice; the registration + separation mode is used to perform extended registration of the new speaker's voiceprint while performing speaker identity recognition, wherein the voice working mode is selected according to the scenario.

[0012] According to another preferred embodiment of the present invention, the voiceprint registration method includes: real-time acquisition of speaker speech information, extraction of voiceprint features from the speaker speech information, and embedding the extracted voiceprint features into a vector. The voiceprint feature extraction formula is as follows:

[0013] ;

[0014] This represents the speech feature vector extracted from the t-th speech frame. This indicates the t-th speech frame. Feature extraction function, Represents the discrete cosine transform. S represents the summation over the frequency dimension f from 1 to F. t (f) represents the short-time Fourier transform coefficients of speech frame t, M(f,k) represents the weight function of the k-th Mel filter, and T indicates that there are T speech frame indices.

[0015] According to another preferred embodiment of the present invention, the formula for calculating the embedding vector is:

[0016] ;

[0017] Let be the embedding vector, where σ() is the Sigmoid activation function and ReLU() is the rectified linear unit. and For different weight parameters, and These are the bias parameters, where T represents the existence of T speech frame indices. The embedding vector is used to aggregate frame-level information to generate a global voiceprint feature vector, and the global voiceprint feature vector is tagged and saved to complete real-time speech registration.

[0018] According to another preferred embodiment of the present invention, the voiceprint registration method includes: reading an input speaker audio file and performing voiceprint analysis on the audio file; extracting voiceprint features and calculating embedding vectors from the speaker's speech information to obtain a global voiceprint feature vector for aggregating frame-level information; and saving the global voiceprint feature vector obtained based on the audio file in a tagged manner to complete the audio file voice registration.

[0019] According to another preferred embodiment of the present invention, when the input voice information is acquired, the input voice information is matched with voiceprints and the voiceprint similarity formula and voiceprint database matching probability of the speaker's identity are determined. Furthermore, the identity label of the speaker corresponding to the current input voice is determined according to the voiceprint similarity formula and the voiceprint database matching probability. And according to the system configuration rules, it is determined whether the input voice information corresponding to the current identity label needs to be transcribed into first-level transcription and second-level transcription.

[0020] According to another preferred embodiment of the present invention, the system performs the following first-level transcription operation on the input speech information of a specific tag in advance, wherein the formula for the first-level transcription operation includes:

[0021] ;

[0022] Where P AM (X|Y) represents the probability that the acoustic model predicts the audio signal X given the word sequence Y, P LM (Y) is the probability of a given speech transcription result from the language model. This represents the final predicted word sequence. P(X|Y) represents the probability of finding the word sequence Y that maximizes the probability of the following words among all possible word sequences Y. Given an audio signal X, P(X|Y) represents the posterior probability of the word sequence Y appearing, thus obtaining the text information after first-level transcription.

[0023] According to another preferred embodiment of the present invention, after the first-level transcription action of the Automatic Speech Recognition (ASR) is completed, the context information of the transcribed text information is obtained, and a second-level transcription action is performed based on the context information, wherein the formula for the second-level transcription action is as follows:

[0024] ;

[0025] Where h i It is a context vector extracted from the text information after primary transcription, g ϕ It is a function of the punctuation restoration model, which predicts whether to insert a punctuation mark at position i, and M represents the total number of punctuation marks to be inserted.

[0026] To achieve at least one of the objectives of this invention, the present invention further provides a multi-strategy fusion speaker recognition and speech-to-text processing system, wherein the system executes the aforementioned multi-strategy fusion speaker recognition and speech-to-text processing method.

[0027] The present invention further provides a computer-readable storage medium storing a computer program, which is executed by a processor to implement the above-described multi-strategy fusion speaker recognition and speech transcription processing method. Attached Figure Description

[0028] Figure 1 The diagram shown is a flowchart of a multi-strategy fusion speaker recognition and speech transcription processing method according to the present invention. Detailed Implementation

[0029] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0030] It is understood that the term "a" should be understood as "at least one" or "one or more," that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "a" should not be understood as a limitation on the number.

[0031] Please combine Figure 1 This invention discloses a multi-strategy fusion speaker recognition and speech-to-text processing method and system, wherein the method mainly includes the following steps:

[0032] S01. Obtain the real-time voice and / or voice file information of the target speaker, select at least one voice working mode, and perform voiceprint analysis on the real-time voice and / or voice file information according to the voice working model;

[0033] S02. Register the voiceprint of the identified target speaker according to the current voice working mode, and perform optional ASR level 1 transcription on the registered voiceprint of the target speaker to obtain the voice text information of the target speaker.

[0034] S03. Based on the context information of the target speaker's speech-text information, perform selectable CTTransformer secondary transcription on the target speaker's speech-text information to recover the punctuation of the target speaker's speech-text information;

[0035] S04. Map and store the target speaker's voice text information, including the second-level transcription with punctuation, the first-level transcription information, or the untranscribed information, according to the registered target speaker identity tag.

[0036] Specifically, this invention includes a system initialization module, which can actively or automatically select one of three working modes based on the current scenario: registration-only mode, separation-only mode, and registration + separation mode. The registration-only mode only performs new user voiceprint collection and storage, and is used solely for registering the identity information of new speakers. The separation-only mode is used to match the input speech in the voiceprint database, and is used solely for determining the identity information of the speaker corresponding to the input speech. The registration + separation mode is used to perform extended registration of the new speaker's voiceprint while simultaneously performing speaker identity recognition. This invention utilizes these three different working modes to achieve flexible scenario adaptability. Furthermore, these three working modes can automatically assign temporary IDs to unregistered users and optionally allow dynamic registration, balancing stability and scalability.

[0037] Furthermore, the present invention also provides multiple types of voiceprint registration modules. These include a dynamic voiceprint registration module based on real-time speech and a voiceprint registration module based on audio files. The dynamic voiceprint registration module based on real-time speech mainly performs the following registration steps:

[0038] First, real-time speech data from the speaker is collected using a microphone or other real-time speech acquisition device. Then, voiceprint analysis is performed on the collected speech data. The voiceprint analysis method includes: performing voiceprint feature extraction; performing feature modeling on the feature vectors; and storing the generated features in a voiceprint database. The voiceprint feature extraction formula is as follows:

[0039] ;

[0040] This represents the speech feature vector extracted from the t-th speech frame. This indicates the t-th speech frame. Feature extraction function, Represents the discrete cosine transform. S represents the summation over the frequency dimension f from 1 to F. t (f) represents the short-time Fourier transform coefficients of speech frame t, M(f,k) represents the weight function of the k-th Mel filter, and T indicates that there are T speech frame indices.

[0041] Furthermore, feature modeling of the embedding vector is performed on the speech feature vector, wherein the formula for calculating the embedding vector is:

[0042] The formula for calculating the embedding vector is:

[0043] ;

[0044] Let be the embedding vector, where σ() is the Sigmoid activation function and ReLU() is the rectified linear unit. and For different weight parameters, and These are the bias parameters, where T represents the existence of T speech frame indices. The embedding vector is used to aggregate frame-level information to generate a global voiceprint feature vector, and the global voiceprint feature vector is tagged and saved to complete real-time speech registration.

[0045] Similarly, when the input is a speaker's audio file, the input speaker's audio file can be read, and voiceprint analysis can be performed on the audio file. Voiceprint features are extracted and embedding vectors are calculated from the speaker's speech information to obtain a global voiceprint feature vector for aggregating frame-level information. The global voiceprint feature vector obtained based on the audio file is tagged and saved to complete the audio file speech registration. It should be noted that the voiceprint feature extraction and embedding vector calculation based on the audio file can refer to the above-mentioned calculation method based on real-time speech, and this invention will not elaborate on it further.

[0046] Furthermore, the present invention provides a speaker recognition module. When input voice information is acquired, the speaker recognition module performs voiceprint matching on the input voice information and determines the speaker's identity using a voiceprint similarity formula and a voiceprint database matching probability. It further determines the speaker's identity label based on the voiceprint similarity formula and the voiceprint database matching probability. The voiceprint similarity calculation formula is as follows:

[0047] ;

[0048] in i and j It is the embedding vector of two speakers in the database. The angle similarity between them is calculated to determine whether they are the same speaker.

[0049]

[0050] Where P(y=k|) ) represents the probability that speaker k was found in the database;

[0051] w k This represents the weight vector of the k-th speaker in the database.

[0052] w j This represents the weight vector of the j-th speaker in the database;

[0053] T represents the transpose of the matrix, and N represents the total number of speakers.

[0054] It is worth mentioning that the present invention also provides a speech transcription module, which performs the following actions: based on the input speech information of a specific tag pre-defined by the system, it performs the following first-level transcription action of Automatic Speech Recognition (ASR), wherein the formula of the first-level transcription action includes:

[0055] ;

[0056] Where P AM(X|Y) represents the probability that the acoustic model predicts the audio signal X given the word sequence Y, P LM (Y) is the probability of a given speech transcription result from the language model. This represents the final predicted word sequence. P(X|Y) represents the probability of finding the word sequence Y that maximizes the probability of the following words among all possible word sequences Y. Given an audio signal X, P(X|Y) represents the posterior probability of the word sequence Y appearing, thus obtaining the text information after first-level transcription.

[0057] After completing the first-level transcription of the Automatic Speech Recognition (ASR), the context information of the transcribed text is obtained, and a second-level transcription is performed based on the context information. The formula for the second-level transcription is as follows:

[0058] ;

[0059] Where h i It is a context vector extracted from the text information after primary transcription, g ϕ This is a function of the punctuation restoration model that predicts whether to insert punctuation at position i. M represents the total number of punctuation marks to be inserted. For example, when the text output after the first-level transcribing is only a single sentence, the context and the corresponding punctuation restoration model function can determine that the current output text does not need to be punctuated, so the corresponding second-level transcribing action can be skipped.

[0060] When the system determines that ASR needs to be enabled: it calls the FunASR engine to perform first-level transcription of the input speech; when ASR is not enabled: the system only outputs the timestamp and speaker tag, without generating text content. Therefore, the corresponding output results are as follows:

[0061] When ASR is enabled: SegmentX[time]speaker_Y:"text content"

[0062] When ASR is not enabled: SegmentX[time]speaker_Y: "No text".

[0063] The processes described in the flowcharts above, as disclosed in the embodiments of this invention, can be implemented as computer software programs. Embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the methods of this application are not limited to the aforementioned functions. It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wire segments, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless segments, wire segments, optical fibers, RF, etc., or any suitable combination thereof.

[0064] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0065] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functions and structural principles of the present invention have been shown and explained in the embodiments. Without departing from the stated principles, the implementation of the present invention may have any variations or modifications.

Claims

1. A multi-strategy fusion speaker recognition and speech transcription processing method, characterized in that, The method comprises: Obtaining real-time speech and / or speech file information of a target speaker, and selecting at least one speech working mode, and performing voiceprint analysis on the real-time speech and / or speech file information according to the speech working mode; According to the current voice working mode, the voiceprint of the identified target speaker is registered, and the registered voiceprint of the target speaker is selectively subjected to ASR first-level transcription to obtain the voice text information of the target speaker; According to the context information of the voice text information of the target speaker, the voice text information of the target speaker is selectively subjected to CTTransformer second-level transcription to restore the punctuation of the voice text information of the target speaker; The voice text information of the target speaker after the second-level transcription, the first-level transcription information or the non-transcription information are mapped and stored according to the registered target speaker identity label.

2. The multi-strategy fused speaker recognition and speech transcription processing method of claim 1, wherein, The speech working mode comprises: only registration mode, only separation mode and registration + separation mode, wherein the only registration mode only performs new user voiceprint collection and storage, and is only used for registering new speaker identity information; the only separation mode is used for matching input speech in a voiceprint database, and is only used for judging the identity information of the speaker corresponding to the input speech; the registration + separation mode is used for expanding the registration of the voiceprint of a new speaker while performing identity recognition of the speaker, wherein the speech working mode is selected according to the scene.

3. The multi-strategy fused speaker recognition and speech transcription processing method of claim 1, wherein, The voiceprint registration method comprises: collecting speaker speech information in real time, and extracting voiceprint features from the speaker speech information, and embedding the extracted voiceprint features into a vector, and the voiceprint feature extraction formula is: ； denotes the speech feature vector extracted from the t-th speech frame, denotes a feature extraction function for the t-th speech frame denotes a discrete cosine transform, denotes a summation over the frequency dimension f from 1 to F, S t (f) is the short-time Fourier transform coefficient for the speech frame t, M(f, k) is the weighting function for the k-th mel filter, and T denotes that there are T speech frame indices.​ 4. The multi-strategy fused speaker recognition and speech transcription processing method of claim 3, wherein, The calculation formula of the embedding vector is: ； is an embedding vector, where σ() is a sigmoid activation function, ReLU() is a rectified linear unit, and are different weight parameters, respectively, and are bias parameters, respectively, T represents that there are T speech frame indexes, the embedding vector is used to aggregate frame-level information to generate a global voiceprint feature vector, and the global voiceprint feature vector is labeled and saved to complete real-time voice registration.

5. The multi-strategy fused speaker recognition and speech transcription processing method of claim 1, wherein, The voiceprint registration method comprises: reading an input speaker audio file, and performing voiceprint analysis on the audio file, and extracting voiceprint features from the speaker speech information and calculating embedding vectors to obtain a global voiceprint feature vector for aggregating frame-level information, and saving the global voiceprint feature vector based on the audio file to complete audio file voice registration.

6. The multi-strategy fused speaker recognition and speech transcription processing method of claim 1, wherein, When input speech information is obtained, a voiceprint matching and speaker identity judgment voiceprint similarity formula and a voiceprint database matching probability are used to further judge the identity label of the speaker corresponding to the current input speech according to the voiceprint similarity formula and the voiceprint database matching probability, and to judge whether the input speech information corresponding to the current identity label needs to be subjected to first-level transcription and second-level transcription according to system configuration rules.

7. The multi-strategy fused speaker recognition and speech transcription processing method of claim 1, wherein, According to the system, the input speech information of a specific label is subjected to the following automatic speech recognition ASR first-level transcription action, wherein the formula of the first-level transcription action comprises: ； where P(X|Y) denotes the probability of the acoustic model predicting the audio signal X given the word sequence Y, AM (X|Y) denotes the probability of the acoustic model predicting the audio signal X given the word sequence Y, P(Y) denotes the probability of the language model given the speech transcription result, LM (Y) is the probability of the language model given the speech transcription result, denotes the final predicted word sequence, denotes the probability of finding the Y that maximizes the posterior probability given all possible word sequences Y, P(X|Y) denotes the posterior probability of the word sequence Y given the audio signal X, resulting in the text information after the first transcription.

8. The multi-strategy fused speaker recognition and speech transcription processing method of claim 7, wherein, After the automatic speech recognition ASR first-level transcription action is completed, the context information of the first-level transcribed text information is obtained, and a second-level transcription action is performed according to the context information, wherein the formula of the second-level transcription action is as follows: ； where h i is the context vector extracted from the text information after the first transcription, g ϕ is the function of the punctuation restoration model that predicts whether to insert a punctuation at position i, and M denotes the total number of predictions of inserting a punctuation.

9. A multi-strategy fused speaker recognition and speech transcription processing system, characterized by, The system performs a multi-strategy fusion speaker recognition and speech transcription processing method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the multi-strategy fusion speaker recognition and speech transcription processing method in any one of claims 1-8.

Images