Speech recognition method and device

Abstract

The invention discloses a voice recognition method and device, and the method comprises the steps: obtaining a to-be-processed voice signal which needs to be processed; obtaining a pre-trained speaker detection neural network model; speaker recognition is carried out on the to-be-processed voice signal based on the speaker detection neural network model, a first predicted speaker identification sequence is obtained, and the first predicted speaker identification sequence comprises predicted speaker identification of each piece of voice frame data on the to-be-processed voice signal. According to the method, the speaker detection neural network model is trained in advance to directly carry out frame-level speaker change detection, the first predicted speaker identification sequence for frame-level speaker change detection is obtained, and compared with a traditional clustering mode, the accuracy of speech recognition is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech recognition method and device. Background Art

[0002] With the continuous development of mobile terminal technology, consumers' demand for mobile terminal albums is no longer limited to reviewing photos and videos, and the demand for multimodal intelligent editing is also increasing. In this context, intelligent album editing technology based on voice information in album videos is becoming increasingly important. The current mainstream speech processing process mainly includes: speaker segmentation and clustering (SD). The problem solved by speaker segmentation and clustering technology is "who spoke and when". Given a speech containing multiple people speaking alternately, the "speaker segmentation and clustering" technology needs to determine the start time and end time of each sentence and determine who said this sentence. However, in order to achieve a good speaker clustering effect, a good voiceprint model needs to be pre-trained to ensure that the extracted voiceprint information vector has strong distinguishability. This will make the process complicated and increase the risk of information loss. In addition, the solution based on speaker segmentation and clustering needs to ensure that the corpus of each speaker is not too small. If the corpus of a certain speaker in the speech segment is small, the segmentation and clustering effect will be seriously reduced, resulting in low accuracy of speech recognition.

[0003] That is, the accuracy of speech recognition in the prior art is low. Summary of the invention

[0004] The embodiments of the present application provide a speech recognition method and device, which can improve the accuracy of speech recognition.

[0005] In a first aspect, the speech recognition method provided by the present application comprises:

[0006] Obtaining a speech signal to be processed;

[0007] Get a pre-trained speaker detection neural network model;

[0008] Speaker recognition is performed on the speech signal to be processed based on the speaker detection neural network model to obtain a first predicted speaker identification sequence, wherein the first predicted speaker identification sequence includes the predicted speaker identification of each speech frame data on the speech signal to be processed.

[0009] In a second aspect, the speech recognition device provided by the present application includes:

[0010] A voice acquisition module is used to acquire the voice signal to be processed;

[0011] A model acquisition module, used to acquire a pre-trained speaker detection neural network model;

[0012] The recognition module is used to perform speaker recognition on the speech signal to be processed based on the speaker detection neural network model to obtain a first predicted speaker identification sequence, wherein the first predicted speaker identification sequence includes the predicted speaker identification of each speech frame data on the speech signal to be processed.

[0013] In a third aspect, the electronic device provided in the present application includes a memory and a processor, the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the speech recognition method provided in the present application.

[0014] In a fourth aspect, the computer-readable storage medium provided in the present application stores a plurality of instructions, which are suitable for loading by a processor to implement the steps in the speech recognition method provided in the present application.

[0015] In a fifth aspect, the computer program product provided in the present application includes a computer program or instructions, which, when executed by a processor, implement the steps in the speech recognition method provided in the present application.

[0016] In the present application, compared with the related art, a speech signal to be processed is obtained; a pre-trained speaker detection neural network model is obtained; speaker recognition is performed on the speech signal to be processed based on the speaker detection neural network model to obtain a first predicted speaker identification sequence, wherein the first predicted speaker identification sequence includes predicted speaker identifications of each speech frame data on the speech signal to be processed. The present application pre-trains the speaker detection neural network model to directly perform frame-level speaker change detection, and obtains a first predicted speaker identification sequence for frame-level speaker change detection, which greatly improves the accuracy of speech recognition compared to traditional clustering methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 is a schematic diagram of a scenario of a speech recognition system provided in an embodiment of the present application;

[0019] Figure 2 It is a flowchart of an embodiment of the speech recognition method provided by the embodiment of the present application;

[0020] Figure 3This is a schematic diagram of the network structure of resnet34 in one embodiment of the speech recognition method provided in the embodiment of the present application;

[0021] Figure 4 is a flowchart of another embodiment of the speech recognition method provided in the embodiment of the present application;

[0022] Figure 5 is a schematic diagram of a training corpus in another embodiment of the speech recognition method provided in an embodiment of the present application;

[0023] Figure 6 is a structural diagram of an embodiment of a speech recognition device provided in an embodiment of the present application;

[0024] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] It should be noted that the principles of the present application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application and should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0026] In the following description of the present application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0027] In the following description of the present application, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0029] The existing speaker segmentation and clustering process is as follows: First, for a whole segment of speech audio to be separated, we need to segment the audio into fixed lengths. For example, we segment the whole segment of speech audio into segments of 1.0s each, where the overlap between two adjacent segments is 0.5s. Then we need to use the pre-trained voiceprint recognition model, take out the voiceprint feature extraction module in the voiceprint recognition model separately, and then put all the segmented audio segments into the voiceprint extraction module for processing. Through the voiceprint extraction module, the voiceprint vector information of each audio segment is extracted. Next, we need to splice the voiceprint vectors of each segment into a matrix. The horizontal axis of the matrix is ​​the time dimension, the unit is the size of a window, and the vertical axis is the dimension of the voiceprint vector. The spliced ​​matrix is ​​called the voiceprint vector matrix of the audio. Next, we need to construct a similarity matrix for the audio, which is used to describe the similarity between each segment of the audio. We use the transposed voiceprint vector matrix of the audio to obtain the audio similarity matrix. In the similarity matrix, the value of each element represents the similarity between the audio segment voiceprint information corresponding to the index of the row and the audio segment voiceprint information corresponding to the index of the column. After we obtain the similarity matrix, we can use the similarity matrix and hierarchical clustering to cluster each audio segment. Through hierarchical clustering, we can cluster two segments whose similarity meets the set threshold into one category, and then combine audio segments of the same category into one audio, thereby saving the voices of different speakers in one audio separately as one audio.

[0030] From the above process, we can see that the current speaker segmentation and clustering technology has the following problems:

[0031] The number of speakers in each speech segment is uncertain. If the task is to segment and cluster speakers with an uncertain number of speakers, it is difficult to ensure accuracy by clustering in this unsupervised way. In actual user videos, the duration of each speaker in each speech segment is uncertain. If the duration of a speaker's speech is much shorter than that of other speakers, it is difficult to cluster the speech segments of speakers with shorter speech durations well during the clustering process. In order to achieve a good speaker clustering effect, a good voiceprint model needs to be pre-trained to ensure that the extracted voiceprint information vector has strong distinguishability. This will complicate the process and increase the risk of information loss. In the speaker segmentation process, a fixed speech segment length and the overlapping length of adjacent speech segments are usually set, which will result in very low accuracy of speaker segmentation. The minimum segmentation accuracy is the overlapping length of adjacent speech segments.

[0032] In order to improve the accuracy of speech recognition, the embodiments of the present application provide a speech recognition method, a speech recognition device, an electronic device, a computer-readable storage medium, and a computer program product. The speech recognition method can be performed by the speech recognition device, or by an electronic device integrated with the speech recognition device.

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0034] Please refer to Figure 1 , the present application also provides a speech recognition system, such as Figure 1 As shown, the speech recognition system includes an electronic device 100, in which the speech recognition device provided by the present application is integrated.

[0035] Among them, the electronic device 100 can be any device equipped with a processor and has processing capabilities, such as mobile electronic devices with processors such as smart phones, tablet computers, PDAs, laptops, smart speakers, or fixed electronic devices with processors such as desktop computers, televisions, servers, industrial equipment, etc.

[0036] In addition, if Figure 1 As shown, the speech recognition system may further include a memory 200 for storing original data, intermediate data and result data.

[0037] In the embodiment of the present application, the memory may be a cloud memory. Cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as the storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0038] At present, the storage method of the storage system is: create a logical volume, and when creating a logical volume, allocate physical storage space for each logical volume. The physical storage space may be composed of disks of a storage device or several storage devices. The client stores data on a logical volume, that is, stores the data on the file system. The file system divides the data into many parts, each of which is an object. The object contains not only data but also additional information such as data identification (ID, ID entity). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.

[0039] The process of the storage system allocating physical storage space to a logical volume is as follows: based on the estimated capacity of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of independent redundant disk arrays (RAID, Redundant Array of Independent Disks), the physical storage space is pre-divided into stripes. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0040] It should be noted that Figure 1 The scenario diagram of the speech recognition system shown is only an example. The speech recognition system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the speech recognition system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0041] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0042] Please refer to Figure 2 , Figure 2 FIG. 1 is a flow chart of an embodiment of the speech recognition method provided in the present application. Figure 2 As shown, the process of the speech recognition method provided by this application is as follows:

[0043] 201. Obtain a speech signal to be processed.

[0044] The voice signal to be processed may be an audio signal collected by a microphone or a pre-processed signal. In this case, the voice signal to be processed may be read.

[0045] In a specific embodiment, obtaining a speech signal to be processed includes:

[0046] (1) Obtain audio signal.

[0047] In the embodiment of the present application, the audio signal is an audio signal collected by a microphone. The audio signal may contain non-speech signals such as voice signals and background noise signals. The audio signal may be an audio signal in a user's album video. Of course, in other scenarios, the audio signal may be a customer service call recording, etc.

[0048] (2) Perform voice activity detection on the audio signal to obtain the voice signal in the audio signal.

[0049] Among them, Voice Activity Detection (VAD) technology, as a preprocessing technology for intelligent speech processing, automatically divides the frames of audio signals into speech and non-speech.

[0050] Speech is an important means of communication for humans, and it is also an important tool for communication between humans and machines. In the highly developed information age, digital speech transmission, recognition, synthesis, and enhancement are also one of the most important and basic components of the entire digital communication network. Digital speech processing has been widely used in many fields, the most important of which include speech compression, speech synthesis, speech recognition, and speech enhancement.

[0051] In digital speech processing, the purpose of voice activity detection (VAD) technology is to detect whether the current speech signal contains a voice signal, that is, to judge the input signal, distinguish the voice signal from various background noise signals, and use different processing methods for the two signals. For voice signals, it can be used for speech coding, speech recognition, speech synthesis, etc. For background noise, its background noise parameters can be extracted to generate comfortable noise with similar characteristics to the background noise. Voice activity detection is the basic link of digital speech signal processing. In many practical application systems, voice signal detection must be performed first, so that the data processed later is the actual valid voice signal data, thereby reducing the amount of data and the amount of calculation, and then reducing the processing time of the system.

[0052] (3) Each speech signal in the audio signal is determined as a speech signal to be processed.

[0053] Among them, multiple voice signals may be detected in the audio signal, and each voice signal in the audio signal is determined as a to-be-processed voice signal that needs to be processed.

[0054] 202. Obtain a pre-trained speaker detection neural network model.

[0055] Resnet (Residual Neural Network) was proposed by He Kaiming and four other Chinese people from Microsoft Research. The structure of resnet can accelerate the training of neural networks very quickly, and the accuracy of the model is also well guaranteed. Traditional convolutional neural networks (CNN) or long short-term memory networks (LSTM) will have more or less information loss and loss when transmitting information, and will also cause gradient disappearance or gradient explosion, making it impossible to train very deep networks. Resnet solves this problem to a certain extent by directly bypassing the input information to the output to protect the integrity of the information. The entire network only needs to learn the difference between the input and output, simplifying the learning objectives and difficulty.

[0056] In the embodiment of the present application, the speaker detection neural network model may be resnet34. Of course, in other embodiments, the speaker detection neural network model may also be resnet18, resnet50, etc., which can be set according to the specific situation. Among them, the structure of resnet34 is as follows Figure 3 shown.

[0057] 203. Perform speaker recognition on the speech signal to be processed based on the speaker detection neural network model to obtain a first predicted speaker identification sequence.

[0058] The first predicted speaker identification sequence includes predicted speaker identifications of each speech frame data on the speech signal to be processed.

[0059] In a specific embodiment, the first predicted speaker identification sequence may be: {1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0}.

[0060] Furthermore, the speech recognition method includes:

[0061] (1) Split the first predicted speaker identification sequence into multiple speaker identification subsequences.

[0062] The predicted speaker identifications of each speech frame data in the same speaker identification subsequence all belong to the same predicted speaker identification. Furthermore, the predicted speaker identifications of two adjacent speaker identification subsequences are different.

[0063] In a specific embodiment, the first predicted speaker identification sequence may be: {1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0}, and the multiple speaker identification subsequences obtained by splitting are: {1, 1, 1, 1}, {0, 0}, {1, 1}, {0, 0, 0, 0, 0}.

[0064] (2) Generate a target audio file based on the speech frame data corresponding to the speaker identification subsequence belonging to a preset type.

[0065] Among them, the preset type is the child voice type. Of course, the preset type can also be other types, such as male type, female type, elderly type, etc., which can be set according to the specific situation.

[0066] Specifically, multiple speech signals to be processed are processed respectively to obtain multiple speaker identification subsequences, multiple speaker identification subsequences are obtained, child voice detection is performed on each speaker identification subsequence, and it is determined whether each speaker identification subsequence belongs to a preset type, and a target audio file is generated based on speech frame data corresponding to the speaker identification subsequence belonging to the preset type. The speech frame data corresponding to the speaker identification subsequence belonging to the preset type can be edited to generate a target audio file.

[0067] For further information, please refer to Figure 4 , Figure 4 FIG. 1 is a flow chart of another embodiment of the speech recognition method provided in the embodiment of the present application. Figure 4 As shown, the process of the speech recognition method provided by this application is as follows:

[0068] 301. Obtain a speech signal to be processed.

[0069] In a specific embodiment, obtaining a speech signal to be processed includes:

[0070] (1) Obtain audio signal.

[0071] In the embodiment of the present application, the audio signal is an audio signal collected by a microphone. The audio signal may contain non-speech signals such as voice signals and background noise signals. The audio signal may be an audio signal in a user's album video. Of course, in other scenarios, the audio signal may be a customer service call recording, etc.

[0072] (2) Perform voice activity detection on the audio signal to obtain the voice signal in the audio signal.

[0073] (3) Each speech signal in the audio signal is determined as a speech signal to be processed.

[0074] Among them, multiple voice signals may be detected in the audio signal, and each voice signal in the audio signal is determined as a to-be-processed voice signal that needs to be processed.

[0075] 302. Obtain a pre-trained speaker detection neural network model.

[0076] In a specific embodiment, obtaining a pre-trained speaker detection neural network model includes:

[0077] (1) Obtain the preset training set.

[0078] The preset training set includes training corpus and corresponding corpus labels, the training corpus includes at least two speech segments, and the at least two training corpora each contain a different number of speech segments.

[0079] In the embodiment of the present application, multiple audio corpora in a preset prediction library are obtained, and the VAD model is used to segment the multiple audio corpora in the preset prediction library into speech signals and non-speech signals, and the speaker identification of multiple speech segments on each speech signal is recorded, and the speaker identification can be manually annotated. Multiple speech segments are simulated and combined to simulate the training corpora and corresponding corpus labels, thereby obtaining a preset training set.

[0080] In a specific embodiment, multiple speech segments and corresponding speaker identifiers are obtained, a random number of speech segments are randomly obtained from the multiple speech segments, the random number of speech segments are spliced ​​together to obtain training corpus, and the speaker identifiers of the various speech segments are correspondingly spliced ​​together to obtain corpus labels. Multiple random processes are performed to obtain multiple training corpora.

[0081] In another specific embodiment, multiple speech segments and corresponding speaker identifiers are obtained, a random number of speech segments of different lengths are randomly obtained from the multiple speech segments, the random number of speech segments of different lengths are spliced ​​together to obtain training corpus, and the speaker identifiers of the various speech segments are spliced ​​together to obtain corpus labels. Multiple random processes are performed to obtain multiple training corpora.

[0082] Furthermore, a random number of speech segments of different lengths are randomly obtained from the multiple speech segments, and the random number of speech segments of different lengths are randomly arranged to obtain a training corpus, and the speaker identifiers of the various speech segments are correspondingly spliced ​​to obtain a corpus label. Multiple random processing is performed to obtain multiple training corpora. The training corpus consists of multiple speech segments of different lengths, and the multiple speech segments of different lengths are randomly arranged, and the number of the multiple speech segments of different lengths is randomly generated. For example, one training corpus includes 5 speech segments, and the other training corpus includes 4 speech segments.

[0083] According to the task requirements, each training corpus generated by this application contains K speakers, K is a positive integer, and the position and duration of each speaker's corpus are random. Figure 5 As shown in the figure, a possible simulated training corpus contains 2 speakers, and the speaker identification can be manually annotated. The training corpus contains 5 speech segments, and the lengths of the 5 speech segments are different. The speakers of the 5 speech segments are: speaker 1, speaker 2, speaker 1, speaker 2, speaker 1. If a speech frame data is spoken by speaker 1, the corpus label of the speech frame data is recorded as 0. If a speech frame data is spoken by speaker 2, the corpus label of the speech frame data is recorded as 1. When the 5 speech segments are randomly arranged, another training corpus can be obtained. For example, after the 5 speech segments are randomly arranged, the speakers of the 5 speech segments are: speaker 2, speaker 1, speaker 2, speaker 1, speaker 2, thus forming a new training corpus.

[0084] At the data preprocessing level, in order to simulate the situation of an indefinite number of speakers, speech data of an indefinite number of people will be generated when simulating the generation of training data, and the length, number and position of speech segments of different speakers are guaranteed to be random. This allows the data to simulate the complex environment of actual user scenarios and improve the robustness of the model.

[0085] (2) Train the speaker detection neural network model based on the preset training set.

[0086] In a specific embodiment, a preset training set is input into the speaker detection neural network model for iterative training until the loss value of the speaker detection neural network model is lower than a preset loss value, wherein the preset loss value can be set according to specific circumstances.

[0087] 303. Perform speaker recognition on the speech signal to be processed based on the speaker detection neural network model to obtain a first predicted speaker identification sequence.

[0088] In the embodiment of the present application, speaker recognition is performed on the speech signal to be processed based on the speaker detection neural network model to obtain a first predicted speaker identification sequence, including:

[0089] (1) Inputting the speech signal to be processed into the speaker detection neural network model, the probability that each speech frame data of the speech signal to be processed belongs to each speaker identifier is obtained.

[0090] For example, the probability that each speech frame data in the speech signal to be processed belongs to speaker 0 is:

[0091] {0.01, 0.05, 0.02, 0.12, 0.54, 0.64, 0.01, 0.02, 0.96, 0.95, 0.92, 0.90}.

[0092] The probability that each speech frame data in the speech signal to be processed belongs to speaker 1 is:

[0093] {0.99, 0.95, 0.98, 0.88, 0.46, 0.36, 0.99, 0.98, 0.04, 0.05, 0.08, 0.10}.

[0094] (2) The speaker identifier corresponding to the maximum value among the probabilities that the speech frame data belongs to each speaker identifier is determined as the predicted speaker identifier of the speech frame data, thereby obtaining a first predicted speaker identifier sequence.

[0095] For example, the obtained first predicted speaker identification sequence is: {1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0}.

[0096] 304. Perform smoothing on the first predicted speaker identification sequence to obtain a second predicted speaker identification sequence.

[0097] In a specific embodiment, smoothing the first predicted speaker identification sequence to obtain the second predicted speaker identification sequence includes:

[0098] (1) Obtain a speaker change frame of a first predicted speaker identification sequence.

[0099] The speaker identifier of the speaker change frame is different from the speaker identifier of the frame before the speaker change frame.

[0100] For example, the first predicted speaker identification sequence is: {1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0}, then the speaker change frame is the 5th frame, the speaker identification of the speaker change frame is 0, and the speaker identification of the frame before the speaker change frame is 1.

[0101] (2) Obtain the number of consecutive identical frames of speech frame data that are consecutive and subsequent to the speaker change frame and have the same speaker identifier as the speaker change frame.

[0102] For example, the speaker change frame is the 5th frame, and after the speaker change frame, only the 5th frame and the 6th frame are continuous and have the same speaker identifier, so the number of continuous identical frames is 2.

[0103] (3) When the number of consecutive identical frames is not greater than a preset number of frames, the speaker identifier corresponding to the speech frame data of the consecutive identical frames is updated to the speaker identifier of the frame before the speaker change frame, to obtain an updated predicted speaker identifier sequence.

[0104] Among them, the preset frame number can be 2, 3, 4, etc., depending on the specific settings.

[0105] For example, the preset number of frames is 3, the speaker change frame is the 5th frame, and the number of consecutive identical frames is 2. At this time, the number of consecutive identical frames is not greater than the preset number of frames, and the speaker ID corresponding to the speech frame data of the consecutive identical frames is updated to the speaker ID of the frame before the speaker change frame to obtain an updated predicted speaker ID sequence. That is, the speaker IDs of the 5th and 6th frames are updated from 0 to 1, and the updated predicted speaker ID sequence is: {1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0}.

[0106] (4) reacquiring the speaker change frames in the updated predicted speaker identification sequence and performing iterative updating to obtain a second predicted speaker identification sequence, wherein the number of consecutive identical frames corresponding to any speaker change frame in the second predicted speaker identification sequence is greater than a preset number of frames.

[0107] After obtaining the updated predicted speaker identification sequence, the speaker change frame in the updated predicted speaker identification sequence is re-acquired for iterative updating to obtain a second predicted speaker identification sequence.

[0108] For example, the updated predicted speaker identification sequence is: {1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0}, the new speaker change frame is the 9th frame, and after the speaker change frame, the 9th frame, the 10th frame, the 11th frame, the 12th frame, and the 13th frame are continuous and have the same speaker identification, so the number of continuous identical frames is 5. If the number of continuous identical frames is greater than the preset number of frames, the updated predicted speaker identification sequence is determined as the second predicted speaker identification sequence.

[0109] Therefore, the second predicted speaker identification sequence is: {1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0}.

[0110] Without other processing, the first predicted speaker identification sequence corresponding to a speech segment will usually be non-smooth, because the speaker detection neural network model outputs the corresponding judgment result for each frame. This will cause a segment of speech to be cut very discretely by the speaker detection neural network model, and inevitably there will be some inaccurate model judgment results. Therefore, this application proposes a post-processing method for this non-smooth result, thereby improving the accuracy of the speaker detection neural network model and making the length of the cut speech segment more reasonable.

[0111] 305. Split the second predicted speaker identification sequence into multiple speaker identification subsequences.

[0112] The speaker identifications of the speech frame data in the same speaker identification subsequence all belong to the same speaker identification. Furthermore, the speaker identifications of two adjacent speaker identification subsequences are different.

[0113] In a specific embodiment, the second predicted speaker identification sequence is: {1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0}, and the multiple speaker identification subsequences obtained by splitting are: {1, 1, 1, 1, 1, 1, 1}, {0, 0, 0, 0, 0}.

[0114] 306. Generate a target audio file based on the speech frame data corresponding to the speaker identification subsequence belonging to a preset type.

[0115] Among them, the preset type is the children's voice type, and the target audio file is the video children's voice wonderful memories. Of course, the preset type can also be other types, such as male type, female type, elderly type, etc., which can be set according to the specific situation.

[0116] Specifically, multiple speech signals to be processed are processed respectively to obtain multiple speaker identification subsequences, multiple speaker identification subsequences are obtained, child voice detection is performed on each speaker identification subsequence, and it is determined whether each speaker identification subsequence belongs to a preset type, and a target audio file is generated based on speech frame data corresponding to the speaker identification subsequence belonging to the preset type. The speech frame data corresponding to the speaker identification subsequence belonging to the preset type can be edited to generate a target audio file.

[0117] In order to better implement the speech recognition method provided in the embodiment of the present application, the embodiment of the present application also provides a speech recognition device based on the above speech recognition method. The meaning of the nouns is the same as that in the above speech recognition method. For specific implementation details, please refer to the description in the above method embodiment.

[0118] Please refer to Figure 6 , Figure 6 The structure diagram of the speech recognition device provided in the embodiment of the present application is as follows. The speech recognition device may include a speech acquisition module 60, a model acquisition module 602, and a recognition module 603, wherein:

[0119] The speech acquisition module 601 is used to acquire the speech signal to be processed;

[0120] A model acquisition module 602, used to acquire a pre-trained speaker detection neural network model;

[0121] The recognition module 603 is used to perform speaker recognition on the speech signal to be processed based on the speaker detection neural network model to obtain a first predicted speaker identification sequence, wherein the first predicted speaker identification sequence includes the predicted speaker identification of each speech frame data on the speech signal to be processed.

[0122] The specific implementation of each of the above modules can be found in the previous embodiments and will not be described in detail here.

[0123] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the processor is used to execute the steps in the speech recognition method provided in this embodiment by calling a computer program stored in the memory.

[0124] Please refer to Figure 7 , Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0125] The electronic device may include components such as a processor 101 with one or more processing cores, a memory 102 with one or more computer-readable storage media, a power supply 103, and an input unit 104. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0126] The processor 101 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 102, and calling data stored in the memory 102. Optionally, the processor 101 may include one or more processing cores; optionally, the processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 101.

[0127] The memory 102 can be used to store software programs and modules. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 102 may also include a memory controller to provide the processor 101 with access to the memory 102.

[0128] The electronic device also includes a power supply 103 for supplying power to each component. Optionally, the power supply 103 can be logically connected to the processor 101 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 103 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0129] The electronic device may further include an input unit 104, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0130] Although not shown, the electronic device may also include a display unit, an image acquisition component, etc., which will not be described in detail here. Specifically in this embodiment, the processor 101 in the electronic device will load the executable code corresponding to one or more computer programs into the memory 102 according to the following instructions, and the processor 101 will execute the steps in the speech recognition method provided in this application, such as:

[0131] Acquire a speech signal to be processed; acquire a pre-trained speaker detection neural network model; perform speaker recognition on the speech signal to be processed based on the speaker detection neural network model to obtain a first predicted speaker identification sequence, wherein the first predicted speaker identification sequence includes predicted speaker identifications of each speech frame data on the speech signal to be processed.

[0132] It should be noted that the electronic device provided in the embodiment of the present application and the voice recognition method in the above embodiment belong to the same concept, and its specific implementation process is detailed in the above related embodiments and will not be repeated here.

[0133] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program stored therein is executed on a processor of an electronic device provided in an embodiment of the present application, the processor of the electronic device executes the steps in the speech recognition method provided in the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0134] The present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs various optional implementations of the above-mentioned speech recognition method.

[0135] The above is a detailed introduction to a speech recognition method and device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0136] It should be noted that when the above embodiments of the present application are applied to specific products or technologies, the relevant data of the user is involved, and the user's permission or consent is required, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A speech recognition method, characterized in that: include: Obtaining a speech signal to be processed; Get a pre-trained speaker detection neural network model; Speaker recognition is performed on the speech signal to be processed based on the speaker detection neural network model to obtain a first predicted speaker identification sequence, wherein the first predicted speaker identification sequence includes the predicted speaker identification of each speech frame data on the speech signal to be processed.

2. The speech recognition method according to claim 1, characterized in that: The speech recognition method further comprises: smoothing the first predicted speaker identification sequence to obtain a second predicted speaker identification sequence; Splitting the second predicted speaker identification sequence into a plurality of speaker identification subsequences, wherein the predicted speaker identifications of the speech frame data in the same speaker identification subsequence all belong to the same predicted speaker identification; A target audio file is generated based on the speech frame data corresponding to the speaker identification subsequence belonging to a preset type.

3. The speech recognition method according to claim 2, characterized in that: The step of smoothing the first predicted speaker identification sequence to obtain a second predicted speaker identification sequence includes: Acquire a speaker change frame of the first predicted speaker identification sequence, wherein the predicted speaker identification of the speaker change frame is different from the predicted speaker identification of a frame before the speaker change frame; Acquire the number of consecutive identical frames of speech frame data that are consecutive and subsequent to the speaker change frame and have the same predicted speaker identifier as the speaker change frame; When the number of consecutive identical frames is not greater than a preset number of frames, the predicted speaker identifier corresponding to the speech frame data of the consecutive identical frames is updated to the predicted speaker identifier of the frame before the speaker change frame, to obtain an updated predicted speaker identifier sequence; The speaker change frames in the updated predicted speaker identification sequence are re-acquired for iterative updating to obtain the second predicted speaker identification sequence, wherein the number of consecutive identical frames corresponding to any speaker change frame in the second predicted speaker identification sequence is greater than a preset number of frames.

4. The speech recognition method according to claim 1, characterized in that: The performing speaker recognition on the speech signal to be processed based on the speaker detection neural network model to obtain a first predicted speaker identification sequence includes: Inputting the speech signal to be processed into the speaker detection neural network model to obtain the probability that each of the speech frame data in the speech signal to be processed belongs to each speaker identifier; The speaker identifier corresponding to the maximum value among the probabilities that the speech frame data belongs to each speaker identifier is determined as the predicted speaker identifier of the speech frame data, so as to obtain a first predicted speaker identifier sequence.

5. The speech recognition method according to claim 1, characterized in that: The step of obtaining the speech signal to be processed includes: Get audio signal; Performing voice activity detection on the audio signal to obtain a voice signal in the audio signal; Each of the speech signals in the audio signal is determined as the to-be-processed speech signal that needs to be processed.

6. The speech recognition method according to claim 1, characterized in that: The step of obtaining a pre-trained speaker detection neural network model comprises: Obtaining a preset training set, wherein the preset training set includes training corpus and corresponding corpus labels, the training corpus includes at least two speech segments, and at least two of the training corpora each contain a different number of speech segments; The speaker detection neural network model is trained based on the preset training set.

7. A speech recognition device, characterized in that: include: A voice acquisition module is used to acquire the voice signal to be processed; A model acquisition module, used to acquire a pre-trained speaker detection neural network model; The recognition module is used to perform speaker recognition on the speech signal to be processed based on the speaker detection neural network model to obtain a first predicted speaker identification sequence, wherein the first predicted speaker identification sequence includes the predicted speaker identification of each speech frame data on the speech signal to be processed.

8. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the steps in the speech recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the speech recognition method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps in the speech recognition method according to any one of claims 1 to 6 are implemented.

Images