Voice activity detection method and device, computer device and storage medium

Abstract

The scheme relates to a voice activity detection method and device, computer equipment and a storage medium. The method comprises the following steps: randomly sampling voice segments and adding noise data to splice into multi-person dialogue simulation recording; inputting the simulation recording into a gate cycle unit after coding processing by a learnable self-supervised voice representation model to obtain an active voice detection feature; obtaining a target speaker registration voice and inputting the target speaker registration voice into a channel attention and propagation aggregation network to obtain a target speaker voiceprint feature; collecting a voice to be detected and inputting the voice to be detected and the target speaker registration voice into a bidirectional encoder network to obtain a verification feature; and inputting the active voice detection feature, the target speaker voiceprint feature and the verification feature into a voice activity detection model in sequence to obtain a detection result. Since the input features are diversified, the detection result of the voice activity detection model can be improved, thereby improving the accuracy of voice activity detection.

Description

Technical Field

[0001] This invention relates to the field of voice information processing technology, and in particular to a voice activity detection method, apparatus, computer device, and storage medium. Background Technology

[0002] With social development and the continuous improvement of people's living standards, various intelligent voice products have emerged. Voice activity detection is an indispensable part of the operation of these products. Voice activity detection refers to a technique that identifies the start and end positions of valid speech segments in a given audio stream. When using voice activity detection technology, the speech signal is often pre-processed frame by frame to extract relevant features, and then the frame is determined to be a speech frame based on these features. However, traditional voice activity detection methods suffer from low accuracy. Summary of the Invention

[0003] Therefore, in order to solve the above-mentioned technical problems, a method, apparatus, computer device and storage medium for voice activity detection are provided, which can improve the accuracy of voice activity detection.

[0004] A method for detecting speech activity, the method comprising:

[0005] Randomly sample speech segments and add noise data to the speech segments. Then, splice the speech segments with added noise data into a multi-person dialogue simulation recording.

[0006] The simulated recording is encoded using a learnable self-supervised speech representation model, and the encoded simulated recording is input into a gated loop unit to obtain active speech detection features.

[0007] The target speaker's registered voice is obtained and input into the channel attention and propagation aggregation network to obtain the target speaker's voiceprint features;

[0008] The speech to be detected is collected, and the speech to be detected and the target speaker's registered speech are input into a bidirectional encoder network to obtain verification features;

[0009] The active speech detection features, the target speaker's voiceprint features, and the verification features are sequentially input into the speech activity detection model to obtain the detection results.

[0010] In one embodiment, the random sampling of speech segments, the addition of noise data to the speech segments, and the splicing of the speech segments with the added noise data into a multi-person dialogue simulation recording include:

[0011] Speakers are randomly selected from a database with speaker tags, and recordings are randomly extracted from the corpus data of the selected speakers.

[0012] Silence segments are inserted between the recordings of the preceding and following speakers to form speech segments. Additive noise data and convolutional noise data are added to the speech segments and spliced ​​together to form a multi-person dialogue simulation recording.

[0013] In one embodiment, speakers are randomly selected from a database with speaker tags, and recordings are randomly extracted from the corpus data of the selected speakers, including:

[0014] Speakers are randomly selected from a database with speaker tags, and target speakers and interfering speakers are identified from them;

[0015] Each recording is randomly selected from the corpus of the chosen speakers, and each recording is divided into frames;

[0016] Each recording is labeled as a target speaker frame, a distractor speaker frame, or a silent frame.

[0017] In one embodiment, the simulated recording is encoded using a learnable self-supervised speech representation model, and the encoded simulated recording is input into a gated loop unit to obtain active speech detection features, including:

[0018] The feature representation of each frame of the simulated recording is extracted using a learnable self-supervised speech representation model.

[0019] Each of the aforementioned feature representations is sequentially input into a gated loop unit to obtain the corpus-level features for each frame, and the corpus-level features are used as active speech detection features.

[0020] In one embodiment, the bidirectional encoder network consists of a convolutional network and a deep self-attention network.

[0021] In one embodiment, the speech activity detection model consists of a three-layer long short-term memory network structure and a two-layer fully connected layer structure;

[0022] The active speech detection features, the target speaker's voiceprint features, and the verification features are sequentially input into the speech activity detection model to obtain the detection results, including:

[0023] The active speech detection features are input into the first layer of the long short-term memory network structure, the target speaker voiceprint features are input into the second layer of the long short-term memory network structure, and the verification features are input into the third layer of the long short-term memory network structure.

[0024] Feature fusion is performed using a three-layer long short-term memory network structure, and the fused features are then input into two fully connected layers to obtain the detection results.

[0025] In one embodiment, the training method of the voice activity detection model includes:

[0026] Acquire audio training data, and obtain training activity speech detection features, training voiceprint features, and training verification features based on the audio training data;

[0027] The training activity speech detection features, the training voiceprint features, and the training verification features are fed into the initial speech activity detection model to obtain the prediction results.

[0028] The prediction results are compared with the label data, and the loss function is calculated;

[0029] The parameters of the initial speech activity detection model are adjusted according to the loss function to obtain the speech activity detection model.

[0030] A voice activity detection device, the device comprising:

[0031] The speech splicing module is used to randomly sample speech segments, add noise data to the speech segments, and splice the speech segments with added noise data into a multi-person dialogue simulation recording.

[0032] The active speech detection feature extraction module is used to encode the simulated recording through a learnable self-supervised speech representation model, and input the encoded simulated recording into a gated loop unit to obtain active speech detection features;

[0033] The target speaker voiceprint feature extraction module is used to acquire the target speaker's registered speech and input the target speaker's registered speech into the channel attention and propagation aggregation network to obtain the target speaker's voiceprint features;

[0034] The verification feature extraction module is used to collect the speech to be detected and input the speech to be detected and the target speaker's registered speech into the bidirectional encoder network to obtain verification features;

[0035] The detection module is used to sequentially input the active speech detection features, the target speaker's voiceprint features, and the verification features into the speech activity detection model to obtain the detection results.

[0036] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0037] Randomly sample speech segments and add noise data to the speech segments. Then, splice the speech segments with added noise data into a multi-person dialogue simulation recording.

[0038] The simulated recording is encoded using a learnable self-supervised speech representation model, and the encoded simulated recording is input into a gated loop unit to obtain active speech detection features.

[0039] The target speaker's registered voice is obtained and input into the channel attention and propagation aggregation network to obtain the target speaker's voiceprint features;

[0040] The speech to be detected is collected, and the speech to be detected and the target speaker's registered speech are input into a bidirectional encoder network to obtain verification features;

[0041] The active speech detection features, the target speaker's voiceprint features, and the verification features are sequentially input into the speech activity detection model to obtain the detection results.

[0042] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0043] Randomly sample speech segments and add noise data to the speech segments. Then, splice the speech segments with added noise data into a multi-person dialogue simulation recording.

[0044] The simulated recording is encoded using a learnable self-supervised speech representation model, and the encoded simulated recording is input into a gated loop unit to obtain active speech detection features.

[0045] The target speaker's registered voice is obtained and input into the channel attention and propagation aggregation network to obtain the target speaker's voiceprint features;

[0046] The speech to be detected is collected, and the speech to be detected and the target speaker's registered speech are input into a bidirectional encoder network to obtain verification features;

[0047] The active speech detection features, the target speaker's voiceprint features, and the verification features are sequentially input into the speech activity detection model to obtain the detection results.

[0048] The aforementioned speech activity detection method, apparatus, computer equipment, and storage medium randomly sample speech segments and add noise data to these segments. The speech segments with added noise data are then concatenated into a multi-person dialogue simulation recording. This simulation recording is encoded using a learnable self-supervised speech representation model, and the encoded recording is input into a gated recurrent unit to obtain active speech detection features. The registered speech of the target speaker is acquired and input into a channel attention and propagation aggregation network to obtain the target speaker's voiceprint features. The speech to be detected is collected, and the speech to be detected and the registered speech of the target speaker are input into a bidirectional encoder network to obtain verification features. Finally, the active speech detection features, the target speaker's voiceprint features, and the verification features are sequentially input into the speech activity detection model to obtain the detection result. Active speech detection features are obtained through a learnable self-supervised speech representation model and gated recurrent units. Target speaker voiceprint features are obtained through a channel attention and propagation aggregation network. Verification features are obtained through a bidirectional encoder network. Finally, these features are input into the speech activity detection model for fusion to obtain the detection result. Due to the diversity of input features, the detection results of the speech activity detection model can be improved, thereby increasing the accuracy of speech activity detection. Attached Figure Description

[0049] Figure 1 This is a diagram illustrating the application environment of a voice activity detection method in one embodiment;

[0050] Figure 2 This is a flowchart illustrating a speech activity detection method in one embodiment;

[0051] Figure 3 This is a schematic diagram of the processing flow for obtaining detection results from a speech activity detection model in one embodiment;

[0052] Figure 4 This is a structural block diagram of a voice activity detection device in one embodiment;

[0053] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] The voice activity detection method provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1As shown, the application environment includes computer device 110. Computer device 110 can randomly sample speech segments and add noise data to them, then concatenate the speech segments with added noise data into a multi-person dialogue simulation recording. Computer device 110 can encode the simulation recording using a learnable self-supervised speech representation model, and input the encoded simulation recording into a gated recurrent unit to obtain active speech detection features. Computer device 110 can acquire the target speaker's registered speech and input it into a channel attention and propagation aggregation network to obtain the target speaker's voiceprint features. Computer device 110 can collect the speech to be detected and input the speech to be detected and the target speaker's registered speech into a bidirectional encoder network to obtain verification features. Computer device 110 can sequentially input the active speech detection features, target speaker voiceprint features, and verification features into a speech activity detection model to obtain the detection result. The computer device 110 can be, but is not limited to, various personal computers, embedded microcomputers, robots, unmanned aerial vehicles, smart speakers, and other devices.

[0056] In one embodiment, such as Figure 2 As shown, a method for detecting speech activity is provided, including the following steps:

[0057] Step 202: Randomly sample speech segments and add noise data to the speech segments. Then, splice the speech segments with added noise data into a multi-person dialogue simulation recording.

[0058] Computer devices can contain a speech database that stores different speech segments from different speakers. The computer can randomly sample speech segments from the database, add noise data to these segments, and then stitch them together to create a simulated multi-person dialogue recording. By adding noise data to the speech segments, the channel consistency between different segments and the realism of the simulated recording can be enhanced.

[0059] Step 204: Encode the simulated recording using a learnable self-supervised speech representation model, and input the encoded simulated recording into a gated recurrent unit to obtain active speech detection features.

[0060] The learnable self-supervised speech representation model is composed of 12 deep self-attention networks as basic modules. It is trained by masking unlabeled speech data in the time, frequency, and amplitude domains, and then predicting the masked data. The learnable self-supervised speech representation model can be used to encode simulated recordings. These encoded simulated recordings are then input into a gated recurrent unit to synthesize active speech detection features.

[0061] Step 206: Obtain the target speaker's registered voice and input the target speaker's registered voice into the channel attention and propagation aggregation network to obtain the target speaker's voiceprint features.

[0062] In the Channel Attention and Propagation Aggregation Network, the first three layers consist of one-dimensional convolutions, rectified linear units, and batch normalization. The subsequent fourth, fifth, and sixth layers all employ compression-excitation residual modules. The network first compresses the input target speaker's registered speech into a one-dimensional vector of size C, then upsamples it to its original size and directly adds it to the original vector using residual connections. The seventh layer concatenates the outputs of layers four, five, and six, and the eighth layer performs attention statistical pooling. Finally, a fully connected layer outputs the final target speaker's voiceprint features. The more effective the extracted target speaker's voiceprint features, the better the detection performance of target speaker activity.

[0063] Step 208: Collect the speech to be detected, and input the speech to be detected and the target speaker's registered speech into the bidirectional encoder network to obtain the verification features.

[0064] The bidirectional encoder network uses a pre-trained bidirectional encoder as the base model. An average temporal pooling layer is added on top, and finally, a fully connected layer is used to obtain validation features. Specifically, the validation features are calculated using cosine values. The training method for the bidirectional encoder network involves freezing the bidirectional encoder convolutional modules and fine-tuning the deep self-attention network of the bidirectional encoder.

[0065] Step 210: Input the active speech detection features, target speaker voiceprint features, and verification features into the speech activity detection model in sequence to obtain the detection results.

[0066] After obtaining the active speech detection features, target speaker voiceprint features, and verification features, the obtained features can be sequentially input into the speech activity detection model for feature fusion to obtain the detection results.

[0067] In this embodiment, the computer device obtains active speech detection features through a learnable self-supervised speech representation model and a gated recurrent unit, obtains target speaker voiceprint features through a channel attention and propagation aggregation network, obtains verification features through a bidirectional encoder network, and finally inputs each feature into the speech activity detection model for fusion to obtain the detection result. Due to the diversification of input features, the detection result of the speech activity detection model can be improved, thereby improving the accuracy of speech activity detection.

[0068] In one embodiment, a speech activity detection method may further include a process of adding noise data, specifically including: randomly selecting speakers from a database with speaker tags, and randomly extracting each recording from the corpus data of the selected speakers; inserting silent segments between each recording of the preceding and following speakers to form speech segments, and adding additive noise data and convolutional noise data to the speech segments to splice them into a multi-person dialogue simulation recording.

[0069] Given a database with speaker labels, a computer device can obtain the number of speakers n ~ U(a,b) based on uniform distribution sampling, where n is the number of speakers sampled, a is the lower limit of the number of samples, and b is the upper limit of the number of samples. n speakers are randomly selected from the database, and a recording is randomly selected from the corpus data of each speaker. These recordings are then spliced ​​together according to random timing and speech overlap ratios to obtain a simulated multi-person dialogue recording. Silent segments are then randomly inserted between consecutive speakers with a certain probability to form speech segments. Additive and convolutional noise data can then be added to the speech segments, and the resulting spliced ​​recordings are used to create a simulated multi-person dialogue recording.

[0070] In one embodiment, a speech activity detection method may further include a process of tagging each recording, specifically including: randomly selecting speakers from a database with speaker tags, and identifying target speakers and interfering speakers from them; randomly extracting each recording from the corpus of the selected speakers, and dividing each recording into frames; and tagging each recording as a target speaker frame, an interfering speaker frame, or a silent frame.

[0071] In this process, for each of the selected n speakers, one is randomly designated as the target speaker, and the others are designated as interfering speakers. The recording is divided into frames of a certain duration, for example, a 25ms frame with a 10ms frame jump. Each frame is labeled with a speaker, and the frame is marked as a target speaker frame, an interfering speaker frame, or a silent frame based on the speaker it corresponds to.

[0072] In one embodiment, a speech activity detection method may further include a process of obtaining active speech detection features, specifically including: extracting feature representations of each frame of a simulated recording through a learnable self-supervised speech representation model; sequentially inputting each feature representation into a gated recurrent unit to obtain corpus-level features for each frame; and using the corpus-level features as active speech detection features.

[0073] Specifically, the computer device can extract the feature representation of each frame of the simulated recording through a learnable self-supervised speech representation model, and on this basis, the feature representation of each frame is extracted and input into the gated recurrent unit in chronological order to obtain the corpus-level features of each frame as the active speech detection features.

[0074] In one embodiment, the bidirectional encoder network consists of a convolutional network and a deep self-attention network. The bidirectional encoder network is trained using a self-supervised method that randomly masks the input speech and then predicts it.

[0075] In one embodiment, such as Figure 3 As shown, the provided speech activity detection method may further include a process for obtaining detection results. The specific process includes: inputting the active speech detection features into a first-layer long short-term memory network structure, inputting the target speaker's voiceprint features into a second-layer long short-term memory network structure, and inputting the verification features into a third-layer long short-term memory network structure; performing feature fusion through the three-layer long short-term memory network structure, and inputting the fused features into two fully connected layers to obtain the detection results.

[0076] The speech activity detection model consists of a three-layer long short-term memory network structure and a two-layer fully connected layer structure. After inputting the fused features into the two fully connected layers, three probabilities are obtained: the probability of silence, the probability of non-target speaker speech, and the probability of target speaker speech.

[0077] In one embodiment, a speech activity detection method may further include a training process for a speech activity detection model, specifically including: acquiring recorded training data, and obtaining training activity speech detection features, training voiceprint features, and training verification features based on the recorded training data; inputting the training activity speech detection features, training voiceprint features, and training verification features into an initial speech activity detection model to obtain a prediction result; comparing the prediction result with label data and calculating a loss function; and adjusting the parameters of the initial speech activity detection model based on the loss function to obtain the speech activity detection model.

[0078] Specifically, when calculating the loss function, an exponential normalization method with added differences is used. The calculation method is as follows:

[0079]

[0080]

[0081]

[0082] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0083] In one embodiment, such as Figure 4 As shown, a speech activity detection device is provided, including: a speech splicing module 410, an active speech detection feature extraction module 420, a target speaker voiceprint feature extraction module 430, a verification feature extraction module 440, and a detection module 450, wherein:

[0084] The speech splicing module 410 is used to randomly sample speech segments, add noise data to the speech segments, and splice the speech segments with added noise data into a multi-person dialogue simulation recording.

[0085] The active speech detection feature extraction module 420 is used to encode the simulated recording through a learnable self-supervised speech representation model, and input the encoded simulated recording into a gated loop unit to obtain active speech detection features.

[0086] The target speaker voiceprint feature extraction module 430 is used to obtain the target speaker's registered speech and input the target speaker's registered speech into the channel attention and propagation aggregation network to obtain the target speaker's voiceprint features;

[0087] The verification feature extraction module 440 is used to collect the speech to be detected and input the speech to be detected and the target speaker's registered speech into the bidirectional encoder network to obtain verification features;

[0088] The detection module 450 is used to input the active speech detection features, target speaker voiceprint features, and verification features into the speech activity detection model in sequence to obtain the detection results.

[0089] In one embodiment, the speech splicing module 410 is further configured to randomly select a speaker from a database with speaker tags, and randomly extract each recording from the corpus data of the selected speaker; insert silent segments between the recordings of the preceding and following speakers to form speech segments, and add additive noise data and convolutional noise data to the speech segments to splice them into a multi-person dialogue simulation recording.

[0090] In one embodiment, the speech splicing module 410 is further configured to randomly select speakers from a database with speaker tags, and determine the target speaker and the interfering speaker from it; randomly extract each recording from the corpus of the selected speaker, and divide each recording into frames; and mark each recording as a target speaker frame, an interfering speaker frame, or a silent frame according to the frame.

[0091] In one embodiment, the active speech detection feature extraction module 420 is further configured to extract the feature representation of each frame of the simulated recording through a learnable self-supervised speech representation model; input each feature representation sequentially into a gated loop unit to obtain the corpus-level features of each frame, and use the corpus-level features as active speech detection features.

[0092] In one embodiment, the bidirectional encoder network consists of a convolutional network and a deep self-attention network.

[0093] In one embodiment, the speech activity detection model consists of a three-layer long short-term memory network structure and a two-layer fully connected layer structure. The detection module 450 is further configured to input the active speech detection features into the first layer long short-term memory network structure, input the target speaker's voiceprint features into the second layer long short-term memory network structure, and input the verification features into the third layer long short-term memory network structure; perform feature fusion through the three-layer long short-term memory network structure, and input the fused features into the two fully connected layers to obtain the detection result.

[0094] In one embodiment, the provided speech activity detection device may further include a model training module for acquiring recorded training data and obtaining training activity speech detection features, training voiceprint features, and training verification features based on the recorded training data; inputting the training activity speech detection features, training voiceprint features, and training verification features into an initial speech activity detection model to obtain a prediction result; comparing the prediction result with label data and calculating a loss function; and adjusting the parameters of the initial speech activity detection model according to the loss function to obtain a speech activity detection model.

[0095] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a voice activity detection method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0096] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0097] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0098] Randomly sample speech segments and add noise data to the speech segments. Then, splice the speech segments with added noise data into a multi-person dialogue simulation recording.

[0099] The simulated recordings are encoded using a learnable self-supervised speech representation model. The encoded simulated recordings are then input into a gated recurrent unit to obtain active speech detection features.

[0100] The target speaker's registered voice is obtained and input into the channel attention and propagation aggregation network to obtain the target speaker's voiceprint features;

[0101] The speech to be detected is collected, and the speech to be detected and the target speaker's registered speech are input into a bidirectional encoder network to obtain verification features;

[0102] The activity speech detection features, target speaker voiceprint features, and verification features are sequentially input into the speech activity detection model to obtain the detection results.

[0103] In one embodiment, when the processor executes the computer program, it further performs the following steps: randomly selecting speakers from a database with speaker tags, and randomly extracting each recording from the corpus data of the selected speakers; inserting silent segments between each recording of the preceding and following speakers to form speech segments, and adding additive noise data and convolutional noise data to the speech segments to splice them into a multi-person dialogue simulation recording.

[0104] In one embodiment, when the processor executes the computer program, it further performs the following steps: randomly selecting speakers from a database with speaker tags, and identifying target speakers and interfering speakers from them; randomly extracting each recording from the corpus of the selected speakers, and dividing each recording into frames; and labeling each recording as a target speaker frame, an interfering speaker frame, or a silent frame.

[0105] In one embodiment, when the processor executes the computer program, it further performs the following steps: extracting feature representations of each frame of the simulated recording through a learnable self-supervised speech representation model; sequentially inputting each feature representation into a gated loop unit to obtain corpus-level features for each frame; and using the corpus-level features as active speech detection features.

[0106] In one embodiment, the bidirectional encoder network consists of a convolutional network and a deep self-attention network.

[0107] In one embodiment, the speech activity detection model consists of a three-layer long short-term memory network structure and a two-layer fully connected layer structure. When the processor executes the computer program, it also performs the following steps: inputting the active speech detection features into the first-layer long short-term memory network structure, inputting the target speaker's voiceprint features into the second-layer long short-term memory network structure, and inputting the verification features into the third-layer long short-term memory network structure; performing feature fusion through the three-layer long short-term memory network structure, and inputting the fused features into the two-layer fully connected layer to obtain the detection result.

[0108] In one embodiment, when the processor executes the computer program, it further performs the following steps: acquiring recorded training data, and obtaining training activity speech detection features, training voiceprint features, and training verification features based on the recorded training data; inputting the training activity speech detection features, training voiceprint features, and training verification features into an initial speech activity detection model to obtain a prediction result; comparing the prediction result with the label data and calculating the loss function; adjusting the parameters of the initial speech activity detection model based on the loss function to obtain a speech activity detection model.

[0109] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0110] Randomly sample speech segments and add noise data to the speech segments. Then, splice the speech segments with added noise data into a multi-person dialogue simulation recording.

[0111] The simulated recordings are encoded using a learnable self-supervised speech representation model. The encoded simulated recordings are then input into a gated recurrent unit to obtain active speech detection features.

[0112] The target speaker's registered voice is obtained and input into the channel attention and propagation aggregation network to obtain the target speaker's voiceprint features;

[0113] The speech to be detected is collected, and the speech to be detected and the target speaker's registered speech are input into a bidirectional encoder network to obtain verification features;

[0114] The activity speech detection features, target speaker voiceprint features, and verification features are sequentially input into the speech activity detection model to obtain the detection results.

[0115] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: randomly selecting speakers from a database with speaker tags, and randomly extracting each recording from the corpus data of the selected speakers; inserting silent segments between each recording of the preceding and following speakers to form speech segments, and adding additive noise data and convolutional noise data to the speech segments to splice them into a multi-person dialogue simulation recording.

[0116] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: randomly selecting speakers from a database with speaker tags, and identifying target speakers and interfering speakers from them; randomly extracting each recording from the corpus of the selected speakers, and dividing each recording into frames; and labeling each recording as a target speaker frame, an interfering speaker frame, or a silent frame.

[0117] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: extracting feature representations of each frame of the simulated recording through a learnable self-supervised speech representation model; sequentially inputting each feature representation into a gated loop unit to obtain corpus-level features for each frame; and using the corpus-level features as active speech detection features.

[0118] In one embodiment, the bidirectional encoder network consists of a convolutional network and a deep self-attention network.

[0119] In one embodiment, the speech activity detection model consists of a three-layer long short-term memory network structure and a two-layer fully connected layer structure. When the computer program is executed by the processor, it also performs the following steps: inputting the active speech detection features into the first layer long short-term memory network structure, inputting the target speaker's voiceprint features into the second layer long short-term memory network structure, and inputting the verification features into the third layer long short-term memory network structure; performing feature fusion through the three-layer long short-term memory network structure, and inputting the fused features into the two fully connected layers to obtain the detection result.

[0120] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: acquiring recorded training data, and obtaining training activity speech detection features, training voiceprint features, and training verification features based on the recorded training data; inputting the training activity speech detection features, training voiceprint features, and training verification features into an initial speech activity detection model to obtain a prediction result; comparing the prediction result with the label data and calculating the loss function; adjusting the parameters of the initial speech activity detection model according to the loss function to obtain a speech activity detection model.

[0121] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0122] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for detecting speech activity, characterized in that, The method includes: Randomly sample speech segments and add noise data to the speech segments. Then, splice the speech segments with added noise data into a multi-person dialogue simulation recording. The simulated recording is encoded using a learnable self-supervised speech representation model, and the encoded simulated recording is input into a gated loop unit to obtain active speech detection features. The target speaker's registered voice is obtained and input into the channel attention and propagation aggregation network to obtain the target speaker's voiceprint features; The speech to be detected is collected, and the speech to be detected and the target speaker's registered speech are input into a bidirectional encoder network to obtain verification features; The speech activity detection model consists of a three-layer long short-term memory network structure and a two-layer fully connected layer structure. The active speech detection features, the target speaker's voiceprint features, and the verification features are sequentially input into the speech activity detection model to obtain the detection result. This includes: inputting the active speech detection features into the first layer of the long short-term memory network structure, inputting the target speaker's voiceprint features into the second layer of the long short-term memory network structure, and inputting the verification features into the third layer of the long short-term memory network structure; performing feature fusion through the three-layer long short-term memory network structure, and inputting the fused features into the two fully connected layers to obtain the detection result.

2. The speech activity detection method according to claim 1, characterized in that, The process involves randomly sampling speech segments, adding noise data to these speech segments, and then concatenating the speech segments with the added noise data into a multi-person dialogue simulation recording. Speakers are randomly selected from a database with speaker tags, and recordings are randomly extracted from the corpus data of the selected speakers. Silence segments are inserted between the recordings of the preceding and following speakers to form speech segments. Additive noise data and convolutional noise data are added to the speech segments and spliced ​​together to form a multi-person dialogue simulation recording.

3. The speech activity detection method according to claim 2, characterized in that, Speakers are randomly selected from a database with speaker tags, and recordings are randomly extracted from the corpus data of the selected speakers, including: Speakers are randomly selected from a database with speaker tags, and target speakers and interfering speakers are identified from them; Each recording is randomly selected from the corpus of the chosen speakers, and each recording is divided into frames; Each recording is labeled as a target speaker frame, a distractor speaker frame, or a silent frame.

4. The speech activity detection method according to claim 1, characterized in that, The simulated recording is encoded using a learnable self-supervised speech representation model. The encoded simulated recording is then input into a gated recurrent unit to obtain active speech detection features, including: The feature representation of each frame of the simulated recording is extracted using a learnable self-supervised speech representation model. Each of the aforementioned feature representations is sequentially input into a gated loop unit to obtain the corpus-level features for each frame, and the corpus-level features are used as active speech detection features.

5. The speech activity detection method according to claim 1, characterized in that, The bidirectional encoder network consists of a convolutional network and a deep self-attention network.

6. The speech activity detection method according to claim 1, characterized in that, The training methods for the speech activity detection model include: Acquire audio training data, and obtain training activity speech detection features, training voiceprint features, and training verification features based on the audio training data; The training activity speech detection features, the training voiceprint features, and the training verification features are fed into the initial speech activity detection model to obtain the prediction results. The prediction results are compared with the label data, and the loss function is calculated; The parameters of the initial speech activity detection model are adjusted according to the loss function to obtain the speech activity detection model.

7. A voice activity detection device, characterized in that, The device includes: The speech splicing module is used to randomly sample speech segments, add noise data to the speech segments, and splice the speech segments with added noise data into a multi-person dialogue simulation recording. The active speech detection feature extraction module is used to encode the simulated recording through a learnable self-supervised speech representation model, and input the encoded simulated recording into a gated loop unit to obtain active speech detection features; The target speaker voiceprint feature extraction module is used to acquire the target speaker's registered speech and input the target speaker's registered speech into the channel attention and propagation aggregation network to obtain the target speaker's voiceprint features; The verification feature extraction module is used to collect the speech to be detected and input the speech to be detected and the target speaker's registered speech into the bidirectional encoder network to obtain verification features; The detection module is used to sequentially input the active speech detection features, the target speaker's voiceprint features, and the verification features into the speech activity detection model to obtain the detection result. The speech activity detection model consists of a three-layer long short-term memory network structure and a two-layer fully connected layer structure. Inputting the active speech detection features, the target speaker's voiceprint features, and the verification features into the speech activity detection model to obtain the detection result includes: inputting the active speech detection features into the first layer of the long short-term memory network structure, inputting the target speaker's voiceprint features into the second layer of the long short-term memory network structure, and inputting the verification features into the third layer of the long short-term memory network structure; performing feature fusion through the three-layer long short-term memory network structure, and inputting the fused features into the two fully connected layers to obtain the detection result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images