A voice activity detection method, device, apparatus and storage medium

Abstract

The application discloses a voice activity detection method, device and equipment and a storage medium. The method comprises the following steps: acquiring mixed voice and registered voice of a first speaker; performing feature extraction on the mixed voice and the registered voice by using a first personalized voice activity detection (PVAD) model, and obtaining a probability distribution of the first speaker on each frame of sub-mixed voice based on frame-level acoustic features of the mixed voice and frame-level acoustic features of the registered voice. In this way, by using the frame-level acoustic features of the registered voice as the speaker information of the first speaker, the more fine voice signal characteristics of the short voice (such as the wake-up word voice) can be extracted when the short voice is registered, the accuracy of the output result of the PVAD model is further ensured, and the recall rate of the PVAD model is improved.

Description

Technical Field

[0001] This application relates to speech detection technology, and more particularly to a method, apparatus, device, and storage medium for speech activity detection. Background Technology

[0002] Currently, Personalized Voice Activity Detection (PVAD) can detect the start-time segment of a specific target speaker in noisy multi-speaker speech. A PVAD model consists of a pre-trained speaker recognition model, a PVAD backbone network, and linear layers. Since PVAD models largely rely on the quality of the segment-level speaker embeddings extracted by the pre-trained speaker recognition model, the quality of these embeddings decreases when short speech (such as wake words) is used for registration, leading to a drop in the recall rate of the PVAD model. Summary of the Invention

[0003] This application aims to provide a method, apparatus, device, and storage medium for detecting voice activity.

[0004] The technical solution of this application is implemented as follows:

[0005] Firstly, a method for detecting speech activity is provided, the method comprising:

[0006] Obtain the mixed speech and the registered speech of the first speaker;

[0007] The first personalized speech activity detection (PVAD) model is used to extract features from the mixed speech and the registered speech. Based on the extracted frame-level acoustic features of the mixed speech and the frame-level acoustic features of the registered speech, the probability distribution of the first speaker in each frame of the sub-mixed speech is obtained.

[0008] Secondly, a voice activity detection device is provided, the device comprising:

[0009] The acquisition unit is used to acquire the mixed speech and the registered speech of the first speaker;

[0010] The processing unit is configured to extract features from the mixed speech and the registered speech using a first personalized speech activity detection (PVAD) model, and to obtain the probability distribution of the first speaker in each frame of sub-mixed speech based on the extracted frame-level acoustic features of the mixed speech and the frame-level acoustic features of the registered speech.

[0011] Thirdly, an electronic device is provided, comprising: a processor and a memory configured to store a computer program capable of running on the processor, wherein the processor is configured to perform the steps of the method described in the first aspect when running the computer program.

[0012] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0013] This application discloses a method, apparatus, device, and storage medium for voice activity detection. It acquires mixed speech and the registered speech of a first speaker; uses a first personalized voice activity detection (PVAD) model to extract features from the mixed speech and the registered speech; and obtains the probability distribution of the first speaker in each frame of sub-mixed speech based on the extracted frame-level acoustic features of the mixed speech and the registered speech. Thus, this application uses the frame-level features of the registered speech as the speaker information of the first speaker. This not only ensures the overall performance of the PVAD model but also effectively improves its recall rate when using short speech (such as wake-up words) for registration. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0015] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0016] Figure 1 This is a schematic diagram of the first process of the voice activity detection method in the embodiments of this application;

[0017] Figure 2 This is a schematic diagram of the framework of the first PVAD model in the embodiments of this application;

[0018] Figure 3 This is a schematic diagram of the second process of the voice activity detection method in the embodiments of this application;

[0019] Figure 4 This is a schematic diagram of a coding sublayer structure provided in an embodiment of this application;

[0020] Figure 5 This is a schematic diagram of the third process of the voice activity detection method in the embodiments of this application;

[0021] Figure 6 This is a schematic diagram of the fourth process of the voice activity detection method in the embodiments of this application;

[0022] Figure 7 This is a schematic diagram of the composition of the voice activity detection device in the embodiments of this application;

[0023] Figure 8 This is a schematic diagram of the electronic device structure in an embodiment of this application. Detailed Implementation

[0024] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing the embodiments only and is not intended to limit the scope of this application.

[0026] In the following description, references to "some embodiments," "this embodiment," "this embodiment," and examples, etc., describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subset of all possible embodiments and may be combined with each other without conflict.

[0027] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0028] In this embodiment, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0029] This application provides a method for detecting voice activity. Figure 1 This is a schematic diagram of the first process of the voice activity detection method in the embodiments of this application, such as... Figure 1As shown, the speech activity detection method may include:

[0030] Step 101: Obtain the mixed speech and the registered speech of the first speaker.

[0031] In this embodiment, the mixed speech includes the registered speech of the first speaker. Mixed speech refers to a complex sound signal formed by the simultaneous speaking or singing of multiple people, including the first speaker. The registered speech refers to the speech signal of the first speaker.

[0032] Step 102: Use the first personalized speech activity detection (PVAD) model to extract features from the mixed speech and the registered speech, and based on the extracted frame-level acoustic features of the mixed speech and the frame-level acoustic features of the registered speech, obtain the probability distribution of the first speaker in each frame of the sub-mixed speech.

[0033] In this embodiment, a first Personal Voice Activity Detection (PVAD) model is used to extract frame-level features from the mixed speech, obtaining frame-level acoustic features of the mixed speech. Similarly, frame-level features are extracted from the registered speech, obtaining frame-level acoustic features of the registered speech. Based on the frame-level acoustic features of the mixed speech and the registered speech, the probability distribution of the first speaker in each frame of the mixed speech is obtained. This probability distribution of the first speaker in each frame of the mixed speech can be understood as the probability of the first speaker appearing in each frame of the mixed speech. Furthermore, operations such as separation, recognition, or extraction of the mixed speech can be performed based on this probability distribution.

[0034] In some embodiments, step 102 may include:

[0035] Based on the frame-level acoustic features of the mixed speech and the frame-level acoustic features of the registered speech, the component representation of the first speaker in each frame of sub-mixed speech is obtained;

[0036] The component representations of the first speaker in each frame of mixed speech are linearly processed to obtain the probability distribution of the first speaker in each frame of mixed speech.

[0037] In other words, the first PVAD model first obtains the component representation of the first speaker in each frame of mixed speech based on the frame-level acoustic features of the mixed speech and the frame-level acoustic features of the registered speech. Then, it performs linear processing on the component representation of the first speaker in each frame of mixed speech to obtain the probability distribution of the first speaker in each frame of mixed speech.

[0038] Here, the entity executing steps 101 to 102 can be the processor of an electronic device.

[0039] This application utilizes the frame-level acoustic features of registered speech as speaker information for the first speaker. This allows for the extraction of more refined speech signal characteristics when registering using short speech (such as wake word speech), further ensuring the accuracy of the PVAD model's output and thus improving the recall rate of the PVAD model.

[0040] Here, short speech refers to relatively short speech segments, generally only a few seconds to tens of seconds in length. Short speech includes at least wake-up words, command words, and keyword words.

[0041] To better illustrate the purpose of this application, based on the above embodiments, this application provides a schematic diagram of the framework of a first PVAD model. Figure 2 This is a schematic diagram of the framework of the first PVAD model in the embodiments of this application, as shown below. Figure 2 As shown, the first PVAD model includes a feature extractor 201, a first Conformer encoder 202, a cross-attention mechanism module 203, a feature linear modulation layer 204, a second Conformer encoder 205, and a linear layer 206.

[0042] The output of the feature extractor is connected to the input of the first Conformer encoder. The output of the first Conformer encoder is connected to the input of the Cross-Attention Mechanism Module (CA) and the input of the Feature-wise Linear Modulation (FiLM) layer. The output of the Feature-wise Linear Modulation layer is connected to the input of the second Conformer encoder. The output of the second Conformer encoder is connected to the Linear layer.

[0043] It should be noted that, Figure 2 The first Conformer encoder, which indicates that the two parameters are shared, is used to express that the first Conformer encoder performs encoding operations on the frame-level acoustic features of the registered speech and the frame-level acoustic features of the mixed speech, respectively.

[0044] based on Figure 2 The first PVAD model framework diagram is shown below. This application further provides a speech activity detection method. Figure 3 This is a schematic diagram of the second process of the voice activity detection method in the embodiments of this application, as shown below. Figure 3 As shown, the speech activity detection method may include:

[0045] Step 301: Obtain the mixed speech and the registered speech of the first speaker.

[0046] In this embodiment, the mixed speech includes the registered speech of the first speaker. Mixed speech refers to a complex sound signal formed by the simultaneous speaking or singing of multiple people, including the first speaker. The registered speech refers to the speech signal of the first speaker.

[0047] Step 302: Use the mixed speech as input to the feature extractor to obtain the frame-level acoustic features of the mixed speech; use the registered speech as input to the feature extractor to obtain the frame-level acoustic features of the registered speech.

[0048] In this embodiment, the feature extractor is used to extract frame-level acoustic features that represent the mixed speech from the mixed speech, and to extract frame-level acoustic features that represent the registered speech from the registered speech. Frame-level acoustic features of the mixed speech Represent a matrix, NF express The number of rows in a matrix Tm express The number of columns in a matrix Tm The number of frames after mixed speech framing. NF The dimension of each frame of sub-mixed speech (i.e., the dimension of a vector). Register the frame-level acoustic features of the speech. Represent a matrix, NF express The number of rows in a matrix Tr express The number of columns in a matrix Tr To register the number of frames after voice framing processing. NF Register the dimension of speech for each frame.

[0049] Step 303: Use the frame-level acoustic features of the mixed speech as input to the first Conformer encoder to obtain the frame-level coding result of the mixed speech.

[0050] In this embodiment, the first Conformer encoder is used to encode the frame-level acoustic features of the mixed speech to obtain the frame-level coding result of the mixed speech. Represent a matrix, NE express The number of rows in a matrix Tm express The number of columns in a matrix Tm The number of frames after mixed speech framing. NE The dimension of the sub-mixed speech in each frame.

[0051] Step 304: Use the frame-level acoustic features of the registered speech as input to the first Conformer encoder to obtain the frame-level encoding result of the registered speech.

[0052] In this embodiment, the first Conformer encoder is used to encode the frame-level acoustic features of the registered speech to obtain the frame-level encoding result of the registered speech. Represent a matrix, NE express The number of rows in a matrix Tr express The number of columns in a matrix Tr The number of frames after mixed speech framing. NE The dimension of the sub-mixed speech in each frame.

[0053] Step 305: Use the frame-level coding results of the mixed speech and the frame-level coding results of the registered speech as input to the cross-attention mechanism module to obtain the attention weight of the first speaker on each frame of sub-mixed speech.

[0054] In this embodiment, the cross-attention mechanism module performs a cross-attention operation on the frame-level encoding results of the mixed speech and the frame-level encoding of the registered speech to obtain the mapping of the frame-level acoustic features of the registered speech onto the frame-level acoustic features of the mixed speech, i.e., the attention weight of the first speaker in each frame of the mixed speech. The attention weight of the first speaker in each frame of the mixed speech can be understood as the contribution or importance of the first speaker in each frame of the mixed speech.

[0055] Step 306: Use the attention weights of the first speaker in each frame of mixed speech and the frame-level coding result of the mixed speech as input to the feature linear modulation layer to obtain the component representation of the first speaker in each frame of mixed speech.

[0056] In this embodiment, the feature linear modulation layer is used to fuse the attention weight of the first speaker in each frame of sub-mixed speech with the frame-level coding result of the mixed speech to obtain the component representation of the first speaker in each frame of sub-mixed speech.

[0057] Step 307: Use the component representation of the first speaker in each frame of sub-mixed speech as the input of the second Conformer encoder to obtain the encoding result of the first speaker in each frame of sub-mixed speech.

[0058] In this embodiment, the second Conformer encoder is used to encode the component representation of the first speaker in each frame of sub-mixed speech, obtaining the encoding result of the first speaker in each frame of sub-mixed speech. The second Conformer encoder is used to improve the generalization performance of the network model.

[0059] Step 308: Use the encoding result of the first speaker in each frame of sub-mixed speech as the input of the linear layer to obtain the probability distribution of the first speaker in each frame of sub-mixed speech.

[0060] The probability distribution of the first speaker in each frame of the mixed speech can be understood as the probability of the first speaker appearing in each frame of the mixed speech. Furthermore, this probability distribution can be used to perform operations such as separation, recognition, or extraction of the mixed speech.

[0061] In some embodiments, the cross-attention mechanism module includes a Transformer encoder with multiple encoding sub-layers, each encoding sub-layer including a multi-head attention layer and a fully connected layer; based on this, step 305 may include:

[0062] The frame-level encoding result of the mixed speech is used as the query matrix of the first multi-head attention layer, and the frame-level encoding result of the registered speech is used as the key matrix and value matrix of the first multi-head attention layer.

[0063] The Transformer encoder, consisting of multiple multi-head attention layers and fully connected layers, processes the query matrix, the key matrix, and the value matrix to obtain the attention weights of the first speaker in each frame of sub-mixed speech.

[0064] In this embodiment, the cross-attention mechanism module is completed by a Transformer encoder. The Transformer encoder has multiple coding sub-layers. Each coding sub-layer includes a multi-head attention layer and a fully connected layer. A normalization layer is added after the multi-head attention layer, and a normalization layer is added after the fully connected layer.

[0065] Figure 4 This is a schematic diagram of the composition structure of an encoding sublayer provided in an embodiment of this application, such as... Figure 4 As shown, the Transformer encoder includes N coding sub-layers. Each coding sub-layer includes a multi-head attention layer and a fully connected layer. A normalization layer is added after the multi-head attention layer, and a normalization layer is added after the fully connected layer.

[0066] Based on this, the output O of the cross-attention mechanism module CA It can be calculated using formula (1):

[0067] O CA =Transformer(q=E) mix ;k,v=E ref (1)

[0068] The frame-level encoding result of the mixed speech is used as the query matrix (q) of the first multi-head attention layer, and the frame-level encoding result of the registered speech is used as the key matrix (k) and value matrix (v) of the first multi-head attention layer. The query matrix, key matrix and value matrix are processed by the Transformer encoder to obtain the attention weight of the first speaker on each frame of the mixed speech.

[0069] This application utilizes frame-level acoustic features of registered speech as speaker information for the first speaker. This allows for the extraction of finer speech signal characteristics from short speech samples (such as wake-up words) during registration, further ensuring the accuracy of the PVAD model's output and thus improving its recall. The improved recall of the PVAD model ensures that significant speaker-specific information is not lost before the user's speech input reaches the downstream model, preventing the downstream model from crashing due to inaccurate PVAD results (especially when recall is low).

[0070] In addition, the PVAD model in this application has a small number of parameters and only uses the Conformer encoder, which can be easily extended into a streaming model and deployed to the edge.

[0071] It should be noted that the first PVAD model is a pre-trained model. Therefore, model training is required before performing the steps of obtaining the mixed speech and the registered speech of the first speaker, which may include... Figure 5 The speech activity detection method shown:

[0072] Step 501: Obtain a first mixed speech sample set, a first registered speech sample set of multiple second speakers, and a first sample tag set. Each first sample tag includes the true probability distribution of the corresponding second speaker in each frame of sub-mixed speech sample. The length of the first registered speech sample is greater than or equal to a first threshold.

[0073] In this embodiment, the first mixed speech sample set corresponds one-to-one with the first registered speech sample sets of multiple second speakers. The first mixed speech sample includes the first registered speech sample corresponding to the second speaker. Each first sample tag includes the true probability distribution of the corresponding second speaker in each frame of sub-mixed speech sample, which can be understood as the true probability of the corresponding second speaker appearing in each frame of sub-mixed speech sample.

[0074] The length of the first registered speech sample is greater than or equal to the first threshold, which means that the first registered speech sample is a long speech, such as a sentence or a paragraph.

[0075] Step 502: Based on the first mixed speech sample set, the first registered speech sample set, and the first sample label set, train the preset PVAD model to obtain the second PVAD model.

[0076] In this embodiment, the first mixed speech sample and the first registered speech sample are used as inputs to a preset PVAD model to obtain the predicted probability distribution of the second speaker in each frame of the mixed speech sample. Then, based on a preset loss function, the true probability distribution (i.e., the first sample label) and the predicted probability distribution of the second speaker in each frame of the mixed speech sample, the loss value is determined. The model parameters of the preset PVAD model are then adjusted according to the loss value until the preset PVAD model meets the convergence condition. This step is repeated to finally obtain the second PVAD model.

[0077] Here, the pre-defined PVAD model includes a feature extractor, a first Conformer encoder, a cross-attention mechanism module, a feature linear modulation layer, a second Conformer encoder, and a linear layer.

[0078] Step 503: Obtain the second mixed speech sample set, the second registered speech sample set of multiple third speakers, and the second sample label set. Each second sample label includes the true probability distribution of the corresponding third speaker in each frame of sub-mixed speech sample. The length of the second registered speech sample is less than the first threshold.

[0079] In this embodiment, the second mixed speech sample set corresponds one-to-one with the second registered speech sample sets of multiple third speakers, and the second mixed speech sample includes the second registered speech sample corresponding to the third speaker. Each second sample label includes the true probability distribution of the corresponding third speaker in each frame of sub-mixed speech sample, which can be understood as the true probability of the corresponding third speaker appearing in each frame of sub-mixed speech sample.

[0080] The length of the second registered speech sample is less than the first threshold, which means that the second registered speech sample is a short speech. Short speech refers to a relatively short speech segment, generally only a few seconds to tens of seconds in length. Short speech includes at least wake word speech, command word speech and keyword speech.

[0081] Step 504: Based on the second mixed speech sample set, the second registered speech sample set, and the second sample label set, train the second PVAD model to obtain the first PVAD model.

[0082] In this embodiment, the second mixed speech sample and the second registered speech sample are used as inputs to the second PVAD model to obtain the predicted probability distribution of the third speaker in each frame of the mixed speech sample. Then, based on a preset loss function, the true probability distribution (i.e., the second sample label) of the third speaker in each frame of the mixed speech sample, and the predicted probability distribution, a loss value is determined. The model parameters of the second PVAD model are then adjusted according to the loss value until the second PVAD model meets the convergence condition. This step is repeated to finally obtain the first PVAD model.

[0083] First, a pre-defined PVAD model is trained using common interactive speech (i.e., long registered speech) to obtain a second PVAD model. Then, a short registered speech is used to train the second PVAD model to obtain a first PVAD model. In this way, the first PVAD model can recognize both long and short registered speech.

[0084] Before performing the steps of acquiring the mixed speech and the registered speech of the first speaker, model training is required, and may also include... Figure 6 The speech activity detection method shown:

[0085] Step 601: Obtain a second mixed speech sample set, a second registered speech sample set of multiple third speakers, and a second sample tag set. Each second sample tag includes the true probability distribution of the corresponding third speaker in each frame of sub-mixed speech sample. The length of the second registered speech sample is less than a first threshold.

[0086] In this embodiment, the second mixed speech sample set corresponds one-to-one with the second registered speech sample sets of multiple third speakers, and the second mixed speech sample includes the second registered speech sample corresponding to the third speaker. Each second sample label includes the true probability distribution of the corresponding third speaker in each frame of sub-mixed speech sample, which can be understood as the true probability of the corresponding third speaker appearing in each frame of sub-mixed speech sample.

[0087] The length of the second registered speech sample is less than the first threshold, which means that the second registered speech sample is a short speech. Short speech refers to a relatively short speech segment, generally only a few seconds to tens of seconds in length. Short speech includes at least wake word speech, command word speech and keyword speech.

[0088] Step 602: Based on the second mixed speech sample set, the second registered speech sample set, and the second sample label set, train the preset PVAD model to obtain the first PVAD model.

[0089] In this embodiment, the second mixed speech sample and the second registered speech sample are used as inputs to a preset PVAD model to obtain the predicted probability distribution of the third speaker in each frame of the mixed speech sample. Then, based on a preset loss function, the true probability distribution (i.e., the second sample label) of the third speaker in each frame of the mixed speech sample, and the predicted probability distribution, a loss value is determined. The model parameters of the preset PVAD model are then adjusted according to the loss value until the preset PVAD model meets the convergence condition. This step is repeated to finally obtain the first PVAD model.

[0090] Here, the first PVAD model includes a feature extractor, a first Conformer encoder, a cross-attention mechanism module, a feature linear modulation layer, a second Conformer encoder, and a linear layer.

[0091] Even if the preset PVAD model is trained using short registered speech to obtain the first PVAD model, the first PVAD model can only recognize short registered speech.

[0092] To implement the method of the embodiments of this application, based on the same inventive concept, the embodiments of this application also provide a voice activity detection device. Figure 7 This is a schematic diagram of the composition of the voice activity detection device in the embodiments of this application, as shown below. Figure 7 As shown, the voice activity detection device includes:

[0093] Acquisition unit 701 is used to acquire the mixed speech and the registered speech of the first speaker;

[0094] The processing unit 702 is used to extract features from the mixed speech and the registered speech using a first personalized speech activity detection (PVAD) model, and to obtain the probability distribution of the first speaker in each frame of sub-mixed speech based on the extracted frame-level acoustic features of the mixed speech and the frame-level acoustic features of the registered speech.

[0095] This application utilizes the frame-level acoustic features of registered speech as speaker information for the first speaker. This allows for the extraction of more refined speech signal characteristics when registering using short speech (such as wake word speech), further ensuring the accuracy of the PVAD model's output and thus improving the recall rate of the PVAD model.

[0096] In some embodiments, the processing unit 702 is further configured to obtain the component representation of the first speaker in each frame of sub-mixed speech based on the frame-level acoustic features of the mixed speech and the frame-level acoustic features of the registered speech;

[0097] Linear processing is performed on the component representation of the first speaker in each frame of mixed speech to obtain the probability distribution of the first speaker in each frame of mixed speech.

[0098] In some embodiments, the first PVAD model includes a first Conformer encoder, a cross-attention mechanism module, and a feature linear modulation layer; the processing unit 702 is further configured to use the frame-level acoustic features of the mixed speech as input to the first Conformer encoder to obtain the frame-level encoding result of the mixed speech.

[0099] The frame-level acoustic features of the registered speech are used as input to the first Conformer encoder to obtain the frame-level encoding result of the registered speech.

[0100] The frame-level encoding results of the mixed speech and the frame-level encoding results of the registered speech are used as inputs to the cross-attention mechanism module to obtain the attention weight of the first speaker on each frame of sub-mixed speech.

[0101] The attention weights of the first speaker in each frame of sub-mixed speech and the frame-level coding result of the mixed speech are used as inputs to the feature linear modulation layer to obtain the component representation of the first speaker in each frame of sub-mixed speech.

[0102] In some embodiments, the cross-attention mechanism module includes a Transformer encoder with multiple coding sub-layers, each coding sub-layer including a multi-head attention layer and a fully connected layer; the processing unit 702 is further configured to use the frame-level coding result of the mixed speech as the query matrix of the first multi-head attention layer, and use the frame-level coding result of the registered speech as the key matrix and value matrix of the first multi-head attention layer;

[0103] The Transformer encoder, consisting of multiple multi-head attention layers and fully connected layers, processes the query matrix, the key matrix, and the value matrix to obtain the attention weights of the first speaker in each frame of sub-mixed speech.

[0104] In some embodiments, the processing unit 702 is further configured to use the component representation of the first speaker in each frame of sub-mixed speech as the input of the second Conformer encoder to obtain the encoding result of the first speaker in each frame of sub-mixed speech;

[0105] The encoding result of the first speaker in each frame of sub-mixed speech is used as the input of the linear layer to obtain the probability distribution of the first speaker in each frame of sub-mixed speech.

[0106] In some embodiments, the acquisition unit 701 is further configured to acquire a first mixed speech sample set, a first registered speech sample set of multiple second speakers, and a first sample tag set, wherein each first sample tag includes the true probability distribution of the corresponding second speaker in each frame of sub-mixed speech sample, and the length of the first registered speech sample is greater than or equal to a first threshold.

[0107] Processing unit 702 is further configured to train a preset PVAD model based on the first mixed speech sample set, the first registered speech sample set and the first sample label set to obtain a second PVAD model;

[0108] The acquisition unit 701 is also used to acquire a second mixed speech sample set, a second registered speech sample set of multiple third speakers and a second sample tag set, wherein each second sample tag includes the true probability distribution of the corresponding third speaker in each frame of sub-mixed speech sample, and the length of the second registered speech sample is less than the first threshold.

[0109] The processing unit 702 is further configured to train the second PVAD model based on the second mixed speech sample set, the second registered speech sample set, and the second sample label set to obtain the first PVAD model.

[0110] In some embodiments, the acquisition unit 701 is further configured to acquire a second mixed speech sample set, a second registered speech sample set of multiple third speakers, and a second sample tag set, wherein each second sample tag includes the true probability distribution of the corresponding third speaker in each frame of sub-mixed speech sample, and the length of the second registered speech sample is less than a first threshold.

[0111] The processing unit 702 is further configured to train a preset PVAD model based on the second mixed speech sample set, the second registered speech sample set and the second sample label set to obtain a first PVAD model.

[0112] This application also provides another electronic device. Figure 8 This is a schematic diagram of the electronic device composition structure in the embodiments of this application, such as... Figure 8 As shown, the electronic device 80 includes: a processor 801 and a memory 802 configured to store computer programs capable of running on the processor;

[0113] When the processor 801 is configured to run a computer program, it executes the method steps described in the foregoing embodiments.

[0114] Of course, in practical applications, such as Figure 8 As shown, the various components in the electronic device 80 are coupled together via a bus system 803. It can be understood that the bus system 803 is used to implement communication between these components. In addition to a data bus, the bus system 803 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 8 The general labeled all buses as Bus System 803.

[0115] In practical applications, the aforementioned processor can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic devices used to implement the functions of the aforementioned processor can also be other types, and the embodiments of this application do not specifically limit this.

[0116] The aforementioned memory can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor.

[0117] In an exemplary embodiment, this application also provides a computer-readable storage medium for storing a computer program.

[0118] Optionally, the computer-readable storage medium can be applied to any of the methods in the embodiments of this application, and the computer program causes the computer to execute the corresponding processes implemented by the processor in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0120] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0121] Furthermore, in the various embodiments of the present invention, all functional units can be integrated into one processing module, or each unit can be a separate unit, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units. Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0122] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0123] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0124] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0125] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting speech activity, characterized in that, The method includes: Obtain the mixed speech and the registered speech of the first speaker; The first personalized speech activity detection (PVAD) model is used to extract features from the mixed speech and the registered speech. Based on the extracted frame-level acoustic features of the mixed speech and the frame-level acoustic features of the registered speech, the probability distribution of the first speaker in each frame of the sub-mixed speech is obtained. The step of obtaining the probability distribution of the first speaker in each sub-frame of mixed speech based on the extracted frame-level acoustic features of the mixed speech and the frame-level acoustic features of the registered speech includes: obtaining the component representation of the first speaker in each sub-frame of mixed speech based on the frame-level acoustic features of the mixed speech and the frame-level acoustic features of the registered speech; and performing linear processing on the component representation of the first speaker in each sub-frame of mixed speech to obtain the probability distribution of the first speaker in each sub-frame of mixed speech. The first PVAD model includes a first Conformer encoder, a cross-attention mechanism module, and a feature linear modulation layer. The step of obtaining the component representation of the first speaker in each sub-frame of mixed speech based on the frame-level acoustic features of the mixed speech and the frame-level acoustic features of the registered speech includes: using the frame-level acoustic features of the mixed speech as input to the first Conformer encoder to obtain the frame-level encoding result of the mixed speech; using the frame-level acoustic features of the registered speech as input to the first Conformer encoder to obtain the frame-level encoding result of the registered speech; using the frame-level encoding result of the mixed speech and the frame-level encoding result of the registered speech as input to the cross-attention mechanism module to obtain the attention weight of the first speaker in each sub-frame of mixed speech; and using the attention weight of the first speaker in each sub-frame of mixed speech and the frame-level encoding result of the mixed speech as input to the feature linear modulation layer to obtain the component representation of the first speaker in each sub-frame of mixed speech. The first PVAD model includes a second Conformer encoder and a linear layer. The step of linearly processing the component representation of the first speaker in each frame of sub-mixed speech to obtain the probability distribution of the first speaker in each frame of sub-mixed speech includes: using the component representation of the first speaker in each frame of sub-mixed speech as the input of the second Conformer encoder to obtain the encoding result of the first speaker in each frame of sub-mixed speech; and using the encoding result of the first speaker in each frame of sub-mixed speech as the input of the linear layer to obtain the probability distribution of the first speaker in each frame of sub-mixed speech.

2. The method according to claim 1, characterized in that, The cross-attention mechanism module includes a Transformer encoder with multiple coding sub-layers, each coding sub-layer including a multi-head attention layer and a fully connected layer; the step of using the frame-level coding results of the mixed speech and the frame-level coding results of the registered speech as inputs to the cross-attention mechanism module to obtain the attention weights of the first speaker on each frame of sub-mixed speech includes: The frame-level encoding result of the mixed speech is used as the query matrix of the first multi-head attention layer, and the frame-level encoding result of the registered speech is used as the key matrix and value matrix of the first multi-head attention layer. The Transformer encoder, consisting of multiple multi-head attention layers and fully connected layers, processes the query matrix, the key matrix, and the value matrix to obtain the attention weights of the first speaker in each frame of sub-mixed speech.

3. The method according to claim 1, characterized in that, Before acquiring the mixed speech and the registered speech of the first speaker, the method further includes: Obtain a first mixed speech sample set, a first registered speech sample set of multiple second speakers, and a first sample tag set. Each first sample tag includes the true probability distribution of the corresponding second speaker in each frame of sub-mixed speech sample. The length of the first registered speech sample is greater than or equal to a first threshold. Based on the first mixed speech sample set, the first registered speech sample set, and the first sample label set, a preset PVAD model is trained to obtain a second PVAD model. Obtain a second mixed speech sample set, a second registered speech sample set of multiple third speakers, and a second sample tag set. Each second sample tag includes the true probability distribution of the corresponding third speaker in each frame of sub-mixed speech sample. The length of the second registered speech sample is less than the first threshold. Based on the second mixed speech sample set, the second registered speech sample set, and the second sample label set, the second PVAD model is trained to obtain the first PVAD model.

4. The method according to claim 1, characterized in that, Before acquiring the mixed speech and the registered speech of the first speaker, the method further includes: Obtain a second mixed speech sample set, a second registered speech sample set of multiple third speakers, and a second sample tag set. Each second sample tag includes the true probability distribution of the corresponding third speaker in each frame of sub-mixed speech sample. The length of the second registered speech sample is less than a first threshold. Based on the second mixed speech sample set, the second registered speech sample set, and the second sample label set, a preset PVAD model is trained to obtain the first PVAD model.

5. A voice activity detection device, characterized in that, The apparatus is used to perform the method according to any one of claims 1 to 4.

6. An electronic device, characterized in that, The electronic device includes: a processor and a memory configured to store computer programs capable of running on the processor. Wherein, when the processor is configured to run the computer program, it performs the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 4.

Images