Voice activity detection model construction method and device, and electronic device

Abstract

Embodiments of the present disclosure relate to a voice activity detection model construction method and device and electronic equipment. The method comprises: acquiring multi-modal data; performing feature extraction on each modal data in the multi-modal data respectively to obtain modal features corresponding to each modal data respectively; when the modal features corresponding to one or more modal data in the multi-modal data do not conform to a preset alignment standard, performing feature alignment processing on the modal features corresponding to the one or more modal data; fusing all the modal features after feature alignment to obtain fused features; and iteratively training an initially constructed model using the fused features until the model after training meets a preset condition, and determining the model after meeting the preset condition as a final voice activity detection model. Through the model, the sensitivity and accuracy of voice activity detection can be improved, and the false detection rate and the missed detection rate can be reduced.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, and electronic device for constructing a voice activity detection model. Background Technology

[0002] Voice activity detection (VAD) is used to detect speech within an audio clip. In far-field voice interaction scenarios, current VAD technologies face several challenges, including successfully detecting the lowest-energy speech, i.e., improving VAD's sensitivity and accuracy, and successfully detecting speech in noisy environments, i.e., reducing the false positive / false negative rate of VAD.

[0003] It is precisely because the current VAD technology still has the above-mentioned problems that the application of VAD is not widespread, and the application effect in many scenarios with very high requirements for speech detection is not ideal. Summary of the Invention

[0004] This disclosure provides a method, apparatus, and electronic device for constructing a speech activity detection model to solve some or all of the technical problems in the prior art.

[0005] Firstly, this disclosure provides a method for constructing a speech activity detection model, including:

[0006] Acquire multimodal data, which includes at least audio and video data;

[0007] Feature extraction is performed on each modal data in the multimodal data to obtain the modal features corresponding to each modal data.

[0008] When the modal features corresponding to one or more modal data in a variety of modal data do not conform to the preset alignment standard, feature alignment processing is performed on the modal features corresponding to one or more modal data.

[0009] All modal features after feature alignment are fused to obtain fused features;

[0010] By utilizing fusion features, the pre-constructed initial model is iteratively trained until the trained model meets the preset conditions. The model that meets the preset conditions is then determined as the final speech activity detection model.

[0011] The method provided in this disclosure acquires multimodal data, which includes at least audio data and video data, and may also include at least audio data and image data. Features are extracted from each modality of the multimodal data, or modal features corresponding to each modality. Then, feature alignment is performed on the modal features corresponding to one or more modalities in the multimodal data to align all modal features. The aligned modal features are then fused to obtain fused features. A pre-constructed initial model is iteratively trained using these fused features until the trained model meets preset conditions. The model that meets the preset conditions is then determined as the final speech activity detection model. Throughout this process, because both speech and video data are involved, the speech recognition and image recognition capabilities of the model will improve after continuous iterative training of the initial model. Therefore, the improved speech recognition capability can indirectly improve the sensitivity and accuracy of VAD (Voice Activity Detection). Furthermore, by detecting video data, such as facial muscle movements, it is possible to further verify whether a user is speaking, thus further verifying the presence of speech data and improving the sensitivity and accuracy of VAD. Furthermore, it can reduce the false positive rate of VAD. Even in noisy environments where VAD cannot detect speech through voice data, it can still identify that a user is speaking through image features, thus reducing the false positive rate. Moreover, even if speech is detected through voice data, if image recognition determines that no facial muscles are moving at the current moment, it can still identify that no one is actually speaking; the voice may be coming from another device. In this case, VAD can still determine that no one is speaking, thus reducing the false positive rate. Of course, if the multimodal data includes other modal data, the detection accuracy of VAD can be further improved. Therefore, in this application, the multimodal data includes at least audio and video data.

[0012] Secondly, this disclosure provides a speech activity detection model construction apparatus, the apparatus comprising:

[0013] The acquisition module is used to acquire multimodal data, which includes at least audio data and video data;

[0014] The extraction module is used to extract features from each modality of the multimodal data to obtain the modal features corresponding to each modality.

[0015] The processing module is used to perform feature alignment processing on the modal features corresponding to one or more modal data when the modal features corresponding to one or more modal data do not conform to the preset alignment standard.

[0016] The fusion module is used to fuse all modal features after feature alignment to obtain fused features;

[0017] The training module is used to iteratively train the pre-built initial model using fused features until the trained model meets preset conditions. The model that meets the preset conditions is then determined as the final speech activity detection model.

[0018] The speech activity detection model construction apparatus provided in this disclosure acquires multimodal data, which includes at least audio data and video data, and may also include at least audio data and image data. Features are extracted from each modality of the multimodal data, or modal features corresponding to each modality. Then, feature alignment processing is performed on the modal features corresponding to one or more modal data in the multimodal data to align all modal features. The aligned modal features are then fused to obtain fused features. The fused features are then used to iteratively train a pre-constructed initial model until the trained model meets preset conditions. The model that meets the preset conditions is then determined as the final speech activity detection model. Throughout this process, because both speech and video data are involved, the speech recognition and image recognition capabilities of the model will improve after continuous iterative training of the initial model. Therefore, the improved speech recognition capability can indirectly improve the sensitivity and accuracy of VAD (Voice Activity Detection). Furthermore, by detecting video data, such as facial muscle movements, it is possible to further verify whether a user is speaking, thus further verifying the presence of speech data and improving the sensitivity and accuracy of VAD. Furthermore, it can reduce the false positive rate of VAD. Even in noisy environments where VAD cannot detect speech through voice data, it can still identify that a user is speaking through image features, thus reducing the false positive rate. Moreover, even if speech is detected through voice data, if image recognition determines that no facial muscles are moving at the current moment, it can still identify that no one is actually speaking; the voice may be coming from another device. In this case, VAD can still determine that no one is speaking, thus reducing the false positive rate. Of course, if the multimodal data includes other modal data, the detection accuracy of VAD can be further improved. Therefore, in this application, the multimodal data includes at least audio and video data.

[0019] Thirdly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0020] Memory, used to store computer programs;

[0021] When a processor executes a program stored in memory, it implements the steps of the speech activity detection model construction method according to any embodiment of the first aspect.

[0022] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by an electronic device, implements the steps of the voice activity detection model construction method, apparatus, and electronic device as described in any embodiment of the first aspect. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of a method for constructing a speech activity detection model provided in this disclosure;

[0024] Figure 2 This is a schematic diagram of a twin-tower structure provided in this disclosure;

[0025] Figure 3 A schematic diagram of another speech activity detection model construction device provided in this disclosure;

[0026] Figure 4 This is a schematic diagram of an electronic device structure provided in an embodiment of the present disclosure. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] To facilitate understanding of the embodiments of this disclosure, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of this disclosure.

[0029] In response to the technical problems mentioned in the background section, this disclosure provides a method, apparatus, and electronic device for constructing a voice activity detection model.

[0030] The specific methods and steps for constructing the speech activity detection model will be described in detail below. Please refer to the accompanying drawings provided in the embodiments of this disclosure. Figure 1 This is a schematic flowchart illustrating a method for constructing a speech activity detection model according to an embodiment of this disclosure. The method includes the following steps:

[0031] Step 110: Obtain multimodal data.

[0032] Currently, mainstream VADs are typically audio-based, meaning they only utilize audio-based information. However, in real-world applications, device sensors can collect a wide variety of signals, not just audio. Therefore, utilizing these multimodal signals can significantly improve the detection accuracy of VADs compared to those based solely on audio signals. Even in noisy environments, this greatly enhances the sensitivity and accuracy of sound detection, while also reducing false negative and false positive rates.

[0033] Multimodal data, as the name suggests, refers to data in multiple different forms. For example, audio data, video data, image data, infrared data, laser data, etc., all belong to different modalities of data. In this disclosure, multimodal data includes at least audio data and video data, or it may also include at least audio data and image data. The following explanation uses audio and video data as an example of multimodal data; the processing method for other modalities is similar and will not be elaborated upon here.

[0034] Step 120: Extract features from each modal data in the multimodal data to obtain the modal features corresponding to each modal data.

[0035] Specifically, assuming multimodal data includes audio and video data, feature extraction from audio data yields audio features. Feature extraction from video data yields image features. The specific feature extraction methods can be implemented using conventional techniques already in the market, so they will not be explained in detail here.

[0036] Step 130: When the modal features corresponding to one or more modal data in multiple modal data do not conform to the preset alignment standard, feature alignment processing is performed on the modal features corresponding to one or more modal data.

[0037] Specifically, during feature alignment, an alignment standard can be pre-defined. If the modal features corresponding to some modal data in the multimodal data already meet the preset alignment standard, then feature alignment is not required. When one or more modal features do not meet the preset alignment standard, feature alignment is required for one or more modal features in the multimodal data.

[0038] Therefore, the first step is to determine whether the modal features corresponding to each type of modal data meet the preset alignment standard, and then determine whether the modal data needs to undergo feature alignment processing.

[0039] Optionally, in a specific example, a preset alignment standard is used to indicate the number of preset features per unit time.

[0040] When determining whether the modal features corresponding to each type of modal data meet the preset alignment standard, the following method can be used:

[0041] Specifically, the number of features corresponding to each modality data per unit time is obtained, and then the number of features is compared with a preset alignment standard. If the two are equal, no feature alignment processing is required; otherwise, feature alignment processing is required.

[0042] The specific feature alignment processing method can be jointly determined based on the number of modal features per unit time and a preset alignment standard.

[0043] After determining the feature alignment processing method for the modal features of the first modal data based on the preset alignment standard and the number of features corresponding to the first modal data per unit time, the feature alignment processing of the modal features corresponding to the first modal data is completed according to the feature alignment processing method corresponding to the modal features of the first modal data. The feature alignment processing method includes upsampling or downsampling of the modal features. The first modal data is any one of one or more modal data.

[0044] Step 140: Fuse all the modal features after feature alignment to obtain the fused features.

[0045] Step 150: Using fusion features, iteratively train the pre-constructed initial model until the trained model meets the preset conditions. Then, determine the model that meets the preset conditions as the final speech activity detection model.

[0046] Specifically, by fusing features from multiple modalities to train a model, the model can better recognize various types of data features and then comprehensively identify whether there is speech activity based on these different types of data features. For example, if a person's mouth is moving and a voice is being heard at the same time, it indicates that there is speech activity and someone is speaking.

[0047] Optionally, in this application, the focus is on extracting human facial features, including but not limited to lip movement detection. This further includes feature extraction of facial muscle movements, facial expressions, etc., to improve the accuracy of speech activity detection.

[0048] In this situation, even if the user's mouth is covered (making it impossible to determine whether the user is speaking by mouth shape) or the head is tilted, features extracted from local facial muscles or local facial expressions can still be used to identify whether the current user is speaking.

[0049] Moreover, by combining multimodal features such as image features, it is also possible to avoid situations where, for example, a mobile phone ringtone may contain "human voices" but is not spoken by people on site, and these may be falsely detected.

[0050] Of course, even in the presence of noise, modal features such as image features can be used for auxiliary identification to prevent noise from being mistaken for "human voice".

[0051] Once the model is built, it can also greatly expand the application scenarios of VAD. It is fully applicable even to application scenarios with high requirements for detection sensitivity, false negative rate and false positive rate.

[0052] The speech activity detection model construction method provided in this disclosure acquires multimodal data, which includes at least audio data and video data, and may also include at least audio data and image data. Features are extracted from each modality of the multimodal data, or modal features corresponding to each modality. Then, feature alignment is performed on the modal features corresponding to one or more modalities in the multimodal data to ensure alignment of all modal features. The aligned modal features are then fused to obtain fused features. The fused features are then used to iteratively train a pre-constructed initial model until the trained model meets preset conditions. The model that meets the preset conditions is then determined as the final speech activity detection model. Throughout this process, because both speech and video data are involved, the speech recognition and image recognition capabilities of the model will improve after continuous iterative training of the initial model. Therefore, the improved speech recognition capability can indirectly improve the sensitivity and accuracy of VAD (Voice Activity Detection). Furthermore, by detecting video data, such as facial muscle movements, it is possible to further verify whether a user is speaking, thus further verifying the presence of speech data and improving the sensitivity and accuracy of VAD. Furthermore, it can reduce the false positive rate of VAD. Even in noisy environments where VAD cannot detect speech through voice data, it can still identify that a user is speaking through image features, thus reducing the false positive rate. Moreover, even if speech is detected through voice data, if image recognition determines that no facial muscles are moving at the current moment, it can still identify that no one is actually speaking; the voice may be coming from another device. In this case, VAD can still determine that no one is speaking, thus reducing the false positive rate. Of course, if the multimodal data includes other modal data, the detection accuracy of VAD can be further improved. Therefore, in this application, the multimodal data includes at least audio and video data.

[0053] Based on the above embodiments, this disclosure also provides another method for constructing a speech activity detection model. The same or similar content as in the above embodiments will not be repeated here. The following will describe in detail how to determine the feature alignment processing method corresponding to the first modality data according to a preset alignment standard.

[0054] See the following for details, including:

[0055] When the number of speech features corresponding to the first modal data per unit time is greater than the preset number of features, the feature alignment processing method for the modal features of the first modal data is determined to be downsampling processing.

[0056] Alternatively, when the number of speech features corresponding to the first modal data in the multimodal data is less than the preset number of features per unit time, the feature alignment processing method for the modal features of the first modal data is determined to be upsampling processing.

[0057] Specifically, let's assume the preset alignment standard is 50 frames of features per unit time, for example, per second;

[0058] For audio signals, such as those collected by a microphone array, assuming a sampling frequency of 16kHz, after extracting audio features using methods such as Discrete Fourier Transform, the frame rate is 100 frames per second.

[0059] For video captured by a camera, after image feature extraction, there are 25 frames of image features per second. To align the two, downsampling can be performed on the audio features, and upsampling can be performed on the image features.

[0060] In a specific example, upsampling can be achieved by linear interpolation or non-linear interpolation between adjacent frames, or other upsampling methods can be used, without any restrictions.

[0061] For image features, random downsampling methods can be used, or upsampling of the data can be performed using a dual-tower structure.

[0062] Of course, in addition to performing sampling processing on image features, the dual-tower structure can also include performing sampling processing on audio features.

[0063] In an optional example, such as when the multimodal data includes first modal data and second modal data, where the first modal data is speech data and the second modal data is video data, the method may further include the following steps when the number of speech features corresponding to the first modal data per unit time is greater than a preset number of features, and the number of video features corresponding to the second modal data per unit time is less than a preset number of features:

[0064] Speech features and video features are respectively input into a pre-built feature alignment model.

[0065] As described above, the method for determining the feature alignment processing method for the first modality data based on a preset alignment standard involves downsampling of speech features and upsampling of video features in the feature alignment model. This aligns the processed speech features with the processed video features. The feature alignment model includes at least two feature alignment processing channels, with speech features and video features respectively input into different feature alignment processing channels within the model.

[0066] In other words, speech features need to be input into one of the feature alignment processing channels in the feature alignment model for downsampling, and video features need to be input into another feature alignment processing channel in the feature alignment model for upsampling.

[0067] Among them, the feature alignment model can be, for example, a dual-tower structure feature extraction model.

[0068] In a specific example, see the twin-tower structure. Figure 2 As shown, Figure 2 The diagram illustrates a structural block diagram of a double-tower structure.

[0069] exist Figure 2 The encoder 1 on the left inputs speech features, and the encoder 2 on the right inputs image features. The encoders perform downsampling on the audio features and upsampling on the image features. Finally, in... Figure 2 The output layers of the two encoders in the code align the frame numbers of the two encoders, which is equivalent to completing the feature alignment operation. This is understandable. Figure 2 The encoder 1 on the left can be understood as one of the feature alignment processing channels in the feature alignment model (dual-tower structure), while Figure 2 The encoder 2 on the right can be understood as another feature alignment processing channel in the feature alignment model. The specific process of using the encoder in the dual-tower structure to complete the feature alignment operation is existing technology, so it will not be described in detail here.

[0070] Alternatively, unlike the above, if the number of audio features per second, such as 100 frames, is used as the alignment standard, then the audio features do not need to be aligned; instead, upsampling can be performed directly on the image.

[0071] Regardless of which method is used, feature alignment can be achieved to facilitate subsequent feature fusion.

[0072] In another embodiment of this disclosure, all modal features after feature alignment are fused to obtain fused features. This can be achieved in various ways, such as any of the possible methods listed in the examples below.

[0073] The first method involves concatenating all the aligned modal features to obtain fused features.

[0074] The second method involves performing dot product on all the modal features after feature alignment to obtain fused features.

[0075] While both implementations can achieve the acquisition of fused features, in a preferred embodiment, it is recommended to use the dot product method to complete the fusion processing of all modal features.

[0076] Specifically, statistical analysis of experimental data shows that dot product operations are more conducive to model convergence. Moreover, the feature dimension of the fused features generated after dot product is smaller than that of the fused features generated after feature concatenation, which greatly reduces the computational load during model training, thereby improving the efficiency of model training.

[0077] Optionally, in another embodiment of this disclosure, another embodiment of the speech activity detection model construction method is also provided, which is the same as or similar to any of the above embodiments and will not be described again here.

[0078] In this embodiment, the method further includes the following steps after fusing all the aligned modal features to obtain the fused features:

[0079] By utilizing the attention mechanism, the attention mechanism is trained on the fused features to determine the weight coefficients of different modal features in the fused features.

[0080] Specifically, attention mechanisms can be used to enhance the weights of certain parts of the different modal features of the input in a neural network, while weakening the weights of other parts. Attention mechanisms are a technique in artificial neural networks that mimics cognitive attention.

[0081] For example, weighting one or more features such as facial muscles, expressions, movements, and lip movements can be increased to predict whether a user is speaking based on changes in these features. Increasing the weight of these speech features improves the sensitivity and accuracy of speech recognition, while also helping to reduce false positive and false negative rates. Conversely, for other features, such as environmental noise, which are less important, their weights should be minimized to avoid impacting the sensitivity and accuracy of speech activity detection and to prevent increased false positive and false negative rates.

[0082] In other words, incorporating an attention mechanism facilitates subsequent iterative training of the initial model based on the fused features. This attention mechanism helps determine the weight coefficients of different modal features within the fused features. Furthermore, based on these weight coefficients, the weights of features that accurately identify speech activities are increased, thereby improving the sensitivity and accuracy of speech activity detection, and reducing false positives and false negatives.

[0083] It should be noted that this disclosure focuses on how to construct a speech activity detection model that can improve the accuracy, sensitivity, and precision of speech activity detection, and reduce the false positive and false negative rates. The subsequent application process is completely similar to the application scenarios of other constructed models; therefore, no further detailed descriptions or explanations of subsequent applications are provided here. Similarly, after constructing a speech activity detection model using any of the embodiments described above, it is not necessary to perform training every time before applying it to a specific application scenario. Instead, once the model is constructed (or passes testing), it can be applied to various application scenarios, such as noisy scenarios, large-scale conference scenarios, noisy outdoor scenarios, or scenarios where the user's face is partially obscured, especially when the mouth is obscured.

[0084] The above are the method embodiments for constructing the speech activity detection model provided in this disclosure. Other embodiments for constructing the speech activity detection model provided in this disclosure will be described below. Please refer to the following for details.

[0085] Figure 3 This is a schematic diagram of a speech activity detection model building device provided in an embodiment of the present disclosure. The device includes: an acquisition module 301, an extraction module 302, a processing module 303, a fusion module 304, and a training module 305.

[0086] The acquisition module 301 is used to acquire multimodal data, wherein the multimodal data includes at least audio data and video data;

[0087] The extraction module 302 is used to extract features from each modal data in the multimodal data and obtain the modal features corresponding to each modal data.

[0088] The processing module 303 is used to perform feature alignment processing on the modal features corresponding to one or more modal data when the modal features corresponding to one or more modal data do not conform to the preset alignment standard.

[0089] The fusion module 304 is used to fuse all the modal features after feature alignment to obtain fused features;

[0090] Training module 305 is used to iteratively train the pre-built initial model using fused features until the trained model meets preset conditions, and then the model that meets the preset conditions is determined as the final speech activity detection model.

[0091] Optionally, the processing module 303 is specifically used for,

[0092] Based on the preset alignment standard and the number of features corresponding to the first modal data per unit time, determine the feature alignment processing method to be performed on the modal features of the first modal data;

[0093] According to the feature alignment processing method corresponding to the modal features of the first modal data, the feature alignment processing of the modal features corresponding to the first modal data is completed. The feature alignment processing method includes upsampling or downsampling of the modal features. The first modal data is any one of one or more modal data.

[0094] Optionally, the processing module 303 is specifically used to determine that the feature alignment processing method performed on the modal features of the first modal data is downsampling processing when the number of features corresponding to the speech features of the first modal data in a unit time is greater than the preset number of features;

[0095] Alternatively, when the number of speech features corresponding to the first modal data in the multimodal data is less than the preset number of features per unit time, the feature alignment processing method for the modal features of the first modal data is determined to be upsampling processing.

[0096] Optionally, the processing module 303 is specifically used to: when the multimodal data includes first modal data and second modal data, and the number of features corresponding to the speech features of the first modal data in a unit time is greater than the preset number of features, and the number of features corresponding to the video features of the second modal data in a unit time is less than the preset number of features, input the speech features and video features into a pre-constructed feature alignment model respectively, so as to perform downsampling processing on the speech features and upsampling processing on the video features in the feature alignment model, so that the processed speech features are aligned with the processed video features, wherein the feature alignment model includes at least two feature alignment processing channels, and the speech features and video features are respectively input into different feature alignment processing channels in the feature alignment model.

[0097] Optionally, the fusion module 304 is specifically used to stitch together all the modal features after feature alignment to obtain fused features.

[0098] Optionally, the fusion module 304 is specifically used to perform dot product processing on all modal features after feature alignment to obtain fused features.

[0099] Optionally, the training module 305 is also used to perform attention mechanism training on the fused features using an attention mechanism to determine the weight coefficients of different modal features in the fused features.

[0100] The functions performed by each component in the speech activity detection model construction device provided in this disclosure have been described in detail in the above method embodiments, and therefore will not be repeated here.

[0101] This disclosure provides a speech activity detection model construction device that acquires multimodal data, including at least audio data and image data. Features are extracted from each modality of the multimodal data, or modal features corresponding to each modality. Then, the modal features corresponding to one or more modalities in the multimodal data are aligned to ensure all modal features are aligned. Finally, all aligned modal features are fused to obtain fused features. These fused features are then used to iteratively train a pre-constructed initial model until the trained model meets preset conditions. The model that meets these preset conditions is then identified as the final speech activity detection model. Throughout this process, because both audio and video data are involved, the model's speech recognition and image recognition capabilities improve through continuous iterative training. Improved speech recognition indirectly enhances the sensitivity and accuracy of VAD (Voice Activity Detection). Furthermore, detection of video data, such as facial muscle movements, further corroborates the presence of user speech, thus verifying the existence of audio data and further improving the sensitivity and accuracy of VAD. It also reduces the false negative rate of VAD. Even in noisy environments where the VAD cannot detect speech through voice data, it can still identify that a user is speaking through image features, thus reducing the false negative rate. Furthermore, even if speech is detected through voice data, if image recognition determines that no facial muscles are moving at any given moment, it can still identify that no one is actually speaking; the voice may be coming from another device. In this case, the VAD can still determine that no one is speaking, further reducing the false positive rate. Of course, including other modalities in the multimodal data can further improve the VAD's detection accuracy. Therefore, in this application, the multimodal data includes at least audio and video data.

[0102] like Figure 4 As shown, this embodiment of the present disclosure provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0103] Memory 113 is used to store computer programs;

[0104] In one embodiment of this disclosure, when the processor 111 executes a program stored in the memory 113, it implements the speech activity detection model construction method provided in any of the foregoing method embodiments, the method comprising:

[0105] Acquire multimodal data, which includes at least audio and video data;

[0106] Feature extraction is performed on each modal data in the multimodal data to obtain the modal features corresponding to each modal data.

[0107] When the modal features corresponding to one or more modal data in a variety of modal data do not conform to the preset alignment standard, feature alignment processing is performed on the modal features corresponding to one or more modal data.

[0108] All modal features after feature alignment are fused to obtain fused features;

[0109] By utilizing fusion features, the pre-constructed initial model is iteratively trained until the trained model meets the preset conditions. The model that meets the preset conditions is then determined as the final speech activity detection model.

[0110] Optionally, a preset alignment standard is used to indicate the number of corresponding preset features per unit time. When the modal features corresponding to one or more modal data in multiple modal data do not conform to the preset alignment standard, feature alignment processing is performed on the modal features corresponding to one or more modal data, specifically including:

[0111] Based on the preset alignment standard and the number of features corresponding to the first modal data per unit time, determine the feature alignment processing method to be performed on the modal features of the first modal data;

[0112] According to the feature alignment processing method corresponding to the modal features of the first modal data, the feature alignment processing of the modal features corresponding to the first modal data is completed. The feature alignment processing method includes upsampling or downsampling of the modal features. The first modal data is any one of one or more modal data.

[0113] Optionally, based on the preset alignment standard and the number of features corresponding to the first modal data per unit time, the feature alignment processing method for the modal features of the first modal data is determined, specifically including:

[0114] When the number of speech features corresponding to the first modal data per unit time is greater than the preset number of features, the feature alignment processing method for the modal features of the first modal data is determined to be downsampling processing.

[0115] Alternatively, when the number of speech features corresponding to the first modal data in the multimodal data is less than the preset number of features per unit time, the feature alignment processing method for the modal features of the first modal data is determined to be upsampling processing.

[0116] Optionally, when the multimodal data includes first modal data and second modal data, and the number of features corresponding to the speech features of the first modal data per unit time is greater than a preset number of features, and the number of features corresponding to the video features of the second modal data per unit time is less than a preset number of features, the method includes:

[0117] Speech features and video features are respectively input into a pre-built feature alignment model so that downsampling processing of speech features and upsampling processing of video features are performed in the feature alignment model, so that the processed speech features are aligned with the processed video features. The feature alignment model includes at least two feature alignment processing channels, and speech features and video features are respectively input into different feature alignment processing channels in the feature alignment model.

[0118] Optionally, all modal features after feature alignment are fused to obtain fused features, specifically including:

[0119] All modal features after feature alignment are concatenated to obtain fused features.

[0120] Optionally, all modal features after feature alignment are fused to obtain fused features, specifically including:

[0121] Perform dot product on all the modal features after feature alignment to obtain the fused features.

[0122] Optionally, the method further includes fusing all the aligned modal features to obtain the fused features:

[0123] By utilizing the attention mechanism, the attention mechanism is trained on the fused features to determine the weight coefficients of different modal features in the fused features.

[0124] This disclosure also provides a computer-readable storage medium storing a computer program thereon, which, when executed by an electronic device, implements the steps of the speech activity detection model construction method provided in any of the foregoing method embodiments.

[0125] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0126] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for constructing a speech activity detection model, characterized in that, The method includes: Acquire multimodal data, wherein the multimodal data includes at least audio data and video data; Feature extraction is performed on each modal data in the multimodal data to obtain the modal features corresponding to each modal data. When the modal features corresponding to one or more modal data in a variety of modal data do not conform to the preset alignment standard, feature alignment processing is performed on the modal features corresponding to one or more modal data. All modal features after feature alignment are fused to obtain fused features; Using the fusion features, the pre-constructed initial model is iteratively trained until the trained model meets preset conditions. The model that meets the preset conditions is then determined as the final speech activity detection model. The preset alignment standard is used to indicate the number of corresponding preset features per unit time; when the modal features corresponding to one or more modal data in multiple modal data do not conform to the preset alignment standard, feature alignment processing is performed on the modal features corresponding to one or more modal data, specifically including: Based on the preset alignment standard and the number of features corresponding to the first modal data per unit time, the feature alignment processing method to be performed on the modal features of the first modal data is determined; The feature alignment processing is performed on the modal features corresponding to the modal features of the first modal data according to the feature alignment processing method corresponding to the modal features of the first modal data. The feature alignment processing method includes upsampling the modal features or downsampling the modal features. The first modal data is any one of one or more modal data.

2. The method according to claim 1, characterized in that, The step of determining the feature alignment processing method for the modal features of the first modal data based on the preset alignment standard and the number of features corresponding to the first modal data per unit time specifically includes: When the number of features corresponding to the speech features of the first modal data in a unit time is greater than the preset number of features, the feature alignment processing method performed on the modal features of the first modal data is determined to be downsampling processing. Alternatively, when the number of speech features corresponding to the first modal data in the multimodal data is less than the preset number of features per unit time, the feature alignment processing method performed on the modal features of the first modal data is determined to be upsampling processing.

3. The method according to claim 2, characterized in that, When the multimodal data includes first modal data and second modal data, and the number of features corresponding to the speech features of the first modal data in a unit time is greater than the preset number of features, and the number of features corresponding to the video features of the second modal data in a unit time is less than the preset number of features, the method includes: The speech features and the video features are respectively input into a pre-built feature alignment model so that downsampling processing of the speech features and upsampling processing of the video features are performed in the feature alignment model, so that the processed speech features are aligned with the processed video features. The feature alignment model includes at least two feature alignment processing channels, and the speech features and the video features are respectively input into different feature alignment processing channels in the feature alignment model.

4. The method according to any one of claims 1-3, characterized in that, The step of fusing all the aligned modal features to obtain the fused features specifically includes: All modal features after feature alignment are concatenated to obtain the fused feature.

5. The method according to any one of claims 1-3, characterized in that, The step of fusing all the aligned modal features to obtain the fused features specifically includes: The fused features are obtained by performing dot product on all the modal features after feature alignment.

6. The method according to any one of claims 1-3, characterized in that, After fusing all the aligned modal features to obtain the fused features, the method further includes: An attention mechanism is used to train the fused features and determine the weight coefficients of different modal features in the fused features.

7. A device for constructing a speech activity detection model, characterized in that, The device includes: An acquisition module is used to acquire multimodal data, wherein the multimodal data includes at least audio data and video data; The extraction module is used to extract features from each modal data in the multimodal data to obtain the modal features corresponding to each modal data. The processing module is used to perform feature alignment processing on the modal features corresponding to one or more modal data when the modal features corresponding to one or more modal data do not conform to the preset alignment standard. The fusion module is used to fuse all modal features after feature alignment to obtain fused features; The training module is used to iteratively train the pre-constructed initial model using the fused features until the trained model meets preset conditions, and then the model that meets the preset conditions is determined as the final speech activity detection model; The preset alignment standard is used to indicate the number of corresponding preset features per unit time; the processing module is specifically used for: Based on the preset alignment standard and the number of features corresponding to the first modal data per unit time, the feature alignment processing method to be performed on the modal features of the first modal data is determined; The feature alignment processing is performed on the modal features corresponding to the modal features of the first modal data according to the feature alignment processing method corresponding to the modal features of the first modal data. The feature alignment processing method includes upsampling the modal features or downsampling the modal features. The first modal data is any one of one or more modal data.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the speech activity detection model construction method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by an electronic device, it implements the steps of the speech activity detection model construction method as described in any one of claims 1-6.

Images