A voice endpoint detection method, device, apparatus and storage medium

Abstract

The application provides a speech endpoint detection method, device and equipment and a storage medium. The speech endpoint detection method comprises the following steps: obtaining target data, wherein the target data comprises target audio data; inputting the target data into a pre-trained audio frame classification model to obtain a classification result of an audio frame of the target audio data, wherein the audio frame classification model is trained by taking first audio data with noise in one or more categories of a plurality of noise categories as a training sample, taking a real category of the audio frame of the first audio data in a plurality of dimensions as a sample label, and the plurality of dimensions comprise a speech dimension and noise dimensions corresponding to the plurality of noise categories; and determining a speech endpoint according to the classification result of the audio frame of the target audio data. The speech endpoint detection method provided by the application can detect an accurate speech endpoint.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a speech endpoint detection method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence, speech signal processing technology has become increasingly important, and speech endpoint detection, located at the front end of the speech processing flow, is particularly essential. Speech endpoint detection, also known as Voice Activity Detection (VAD), refers to identifying the start and end times of speech within a continuous sound signal. VAD can filter out speech segments without audio, allowing downstream tasks to focus only on those segments, thereby reducing system power consumption.

[0003] Current speech endpoint detection methods involve extracting features such as energy and zero-crossing rate from the target audio data (i.e., the audio data to be detected), and then determining whether the audio frame of the target audio data is a silent frame or a non-silent frame based on the extracted features. If the audio frame is a silent frame, it is determined to be a non-speech frame; if the audio frame is a non-silent frame, it is determined to be a speech frame. After obtaining the discrimination results of the audio frames of the target audio data, the speech endpoints are determined based on the discrimination results.

[0004] However, in practical applications, target audio data often contains more than just the speaker's voice; it usually also includes noise. In some complex acoustic scenarios, there may even be more than one type of noise. For example, audio data in acoustic scenarios such as live broadcasts and exhibition halls containing background music includes not only the speaker's voice but also non-speaker voice, music noise, and environmental noise. This means that non-silent frames are not necessarily speech frames, and identifying non-silent frames as speech frames will make it difficult to obtain accurate speech endpoint detection results. Summary of the Invention

[0005] In view of this, the present invention provides a voice endpoint detection method, apparatus, device, and storage medium to solve the problem of low detection accuracy in existing voice endpoint detection methods. The technical solution is as follows:

[0006] A voice endpoint detection method, comprising:

[0007] Acquire target data, wherein the target data includes target audio data;

[0008] The target data is input into a pre-trained audio frame classification model to obtain the classification result of the audio frames of the target audio data. The audio frame classification model is trained using first audio data with noise of one or more of several noise categories as training samples and the true categories of the audio frames of the first audio data in multiple dimensions as sample labels. The multiple dimensions include the speech dimension and the noise dimension corresponding to the several noise categories respectively.

[0009] Based on the classification results of the audio frames of the target audio data, the speech endpoints of the target speech in the target audio data are determined.

[0010] Optionally, the training process of the audio frame classification model includes:

[0011] The first audio data is encoded based on an audio frame classification model to obtain the audio features of the first audio data;

[0012] Based on the audio frame classification model, audio features under multiple dimensions are extracted from the audio features of the first audio data. The audio features under the speech dimension are the audio features of the target speech in the first audio data, and the audio features under the noise dimension corresponding to a noise category are the audio features of the noise of that noise type in the first audio data.

[0013] Based on the audio features under the multiple dimensions, the category of the audio frame of the first audio data under the multiple dimensions is predicted by the audio frame classification model to obtain the predicted category probability of the audio frame of the first audio data under the multiple dimensions.

[0014] Based on the predicted class probabilities of the audio frames in the first audio data across the multiple dimensions and the true class of the audio frames in the first audio data across the multiple dimensions, the parameters of the audio frame classification model are updated.

[0015] Optionally, the target data may further include target video data corresponding to the target audio data; the first audio data corresponds to first video data;

[0016] The training process of the audio frame classification model also includes:

[0017] The first video data is encoded based on an audio frame classification model to obtain the video features of the first video data;

[0018] The step of predicting the category of the audio frames of the first audio data in the multiple dimensions based on the audio features under the multiple dimensions and using an audio frame classification model includes:

[0019] Based on the audio features under the multiple dimensions, and supplemented by the video features of the first video data, the category of the audio frame of the first audio data under the multiple dimensions is predicted based on the audio frame classification model.

[0020] Optionally, the step of predicting the category of the audio frame in the first audio data under the multiple dimensions based on the audio features under the multiple dimensions, supplemented by the video features of the first video data, and based on the audio frame classification model, includes:

[0021] For each dimension:

[0022] Based on the audio frame classification model, the correlation between the audio features in this dimension and the video features of the first video data is calculated, which is used as the weight of the audio features in this dimension. The weight of the audio features in this dimension is then used to weight the audio features in this dimension to obtain the weighted audio features in this dimension.

[0023] Based on the weighted audio features under this dimension, the category of the audio frame of the first audio data under this dimension is predicted using an audio frame classification model.

[0024] Optionally, updating the parameters of the audio frame classification model based on the predicted probabilities of the audio frames in the first audio data across the multiple dimensions and the true categories of the audio frames in the first audio data across the multiple dimensions includes:

[0025] For each dimension, the category prediction loss of the audio frame classification model in that dimension is determined based on the category prediction probability of the audio frame in the first audio data in that dimension and the true category of the audio frame in the first audio data in that dimension.

[0026] The category prediction loss of the audio frame classification model under the multiple dimensions is fused to obtain the fused category prediction loss.

[0027] The parameters of the audio frame classification model are updated based on the fused category prediction loss.

[0028] Optionally, the audio frame classification model includes an audio encoder and a video encoder;

[0029] The audio encoder and the video encoder are the audio encoder and video encoder in the pre-trained audio and video reconstruction model;

[0030] The audio-video reconstruction model is trained using second audio data and second video data corresponding to the second audio data, wherein the second audio data is noise-free audio data;

[0031] The training objective of the audio-video reconstruction model includes: making the audio data and video data reconstructed from the audio features obtained by encoding the second audio data using an audio encoder and / or the video features obtained by encoding the second video data using a video encoder consistent with the second audio data and the second video data.

[0032] Optionally, the classification results of the audio frames of the target audio data include: the classification results of the audio frames of the target audio data in the speech dimension;

[0033] The step of determining the endpoints of the target speech in the target audio data based on the classification results of the audio frames of the target audio data includes:

[0034] Based on the classification results of the audio frames of the target audio data in the speech dimension, the speech endpoints of the target speech in the target audio data are determined.

[0035] Optionally, the classification results of the audio frames of the target audio data include: the classification results of the audio frames of the target audio data under the noise dimensions corresponding to the several noise categories respectively;

[0036] The method further includes:

[0037] Based on the classification results of the audio frames of the target audio data under the noise dimension corresponding to the specified noise type, the noise endpoints of the specified type of noise in the target audio data are determined.

[0038] A voice endpoint detection device includes: a data acquisition module, an audio frame classification module, and a voice endpoint determination module;

[0039] The data acquisition module is used to acquire target data, wherein the target data includes target audio data;

[0040] The audio frame classification module is used to input the target data into a pre-trained audio frame classification model to obtain the classification result of the audio frames of the target audio data; wherein, the audio frame classification model is trained using first audio data with noise of one or more of several noise categories as training samples, and using the true categories of the audio frames of the first audio data in multiple dimensions as sample labels, the multiple dimensions including the speech dimension and the noise dimension corresponding to the several noise categories respectively.

[0041] The speech endpoint determination module is used to determine the speech endpoints of the target speech in the target audio data based on the classification results of the audio frames of the target audio data.

[0042] A voice endpoint detection device includes: a memory and a processor;

[0043] The memory is used to store programs;

[0044] The processor is configured to execute the program to implement each step of the voice endpoint detection method described above.

[0045] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the voice endpoint detection method described in any of the preceding claims.

[0046] The speech endpoint detection method, apparatus, device, and storage medium provided by this invention first acquire target data including target audio data, then input the target data into a pre-trained audio frame classification model to obtain the classification results of the audio frames of the target audio data, and finally determine the speech endpoints of the target speech in the target audio data based on the classification results of the audio frames of the target audio data. The audio frame classification model in this invention uses first audio data with one or more types of noise as training samples, and is trained using the true categories of the audio frames of the first audio data in the speech dimension and several noise dimensions as sample labels. On the one hand, the sample labels contain the true categories of the audio frames of the first audio data in the speech dimension, which can specifically train the classification ability of the audio frame classification model in the speech dimension. This training allows the audio frame classification model to focus on speech information in the audio data, thereby accurately classifying audio frames into speech frames and non-speech frames. On the other hand, the sample labels contain the true categories in the noise dimension, which can train the audio frame classification model's classification ability in the noise dimension. By training the audio frame classification model's classification ability in the speech dimension while simultaneously training its classification ability in the noise dimension, the parameters of the audio frame classification model can be corrected more fully and comprehensively, thus making the audio frame classification model more powerful. Consequently, when classifying audio frames of target audio data based on the trained audio frame classification model, accurate classification results can be obtained. Based on the accurate audio frame classification results, accurate speech endpoints can be obtained. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the hardware architecture involved in the present invention;

[0049] Figure 2This is a flowchart illustrating the voice endpoint detection method provided in an embodiment of the present invention.

[0050] Figure 3 A schematic diagram illustrating the process of training an audio frame classification model according to an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the structure of the audio frame classification model provided in an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of the feature extraction module in the audio frame classification model provided in an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram illustrating the processing of audio and video features by the attention module in the audio frame classification model provided in this embodiment of the invention.

[0054] Figure 7 This is a schematic diagram of the structure of the classification module in the audio frame classification model provided in this embodiment of the invention;

[0055] Figure 8 This is a schematic diagram of the structure of the audio and video reconstruction model provided in an embodiment of the present invention;

[0056] Figure 9 This is a schematic diagram of the process for training the audio and video reconstruction model provided in an embodiment of the present invention;

[0057] Figure 10 This is a schematic diagram of the structure of the audio encoder and video encoder in the audio-video reconstruction model provided in the embodiments of the present invention;

[0058] Figure 11 This is a schematic diagram of the decoder structure in the audio-visual reconstruction model provided in this embodiment of the invention;

[0059] Figure 12 This is a schematic diagram of the structure of the voice endpoint detection device provided in an embodiment of the present invention;

[0060] Figure 13 This is a schematic diagram of the structure of the voice endpoint detection device provided in an embodiment of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] In the process of realizing this invention, the inventors discovered that most existing speech endpoint detection methods are feature-based detection methods. The starting point of feature-based detection methods is to find features that can characterize the difference between silent frames and non-silent frames in order to distinguish between silent frames and non-silent frames, and then determine silent frames as non-speech frames and non-silent frames as speech frames, and on this basis, determine the speech endpoint.

[0063] In addition to feature-based detection methods, there are also model-based detection methods. Model-based detection methods distinguish between silent and non-silent audio frames by modeling silent and non-silent frames, and then identify silent frames as non-speech frames and non-silent frames as speech frames, thereby determining the speech endpoints.

[0064] It is evident that both feature-based and model-based detection methods essentially involve detecting silent and non-silent frames within audio data, classifying silent frames as non-speech frames and non-silent frames as speech frames. However, in practical applications, audio data includes not only speaker-focused speech but also non-speech, musical noise, and environmental noise. This means that non-silent frames are not necessarily speech frames. Therefore, detecting silent and non-silent frames within audio data and classifying the detected non-silent frames as speech frames leads to low accuracy in the final obtained speech endpoints.

[0065] Given the low accuracy of existing speech endpoint detection methods, the inventors of this invention conducted research and, through continuous investigation, ultimately proposed a speech endpoint detection method with higher accuracy. Before introducing the speech endpoint detection method provided by this invention, the hardware architecture involved in this invention will be described first.

[0066] In one possible implementation, such as Figure 1 As shown, the hardware architecture involved in this invention may include: electronic device 101 and server 102.

[0067] For example, electronic device 101 can be any electronic product that can interact with a user, such as PC, laptop, tablet, PDA, mobile phone, learning machine, smart TV, etc.

[0068] It should be noted that, Figure 1 This is just one example; there can be many types of electronic devices, not limited to... Figure 1 The laptop in the middle.

[0069] For example, server 102 can be a single server, a server cluster consisting of multiple servers, or a cloud computing server center. Server 102 may include processors, memory, and network interfaces, etc.

[0070] For example, electronic device 101 can establish a connection and communicate with server 102 through a wireless communication network; for example, electronic device 101 can establish a connection and communicate with server 102 through a wired communication network.

[0071] Electronic device 101 can acquire target data (including at least target audio data) and send the target data to server 102. Server 102 performs voice endpoint detection on the target audio data according to the voice endpoint detection method provided by the present invention.

[0072] In another possible implementation, the hardware architecture involved in this invention may include: an electronic device.

[0073] The electronic device is an electronic product with strong data processing capabilities. The electronic device acquires target data (including at least target audio data) and performs voice endpoint detection on the target audio data according to the voice endpoint detection method provided by this invention.

[0074] Those skilled in the art should understand that the above-described electronic devices and servers are merely examples, and other existing or future electronic devices or servers that are applicable to this invention should also be included within the scope of protection of this invention, and are hereby incorporated by reference.

[0075] The speech endpoint detection method provided by the present invention will be described in the following embodiments.

[0076] Please see Figure 2 The diagram illustrates a flowchart of a voice endpoint detection method provided by an embodiment of the present invention. The method may include:

[0077] Step S201: Obtain target data.

[0078] In one possible implementation, the target data may only include the audio data to be detected, i.e., the target audio data. In another possible implementation, the target data may include the target audio data and the target video data corresponding to the target audio data. If the target data only includes the target audio data, then the speech endpoint detection method provided in this embodiment is a single-modal speech endpoint detection method. If the target data includes the target audio data and the target video data corresponding to the target audio data, then the speech endpoint detection method provided in this embodiment is a multimodal speech endpoint detection method.

[0079] Step S202: Input the target data into the pre-trained audio frame classification model to obtain the classification results of the audio frames of the target audio data.

[0080] The audio frame classification model in this embodiment is trained using first audio data containing noise of one or more of several noise categories as training samples, and using the true categories of the audio frames of the first audio data in multiple dimensions as sample labels.

[0081] Among them, multiple dimensions include the speech dimension and the noise dimension corresponding to several noise categories. The true category of an audio frame under the speech dimension is either a speech frame or a non-speech frame. The true category of an audio frame under the noise dimension corresponding to a noise category is either a noise frame of that noise category or a noise frame of a different noise category.

[0082] To improve the classification performance of the audio frame classification model, the training data of the audio frame classification model can include not only the first audio data and the true categories of the audio frames of the first audio data in multiple dimensions, but also the first video data corresponding to the first audio data. That is, the audio frame classification model is trained using the first audio data and the first video data corresponding to the first audio data as training samples, and using the true categories of the audio frames of the first audio data in multiple dimensions as sample labels.

[0083] For example, the audio frame classification model uses first audio data containing one or more types of noise, including music noise, environmental noise, and interfering speech, as well as corresponding video data, as training samples. The model is trained using the true categories of the audio frames in the first audio data under the speech dimension, the noise dimension corresponding to the interfering speech, the noise dimension corresponding to the music noise, and the noise dimension corresponding to the environmental noise as sample labels. The true category of an audio frame under the speech dimension is either a speech frame or a non-speech frame. The true category of an audio frame under the noise dimension corresponding to the music noise is either a music noise frame or a non-music noise frame. The true category of an audio frame under the noise dimension corresponding to the environmental noise is either an environmental noise frame or a non-environmental noise frame. The true category of an audio frame under the noise dimension corresponding to the interfering speech is either an interfering speech frame or a non-interfering speech frame.

[0084] It should be noted that if the training samples of the audio frame classification model do not contain the first video data corresponding to the first audio data, then the target data only contains the target audio data. That is, the target audio data is input into the audio frame classification model, and the audio classification model classifies the audio frames of the target audio data. If the training samples of the audio frame classification model contain the first video data corresponding to the first audio data, then the target data includes both the target audio data and the target video data corresponding to the target audio data. That is, the target audio data and the target video data corresponding to the target audio data are input into the audio frame classification model, and the audio classification model, supplemented by the target video data, classifies the audio frames of the target audio data to obtain the classification result of the audio frames of the target audio data.

[0085] It should be noted that in this embodiment, the video data corresponding to the audio data is collected synchronously with the audio data, and the video data corresponding to the audio data can be video data containing the face of the speaker corresponding to the audio data.

[0086] Step S203: Based on the classification results of the audio frames of the target audio data, determine the speech endpoints of the target speech in the target audio data.

[0087] It should be noted that the target speech is the speech of the speaker.

[0088] Since the audio frame classification model is trained using the true categories of the audio frames of the first audio data in the speech dimension and the noise dimension corresponding to several noise categories as sample labels, the classification results obtained by classifying the audio frames of the target audio data based on the audio frame classification model include the classification results of the audio frames of the target audio data in the speech dimension (classification results of speech frames / non-speech frames). When determining the speech endpoints of the target speech in the target audio data, the speech endpoints of the target speech in the target audio data can be determined according to the classification results of the audio frames of the target audio data in the speech dimension.

[0089] There are several ways to determine the speech endpoints of the target audio data based on the classification results of audio frames in the speech dimension. For example, if N consecutive speech frames (N can be set according to the specific scenario) are present, the target speech segment is considered to have started, and the first speech frame among the N speech frames is determined as the speech beginning point. Similarly, for the speech ending point, if M consecutive non-speech frames (M can be set according to the specific scenario) are present, the target speech segment is considered to have ended, and the frame preceding the first non-speech frame among the M non-speech frames is determined as the speech ending point. Another example is that if a certain proportion of speech frames exist within a set range, the target speech segment is considered to have started, and the first speech frame within the set range is determined as the speech beginning point. Similarly, for the speech ending point, if a certain proportion of non-speech frames exist within a set range, the target speech segment is considered to have ended, and the frame preceding the first non-speech frame within the set range is determined as the speech ending point.

[0090] The classification results of the audio frames of the target audio data include not only the classification results of the audio frames of the target audio data in the speech dimension, but also the classification results of the audio frames of the target audio data in the noise dimension corresponding to several noise types. For example, the classification results of the audio frames of the target audio data in the noise dimension corresponding to interfering speech (classification results of interfering speech frames / non-interfering speech frames), the classification results of the audio frames of the target audio data in the noise dimension corresponding to music noise (classification results of music noise frames / non-music noise frames), and the classification results of the audio frames of the target audio data in the noise dimension corresponding to ambient noise (classification results of ambient noise frames / non-ambient noise frames).

[0091] Considering that some application scenarios require not only obtaining the speech segment of the speaker of interest (i.e., the target speech segment) but also noise segments of a certain type of noise, the noise endpoints of a specified type of noise in the target audio data can be determined based on the classification results of the audio frames of the target audio data under the noise dimension corresponding to the specified noise type. For example, the noise endpoints of the music noise segment in the target audio data can be determined based on the classification results of the audio frames of the target audio data under the noise dimension corresponding to music noise (classification results of music noise frames / non-music noise frames). Similarly, the speech endpoints of the interference speech segment in the target audio data can be determined based on the classification results of the audio frames of the target audio data under the noise dimension corresponding to the interference speech (classification results of interference speech frames / non-interference speech frames). The method for determining the noise endpoints of a specified type of noise in the target audio data is similar to the method for determining the speech endpoints of the target speech in the target audio data, and will not be elaborated upon in this embodiment.

[0092] The speech endpoint detection method provided in this embodiment of the invention first acquires target data including target audio data, then inputs the target data into a pre-trained audio frame classification model to obtain the classification results of the audio frames of the target audio data, and finally determines the speech endpoints of the target speech in the target audio data based on the classification results of the audio frames of the target audio data. The audio frame classification model in this embodiment of the invention uses first audio data with one or more types of noise as training samples, and is trained using the true categories of the audio frames of the first audio data in multiple dimensions as sample labels. On the one hand, the sample labels contain the true categories of the audio frames of the first audio data in the speech dimension, which can specifically train the audio frame classification model's classification ability in the speech dimension. This allows the audio frame classification model to focus on... The audio data contains speech information, which allows for accurate classification of audio frames into speech frames and non-speech frames. On the other hand, the sample labels contain the true categories in the noise dimension, enabling the audio frame classification model to be trained in terms of noise dimension classification ability. By training the audio frame classification model in both the speech and noise dimensions, the parameters of the audio frame classification model can be corrected more fully and comprehensively, resulting in stronger audio frame classification performance. Consequently, when classifying audio frames of target audio data based on the trained audio frame classification model, accurate classification results (including accurate speech / non-speech frame classification results) can be obtained. Based on the accurate audio frame classification results, accurate speech endpoints can be obtained.

[0093] In another embodiment of the present invention, the process of obtaining training samples and sample labels for training an audio frame classification model is described.

[0094] As mentioned in the above embodiments, the training samples may include first audio data and first video data corresponding to the first audio data, or they may only include first audio data. This embodiment takes the example of training samples including first audio data and first video data corresponding to the first audio data to introduce the process of obtaining training samples.

[0095] The process of obtaining training samples may include:

[0096] Step a1: Obtain noise-free audio data and the corresponding video data.

[0097] Optionally, noise-free audio data and corresponding video data can be acquired through recording. During recording, the recording environment should be kept quiet, focusing only on the speaker's voice and avoiding various interfering information such as non-speaker voices, music noise, and environmental noise to ensure clean audio data is obtained.

[0098] Step a2: Randomly select one or more noise types from several noise types.

[0099] For example, several types of noise can include interfering speech, musical noise, and environmental noise. Environmental noise can use common Gaussian white noise and / or environmental noise from multiple scenarios, such as traffic noise on the street, equipment operating noise in a factory, etc. In scenarios such as live broadcasts, exhibition halls, and shopping malls, background music is often present. This background music, relative to the speaker's voice, is a type of noise, i.e., musical noise. Musical noise can be pure music without vocals or songs with vocals. Interfering speech refers to speech not focused on the speaker. In scenarios such as shopping malls, exhibition halls, and hospitals, in addition to the speaker's voice, there are often other speakers' voices present. These other speakers' voices, relative to the speaker's voice, are also a type of noise.

[0100] Step a3: Superimpose the selected noise onto the noise-free audio data to obtain noisy audio data, which will be used as the first audio data.

[0101] For noise-free audio data, the location for adding noise can be randomly selected. When adding the selected noise to the noise-free audio data based on the selected location, the noise can be added according to a random signal-to-noise ratio.

[0102] Step a4: Use the video data corresponding to the noise-free audio data as the first video data corresponding to the first audio data.

[0103] After obtaining the first audio data and the corresponding first video data, the first audio data can be labeled in conjunction with the corresponding first video data to obtain the true categories of the audio frames in multiple dimensions, i.e., sample labels. For the true categories of the audio frames in the speech dimension, if the speaker has both audio and speech gestures, the true category is "speech frame," represented by "1." If the speaker has neither audio nor speech gestures, the true category is "non-speech frame," represented by "0." If the speaker has lip movements but no audio, the true category is "non-speech frame," represented by "0." Similarly, for the true categories of the audio frames in the noise dimension corresponding to a specific noise type, if noise of that type exists, the true category in the noise dimension is a noise frame of that type, represented by "1." If noise of that type does not exist, the true category in the noise dimension is a noise frame of a different noise type, represented by "0."

[0104] For example, if multiple dimensions include a speech dimension and three noise dimensions (noise dimension corresponding to music noise, noise dimension corresponding to ambient noise, and noise dimension corresponding to interfering speech), then the category of an audio frame of the first audio data in the four dimensions can be represented as {speech dimension: 1; noise dimension corresponding to interfering speech: 0; noise dimension corresponding to music noise: 0; noise dimension corresponding to ambient noise: 1}.

[0105] It should be noted that for training samples that only include the first audio data, only noise-free audio data needs to be obtained. Then, one or more types of noise are randomly selected from several noise types, and the selected noise is superimposed on the noise-free audio data to obtain noisy audio data, which serves as the first audio data. Regarding the true category of the audio frame in the speech dimension of the first audio data, if the focus is on the speaker having audio, the true category in the speech dimension is "speech frame"; if the focus is on the speaker not having audio, the true category in the speech dimension is "non-speech frame". Regarding the true category of the audio frame in the noise dimension corresponding to a certain noise type of the first audio data, if noise of that type exists, the true category in the noise dimension corresponding to that noise type is a noise frame of that noise type; if noise of that type does not exist, the true category in the noise dimension corresponding to that noise type is a noise frame of a different noise type.

[0106] Multiple different training samples and sample labels for each training sample can be obtained in the manner described above. Then, these data can be used to train an audio frame classification model.

[0107] In another embodiment of the present invention, the training process of the audio frame classification model is described.

[0108] As mentioned in the above embodiments, the audio frame classification model can be trained using the first audio data as training samples and the true categories of the audio frames in the first audio data under multiple dimensions as sample labels. Alternatively, it can be trained using the first audio data and the corresponding first video data as training samples and the true categories of the audio frames in the first audio data under multiple dimensions as sample labels. This embodiment focuses on introducing the process of training the audio frame classification model using the first audio data and the corresponding first video data as training samples and the true categories of the audio frames in the first audio data under multiple dimensions as sample labels.

[0109] like Figure 3 As shown, the training process for the audio frame classification model may include:

[0110] Step S301a: Encode the first audio data based on the audio frame classification model to obtain the audio features of the first audio data.

[0111] Optional, such as Figure 4 As shown, the audio frame classification model may include an audio encoder, which can input the first audio data into the audio encoder of the audio frame classification model for encoding to obtain the audio features of the first audio data.

[0112] Step S301b: Encode the first video data based on the audio frame classification model to obtain the video features of the first video data.

[0113] Optional, such as Figure 4 As shown, the audio frame classification model may include a video encoder, which can input the first video data into the video encoder of the audio frame classification model for encoding to obtain the video features of the first video data.

[0114] Step S302: Extract audio features from the audio features of the first audio data in multiple dimensions based on the audio frame classification model, and obtain the audio features of the first audio data in multiple dimensions.

[0115] Among them, the audio features under the speech dimension are the audio features of the target speech in the first audio data, and the audio features under the noise dimension corresponding to a noise category are the audio features of the noise of that noise type in the first audio data.

[0116] like Figure 4As shown, the audio frame classification model includes feature extraction modules in multiple dimensions, namely, a feature extraction module for the speech dimension and several feature extraction modules for the noise dimension. The audio features of the first audio data are input into each feature extraction model for feature extraction. The feature extraction module for the speech dimension extracts the audio features of the target speech in the first audio data from the input audio features, and each feature extraction module for the noise dimension extracts the audio features of the corresponding noise type from the input audio features.

[0117] For example, if multiple dimensions include a speech dimension, a noise dimension corresponding to music noise, a noise dimension corresponding to environmental noise, and a noise dimension corresponding to interfering speech, then the audio frame classification model includes four feature extraction modules. The speech dimension feature extraction module extracts the audio features of the target speech from the audio features of the first audio data. The music noise dimension feature extraction module extracts the audio features of music noise from the audio features of the first audio data. The environmental noise dimension feature extraction module extracts the audio features of environmental noise from the audio features of the first audio data. The interfering speech dimension feature extraction module extracts the audio features of interfering speech from the audio features of the first audio data.

[0118] It should be noted that the feature extraction modules in the audio frame classification model have the same structure, but their parameters differ. Optional, such as... Figure 5 As shown, each feature extraction module can include a recurrent neural network (GRU) and a fully connected layer (FC). For each dimension, the audio features of the first audio data are processed sequentially by the GRU and FC in the feature extraction module of that dimension to obtain the audio features of that dimension.

[0119] Step S303: Based on the audio features of the first audio data in multiple dimensions and supplemented by the video features of the first video data, predict the category of the audio frame of the first audio data in multiple dimensions based on the audio frame classification model, and obtain the predicted category probability of the audio frame of the first audio data in multiple dimensions.

[0120] Specifically, the implementation process of step S303 may include:

[0121] Step S3031: For each dimension, calculate the correlation between the audio features in that dimension and the video features of the first video data based on the audio frame classification model, and use the correlation between the audio features in that dimension and the video features in the first video data as the weights. Then, use the weights corresponding to the audio features in that dimension to weight the audio features in that dimension to obtain the weighted audio features in that dimension.

[0122] Optional, such as Figure 4As shown, the audio frame classification model may include an attention module (e.g., a multi-head attention module). The video features of the first video data and the audio features of the first audio data in multiple dimensions are input into the attention module of the audio frame classification model. For each dimension, such as... Figure 6 As shown, the attention module uses video features as the query vector q for the attention mechanism and audio features in this dimension as key-value pairs (k and v). First, it calculates the correlation between the video features and the audio features in this dimension, and uses the calculated correlation as the weight of the audio features in this dimension. It then uses this weight to weight the audio features in this dimension, thus obtaining the weighted audio features in this dimension.

[0123] This invention uses an attention mechanism to enhance or weaken audio features in each dimension. For each dimension, when the audio features and video features are in the same semantic space, the weighted audio features in that dimension will be enhanced compared to the unweighted audio features in that dimension. Conversely, when the audio features and video features in that dimension are not in the same semantic space, the weighted audio features in that dimension will be weakened compared to the unweighted audio features in that dimension.

[0124] Step S3032: Based on the weighted audio features under this dimension, predict the category of the audio frame of the first audio data under this dimension based on the audio frame classification model, and obtain the predicted category probability of the audio frame of the first audio data under this dimension.

[0125] Optional, such as Figure 4 As shown, the audio frame classification model can include classification modules in multiple dimensions, namely, a speech-dimensional classification module (a classification module that classifies audio frames into speech frames and non-speech frames) and several noise-dimensional classification modules (for example, a classification module that classifies audio frames into interfering speech frames and non-interfering speech frames, a classification module that classifies audio frames into music noise frames and non-music noise frames, and a classification module that classifies audio frames into environmental noise frames and non-environmental noise frames).

[0126] The weighted audio features in the speech dimension are input into the speech dimension classification module of the audio frame classification model to obtain the predicted class probability of the audio frame of the first audio data in the speech dimension. For each noise dimension, the weighted audio features in that noise dimension are input into the noise dimension classification module of the audio frame classification model to obtain the predicted class probability of the audio frame of the first audio data in that noise dimension. It should be noted that the predicted class probability of an audio frame of the first audio data in the speech dimension is the probability that the audio frame is a speech frame and the probability that it is not a speech frame. The predicted class probability of an audio frame of the first audio data in the noise dimension is the probability that the audio frame is a noise frame of the corresponding noise type and the probability that it is a noise frame of a different noise type. For example, the predicted class probability of an audio frame of the first audio data in the noise dimension corresponding to music noise is the probability that the audio frame is a music noise frame and the probability that it is a non-music noise frame.

[0127] It should be noted that the classification modules in the audio frame classification model have the same structure, but their parameters differ. Optional, such as... Figure 7 As shown, each classification module can include a Long Short-Term Memory (LSTM) network, a Convolutional Neural Network (CNN), and a Fully Connected (FC) layer. For each dimension, the weighted features under that dimension are processed sequentially by the LSTM, CNN, and FC in the classification module of that dimension to obtain the predicted probability of the category under that dimension.

[0128] The above process can be used to obtain the category prediction probability of the audio frame of the first audio data in multiple dimensions.

[0129] Step S304: Update the parameters of the audio frame classification model based on the predicted probability of the audio frame in multiple dimensions and the true category of the audio frame in multiple dimensions of the first audio data.

[0130] Specifically, the implementation process of step S304 may include:

[0131] Step S3041: For each dimension, based on the predicted category probability of the audio frame in the first audio data in that dimension and the true category of the audio frame in the first audio data in that dimension, determine the category prediction loss of the audio frame classification model in that dimension.

[0132] Optionally, the category prediction loss can be the cross-entropy loss.

[0133] Step S3042: Fuse the category prediction loss of the audio frame classification model in multiple dimensions to obtain the fused category prediction loss.

[0134] There are several ways to fuse the category prediction losses of an audio frame classification model across multiple dimensions. One possible implementation is to directly sum the category prediction losses of the audio frame classification model across multiple dimensions. Another possible implementation is to sum the category prediction losses of the audio frame classification model across multiple dimensions using weighted summation.

[0135] For example, multiple dimensions include the speech dimension, the noise dimension corresponding to the interfering speech, the noise dimension corresponding to the music noise, and the noise dimension corresponding to the environmental noise. The category prediction loss of the audio frame classification model under these four dimensions can be fused using the following formula:

[0136] L total =γ1L1+γ2L2+γ3L3+γ4L4 (1)

[0137] Among them, L total γ1 represents the category prediction loss after fusion, L2 represents the category prediction loss under the speech dimension, L3 represents the category prediction loss under the noise dimension corresponding to the interfering speech, L4 represents the category prediction loss under the noise dimension corresponding to the music noise, and γ1 represents the weight corresponding to L1, γ2 represents the weight corresponding to L2, γ3 represents the weight corresponding to L3, and γ4 represents the weight corresponding to L4. γ1 to γ4 can be set according to the specific scenario.

[0138] Step S3043: Update the parameters of the audio frame classification model based on the fused category prediction loss.

[0139] The audio frame classification model is trained multiple times using different training samples following the above process until the training termination condition is met (such as model convergence, reaching the preset number of training iterations, etc.).

[0140] As mentioned in the above embodiments, the audio frame classification model may include an audio encoder and a video encoder. In order to obtain better audio frame classification results, the audio encoder and video encoder in the initial audio frame classification model may be the audio encoder and video encoder in the pre-trained audio and video reconstruction model.

[0141] like Figure 8 As shown, the audio-video reconstruction model includes an audio encoder, a video encoder, and a decoder. The training data for the audio-video reconstruction model includes second audio data and the corresponding second video data, where the second audio data is noise-free audio data, i.e., clean audio data. The training objective of the audio-video reconstruction model is to make the reconstructed audio and video data, based on the audio features obtained by encoding the second audio data using the audio encoder and / or the video features obtained by encoding the second video data using the video encoder, as consistent with the second audio data and the second video data.

[0142] Optionally, when training the audio-video reconstruction model, the input data format can be one of three types: first, single audio data, i.e., only the second audio data is input; second, single video data, i.e., only the second video data is input; and third, audio-video data pairs, i.e., the second audio data and the corresponding second video data are input. Optionally, during the entire training process of the audio-video reconstruction model, data of all three formats can be randomly input into the model for training.

[0143] The following section will introduce the training process of the audio-video reconstruction model by taking the input of the second audio data and the corresponding second video data into the audio-video reconstruction model as an example.

[0144] like Figure 9 As shown, the training process of the audio-visual reconstruction model may include:

[0145] Step S901a: Input the second audio data into the audio encoder of the audio-video reconstruction model for encoding to obtain the audio features of the second audio data.

[0146] The second audio data is input into the audio encoder of the audio-video reconstruction model. The audio encoder maps the original high-dimensional data to a low-dimensional feature space and obtains the intermediate hidden features as the audio features of the second audio data.

[0147] Please see Figure 10 An example of an audio encoder is shown. Figure 10 The audio encoder in the model consists of the following layers in sequence: a first convolutional layer (CNN), a second convolutional layer (CNN), a first pooling layer, a third convolutional layer (CNN), a fourth convolutional layer (CNN), a second pooling layer, a fifth convolutional layer (CNN), a sixth convolutional layer (CNN), a third pooling layer, a first fully connected layer (FC), a second fully connected layer (FC), and a normalization layer (L2 Norm). Each convolutional layer is followed by a normalization and ReLU activation function, and the first fully connected layer (FC) is followed by a ReLU activation function. Optionally, all pooling layers can use max pooling. The second audio data is input into the audio encoder of the audio-video reconstruction model, and the second audio data is processed through each layer in sequence to finally obtain the audio features of the second audio data.

[0148] Step S901b: Input the second video data into the video encoder of the audio-video reconstruction model for encoding to obtain the video features of the second video data.

[0149] The second video data is input into the video encoder of the audio-video reconstruction model. The video encoder maps the original high-dimensional data to a low-dimensional feature space and obtains intermediate hidden features as the video features of the second video data.

[0150] The structure of a video encoder is the same as that of an audio encoder. For example, a video encoder also includes, in sequence, a first convolutional layer (CNN), a second convolutional layer (CNN), a first pooling layer, a third convolutional layer (CNN), a fourth convolutional layer (CNN), a second pooling layer, a fifth convolutional layer (CNN), a sixth convolutional layer (CNN), a third pooling layer, a first fully connected layer (FC), a second fully connected layer (FC), and a normalization layer (L2 Norm). It should be noted that although the structure of a video encoder is the same as that of an audio encoder, the parameters of each layer are different.

[0151] Step S902: The decoder based on the audio-video reconstruction model fuses the audio features of the second audio data with the video features of the second video data, and reconstructs the audio and video data according to the fused features to obtain the reconstructed audio and video data.

[0152] Any feature fusion method can be used to fuse the audio features of the second audio data with the video features of the second video data. For example, a splicing fusion method can be used, that is, splicing the audio features of the second audio data with the video features of the second video data.

[0153] Optional, such as Figure 11 As shown, the decoder may include a feature fusion module and a multilayer perceptron (MLP). The audio features of the second audio data and the video features of the second video data are input into the feature fusion module for feature fusion. The fused features output by the feature fusion module are input into the multilayer perceptron. The multilayer perceptron outputs the reconstructed audio data and video data.

[0154] It should be noted that when the input data for the audio-video reconstruction model is only audio data, since the video encoder has no input, the video features are set to all zeros and input into the decoder along with the audio features. Similarly, when the input data for the audio-video reconstruction model is only video data, since the audio encoder has no input, the audio features are set to all zeros and input into the decoder along with the video features. Furthermore, it should be noted that when the input data for the audio-video reconstruction model is only audio data, the decoder will still reconstruct both the audio and video data; similarly, when the input data for the audio-video reconstruction model is only video data, the decoder will reconstruct both the audio and video data.

[0155] Step S903: Determine the reconstruction loss of the audio and video reconstruction model based on the reconstructed audio and video data, as well as the second audio data and the second video data.

[0156] Specifically, the audio reconstruction loss of the audio-video reconstruction model is determined based on the reconstructed audio data and the second audio data, and the video reconstruction loss of the audio-video reconstruction model is determined based on the reconstructed video data and the second video data. The audio reconstruction loss of the audio-video reconstruction model and the video reconstruction loss of the audio-video reconstruction model are fused (for example, the audio reconstruction loss and the video reconstruction loss are summed), and the fused reconstruction loss is used as the final reconstruction loss of the audio-video reconstruction model.

[0157] Optionally, the audio reconstruction loss can be the mean squared error loss, and similarly, the video reconstruction loss can also be the mean squared error loss. It should be noted that this embodiment does not limit the audio reconstruction loss and video reconstruction loss to using the mean squared error; other losses that can characterize the difference between the reconstructed data and the original data are also acceptable.

[0158] Step S904: Update the parameters of the audio-visual reconstruction model based on the determined loss.

[0159] In one possible implementation, after obtaining the reconstruction loss of the audio-video reconstruction model, the parameters of the audio-video reconstruction model can be updated directly based on the reconstruction loss of the audio-video reconstruction model.

[0160] In order to ultimately obtain better performing audio and video encoders, in another possible implementation, in addition to determining the reconstruction loss of the audio and video reconstruction model, a first distance loss and / or a second distance loss of the audio and video reconstruction model can also be determined.

[0161] The process of determining the first distance loss of the audio-video reconstruction model may include: determining the distance between the probability distribution of the audio features of the second audio data and the standard Gaussian distribution, and the distance between the probability distribution of the video features of the second video data and the standard Gaussian distribution; fusing the two distances; and using the fused distance as the first distance loss of the audio-video reconstruction model. It should be noted that the probability distribution of the audio features of the second audio data is determined by the audio encoder, and the probability distribution of the video features of the second video data is determined by the video encoder. The first distance loss is used to constrain the latent spaces of the audio encoder and the video encoder to follow a standard Gaussian distribution.

[0162] Optionally, when determining the distance between two distributions, the KL divergence between the two distributions can be calculated as the distance between them. It should be noted that this embodiment is not limited to obtaining the distance between two distributions by calculating their KL divergence; other methods for obtaining the distance between two distributions are also applicable to this invention.

[0163] The process of determining the first distance loss of the audio-video reconstruction model may include: determining the distance between the audio features of the second audio data and the video features of the second video data, which serves as the second distance loss of the audio-video reconstruction model. It should be noted that the second distance loss is used to align the audio features output by the audio encoder with the video features output by the video encoder in the latent space; in other words, it ensures that the audio features output by the audio encoder and the video features output by the video encoder are in the same latent space. When the audio features and video features are in the same latent space, they can be better integrated.

[0164] Optionally, when determining the distance between audio features and video features, the Wasserstein distance between the audio features and video features can be calculated. Of course, this embodiment is not limited to this, and other distances can also be used to determine the distance between audio features and video features.

[0165] After determining the reconstruction loss and distance loss of the audio-visual reconstruction model (assuming the first distance loss and the second distance loss are determined), the reconstruction loss and distance loss can be fused to obtain the total loss of the audio-visual reconstruction model. Then, the parameters of the audio-visual reconstruction model are updated based on the total loss of the audio-visual reconstruction model.

[0166] When fusing reconstruction loss and distance loss, the reconstruction loss and distance loss can be directly summed, or they can be weighted and summed. The weighted summation method is shown in the following formula:

[0167] L′ total =α1L 重构 +α2L 1距离 +α3L 2距离 (2)

[0168] Among them, L′ total L represents the total loss of the audio / video reconstruction model. 重构 L represents the reconstruction loss of the audio / video reconstruction model. 1距离 L represents the first distance loss of the audio / video reconstruction model. 2距离 α1 represents the second distance loss of the audio / video reconstruction model. 重构 The corresponding weights, α2 represents L 1距离 The corresponding weights, α3 represents L 2距离 The corresponding weights, α1 to α3, can be determined according to the specific scenario.

[0169] It should be noted that if the input to the audio-visual reconstruction model is only the second audio data, then only L needs to be determined. 重构 and L 1距离 At this time, L 重构The loss is still the result of fusing the audio reconstruction loss and the video reconstruction loss of the audio-video reconstruction model. At this point, L... 1距离 The distance between the probability distribution of the audio features of the second audio data and the standard Gaussian distribution.

[0170] It should be noted that if the input to the audio-visual reconstruction model is only the second video data, then only L needs to be determined. 重构 and L 1距离 At this time, L 重构 The loss is still the result of fusing the audio reconstruction loss and the video reconstruction loss of the audio-video reconstruction model. At this point, L... 1距离 The distance between the probability distribution of the video features of the second video data and the standard Gaussian distribution.

[0171] As mentioned above, when training the audio-video reconstruction model, there are three formats for the input data: single audio data, single video data, and audio-video data pairs. During the entire training process, the data in these three formats are randomly selected. However, this invention is not limited to these formats. When training the audio-video reconstruction model, two or any one of the above three formats can also be used as input data. For example, only audio-video data pairs can be input during the entire training process, or single audio data and audio-video data pairs can be randomly selected during the entire training process.

[0172] The audio and video reconstruction model is trained multiple times using different training data until the training termination condition is met (such as model convergence, reaching the preset number of training iterations, etc.).

[0173] The audio encoder and video encoder in the audio-video reconstruction model trained in the above manner can obtain audio features and video features with strong correlation.

[0174] After training, the audio encoder and video encoder in the trained audio-video reconstruction model can be used as the audio encoder and video encoder in the audio frame classification model. When training the audio frame classification model, the parameters of the audio encoder and video encoder do not need to be updated. Of course, the parameters of the audio encoder and video encoder can also be updated.

[0175] After obtaining a trained audio frame classification model, target data can be acquired and input into the audio frame classification model to obtain the classification results of the audio frames of the target audio data contained in the target data. Assuming the audio frame classification model uses first audio data with noise of one or more noise categories and corresponding first video data as training samples, and trains with the true categories of the audio frames of the first audio data in multiple dimensions as sample labels, then the target audio data and the corresponding target video data are acquired and input into the audio frame classification model. The audio frame classification model first encodes the target audio data to obtain its audio features, and then encodes the target video data to obtain its video features. Next, it extracts audio features in multiple dimensions from the audio features of the target audio data, obtaining the audio features of the target audio data in multiple dimensions. Then, based on the audio features of the target audio data in multiple dimensions, and supplemented by the video features of the target video data, it predicts the category of the audio frames of the target audio data in multiple dimensions, thus obtaining the classification results of the audio frames of the target audio data in multiple dimensions. For a more detailed process of predicting the category of audio frames of target audio data in multiple dimensions based on the audio frame classification model, please refer to the above-described specific process of predicting the category of audio frames of first audio data in multiple dimensions based on the audio frame classification model. This embodiment will not repeat the details here.

[0176] After obtaining the classification results of the audio frames of the target audio data across multiple dimensions, the endpoints of the target speech within the target audio data can be determined based on the classification results of the audio frames in the speech dimension. Once the endpoints of the target speech are obtained, the target speech segment can be acquired and applied to downstream tasks, such as speech recognition.

[0177] Optionally, the noise endpoints of a specified type of noise in the target audio data can be determined based on the classification results of the audio frames of the target audio data under the noise dimension corresponding to the specified noise type. After obtaining the noise endpoints of the specified type of noise in the target audio data, noise segments of the specified type (such as interference speech segments) can be obtained and then applied to downstream tasks.

[0178] This invention also provides a voice endpoint detection device. The voice endpoint detection device provided in this invention will be described below. The voice endpoint detection device described below can be referred to in correspondence with the voice endpoint detection method described above.

[0179] Please see Figure 12The diagram shows a schematic of the structure of the voice endpoint detection device provided in an embodiment of the present invention, which may include: a data acquisition module 1201, an audio frame classification module 1202, and a voice endpoint determination module 1203.

[0180] The data acquisition module 1201 is used to acquire target data, wherein the target data includes target audio data.

[0181] The audio frame classification module 1202 is used to input the target data into a pre-trained audio frame classification model to obtain the classification result of the audio frames of the target audio data.

[0182] The audio frame classification model is trained using first audio data containing noise of one or more of several noise categories as training samples, and is trained using the true categories of the audio frames of the first audio data in multiple dimensions as sample labels. The multiple dimensions include the speech dimension and the noise dimensions corresponding to the several noise categories respectively. The true category of an audio frame in the speech dimension is either a speech frame or a non-speech frame. The true category of an audio frame in the noise dimension corresponding to a noise category is either a noise frame of that noise category or a noise frame of a different noise category.

[0183] The speech endpoint determination module 1203 is used to determine the speech endpoints of the target speech in the target audio data based on the classification results of the audio frames of the target audio data.

[0184] The speech endpoint detection device provided in this embodiment of the invention may further include an audio frame classification model training module. The audio frame classification model training module is used for:

[0185] The first audio data is encoded based on an audio frame classification model to obtain the audio features of the first audio data;

[0186] Based on the audio frame classification model, audio features under multiple dimensions are extracted from the audio features of the first audio data. The audio features under the speech dimension are the audio features of the target speech in the first audio data, and the audio features under the noise dimension corresponding to a noise category are the audio features of the noise of that noise type in the first audio data.

[0187] Based on the audio features under the multiple dimensions, the category of the audio frame of the first audio data under the multiple dimensions is predicted by the audio frame classification model to obtain the predicted category probability of the audio frame of the first audio data under the multiple dimensions.

[0188] Based on the predicted class probabilities of the audio frames in the first audio data across the multiple dimensions and the true class of the audio frames in the first audio data across the multiple dimensions, the parameters of the audio frame classification model are updated.

[0189] Optionally, the target data may further include target video data corresponding to the target audio data; the first audio data corresponds to the first video data.

[0190] The audio frame classification model training module is also used to encode the first video data based on the audio frame classification model to obtain the video features of the first video data.

[0191] The audio frame classification model training module, based on the audio features across the multiple dimensions, predicts the category of the audio frames in the first audio data across these multiple dimensions using the audio frame classification model. Specifically, it is used for:

[0192] Based on the audio features under the multiple dimensions, and supplemented by the video features of the first video data, the category of the audio frame of the first audio data under the multiple dimensions is predicted based on the audio frame classification model.

[0193] The audio frame classification model training module, based on the audio features across the multiple dimensions and supplemented by the video features of the first video data, predicts the category of the audio frames in the first audio data across the multiple dimensions using the audio frame classification model. Specifically, it is used for:

[0194] For each dimension:

[0195] Based on the audio frame classification model, the correlation between the audio features in this dimension and the video features of the first video data is calculated, which is used as the weight of the audio features in this dimension. The weight of the audio features in this dimension is then used to weight the audio features in this dimension to obtain the weighted audio features in this dimension.

[0196] Based on the weighted audio features under this dimension, the category of the audio frame of the first audio data under this dimension is predicted using an audio frame classification model.

[0197] When the audio frame classification model training module updates the parameters of the audio frame classification model based on the predicted class probabilities of the audio frames in the first audio data across the multiple dimensions and the true class of the audio frames in the first audio data across the multiple dimensions, it is specifically used for:

[0198] For each dimension, the category prediction loss of the audio frame classification model in that dimension is determined based on the category prediction probability of the audio frame in the first audio data in that dimension and the true category of the audio frame in the first audio data in that dimension.

[0199] The category prediction loss of the audio frame classification model under the multiple dimensions is fused to obtain the fused category prediction loss.

[0200] The parameters of the audio frame classification model are updated based on the fused category prediction loss.

[0201] Optionally, the audio frame classification model includes an audio encoder and a video encoder; the audio encoder and the video encoder are the audio encoder and video encoder in a pre-trained audio-visual reconstruction model;

[0202] The audio-video reconstruction model is trained using second audio data and second video data corresponding to the second audio data, wherein the second audio data is noise-free audio data;

[0203] The training objective of the audio-video reconstruction model includes: making the audio data and video data reconstructed from the audio features obtained by encoding the second audio data using an audio encoder and / or the video features obtained by encoding the second video data using a video encoder consistent with the second audio data and the second video data.

[0204] Optionally, when training the audio-video reconstruction model, the input data of the audio-video reconstruction model is in one or more of the following three formats: single audio data, single video data, and audio-video data pairs;

[0205] The speech endpoint detection device provided in this embodiment of the invention may further include an audio / video reconstruction model training module. The audio / video reconstruction model training module is used for:

[0206] The second audio data is input into the audio encoder of the audio-video reconstruction model for encoding to obtain the audio features of the second audio data, and the second video data is input into the video encoder of the audio-video reconstruction model for encoding to obtain the video features of the second video data.

[0207] The decoder based on the audio-video reconstruction model fuses the audio features of the second audio data with the video features of the second video data, and reconstructs the audio and video data according to the fused features to obtain the reconstructed audio and video data.

[0208] Based on the reconstructed audio and video data, as well as the second audio and video data, determine the reconstruction loss of the audio and video reconstruction model;

[0209] Based on the reconstruction loss, the parameters of the audio-visual reconstruction model are updated.

[0210] Optionally, the audio / video reconstruction model training module is also used for:

[0211] Determine the distance between the probability distribution of the audio features of the second audio data and the standard Gaussian distribution, and the distance between the probability distribution of the video features of the second video data and the standard Gaussian distribution. Then fuse the two distances and use the fused distance as the first distance loss of the audio-video reconstruction model.

[0212] And / or, determine the distance between the audio features of the second audio data and the video features of the second video data, as the second distance loss of the audio-video reconstruction model;

[0213] When updating the parameters of the audio / video reconstruction model based on the reconstruction loss, the audio / video reconstruction model training module is specifically used for:

[0214] The parameters of the audio-visual reconstruction model are updated based on the first distance loss and / or the second distance loss, as well as the reconstruction loss.

[0215] The classification results of the audio frames of the target audio data include: the classification results of the audio frames of the target audio data in the speech dimension.

[0216] When determining the endpoints of the target speech in the target audio data based on the classification results of the audio frames of the target audio data, the speech endpoint determination module 1203 is specifically used to determine the speech endpoints of the target speech in the target audio data based on the classification results of the audio frames of the target audio data in the speech dimension.

[0217] The classification results of the audio frames of the target audio data include: the classification results of the audio frames of the target audio data under the noise dimensions corresponding to the several noise categories.

[0218] Optionally, the voice endpoint detection device provided in this embodiment of the invention may further include a noise endpoint determination module.

[0219] The noise endpoint determination module is used to determine the noise endpoints of a specified type of noise in the target audio data based on the classification results of the audio frames of the target audio data under the noise dimension corresponding to the specified noise type.

[0220] The speech endpoint detection device provided in this embodiment of the invention first acquires target data including target audio data, then inputs the target data into a pre-trained audio frame classification model to obtain the classification results of the audio frames of the target audio data, and finally determines the speech endpoints of the target speech in the target audio data based on the classification results of the audio frames of the target audio data. The audio frame classification model in this embodiment of the invention uses first audio data with one or more types of noise as training samples, and is trained using the true categories of the audio frames of the first audio data in multiple dimensions as sample labels. On the one hand, the sample labels contain the true categories of the audio frames of the first audio data in the speech dimension, which can specifically train the classification ability of the audio frame classification model in the speech dimension. The audio frame classification model focuses on speech information in audio data, thus accurately classifying audio frames into speech frames and non-speech frames. On the other hand, the sample labels contain the true categories in the noise dimension, enabling the audio frame classification model to be trained in terms of noise dimension classification ability. By training the audio frame classification model in both the speech and noise dimensions simultaneously, the parameters of the audio frame classification model can be corrected more fully and comprehensively, resulting in stronger audio frame classification performance. Consequently, when classifying audio frames of target audio data based on the trained audio frame classification model, accurate classification results can be obtained. Based on the accurate audio frame classification results, accurate speech endpoints can be obtained.

[0221] This invention also provides a voice endpoint detection device; please refer to [link to relevant documentation]. Figure 13 The diagram shows the structure of the voice endpoint detection device, which may include: at least one processor 1301, at least one communication interface 1302, at least one memory 1303 and at least one communication bus 1304.

[0222] In this embodiment of the invention, the number of processor 1301, communication interface 1302, memory 1303 and communication bus 1304 is at least one, and processor 1301, communication interface 1302 and memory 1303 communicate with each other through communication bus 1304.

[0223] The processor 1301 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0224] The memory 1303 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0225] The memory stores a program, which the processor can call. The program is used for:

[0226] Acquire target data, wherein the target data includes target audio data;

[0227] The target data is input into a pre-trained audio frame classification model to obtain the classification result of the audio frames of the target audio data. The audio frame classification model is trained using first audio data with noise of one or more of several noise categories as training samples and the true categories of the audio frames of the first audio data in multiple dimensions as sample labels. The multiple dimensions include the speech dimension and the noise dimension corresponding to the several noise categories respectively.

[0228] Based on the classification results of the audio frames of the target audio data, the speech endpoints of the target speech in the target audio data are determined.

[0229] Optionally, the refined and extended functions of the program can be found in the description above.

[0230] This invention also provides a computer-readable storage medium that stores a program suitable for execution by a processor, the program being used for:

[0231] Acquire target data, wherein the target data includes target audio data;

[0232] The target data is input into a pre-trained audio frame classification model to obtain the classification result of the audio frames of the target audio data. The audio frame classification model is trained using first audio data with noise of one or more of several noise categories as training samples and the true categories of the audio frames of the first audio data in multiple dimensions as sample labels. The multiple dimensions include the speech dimension and the noise dimension corresponding to the several noise categories respectively.

[0233] Based on the classification results of the audio frames of the target audio data, the speech endpoints of the target speech in the target audio data are determined.

[0234] Optionally, the refined and extended functions of the program can be found in the description above.

[0235] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0236] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0237] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting speech endpoints, characterized in that, include: Acquire target data, wherein the target data includes target audio data; The target data is input into a pre-trained audio frame classification model to obtain the classification result of the audio frames of the target audio data. The audio frame classification model is trained using first audio data with noise of one or more of several noise categories as training samples, and using the true categories of the audio frames of the first audio data in multiple dimensions as sample labels. The multiple dimensions include a speech dimension and noise dimensions corresponding to the several noise categories respectively. The true category of an audio frame in the speech dimension is either speech or non-speech. The true category of an audio frame in the noise dimension corresponding to a noise category is either that noise category or not that noise category. The audio frame classification model includes multiple feature extraction modules and multiple classification modules that correspond one-to-one with the multiple dimensions, and is configured to independently extract audio features for each dimension and perform binary classification. Based on the classification results of the audio frames of the target audio data, the speech endpoints of the target speech in the target audio data are determined.

2. The voice endpoint detection method according to claim 1, characterized in that, The training process of the audio frame classification model includes: The first audio data is encoded based on an audio frame classification model to obtain the audio features of the first audio data; Based on the audio frame classification model, audio features under multiple dimensions are extracted from the audio features of the first audio data. The audio features under the speech dimension are the audio features of the target speech in the first audio data, and the audio features under the noise dimension corresponding to a noise category are the audio features of the noise of that noise type in the first audio data. Based on the audio features under the multiple dimensions, the category of the audio frame of the first audio data under the multiple dimensions is predicted by the audio frame classification model to obtain the predicted category probability of the audio frame of the first audio data under the multiple dimensions. Based on the predicted class probabilities of the audio frames in the first audio data across the multiple dimensions and the true class of the audio frames in the first audio data across the multiple dimensions, the parameters of the audio frame classification model are updated.

3. The voice endpoint detection method according to claim 2, characterized in that, The target data also includes the target video data corresponding to the target audio data; The first audio data corresponds to the first video data; The training process of the audio frame classification model also includes: The first video data is encoded based on an audio frame classification model to obtain the video features of the first video data; The step of predicting the category of the audio frames of the first audio data in the multiple dimensions based on the audio features under the multiple dimensions and using an audio frame classification model includes: Based on the audio features under the multiple dimensions, and supplemented by the video features of the first video data, the category of the audio frame of the first audio data under the multiple dimensions is predicted based on the audio frame classification model.

4. The voice endpoint detection method according to claim 3, characterized in that, The step of predicting the category of the audio frames in the first audio data under the multiple dimensions based on the audio features of the multiple dimensions, supplemented by the video features of the first video data, and based on the audio frame classification model, includes: For each dimension: Based on the audio frame classification model, the correlation between the audio features in this dimension and the video features of the first video data is calculated, which is used as the weight of the audio features in this dimension. The weight of the audio features in this dimension is then used to weight the audio features in this dimension to obtain the weighted audio features in this dimension. Based on the weighted audio features under this dimension, the category of the audio frame of the first audio data under this dimension is predicted using an audio frame classification model.

5. The voice endpoint detection method according to claim 2, characterized in that, The step of updating the parameters of the audio frame classification model based on the predicted probabilities of the audio frames in the first audio data across multiple dimensions and the true categories of the audio frames in the first audio data across multiple dimensions includes: For each dimension, the category prediction loss of the audio frame classification model in that dimension is determined based on the category prediction probability of the audio frame in the first audio data in that dimension and the true category of the audio frame in the first audio data in that dimension. The category prediction loss of the audio frame classification model under the multiple dimensions is fused to obtain the fused category prediction loss. The parameters of the audio frame classification model are updated based on the fused category prediction loss.

6. The voice endpoint detection method according to claim 3, characterized in that, The audio frame classification model includes an audio encoder and a video encoder; The audio encoder and the video encoder are the audio encoder and video encoder in the pre-trained audio and video reconstruction model; The audio-video reconstruction model is trained using second audio data and second video data corresponding to the second audio data, wherein the second audio data is noise-free audio data; The training objective of the audio-video reconstruction model includes: making the audio data and video data reconstructed from the audio features obtained by encoding the second audio data using an audio encoder and / or the video features obtained by encoding the second video data using a video encoder consistent with the second audio data and the second video data.

7. The speech endpoint detection method according to any one of claims 1 to 6, characterized in that, The classification results of the audio frames of the target audio data include: the classification results of the audio frames of the target audio data in the speech dimension; The step of determining the endpoints of the target speech in the target audio data based on the classification results of the audio frames of the target audio data includes: Based on the classification results of the audio frames of the target audio data in the speech dimension, the speech endpoints of the target speech in the target audio data are determined.

8. The speech endpoint detection method according to any one of claims 1 to 6, characterized in that, The classification results of the audio frames of the target audio data include: the classification results of the audio frames of the target audio data under the noise dimensions corresponding to the several noise categories respectively; The method further includes: Based on the classification results of the audio frames of the target audio data under the noise dimension corresponding to the specified noise type, the noise endpoints of the specified type of noise in the target audio data are determined.

9. A voice endpoint detection device, characterized in that, include: Data acquisition module, audio frame classification module, and voice endpoint determination module; The data acquisition module is used to acquire target data, wherein the target data includes target audio data; The audio frame classification module is used to input the target data into a pre-trained audio frame classification model to obtain the classification result of the audio frames of the target audio data. The audio frame classification model is trained using first audio data containing noise of one or more noise categories as training samples, and using the true categories of the audio frames of the first audio data in multiple dimensions as sample labels. These multiple dimensions include a speech dimension and noise dimensions corresponding to the various noise categories. The true category of an audio frame in the speech dimension is either a speech frame or a non-speech frame. The true category of an audio frame in the noise dimension corresponding to a noise category is either that noise category or a non-noise category. The audio frame classification model includes multiple feature extraction modules and multiple classification modules corresponding one-to-one with the multiple dimensions, and is configured to independently extract audio features for each dimension and perform binary classification. The speech endpoint determination module is used to determine the speech endpoints of the target speech in the target audio data based on the classification results of the audio frames of the target audio data.

10. A voice endpoint detection device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the voice endpoint detection method as described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the voice endpoint detection method as described in any one of claims 1 to 8.

Images