End point detection correlation method and device

Abstract

The invention provides an endpoint detection related method and device, and the method comprises the steps: training a first model based on first training data, determining a first hidden layer feature vector outputted by a hidden layer of the first model, and enabling the first training data to comprise first voice data; a second model is trained based on second training data, a first loss function is constructed based on a first hidden layer feature vector and a second hidden layer feature vector, the second hidden layer feature vector is obtained by outputting a hidden layer of the second model, and the second training data comprises second voice data; the second voice data and the first voice data have a corresponding relation; parameters of the second model are adjusted based on the first loss function. Therefore, the accuracy of voice endpoint detection in a noise scene can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of voice endpoint detection, and in particular to a method and device related to endpoint detection. Background Art

[0002] Voice endpoint detection (VAD), also known as Voice Activity Detection (VAD), is a basic technology for voice recognition in intelligent interaction. It aims to find the starting and ending points of a voice segment from a signal containing voice, that is, to determine the start and end positions of the voice signal. At present, voice endpoint detection in practical applications will be affected by noise, resulting in reduced detection performance. Therefore, how to improve the accuracy of voice endpoint detection in noisy scenarios is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0003] The present application provides a method and device related to endpoint detection, which adjusts the hidden layer output of the second voice data during model training by adjusting the hidden layer output of the first voice data whose signal-to-noise ratio is greater than that of the second voice data during model training, thereby improving the accuracy of voice endpoint detection in noisy scenarios.

[0004] In a first aspect, the present application provides a model training method, the method comprising:

[0005] Training a first model based on first training data to determine a first hidden layer feature vector output by a hidden layer of the first model, wherein the first training data includes first speech data;

[0006] The second model is trained based on the second training data, and a first loss function is constructed based on the first hidden layer feature vector and the second hidden layer feature vector, where the second hidden layer feature vector is obtained by the hidden layer output of the second model, the second training data includes second speech data, the second speech data corresponds to the first speech data, and a signal to noise ratio (SNR) corresponding to the first speech data is higher than a signal to noise ratio corresponding to the second speech data;

[0007] Parameters of the second model are adjusted based on the first loss function.

[0008] It can be seen that in the present application, for the process of model training, when the second voice data used in training corresponds to the first voice data, and the signal-to-noise ratio corresponding to the second voice data is lower than the signal-to-noise ratio corresponding to the first voice data, the first hidden layer feature vector obtained when the model is trained with the first voice data is used to adjust the second hidden layer feature vector obtained when the model is trained with the second voice data; since the noise influence of model training with the first voice data is smaller than that of model training with the second voice data, the first hidden layer feature vector obtained is more of a feature vector related to the voice signal, and by bringing the second hidden layer feature vector closer to the first hidden layer feature vector, the output second hidden layer feature vector can include more feature vectors related to the voice signal, and thus the accuracy of the predicted endpoint obtained according to the second hidden layer feature vector will be higher. Therefore, the present application constructs a loss function based on the second hidden layer feature vector and the first hidden layer feature vector, which can improve the accuracy of the second model obtained by model training for voice endpoint detection in a noisy scene.

[0009] In a second aspect, the present application provides an endpoint detection method, the method comprising:

[0010] The third voice data is input into the second model to obtain a first output result, and the first output result includes a first endpoint; the second model is trained according to the steps described in the first aspect.

[0011] It can be seen that in the present application, since the second model is obtained by adjusting the model parameters according to the first loss function, performing speech endpoint detection through the second model can improve the accuracy of speech endpoint detection in noisy scenarios.

[0012] In a third aspect, the present application provides a model training device, the device comprising:

[0013] A training unit, used for training a first model based on first training data, a determining unit, used for determining a first hidden layer feature vector output by a hidden layer of the first model, wherein the first training data includes first speech data;

[0014] The training unit is further used to train the second model based on the second training data, and construct a first loss function based on the first hidden layer feature vector and the second hidden layer feature vector, where the second hidden layer feature vector is obtained by the hidden layer output of the second model, the second training data includes second speech data, the second speech data corresponds to the first speech data, and the signal-to-noise ratio corresponding to the first speech data is higher than the signal-to-noise ratio corresponding to the second speech data;

[0015] The training unit is further used to adjust the parameters of the second model based on the first loss function.

[0016] In a fourth aspect, the present application provides an endpoint detection device, the device comprising:

[0017] The processing unit is used to input the third voice data into the second model to obtain a first output result, the first output result includes a first endpoint, and the second model is obtained by training the device described in the third aspect.

[0018] In a fifth aspect, the present application provides an electronic device, which includes a processor, a memory, and a communication interface. The processor, the memory, and the communication interface are interconnected and perform communication with each other. The memory stores executable program code, the communication interface is used for wireless communication, and the processor is used to call the executable program code stored in the memory to execute part or all of the steps described in any method of the first aspect and / or the second aspect.

[0019] In a sixth aspect, the present application provides a computer-readable storage medium, in which electronic data is stored. When the electronic data is executed by a processor, it is used to execute the electronic data to implement some or all of the steps described in the first aspect and / or second aspect of the present application.

[0020] In a seventh aspect, the present application provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect and / or the second aspect of the present application. The computer program product may be a software installation package. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A schematic diagram of the structure of a voice endpoint detection system provided in an embodiment of the present application;

[0023] Figure 2 A flowchart of a model training method provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of the structure of a second model is provided for an embodiment of the present application;

[0025] Figure 4 A schematic diagram of a loss function calculation process provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of a flow chart of an endpoint detection method provided in an embodiment of the present application;

[0027] Figure 6 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0028] Figure 7 A block diagram of the functional units of a model training device provided in an embodiment of the present application;

[0029] Figure 8 A block diagram of the functional units of an endpoint detection device provided in an embodiment of the present application;

[0030] Fig. 9 A block diagram of the functional units of a speech processing device provided in an embodiment of the present application;

[0031] Fig.10 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps is not limited to the listed steps, but optionally includes steps that are not listed, or optionally includes other steps inherent to these processes, methods, products or devices.

[0034] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0035] See also Figure 1 , Figure 1A schematic diagram of the structure of a voice endpoint detection system provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the voice endpoint detection system 100 includes a voice processing device 101 and a voice collection device 102 .

[0036] The voice collection device 102 is used to collect voices that need to be detected by voice endpoints, and the voices can be noisy voices or clean voices. The voice collection device 102 can be a microphone, a sound card, or other devices that can realize voice collection.

[0037] The speech processing device 101 is used to implement speech endpoint detection and subsequent speech recognition for the speech collected by the speech collection device 102. The speech processing device 101 can be a processor, a server, or other forms of devices with computing capabilities. The speech collection device 102 can be integrated into the speech processing device 101 to form a device with speech collection function and computing capabilities, such as a smart phone, a tablet computer, and other terminals.

[0038] In the present application, the speech processing device 101 can also be used to train a speech endpoint detection model before performing speech endpoint detection. Specifically, it includes: training the first model based on the first speech data, determining the first hidden layer feature vector output by the hidden layer of the first model, and training the second model based on the second speech data corresponding to the first speech data. In the process of training the second model, it is necessary to adjust the second hidden layer feature vector output by the hidden layer of the second model according to the first hidden layer feature vector, and after obtaining the second model for detecting the speech endpoint of the speech data after training, the second model is stored locally, and the signal-to-noise ratio corresponding to the first speech data is higher than the signal-to-noise ratio corresponding to the second speech data.

[0039] After obtaining the trained second model, the voice collection device 102 collects the third voice data that needs to be detected by voice endpoints, and the voice processing device 101 inputs the third voice data into the second model stored locally to obtain the first endpoint corresponding to the third voice data.

[0040] It can be seen that during model training, since the second voice data corresponds to the first voice data, and the signal-to-noise ratio corresponding to the first voice data is greater than the signal-to-noise ratio corresponding to the second voice data, the second hidden layer feature vector obtained when the second voice data is trained is adjusted by using the first hidden layer feature vector obtained when the model is trained with the first voice data. This can make the features for the output endpoints obtained by the hidden layer during the latter model training closer to the features for the output endpoints obtained by the hidden layer during the former model training, thereby making the endpoint prediction results output by the latter model training also closer to the endpoint prediction results output by the former model training, thereby reducing the impact of noise scenes on voice endpoint detection and improving the recognition performance of the voice endpoint detection model in noise scenes.

[0041] Based on this, an embodiment of the present application provides an endpoint detection and model training method, and the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0042] See also Figure 2 , Figure 2 A flow chart of a model training method provided in an embodiment of the present application is provided, and the method is applied to the above-mentioned speech processing device, such as Figure 2 As shown, the method comprises the following steps:

[0043] Step S201: Train a first model based on first training data to determine a first hidden layer feature vector output by a hidden layer of the first model.

[0044] The first training data includes the first speech data and may also include a preset endpoint (label). Optionally, the first speech data may be clean speech data, which may be speech data without noise or with a signal-to-noise ratio greater than a target signal-to-noise ratio, which is greater than a signal-to-noise ratio corresponding to the second speech data.

[0045] Step S202: training the second model based on the second training data, and constructing a first loss function based on the first hidden layer feature vector and the second hidden layer feature vector.

[0046] The second hidden layer feature vector is obtained by the hidden layer output of the second model, the second training data includes second speech data, the second speech data corresponds to the first speech data, and the signal-to-noise ratio corresponding to the first speech data is greater than the signal-to-noise ratio corresponding to the second speech data. Optionally, the first speech data and the second speech data can be used to characterize clean speech data and noisy speech data, respectively, and the noisy speech data can refer to speech data with noise or a signal-to-noise ratio less than a target signal-to-noise ratio.

[0047] Step S203: adjusting parameters of the second model based on the first loss function.

[0048] The second model is used to detect the voice endpoints of the voice data. The second training data includes the second voice data, and may also include a preset endpoint corresponding to the second voice data, that is, a target value for model training. Moreover, the voice content included in the first voice data and the second voice data is the same. After preparing the first training data and the second training data, it is necessary to extract audio features from the first training data and the second training data, such as filter bank features (fbank), Mel frequency cepstral coefficients (mfcc), or features extracted by a feature extractor based on a neural network.

[0049] The model training for the second model includes a process of adjusting model parameters based on a loss function, where the loss function includes a first loss function, and the first loss function is determined based on the second hidden layer feature vector and the first hidden layer feature vector.

[0050] It can be understood that the network structure of the first model is the same as that of the second model. The model may include a downsampling layer, N transformer layers (hidden layers), a linear layer, and a normalization layer (softmax layer). Among them, the downsampling layer (also called a downsampling layer or a pooling layer) is used to reduce the dimension of the input data while retaining important information. This step can reduce the amount of calculation, avoid overfitting, and improve the generalization ability of the model. Downsampling can be achieved through a variety of methods, such as maximum pooling, average pooling, or convolution operations with a step size greater than 1. The N-layer transformer layer consists of a self-attention mechanism and positional encoding, which can capture long-term dependencies in the input sequence. In this network structure, N layers of transformer layers are used, each of which contains multiple self-attention heads and feedforward neural networks to further process and extract deep features in the input data.

[0051] The linear layer (also called a fully connected layer or dense layer) is used to transform the output of the Transformer layer into the required dimension for subsequent processing. In the present application, the linear layer may be used to prepare data for input to the softmax layer for classification, or to generate a continuous value output for signal-to-noise ratio (SNR) prediction. For classification tasks such as endpoint prediction, the softmax layer is used to convert the output of the linear layer into a probability distribution. The softmax function calculates the ratio of the index of each category to the sum of the indices of all categories to obtain the predicted probability of each category. This allows the network to output the probability of endpoint detection (such as the start / end of speech). It can be understood that the highest point in the output probability can be used as the endpoint.

[0052] For example, see Figure 3 , Figure 3A schematic diagram of the structure of a second model is provided for the embodiment of the present application, such as Figure 3 As shown in the figure, after the speech data is input into the second model, the second model will extract the audio features of the speech data. Then after passing through the downsampling layer and N layers of transformer layers (hidden layers), a hidden layer feature vector is output. After passing through the linear layer and normalization layer on one side, the predicted endpoint is obtained, and after passing through the linear layer on the other side, the predicted signal-to-noise ratio can be obtained.

[0053] In addition, the model training for the second model may be a multi-task model training, which is described in detail below:

[0054] Specifically, in a feasible embodiment, the second model is also used to detect the signal-to-noise ratio of speech data, the second training data also includes a preset signal-to-noise ratio and a preset endpoint, and the parameters of the second model are adjusted based on the first loss function, including: constructing a second loss function based on the preset signal-to-noise ratio and the predicted signal-to-noise ratio; constructing a third loss function based on the preset endpoint and the predicted endpoint, the predicted signal-to-noise ratio and the predicted endpoint are respectively output by the second model during model training; constructing a loss function based on the first loss function, the second loss function and the third loss function; and adjusting the parameters of the second model based on the loss function.

[0055] Among them, the second model can essentially be a multi-task learning (MTL) framework, which can simultaneously learn multiple related tasks, namely, the tasks corresponding to the first loss function, the second loss function, and the third loss function. Through the N-layer Transformer_layer, the second model can learn a more general feature representation, thereby improving performance on various tasks. The predicted signal-to-noise ratio can be the output of the linear layer of the model during model training, and the predicted endpoint can be the output of the softmax layer of the model during model training. The preset signal-to-noise ratio and the preset endpoint can refer to the true signal-to-noise ratio and the true endpoint, and the preset signal-to-noise ratio and the preset endpoint are used as the training targets of each task for model training.

[0056] During the model training process, a second loss function is constructed based on the preset signal-to-noise ratio and the predicted signal-to-noise ratio, and a third loss function is constructed based on the preset endpoint and the predicted endpoint. Finally, a loss function is constructed based on the first loss function, the second loss function and the third loss function, and the parameters of the second model are adjusted based on the loss function.

[0057] In the present application, by introducing the signal-to-noise ratio and the endpoint for joint training to form a multi-task learning framework, the generalization ability of the second model can be improved, which helps the model learn a more comprehensive and rich speech feature representation.

[0058] Furthermore, in a feasible embodiment, a loss function is constructed based on the first loss function, the second loss function and the third loss function, including: according to the weights corresponding to the first loss function, the second loss function and the third loss function respectively, the first loss function, the second loss function and the third loss function are weightedly summed to obtain the loss function.

[0059] Specifically, the loss function constructed by the first loss function, the second loss function and the third loss function can be:

[0060] Loss=α*Loss_VAD+β*Loss_SNR+γ*Loss_EMB.

[0061] Among them, Loss is the loss function, Loss_VAD is the third loss function, Loss_SNR is the second loss function, Loss_EMB is the first loss function, α, β and γ are the weights of the third loss function, the second loss function and the first loss function respectively. α, β and γ can be set differently according to the actual target.

[0062] At the same time, the signal-to-noise ratio corresponding to the first voice data is higher than the signal-to-noise ratio corresponding to the second voice data. When the first voice data is used as the model input, the noise impact on the model will be less than the noise impact on the model when the second voice data is used as the model input, and the accuracy of its endpoint prediction will be higher. When the model is trained with the first voice data, the first hidden layer feature vector output by the model includes more speech-related feature vectors, and the speech-related feature vectors are more effective in endpoint prediction. The first loss function is constructed by the second hidden layer feature vector and the first hidden layer feature vector, which is more effective in endpoint prediction. Similarly, the third loss function is constructed based on the preset endpoint and the predicted endpoint, so the third loss function also acts on endpoint prediction. The second loss function is constructed based on the preset signal-to-noise ratio and the predicted signal-to-noise ratio, so the second loss function mainly acts on signal-to-noise ratio prediction.

[0063] Therefore, when constructing the loss function corresponding to the second model based on the first loss function, the second loss function and the third loss function, it is necessary to set α, β and γ differently according to the actual goal. For example, the second model is mainly for endpoint detection, which focuses on the accuracy of endpoint detection, so the weight setting can focus on the third loss function, based on which, α, β and γ can be set to 0.9, 0.05 and 0.05 respectively.

[0064] In this application, the loss function corresponding to the second model is constructed by the first loss function, the second loss function and the third loss function, so that the model can receive supervision signals from different angles and levels. This multi-perspective supervision helps the model to understand the data more comprehensively and learn more accurate mapping relationships. At the same time, the composition and weight coefficient of the loss function can also be flexibly adjusted to meet more practical application needs.

[0065] In addition, the following method can be used to determine whether the weight setting for endpoint detection is appropriate:

[0066] In a feasible embodiment, the first model is obtained by training based on the first voice data, the signal-to-noise ratio corresponding to the first voice data is higher than the signal-to-noise ratio corresponding to the second voice data, and the first model is less affected by noise during the model training process than the second model (or when the first voice data is clean voice data, it will not be affected by noise). Since it is difficult to judge whether the weight setting of the loss function corresponding to the second model is appropriate through the model output results and preset results in the middle of the training, when the training progress corresponding to the first model is the same as the training progress corresponding to the second model, the output results between the first model and the second model can be used to judge whether the weight setting is appropriate.

[0067] Specifically, a first voice data and a second voice data corresponding to the first voice data are also obtained; the first voice data is input into the first model to obtain a fourth endpoint, and the second voice data is input into the second model to obtain a fifth endpoint; the target difference in accuracy between the fourth endpoint and the fifth endpoint is determined, and it is determined whether the target difference is less than a third preset threshold; if the target difference is less than the third preset threshold, it can be determined that the weight setting of the loss function of the first model is reasonable; if the target difference is not less than the third preset threshold, it can be determined that the weight setting of the loss function of the first model is unreasonable. When it is determined that the loss function setting of the first model is unreasonable, the weight value needs to be adjusted, and at this time, an adjustment prompt instruction can be sent to the user to facilitate the user to make setting changes.

[0068] Furthermore, the above three loss functions are described in detail below:

[0069] In a feasible embodiment, constructing a first loss function based on the first hidden layer feature vector and the second hidden layer feature vector includes: constructing the first loss function based on the mean square error loss between the first hidden layer feature vector and the second hidden layer feature vector.

[0070] Specifically, the first loss function and the second loss function are calculated based on the mean square error loss function, and the third loss function is calculated based on the binary cross entropy loss function. Since the hidden layer feature vector is a vector formed based on speech data, it may contain a complex structure and can correspond to a regression problem. At the same time, since the signal-to-noise ratio is a continuous value, representing the power ratio of the signal to the noise, it can also correspond to a regression problem. The loss function for the regression problem can include the mean square error loss function (MSELoss), so the loss function for the hidden layer feature vector and the signal-to-noise ratio can be constructed based on the mean square error loss function.

[0071] Based on this, the formula corresponding to the first loss function can be Loss_EMB=MSELoss(embedding Clean, embedding Noisy), where MSELoss is used to represent the mean square error loss function, embedding Clean is used to represent the first hidden layer feature vector, and embedding Noisy is used to represent the second hidden layer feature vector. The formula corresponding to the second loss function can be Loss_SNR=MSELoss(Predicted SNR, SNR Lable), where Predicted SNR is used to represent the predicted signal-to-noise ratio, and SNR Lable is used to represent the preset signal-to-noise ratio.

[0072] For speech endpoint detection, the goal is to determine whether each time step (or frame) contains speech activity, which can correspond to a binary classification problem, where speech activity is considered a positive class and non-speech activity (such as noise or silence) is considered a negative class. The loss function for the binary classification problem can include a binary cross-entropy loss function (BCELoss), so the loss function for speech endpoints can be constructed based on the binary cross-entropy loss function.

[0073] Specifically, the formula corresponding to the third loss function can be: Loss_VAD=BCELoss(Predicted VAD, VAD Lable), where BCELoss is used to represent the binary cross entropy loss function, Predicted VAD is used to represent the predicted endpoint, and VAD Lable is used to represent the preset endpoint.

[0074] In the present application, a calculation method of mean square error loss is applied to the first loss function, so that the calculation of the first loss function is more in line with the inherent characteristics of the hidden layer feature vector, thereby improving the accuracy of the loss function calculation.

[0075] For example, see Figure 4 , Figure 4 A schematic diagram of a loss function calculation process provided in an embodiment of the present application, such as Figure 4 As shown, the second speech data is input into the second model for training, and the first speech data is input into the first model for training. The second model constructs the third loss function and the second loss function based on the output predicted endpoint and predicted signal-to-noise ratio as well as the preset endpoint and the preset signal-to-noise ratio, and constructs the first loss function based on the second hidden layer feature vector output by the hidden layer of the second model and the first hidden layer feature vector output by the first model. It can be understood that the network structure of the second model is the same as that of the first model, so the first model will also output the predicted endpoint and the predicted signal-to-noise ratio.

[0076] Correspondingly, in a feasible embodiment, before training the second model based on the second training data, the method further includes: determining the second speech data in the second training data based on noise, a preset signal-to-noise ratio and the first speech data in the first training data.

[0077] The second voice data may be determined by the first voice data, the first voice data may be clean voice data, the first voice data may be obtained from a public database or recorded in a noise-free environment, and the noise may also be collected from a public database or an actual environment in different types of noise. The noise may be dynamic noise, i.e., background noise that changes over time, so that a more realistic scene can be simulated. The signal-to-noise ratio corresponding to the second voice data may be adjusted by presetting the change in the signal-to-noise ratio, so that the second voice data under different signal-to-noise ratios can be simulated, so that model training based on this may improve the generalization ability of the model.

[0078] The second voice data is generated by the first voice data, noise and preset signal-to-noise ratio, and the first voice data and noise samples are mixed, and the preset signal-to-noise ratio is simulated by adjusting the amplitude of the noise. Specifically, the power of the first voice data can be calculated first, and the power can be estimated by summing the squares of the signal samples and then dividing by the number of samples. Then, the required noise power is calculated according to the preset signal-to-noise ratio and the power of the first voice data, and then the amplitude of the noise signal is adjusted according to the calculated noise power, so that its power matches the calculated noise power. Finally, the noise signal with the adjusted amplitude is superimposed on the first voice data to generate the second voice data.

[0079] In the present application, since the second voice data is generated based on the first voice data and noise, the second voice data with different signal-to-noise ratios can be simulated by adjusting the preset signal-to-noise ratio, and model training based on this can improve the generalization ability of the model.

[0080] The following is a summary of the training process of the second model:

[0081] Data preparation: simulate the second voice data with different signal-to-noise ratios through the first voice data and noise. Feature extraction: extract the audio features of the second voice data and the first voice data. Network calculation: input the obtained audio features of the second voice data and the audio features of the first voice data into the second model and the first model respectively. The first model and the second model have the same structure, so the input first voice data and the second voice data will pass through the downsampling layer, N layers of transformer_layer, linear layer and softmax layer of the two models respectively, so that the first model and the second model will obtain three network outputs - predicted endpoints, predicted signal-to-noise ratio and hidden layer feature vectors. Calculate the loss and update the model. Construct the third loss function through the preset endpoint and the predicted endpoint output by the second model; construct the second loss function through the preset signal-to-noise ratio and the predicted signal-to-noise ratio output by the second model; construct the first loss function through the second hidden layer feature vector output by the second model and the first hidden layer feature vector output by the first model; perform weighted summation based on the first loss function, the second loss function and the third loss function to obtain the loss function corresponding to the second model, and adjust the model parameters of the second model according to the loss function to obtain the final second model.

[0082] It is understandable that after the initial training is completed to obtain the second model, the model parameters of the second model can also be adjusted according to actual conditions during use, for example, online learning or batch learning can be performed in combination with real-time feedback data.

[0083] The specific application of the second model is described in detail below:

[0084] See also Figure 5 , Figure 5 A flowchart of an endpoint detection method provided in an embodiment of the present application is provided, and the method is applied to the aforementioned voice processing device, such as Figure 5 As shown, the following steps are included:

[0085] Step S501, determining third voice data.

[0086] The third voice data may be collected and acquired by a voice collection device. After acquiring the third voice data, the voice collection device transmits the third voice data to the voice processing device. Correspondingly, the voice processing device receives the third voice data.

[0087] Step S502: input the third speech data into the second model to obtain a first output result.

[0088] The first output result includes the first endpoint. The second model is the voice processing device according to Figure 2The model is trained by the method shown in the corresponding embodiment. After obtaining the second model, the speech processing device will store it in the memory corresponding to the speech processing device. After the speech acquisition device collects the third voice data, the speech processing device will call the second model to perform endpoint detection and signal-to-noise ratio detection on the third voice data.

[0089] Endpoint detection can be a step in speech recognition. Endpoint detection is mainly used to determine the starting point and ending point of a speech signal in a segment of audio data, so that the position of the speech signal in a segment of audio data can be accurately located. Since the audio data may also include other noise data, after accurately locating the position of the speech signal, the influence of the noise in the non-speech signal part on speech recognition can be eliminated, thereby improving the accuracy of speech recognition. Therefore, the accuracy of endpoint detection will affect the accuracy of speech recognition. The higher the accuracy of endpoint detection, the higher the accuracy of speech recognition can be to a certain extent. When applying the present application to a speech recognition scenario, the present application can improve the accuracy of subsequent speech recognition by improving the accuracy of endpoint detection in a noisy scenario.

[0090] Furthermore, in a feasible embodiment, the second training data also includes a preset endpoint. After obtaining the first output result, the method also includes: training the third model based on the second training data, and adjusting the parameters of the third model based on a fourth loss function, the fourth loss function being constructed based on the preset endpoint and the predicted endpoint output by the third model; inputting the third speech data into the third model to obtain the second endpoint; determining that the accuracy of the first endpoint is greater than the accuracy of the second endpoint, the accuracy of the first endpoint being used to characterize the ratio between the number of speech frames corresponding to the first endpoint and the total number of speech frames in the third speech data.

[0091] Among them, after the second model is generated, since the second model includes the first loss function and the third loss function in the corresponding loss function during model training. The first loss function is determined based on the second hidden layer feature vector and the first hidden layer feature vector, and the purpose is to adjust the second hidden layer feature vector obtained when the second model is trained based on the second voice data by the first hidden layer feature vector obtained when the first model is trained based on the first voice data. Since the predicted endpoint output by the second model is determined based on the second hidden layer feature vector, the predicted endpoint output by the second model can be close to the predicted endpoint output by the first model. In addition, since the noise influence when the first model is trained by the first voice data is less than the noise influence when the second model is trained by the second voice data, the speech-related features (first hidden layer feature vector) output by the first model are more accurate than the speech-related features output by the second model, and thus the accuracy of the predicted endpoint output by the first model based on the feature will also be higher. In the case where the second hidden layer feature vector output by the second model is close to the first hidden layer feature vector output by the first model, the accuracy of the predicted endpoint output by the second model will also be higher.

[0092] Based on this, it can be determined that the accuracy of the first endpoint output by the second model is higher than the accuracy of the second endpoint output by the third model. The third model refers to a detection model obtained by training the model based on the same second training data, but adjusting the model parameters of the predicted endpoint output by the model only according to the preset endpoint. The accuracy of the second endpoint output by the third model is used to characterize the ratio between the number of speech frames corresponding to the second endpoint and the total number of speech frames in the third speech data.

[0093] Therefore, in actual application scenarios, after completing the model training of the second model, there may be a comparison of the output endpoint accuracy of the second model and the third model, and it is determined that based on the same voice data input, the accuracy of the first endpoint output by the second model is greater than the accuracy of the second endpoint output by the third model.

[0094] It is understandable that, since the loss function of the second model may include the first loss function, the second loss function and the third loss function, the loss function of the third model may include the second loss function in addition to the first loss function in addition to the third loss function. And since the first loss function mainly acts on the accuracy of the output endpoint, the comparison of its accuracy is only for the comparison of the accuracy of the output endpoint. And, since the first loss function plays an auxiliary role in the accuracy of the output endpoint result, when constructing the loss function of the second model, the weight corresponding to the first loss function is generally not higher than the weight corresponding to the third loss function.

[0095] At the same time, in order to make the comparative experiment more interpretable, the model network structure and the corresponding initial parameters of the second model and the third model can be the same, so that only the difference in loss functions is highlighted during model training, and then the difference in the accuracy of the output endpoint results of the model obtained after training is highlighted, which lies in the different construction of the loss function during model training.

[0096] In the above content, the accuracy of the first endpoint can be used to characterize the ratio between the number of frames of speech corresponding to the first endpoint and the total number of frames of speech in the third speech data. It can be understood that the length of each frame in the number of frames of speech corresponding to the first endpoint is the same as the length of each frame in the total number of frames of speech.

[0097] In the present application, in the process of model inference on the second model after training, the accuracy of the first endpoint output by it can be compared with the accuracy of the second endpoint output by other simple models that do not have the first loss function. When it is determined that the accuracy of the first endpoint is greater than the accuracy of the second endpoint, explanatory content with high accuracy of the model output endpoint can be obtained, thereby improving the interpretability of the model.

[0098] In addition, in a specific embodiment, the first output result also includes a first signal-to-noise ratio. After obtaining the first output result, the method also includes: performing speech enhancement and speech cropping on the third voice data based on the first signal-to-noise ratio and the first endpoint to obtain fourth voice data; inputting the fourth voice data into the second model to obtain a second signal-to-noise ratio and a third endpoint; judging whether the second condition is met; if the second signal-to-noise ratio is greater than the first preset threshold, and the first ratio is greater than the second preset threshold, performing speech recognition according to the fourth voice data to obtain a speech recognition result; if the second signal-to-noise ratio is not greater than the first preset threshold, or the first ratio is not greater than the second preset threshold, performing speech enhancement and speech cropping on the fourth voice data according to the second signal-to-noise ratio and the third endpoint to obtain fifth voice data; wherein the first ratio is the ratio between the number of frames corresponding to the third endpoint and the total number of frames in the fourth voice data.

[0099] Among them, speech enhancement and speech cropping are both pre-steps of speech recognition, and their purpose is to improve the accuracy and reliability of speech recognition. After obtaining the first signal-to-noise ratio, the signal-to-noise ratio of the third speech data can be determined. If it is determined that the first signal-to-noise ratio is low, it is necessary to use a corresponding speech enhancement algorithm or noise suppression algorithm to improve the signal-to-noise ratio of the third speech data. For example, speech enhancement can be achieved by spectral subtraction, specifically, the spectrum of the noise can be estimated, and then the estimated value of the noise spectrum is subtracted from the spectrum of the original signal to obtain the enhanced speech signal. If it is determined that the first signal-to-noise ratio is higher than the first preset threshold, the third speech data does not need to be subjected to speech enhancement processing, so as to avoid unnecessary processing causing speech distortion or other speech loss to the third speech data.

[0100] After obtaining the first endpoint, the voice signal portion in the third voice data can be determined, so that the third voice data can be cropped according to the starting point and the ending point in the first endpoint, and the audio data between the starting point and the ending point can be extracted as the cropping result.

[0101] After performing speech enhancement and speech cropping, in order to ensure the high accuracy of speech recognition of the fourth speech data after speech enhancement and speech cropping, the fourth speech data can be re-input into the second model to obtain the second signal-to-noise ratio and the third endpoint, and the first ratio is determined according to the ratio between the number of frames corresponding to the third endpoint and the total number of frames in the fourth speech data. If the second signal-to-noise ratio is greater than the first preset threshold, and the first ratio is greater than the second preset threshold, speech recognition can be performed based on the fourth speech data; if the second signal-to-noise ratio is not greater than the first preset threshold, or the first ratio is not greater than the second preset threshold, speech enhancement and speech cropping need to be performed again. And so on, until the final speech data meets the above conditions, speech recognition is performed again to obtain the corresponding speech recognition result. Speech recognition can be based on speech recognition technology to convert speech signals into text or instructions, such as hidden Markov model HMM, deep learning model, etc.

[0102] The number of frames corresponding to the voice endpoint is used to indicate the number of frames corresponding to the voice signal in the voice data, and the total number of frames refers to the number of frames of the entire voice data, that is, the ratio between the number of frames corresponding to the voice signal in the voice data and the total number of frames.

[0103] For example, see Figure 6 , Figure 6 A flow chart of a speech recognition method provided in an embodiment of the present application is shown as follows: Figure 6 As shown, it includes: inputting the speech data into the second model to obtain an output result, and the output result may include a signal-to-noise ratio and an endpoint. According to the output result, speech enhancement and speech clipping are performed, and the speech data obtained after speech enhancement and speech clipping are re-input into the second model, and then whether the second condition is met is judged according to the output result obtained at this time. If the second condition is met, speech recognition is performed to obtain a speech recognition result; if the second condition is not met, speech enhancement and speech clipping are performed again, and the obtained speech data is input into the second model again, and whether the second condition is met is judged again according to the output result of the second model, and so on, until the second condition is met, speech recognition is performed to obtain a speech recognition result.

[0104] In the present application, the third voice data input is voice enhanced and cropped based on the output result of the second model, and when the second condition is met according to another output result obtained after inputting the fourth voice data after voice enhancement and cropping into the second model, voice recognition is performed on the fourth voice data to obtain a voice recognition result, and when the second condition is not met, voice enhancement and cropping are performed again until the second condition is met. This can improve the accuracy of subsequent voice recognition.

[0105] It can be seen that in the embodiment of the present application, for the model training process, when the second voice data used in the training corresponds to the first voice data, and the signal-to-noise ratio corresponding to the second voice data is lower than the signal-to-noise ratio corresponding to the first voice data, the first hidden layer feature vector obtained when the model is trained with the first voice data is used to adjust the second hidden layer feature vector obtained when the model is trained with the second voice data; since the noise influence of the model training with the first voice data is smaller than that of the model training with the second voice data, the first hidden layer feature vector obtained is more of a feature vector related to the voice signal, and by bringing the second hidden layer feature vector closer to the first hidden layer feature vector, the output second hidden layer feature vector can include more feature vectors related to the voice signal, and thus the accuracy of the predicted endpoint obtained according to the second hidden layer feature vector will be higher. Therefore, the present application constructs a loss function based on the second hidden layer feature vector and the first hidden layer feature vector, which can improve the accuracy of the second model obtained by the model training for voice endpoint detection in a noisy scene.

[0106] In accordance with the above-mentioned embodiments, please refer to Figure 7 , Figure 7 This is a functional unit block diagram of a model training device provided in an embodiment of the present application. The model training device may be the above-mentioned speech processing device or a part of the speech processing device, such as Figure 7 As shown, the model training device 70 includes:

[0107] The training unit 701 is used to train the first model based on the first training data, and the determination unit 702 is used to determine the first hidden layer feature vector output by the hidden layer of the first model, wherein the first training data includes the first speech data;

[0108] The training unit 701 is further used to train the second model based on the second training data, and construct a first loss function based on the first hidden layer feature vector and the second hidden layer feature vector, where the second hidden layer feature vector is obtained by the hidden layer output of the second model, the second training data includes second speech data, the second speech data corresponds to the first speech data, and the signal-to-noise ratio corresponding to the first speech data is higher than the signal-to-noise ratio corresponding to the second speech data;

[0109] The training unit 701 is further used to adjust the parameters of the second model based on the first loss function.

[0110] In a feasible embodiment, the second model is also used to detect the signal-to-noise ratio of speech data, and the second training data also includes a preset signal-to-noise ratio and a preset endpoint. In terms of adjusting the parameters of the second model based on the first loss function, the training unit 701 is specifically used to: construct a second loss function based on the preset signal-to-noise ratio and the predicted signal-to-noise ratio; construct a third loss function based on the preset endpoint and the predicted endpoint, and the predicted signal-to-noise ratio and the predicted endpoint are respectively output by the second model during model training; construct a loss function based on the first loss function, the second loss function and the third loss function; and adjust the parameters of the second model based on the loss function.

[0111] In a feasible embodiment, in terms of constructing a loss function based on the first loss function, the second loss function and the third loss function, the training unit 701 is specifically used to: perform weighted summation of the first loss function, the second loss function and the third loss function according to the weights corresponding to the first loss function, the second loss function and the third loss function respectively, to obtain the loss function.

[0112] In a feasible embodiment, in terms of constructing a first loss function based on the first hidden layer feature vector and the second hidden layer feature vector, the training unit 701 is specifically used to: construct the first loss function based on the mean square error loss between the first hidden layer feature vector and the second hidden layer feature vector.

[0113] In a feasible embodiment, the determining unit 702 is further configured to determine the second speech data in the second training data based on noise, a preset signal-to-noise ratio, and the first speech data in the first training data.

[0114] It can be understood that since the embodiments corresponding to the model training method and the embodiments corresponding to the model training device are different presentation forms of the same technical concept, the content of the model training method embodiment part in this application should be synchronously adapted to the model training device embodiment part and will not be repeated here.

[0115] In accordance with the above-mentioned embodiment, please refer to Figure 8 , Figure 8 This is a functional unit block diagram of an endpoint detection device provided in an embodiment of the present application. The endpoint detection device can also be the above-mentioned voice processing device or a part of the voice processing device, such as Figure 8 As shown, the endpoint detection device 80 includes:

[0116] The processing unit 801 is used to input the third voice data into the second model to obtain a first output result, wherein the first output result includes a first endpoint, and the second model can be obtained by training through the model training device 70.

[0117] In a feasible embodiment, the first output result also includes a first signal-to-noise ratio. After obtaining the first output result, the processing unit 801 is further used to: perform speech enhancement and speech cropping on the third voice data based on the first signal-to-noise ratio and the first endpoint to obtain fourth voice data; input the fourth voice data into the second model to obtain a second signal-to-noise ratio and a third endpoint; if the second signal-to-noise ratio is greater than a first preset threshold, and the first ratio is greater than the second preset threshold, perform speech recognition based on the fourth voice data to obtain a speech recognition result; if the second signal-to-noise ratio is not greater than the first preset threshold, or the first ratio is not greater than the second preset threshold, perform speech enhancement and speech cropping on the fourth voice data based on the second signal-to-noise ratio and the third endpoint to obtain fifth voice data, where the first ratio is the ratio between the number of frames corresponding to the third endpoint and the total number of frames in the fourth voice data.

[0118] It can be understood that since the embodiments corresponding to the endpoint detection method and the embodiments corresponding to the endpoint detection device are different presentation forms of the same technical concept, the content of the endpoint detection method embodiment part in this application should be synchronously adapted to the endpoint detection device embodiment part and will not be repeated here.

[0119] In the case of integrated units, such as Fig. 9 As shown, Fig. 9 This is a block diagram of the functional units of a speech processing device provided in an embodiment of the present application. Fig. 9 In the embodiment, the speech processing device 101 includes: a processing module 912 and a communication module 911. The processing module 912 is used to control and manage the actions of the speech processing device 101, for example, the steps of the training unit 701, the determination unit 702, and the processing unit 801, and / or other processes for executing the technology described herein. The communication module 911 is used to support the interaction between the speech processing device 101 and other devices. Fig. 9 As shown, the speech processing device 101 may further include a storage module 913 , and the storage module 913 is used to store program codes and data of the speech processing device 101 .

[0120] Among them, the processing module 912 can be a processor or a controller, for example, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication module 911 can be a transceiver, an RF circuit or a communication interface, and the like. The storage module 913 can be a memory.

[0121] All relevant contents of each scenario involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here. Figure 2 and / or Figure 5 The method shown.

[0122] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When a computer instruction or computer program is loaded or executed on a computer, a process or function according to an embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center containing one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0123] Fig.10 This is a structural block diagram of an electronic device provided in an embodiment of the present application. Fig.10As shown, the electronic device 1000 may include one or more of the following components: a processor 1001, a memory 1002 and a communication interface 1003, wherein the processor 1001, the memory 1002 and the communication interface 1003 are interconnected and perform communication work with each other, wherein the memory 1002 may store one or more computer programs, and the one or more computer programs may be configured to implement the methods described in the above embodiments when executed by one or more processors 1001.

[0124] The processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect the various parts of the entire electronic device 1000, and executes various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1002, and calling data stored in the memory 1002. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU) and a modem. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and can be implemented separately through a communication chip.

[0125] The memory 1002 may include a random access memory (RAM) or a read-only memory (ROM). The memory 1002 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 1002 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created by the electronic device 1000 during use, etc.

[0126] It is understandable that the electronic device 1000 may include more or fewer structural elements than those in the above structural block diagram, for example, including a power module, physical buttons, a WiFi (Wireless Fidelity) module, a speaker, a Bluetooth module, a sensor, etc., which are not limited here.

[0127] The electronic device 1000 may be a part of the voice endpoint detection system 100 (it may also be a voice processing device or a part of a voice processing device) or a device independent of the voice endpoint detection system 100 .

[0128] An embodiment of the present application provides a computer-readable storage medium, wherein program data is stored in the computer-readable storage medium, and when the program data is executed by a processor, the program data is used to execute part or all of the steps of any endpoint detection and model training method recorded in the above method embodiments.

[0129] The present application also provides a computer program product, which includes a computer program that can be operated to enable a computer to perform some or all of the steps of any endpoint detection and model training method described in the above method embodiment. The computer program product can be a software installation package.

[0130] It should be noted that for any of the aforementioned endpoint detection and model training method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the described order of actions, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by this application.

[0131] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "one" does not exclude multiple situations. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0132] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the method embodiments of any of the above-mentioned endpoint detection and model training methods can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory may include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a disk or an optical disk, etc.

[0133] The above is a detailed introduction to the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of a method and device related to endpoint detection in the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application; at the same time, for general technical personnel in this field, based on the idea of ​​a method and device related to endpoint detection in the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0134] The present application is described with reference to the flowcharts and / or block diagrams of the methods, hardware products, and computer program products of the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0137] It can be understood that any product that is controlled or configured to execute the processing method of the flowchart described in the method embodiment of an endpoint detection and model training method of the present application, such as the terminal and computer program product in the above flowchart, falls within the scope of the related products described in this application.

[0138] Obviously, those skilled in the art can make various changes and modifications to the endpoint detection related method and device provided by the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A model training method, characterized in that: The method comprises: Training a first model based on first training data to determine a first hidden layer feature vector output by a hidden layer of the first model, wherein the first training data includes first speech data; Training the second model based on second training data, and constructing a first loss function based on the first hidden layer feature vector and the second hidden layer feature vector, wherein the second hidden layer feature vector is obtained by outputting a hidden layer of the second model, the second training data includes second speech data, the second speech data corresponds to the first speech data, and a signal-to-noise ratio corresponding to the first speech data is higher than a signal-to-noise ratio corresponding to the second speech data; Parameters of the second model are adjusted based on the first loss function.

2. The method according to claim 1, characterized in that The second training data further includes a preset signal-to-noise ratio and a preset endpoint, and adjusting the parameters of the second model based on the first loss function includes: Constructing a second loss function based on the preset signal-to-noise ratio and the predicted signal-to-noise ratio; Constructing a third loss function based on the preset endpoint and the predicted endpoint, wherein the predicted signal-to-noise ratio and the predicted endpoint are respectively outputted when the second model is trained; Construct a loss function based on the first loss function, the second loss function and the third loss function; Parameters of the second model are adjusted based on the loss function.

3. The method according to claim 2, characterized in that The constructing a loss function based on the first loss function, the second loss function and the third loss function includes: According to the weights corresponding to the first loss function, the second loss function and the third loss function respectively, the first loss function, the second loss function and the third loss function are weightedly summed to obtain the loss function.

4. The method according to claim 2 or 3, characterized in that: Constructing a first loss function based on the first hidden layer feature vector and the second hidden layer feature vector includes: The first loss function is constructed based on a mean square error loss between the first hidden layer feature vector and the second hidden layer feature vector.

5. The method according to claim 2 or 3, characterized in that: Before training the second model based on the second training data, the method further includes: The second speech data in the second training data is determined based on the noise, the preset signal-to-noise ratio, and the first speech data in the first training data.

6. An endpoint detection method, characterized in that: The method comprises: The third speech data is input into the second model to obtain a first output result, wherein the first output result includes a first endpoint; the second model is obtained by training the model training method according to any one of claims 1 to 5.

7. The method according to claim 6, characterized in that The first output result also includes a first signal-to-noise ratio. After obtaining the first output result, the method further includes: Performing voice enhancement and voice cropping on the third voice data based on the first signal-to-noise ratio and the first endpoint to obtain fourth voice data; Inputting the fourth voice data into the second model to obtain a second signal-to-noise ratio and a third endpoint; If the second signal-to-noise ratio is greater than a first preset threshold value, and the first ratio is greater than a second preset threshold value, performing speech recognition according to the third speech data to obtain a speech recognition result, wherein the first ratio is a ratio between the number of frames corresponding to the third endpoint in the third speech data and the total number of frames; If the second signal-to-noise ratio is not greater than the first preset threshold, or the first ratio is not greater than the second preset threshold, voice enhancement and voice cropping are performed on the third voice data according to the second signal-to-noise ratio and the third endpoint to obtain fifth voice data.

8. A model training device, characterized in that: The device comprises: A training unit, configured to train a first model based on first training data, and a determining unit, configured to determine a first hidden layer feature vector output by a hidden layer of the first model, wherein the first training data includes first speech data; The training unit is further used to train the second model based on second training data, and construct a first loss function based on the first hidden layer feature vector and the second hidden layer feature vector, where the second hidden layer feature vector is obtained by outputting the hidden layer of the second model, and the second training data includes second speech data, and the second speech data corresponds to the first speech data; The training unit is further used to adjust the parameters of the second model based on the first loss function.

9. An endpoint detection device, characterized in that: The device comprises: A processing unit, configured to input the third voice data into the second model to obtain a first output result, wherein the first output result includes a first endpoint; Wherein, the second model is trained by the model training device as described in claim 8.

10. An electronic device, characterized in that: The device processor, memory and computer program stored in the memory are characterized in that the processor is used to call the executable program code computer program stored in the memory to execute the method according to any one of claims 1-7.

11. A computer-readable storage medium, characterized in that: A computer program for electronic data exchange is stored, wherein the computer program enables a computer to execute the method according to any one of claims 1 to 7.

Images