Endpoint detection related methods and apparatuses

Abstract

The application provides an endpoint detection related method and device. A model training method corresponding to endpoint detection comprises: training a first model based on first training data, determining a first hidden layer feature vector of a hidden layer output of the first model, and the first training data comprising first speech data; training a second model based on second training data, and constructing a first loss function based on the first hidden layer feature vector and a second hidden layer feature vector, the second hidden layer feature vector being obtained from a hidden layer output of the second model, and the second training data comprising second speech data, the second speech data having a corresponding relationship with the first speech data; and adjusting parameters of the second model based on the first loss function. In this way, the accuracy of speech endpoint detection in a noisy environment can be improved.

Description

Technical Field

[0001] This application relates to the field of voice endpoint detection technology, and in particular to an endpoint detection-related method and apparatus. Background Technology

[0002] Endpoint detection, also known as Voice Activity Detection (VAD), is a fundamental pre-processing technology for speech recognition in intelligent interaction. It aims to identify the start and end points of a speech segment from a given signal, thus determining the beginning and end positions of the speech signal. Currently, in practical applications, voice endpoint detection performance is degraded due to noise. Therefore, improving the accuracy of voice endpoint detection in noisy environments is a pressing technical challenge in this field. Summary of the Invention

[0003] This application provides an endpoint detection method and apparatus, which adjusts the hidden layer output of the second speech data during model training by using the hidden layer output of the first speech data with a signal-to-noise ratio greater than that of the second speech data, thereby improving the accuracy of speech endpoint detection in noisy scenarios.

[0004] Firstly, this application provides a model training method, which includes:

[0005] The first model is trained based on the first training data to determine the first hidden layer feature vector output by the hidden layer of the first model. The first training data includes the first speech data.

[0006] The second model is trained based on the second training data, and a first loss function is constructed based on the first hidden layer feature vector and the second hidden layer feature vector. The second hidden layer feature vector is obtained from the hidden layer output of the second model. The second training data includes the second speech data. The second speech data corresponds to the first speech data. The signal-to-noise ratio (SNR) of the first speech data is higher than that of the second speech data.

[0007] The parameters of the second model are adjusted based on the first loss function.

[0008] As can be seen, in this application, regarding the model training process, when the second speech data used for training corresponds to the first speech data, and the signal-to-noise ratio (SNR) of the second speech data is lower than that of the first speech data, the first hidden layer feature vector obtained during model training using the first speech data is used to adjust the second hidden layer feature vector obtained during model training using the second speech data. Since training the model using the first speech data involves less noise compared to training it using the second speech data, the resulting first hidden layer feature vector is more related to the speech signal. By aligning the second hidden layer feature vector with the first hidden layer feature vector, the output second hidden layer feature vector includes more speech signal-related feature vectors, thus improving the accuracy of predicting endpoints based on the second hidden layer feature vector. Therefore, this application constructs a loss function based on the second and first hidden layer feature vectors, which can improve the accuracy of the second model trained in noisy scenarios for speech endpoint detection.

[0009] Secondly, this application provides an endpoint detection method, which includes:

[0010] The third speech data is input into the second model to obtain the first output result, which includes the first endpoint; the second model is trained according to the steps described in the first aspect.

[0011] As can be seen, in this application, since the second model is obtained by adjusting the model parameters based on the first loss function, the accuracy of speech endpoint detection in noisy scenarios can be improved by using the second model for speech endpoint detection.

[0012] Thirdly, this application provides a model training apparatus, which includes:

[0013] The training unit is used to train the first model based on the first training data, and the determination unit is used to determine the first hidden layer feature vector output by the hidden layer of the first model. The first training data includes the first speech data.

[0014] The training unit is also used to train the second model based on the second training data and to construct a first loss function based on the first hidden layer feature vector and the second hidden layer feature vector. The second hidden layer feature vector is obtained from the hidden layer output of the second model. The second training data includes the second speech data. The second speech data has a corresponding relationship with the first speech data. The signal-to-noise ratio corresponding to the first speech data is higher than that corresponding to the second speech data.

[0015] The training unit is also used to adjust the parameters of the second model based on the first loss function.

[0016] Fourthly, this application provides an endpoint detection device, which includes:

[0017] The processing unit is used to input third speech data into the second model and obtain a first output result, the first output result including a first endpoint, and the second model is trained by the device described in the third aspect.

[0018] Fifthly, this application provides an electronic device including a processor, a memory, and a communication interface. The processor, memory, and communication interface are interconnected and perform communication with each other. The memory stores executable program code, the communication interface is used for wireless communication, and the processor is used to retrieve the executable program code stored in the memory and execute some or all of the steps described in, for example, the methods of the first aspect and / or the second aspect.

[0019] In a sixth aspect, this application provides a computer-readable storage medium storing electronic data, which, when executed by a processor, is used to perform the electronic data to implement some or all of the steps described in the first and / or second aspects of this application.

[0020] In a seventh aspect, this application provides a computer program product comprising a computer program operable to cause a computer to perform some or all of the steps described in the first and / or second aspects of this application. The computer program product may be a software installation package. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the structure of a voice endpoint detection system provided in an embodiment of this application;

[0023] Figure 2 A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0024] Figure 3 This application provides a schematic diagram of the structure of a second model as an embodiment;

[0025] Figure 4 A schematic diagram illustrating a loss function calculation process provided in an embodiment of this application;

[0026] Figure 5 A flowchart illustrating an endpoint detection method provided in an embodiment of this application;

[0027] Figure 6 A flowchart illustrating a speech recognition method provided in an embodiment of this application;

[0028] Figure 7 A functional unit block diagram of a model training device provided in an embodiment of this application;

[0029] Figure 8 A functional unit block diagram of an endpoint detection device provided in an embodiment of this application;

[0030] Figure 9 A functional unit block diagram of a voice processing device provided in an embodiment of this application;

[0031] Figure 10 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0033] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps is not limited to the steps listed, but may optionally include steps not listed, or may optionally include other steps inherent to these processes, methods, products, or apparatuses.

[0034] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0035] Please see Figure 1 , Figure 1This is a schematic diagram of the structure of a voice endpoint detection system provided in an embodiment of this application, as shown below. Figure 1 As shown, the voice endpoint detection system 100 includes a voice processing device 101 and a voice acquisition device 102.

[0036] The voice acquisition device 102 is used to acquire the voice for which voice endpoint detection is required. The voice can be noisy or clean. The voice acquisition device 102 can be a microphone, sound card, or other device capable of voice acquisition.

[0037] The voice processing device 101 is used to perform functions such as voice endpoint detection and subsequent voice recognition on the voice collected by the voice acquisition device 102. The voice processing device 101 can be a processor, a server, or other form of device with computing capabilities. The voice acquisition device 102 can be integrated into the voice processing device 101 to form a device with voice acquisition function and computing capabilities, such as a smartphone, tablet computer, or other terminal.

[0038] In this application, the speech processing device 101 can also be used to train a speech endpoint detection model before performing speech endpoint detection. Specifically, this includes: training a first model based on first speech data, determining the first hidden layer feature vector output by the hidden layer of the first model, and training a second model based on second speech data corresponding to the first speech data. During the training of the second model, the second hidden layer feature vector output by the hidden layer of the second model needs to be adjusted according to the first hidden layer feature vector. After obtaining the trained second model for detecting speech endpoints of speech data, the second model is stored locally. The signal-to-noise ratio (SNR) corresponding to the first speech data is higher than that corresponding to the second speech data.

[0039] After obtaining the trained second model, the speech acquisition device 102 acquires the third speech data that needs to be detected for speech endpoints, and the speech processing device 101 inputs the third speech data into the locally stored second model to obtain the first endpoint corresponding to the third speech data.

[0040] It can be seen that during model training, since the second speech data corresponds to the first speech data, and the signal-to-noise ratio (SNR) of the first speech data is greater than that of the second speech data, the first hidden layer feature vector obtained during model training of the first speech data is used to adjust the second hidden layer feature vector obtained during model training of the second speech data. This allows the features used for output endpoints in the hidden layer of the latter model training to align with the features used for output endpoints in the hidden layer of the former model training. Consequently, the endpoint prediction results output by the latter model training also align with the endpoint prediction results output by the former model training. This reduces the impact of noisy scenes on speech endpoint detection and improves the recognition performance of the speech endpoint detection model in noisy scenes.

[0041] Based on this, the embodiments of this application provide an endpoint detection and model training method, and the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0042] Please see Figure 2 , Figure 2 This is a flowchart illustrating a model training method provided in an embodiment of this application. This method is applied to the aforementioned speech processing device, such as... Figure 2 As shown, the method includes the following steps:

[0043] Step S201: Train the first model based on the first training data to determine the first hidden layer feature vector output by the hidden layer of the first model.

[0044] The first training data includes first speech data, and may also include preset endpoints (labels). Optionally, the first speech data may be clean speech data, which may be speech data without noise or with a signal-to-noise ratio greater than a target signal-to-noise ratio, where the target signal-to-noise ratio is greater than the signal-to-noise ratio corresponding to the second speech data.

[0045] Step S202: Train the second model based on the second training data, and construct the first loss function based on the first hidden layer feature vector and the second hidden layer feature vector.

[0046] The second hidden layer feature vector is obtained from the hidden layer output of the second model. The second training data includes second speech data, which corresponds to the first speech data. The signal-to-noise ratio (SNR) of the first speech data is greater than that of the second speech data. Optionally, the first and second speech data can be used to represent clean speech data and noisy speech data, respectively. Noisy speech data can refer to speech data containing noise or with an SNR lower than the target SNR.

[0047] Step S203: Adjust the parameters of the second model based on the first loss function.

[0048] The second model is used to detect the speech endpoints of the speech data. The second training data includes the second speech data, and may also include the preset endpoints corresponding to the second speech data, i.e., the target values ​​for model training. Furthermore, the speech content included in the first and second speech data is identical. After preparing the first and second training data, it is necessary to extract audio features from them, such as filter bank features (fbank), Mel-frequency cepstral coefficients (MFCC), or features extracted by a neural network-based feature extractor.

[0049] The model training for the second model includes the process of adjusting model parameters based on a loss function, which includes a first loss function determined based on the second hidden layer feature vector and the first hidden layer feature vector.

[0050] Understandably, the first and second models share the same network structure. The model can include downsampling layers, N transformer layers (hidden layers), linear layers, and a softmax layer. The downsampling layers (also called pooling layers) reduce the dimensionality of the input data while retaining important information. This step reduces computation, avoids overfitting, and improves the model's generalization ability. Downsampling can be achieved through various methods, such as max pooling, average pooling, or convolutional operations with a stride greater than 1. The N transformer layers consist of a self-attention mechanism and positional encoding, capable of capturing long-term dependencies in the input sequence. In this network structure, N transformer layers are used, each containing multiple self-attention heads and feedforward neural networks to further process and extract deep features from the input data.

[0051] Linear layers (also known as fully connected layers or dense layers) are used to transform the output of the Transformer layers into the desired dimensions for subsequent processing. In this application, linear layers may be used to prepare data for input to a softmax layer for classification, or to generate a continuous output for signal-to-noise ratio (SNR) prediction. For classification tasks such as endpoint prediction, softmax layers are used to transform the output of linear layers into a probability distribution. The softmax function calculates the ratio of the exponent for each class to the sum of the exponents for all classes, thus obtaining the predicted probability for each class. This allows the network to output the probability of endpoint detection (such as the start / end of speech). Understandably, the point with the highest output probability can be taken as the endpoint.

[0052] For example, please refer to Figure 3 , Figure 3This application provides a schematic diagram of the structure of a second model, as shown in the embodiment. Figure 3 As shown, after inputting speech data into the second model, the second model extracts the audio features of the speech data. Then, after passing through a downsampling layer and N transformer layers (hidden layers), a hidden feature vector is output. The predicted endpoint is obtained after passing through a linear layer and a normalization layer on one side, and the predicted signal-to-noise ratio is obtained after passing through a linear layer on the other side.

[0053] Furthermore, the training of the second model can be multi-task model training, which will be explained in detail below:

[0054] Specifically, in one feasible embodiment, the second model is further used to detect the signal-to-noise ratio (SNR) of the speech data. The second training data also includes a preset SNR and preset endpoints. The parameters of the second model are adjusted based on a first loss function, including: constructing a second loss function based on the preset SNR and the predicted SNR; constructing a third loss function based on the preset endpoints and the predicted endpoints, where the predicted SNR and the predicted endpoints are outputs of the second model during model training; constructing a loss function based on the first loss function, the second loss function, and the third loss function; and adjusting the parameters of the second model based on the loss function.

[0055] The second model is essentially a multi-task learning (MTL) framework that can simultaneously learn multiple related tasks, namely the tasks corresponding to the first, second, and third loss functions. Through N Transformer layers, the second model can learn more general feature representations, thereby improving performance across various tasks. The predicted signal-to-noise ratio (SNR) can be the output of the linear layers during model training, and the predicted endpoints can be the output of the softmax layers during model training. The preset SNR and preset endpoints can refer to the true SNR and true endpoints, respectively, and are used as training targets for each task during model training.

[0056] During model training, a second loss function is constructed based on the preset signal-to-noise ratio and the predicted signal-to-noise ratio, and a third loss function is constructed based on the preset endpoints and the predicted endpoints. Finally, a loss function is constructed based on the first loss function, the second loss function, and the third loss function, and the parameters of the second model are adjusted based on the loss function.

[0057] In this application, by introducing signal-to-noise ratio and endpoints together for training, a multi-task learning framework is formed, which can improve the generalization ability of the second model and help the model learn more comprehensive and rich speech feature representations.

[0058] Furthermore, in a feasible embodiment, a loss function is constructed based on the first loss function, the second loss function, and the third loss function, including: weighted summation of the first loss function, the second loss function, and the third loss function according to their respective weights, to obtain the loss function.

[0059] Specifically, the loss function constructed from the first loss function, the second loss function, and the third loss function can be:

[0060] Loss=α*Loss_VAD+β*Loss_SNR+γ*Loss_EMB.

[0061] In this context, Loss is the loss function, Loss_VAD is the third loss function, Loss_SNR is the second loss function, Loss_EMB is the first loss function, and α, β, and γ are the weights of the third, second, and first loss functions, respectively. α, β, and γ can be set differently depending on the actual target.

[0062] Meanwhile, the signal-to-noise ratio (SNR) of the first speech data is higher than that of the second speech data. Therefore, the noise impact on the model is less when using the first speech data as input than when using the second speech data, resulting in higher accuracy in endpoint prediction. When training the model using the first speech data, the output first hidden layer feature vector includes more speech-related feature vectors, which primarily contribute to endpoint prediction. The first loss function, constructed using the second and first hidden layer feature vectors, primarily focuses on endpoint prediction. Similarly, the third loss function, built based on preset and predicted endpoints, also primarily affects endpoint prediction. The second loss function, based on the preset and predicted SNR, mainly focuses on SNR prediction.

[0063] Therefore, when constructing the loss function for the second model based on the first, second, and third loss functions, α, β, and γ need to be set differently according to the actual objective. For example, if the second model mainly performs endpoint detection and focuses on the accuracy of endpoint detection, then the weights can be set with emphasis on the third loss function. Based on this, α, β, and γ can be set to 0.9, 0.05, and 0.05, respectively.

[0064] In this application, the loss function corresponding to the second model is constructed by jointly using a first loss function, a second loss function, and a third loss function, allowing the model to receive supervision signals from different angles and levels. This multi-perspective supervision helps the model to understand the data more comprehensively and learn more accurate mapping relationships. Furthermore, the composition of the loss function and the weight coefficients can be flexibly adjusted to meet more practical application needs.

[0065] In addition, the appropriateness of the weight settings for endpoint detection can be determined using the following method:

[0066] In one feasible embodiment, the first model is trained based on first speech data, where the signal-to-noise ratio (SNR) of the first speech data is higher than that of the second speech data. The first model is less affected by noise during training compared to the second model (or, if the first speech data is clean, it is unaffected by noise). Since the appropriateness of the weight settings for the loss function of the second model is difficult to determine by comparing the model output with the preset results during training, but if the training progress of the first and second models is the same, the appropriateness of the weight settings can be determined by comparing the output results of the first and second models.

[0067] Specifically, a first voice data point and a corresponding second voice data point are acquired. The first voice data is input into a first model to obtain a fourth endpoint, and the second voice data is input into a second model to obtain a fifth endpoint. A target difference in accuracy between the fourth and fifth endpoints is determined, and it is judged whether the target difference is less than a third preset threshold. If the target difference is less than the third preset threshold, it can be determined that the weight settings of the loss function of the first model are reasonable. If the target difference is not less than the third preset threshold, it can be determined that the weight settings of the loss function of the first model are unreasonable. When it is determined that the loss function settings of the first model are unreasonable, the weight values ​​need to be adjusted. At this time, an adjustment prompt instruction can be sent to the user to facilitate the user's setting changes.

[0068] Furthermore, the three loss functions mentioned above will be explained in detail below:

[0069] In one feasible embodiment, constructing a first loss function based on the first hidden layer feature vector and the second hidden layer feature vector includes: constructing the first loss function based on the mean squared error loss between the first hidden layer feature vector and the second hidden layer feature vector.

[0070] Specifically, the first and second loss functions are calculated based on the mean squared error loss function, and the third loss function is calculated based on the binary cross-entropy loss function. Since the hidden layer feature vector is a vector formed from speech data and may contain complex structures, it can correspond to regression problems. Simultaneously, since the signal-to-noise ratio (SNR) is a continuous value representing the power ratio of signal to noise, it can also correspond to regression problems. The loss function for regression problems can include the mean squared error loss function (MSELoss). Therefore, the loss functions for the hidden layer feature vector and the SNR can be constructed based on the mean squared error loss function.

[0071] Based on this, the formula for the first loss function can be Loss_EMB = MSELoss(embedding Clean, embedding Noisy), where MSELoss represents the mean squared error loss function, embedding Clean represents the first hidden layer feature vector, and embedding Noisy represents the second hidden layer feature vector. The formula for the second loss function can be Loss_SNR = MSELoss(Predicted SNR, SNR Label), where Predicted SNR represents the predicted signal-to-noise ratio, and SNR Label represents the preset signal-to-noise ratio.

[0072] For speech endpoint detection, the goal is to determine whether each time step (or frame) contains speech activity, which corresponds to a binary classification problem. Speech activity is considered positive, and non-speech activity (such as noise or silence) is considered negative. The loss function for binary classification can include the binary cross-entropy loss function (BCELoss). Therefore, the loss function for speech endpoints can be constructed based on the binary cross-entropy loss function.

[0073] Specifically, the formula for the third loss function can be: Loss_VAD = BCELoss(Predicted VAD, VAD Label), where BCELoss represents the binary cross-entropy loss function, Predicted VAD represents the predicted endpoint, and VAD Label represents the preset endpoint.

[0074] In this application, a mean squared error loss calculation method is applied to the first loss function, which makes the calculation of the first loss function more closely match the characteristics of the hidden layer feature vector, thereby improving the accuracy of the loss function calculation.

[0075] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating a loss function calculation process provided in an embodiment of this application, such as... Figure 4 As shown, the second speech data is input into the second model for training, and the first speech data is input into the first model for training. The second model constructs a third loss function and a second loss function based on the output predicted endpoints and predicted signal-to-noise ratio (SNR), as well as preset endpoints and preset SNR. It also constructs a first loss function based on the second hidden layer feature vector output by the second model and the first hidden layer feature vector output by the first model. It is understandable that the second and first models have the same network structure; therefore, the first model will also output predicted endpoints and predicted SNR.

[0076] Correspondingly, in a feasible embodiment, before training the second model based on the second training data, the method further includes: determining the second speech data in the second training data based on noise, a preset signal-to-noise ratio, and the first speech data in the first training data.

[0077] The second speech data can be determined from the first speech data. The first speech data can be clean speech data, which can be obtained from a public database or recorded in a noise-free environment. Similarly, the noise can be collected from a public database or from a real-world environment, and can be dynamic noise, i.e., background noise that changes over time, thus simulating a more realistic scenario. The signal-to-noise ratio (SNR) of the second speech data can be adjusted by preset variations, simulating second speech data under different SNRs. This allows for model training based on these variations, improving the model's generalization ability.

[0078] Generating second speech data from first speech data, noise, and a preset signal-to-noise ratio (SNR) can be achieved by mixing the first speech data with noise samples and adjusting the noise amplitude to simulate the preset SNR. Specifically, the power of the first speech data can be calculated first, which can be estimated by summing the squares of the signal samples and dividing by the number of samples. Then, based on the preset SNR and the power of the first speech data, the required noise power is calculated. The amplitude of the noise signal is then adjusted to match the calculated noise power. Finally, the adjusted noise signal is superimposed on the first speech data to generate the second speech data.

[0079] In this application, since the second speech data is generated based on the first speech data and noise, the second speech data with different signal-to-noise ratios can be simulated by adjusting the preset signal-to-noise ratio, and the generalization ability of the model can be improved by training the model accordingly.

[0080] The training process of the second model is summarized below:

[0081] Data preparation involves simulating second speech data with different signal-to-noise ratios using first speech data and noise. Feature extraction involves extracting audio features from the second and first speech data. Network computation involves inputting the obtained audio features from the second and first speech data into the second and first models, respectively. Since the first and second models have the same structure, the input first and second speech data will pass through downsampling layers, N transformer layers, linear layers, and softmax layers in each model, resulting in three network outputs for both models: predicted endpoints, predicted signal-to-noise ratio, and hidden layer feature vectors. Loss calculation and model update involve constructing a third loss function using preset endpoints and the predicted endpoints output by the second model; constructing a second loss function using preset signal-to-noise ratios and the predicted signal-to-noise ratios output by the second model; constructing a first loss function using the second hidden layer feature vectors output by the second model and the first hidden layer feature vectors output by the first model; and finally, a weighted sum of the first, second, and third loss functions to obtain the corresponding loss function for the second model. The model parameters of the second model are then adjusted based on this loss function to obtain the final second model.

[0082] Understandably, after the initial training to obtain the second model, the model parameters can also be adjusted according to the actual situation during use. For example, it can be used for online learning or batch learning by incorporating real-time feedback data.

[0083] The specific application of the second model will be explained in detail below:

[0084] Please see Figure 5 , Figure 5 This is a flowchart illustrating an endpoint detection method provided in an embodiment of this application. The method is applied to the aforementioned voice processing device, such as... Figure 5 As shown, it includes the following steps:

[0085] Step S501: Determine the third voice data.

[0086] The third voice data can be acquired by a voice acquisition device. After acquiring the third voice data, the voice acquisition device transmits the third voice data to a voice processing device. Correspondingly, the voice processing device receives the third voice data.

[0087] Step S502: Input the third speech data into the second model to obtain the first output result.

[0088] The first output includes the first endpoint. The second model is based on the speech processing device... Figure 2The method shown in the corresponding embodiment is used to train the model. After obtaining the second model, the voice processing device will store it in the corresponding memory of the voice processing device. After the voice acquisition device collects the third voice data, the voice processing device will call the second model to perform endpoint detection and signal-to-noise ratio detection on the third voice data.

[0089] Endpoint detection can be a step in speech recognition. It primarily determines the start and end points of the speech signal within an audio data segment, thus accurately locating the speech signal's position. Since audio data may also include other noise, accurately locating the speech signal eliminates the influence of noise from non-speech components on speech recognition, thereby improving accuracy. Therefore, the accuracy of endpoint detection affects the accuracy of speech recognition; higher accuracy generally leads to higher accuracy in speech recognition. When applying this application to speech recognition scenarios, it improves the accuracy of endpoint detection in noisy environments, thereby enhancing the accuracy of subsequent speech recognition.

[0090] Furthermore, in a feasible embodiment, the second training data further includes a preset endpoint. After obtaining the first output result, the method further includes: training the third model based on the second training data, and adjusting the parameters of the third model based on a fourth loss function, wherein the fourth loss function is constructed based on the preset endpoint and the predicted endpoint output by the third model; inputting the third speech data into the third model to obtain the second endpoint; determining that the accuracy of the first endpoint is greater than the accuracy of the second endpoint, wherein the accuracy of the first endpoint is used to characterize the ratio between the number of speech frames corresponding to the first endpoint and the total number of speech frames in the third speech data.

[0091] After generating the second model, its training loss function includes both a first loss function and a third loss function. The first loss function is determined based on the second hidden layer feature vector and the first hidden layer feature vector. Its purpose is to adjust the second hidden layer feature vector obtained from training the first model using the first hidden layer feature vector obtained from training the first model using the first speech data. Since the predicted endpoint of the second model is determined based on the second hidden layer feature vector, it allows the predicted endpoint of the second model to converge with the predicted endpoint of the first model. Furthermore, because the noise impact during training the first model using the first speech data is less than the noise impact during training the second model using the second speech data, the speech-related features (first hidden layer feature vector) output by the first model are more accurate than those output by the second model. Consequently, the accuracy of the predicted endpoint output by the first model based on these features is also higher. And when the second hidden layer feature vector output by the second model converges with the first hidden layer feature vector output by the first model, the accuracy of the predicted endpoint output by the second model is also higher.

[0092] Based on this, it can be determined that the accuracy of the first endpoint output by the second model is higher than the accuracy of the second endpoint output by the third model. The third model refers to the detection model obtained by training the model on the same second training data, but adjusting the model parameters only for the predicted endpoints output by the model based on preset endpoints. The accuracy of the second endpoint output by the third model is used to characterize the ratio between the number of frames of speech corresponding to the second endpoint and the total number of frames of speech in the third speech data.

[0093] Therefore, in practical application scenarios, after the training of the second model is completed, there may be a comparison of the accuracy of the output endpoints of the second and third models, and it may be determined that, based on the same voice data input, the accuracy of the first endpoint output by the second model is greater than the accuracy of the second endpoint output by the third model.

[0094] Understandably, since the loss function of the second model can include the first, second, and third loss functions, the loss function of the third model can also include the second loss function (excluding the first loss function) in addition to the third loss function. Furthermore, since the first loss function primarily affects the accuracy of the output endpoint, the accuracy comparison only considers the accuracy of the output endpoint. Moreover, because the first loss function plays a supporting role in the accuracy of the output endpoint results, when constructing the loss function for the second model, the weight corresponding to the first loss function is generally not higher than the weight corresponding to the third loss function.

[0095] Meanwhile, to enhance the interpretability of the comparative experiments, the network structure and corresponding initial parameters of the second and third models can be the same. Thus, during model training, only the difference in the loss function is highlighted, thereby emphasizing that the difference in the accuracy of the output endpoint results of the trained models lies in the different construction of the loss function during model training.

[0096] As mentioned above, the accuracy of the first endpoint can be used to characterize the ratio between the number of frames of speech corresponding to the first endpoint and the total number of frames of speech in the third speech data. It is understood that the length of each frame in the speech corresponding to the first endpoint is the same as the length of each frame in the total number of frames of speech.

[0097] In this application, during the model inference process of the second model after training, the accuracy of its output first endpoint can be compared with the accuracy of the output second endpoint of other simple models that do not have a first loss function. When it is determined that the accuracy of the first endpoint is greater than the accuracy of the second endpoint, the interpretable content with high model output endpoint accuracy can be obtained, thereby improving the interpretability of the model.

[0098] Furthermore, in a specific embodiment, the first output result also includes a first signal-to-noise ratio (SNR). After obtaining the first output result, the method further includes: performing speech enhancement and speech trimming on the third speech data based on the first SNR and the first endpoint to obtain fourth speech data; inputting the fourth speech data into a second model to obtain a second SNR and a third endpoint; determining whether a second condition is met; if the second SNR is greater than a first preset threshold and the first ratio is greater than a second preset threshold, then performing speech recognition based on the fourth speech data to obtain a speech recognition result; if the second SNR is not greater than the first preset threshold, or the first ratio is not greater than the second preset threshold, then performing speech enhancement and speech trimming on the fourth speech data based on the second SNR and the third endpoint to obtain fifth speech data; wherein, the first ratio is the ratio between the number of frames corresponding to the third endpoint in the fourth speech data and the total number of frames.

[0099] Speech enhancement and speech trimming are both preliminary steps in speech recognition, aiming to improve the accuracy and reliability of speech recognition. After obtaining the first signal-to-noise ratio (SNR), the SNR of the third speech data can be determined. If the first SNR is low, the corresponding speech enhancement or noise suppression algorithm needs to be used to improve the SNR of the third speech data. For example, speech enhancement can be achieved through spectral subtraction, specifically by estimating the spectrum of the noise and then subtracting the estimated noise spectrum from the spectrum of the original signal to obtain the enhanced speech signal. If the first SNR is higher than a first preset threshold, speech enhancement processing of the third speech data can be omitted to avoid unnecessary processing that could cause speech distortion or other speech loss.

[0100] After obtaining the first endpoint, the speech signal portion in the third speech data can be determined. Then, based on the start and end points in the first endpoint, the third speech data can be trimmed to extract the audio data between the start and end points as the trimming result.

[0101] After performing speech enhancement and trimming, to ensure high accuracy in speech recognition using the fourth speech data obtained through these processes, the fourth speech data can be re-input into the second model to obtain a second signal-to-noise ratio (SNR) and a third endpoint. A first ratio is determined based on the ratio between the number of frames corresponding to the third endpoint in the fourth speech data and the total number of frames. If the second SNR is greater than a first preset threshold and the first ratio is greater than a second preset threshold, speech recognition can be performed based on the fourth speech data. If the second SNR is not greater than the first preset threshold, or the first ratio is not greater than the second preset threshold, speech enhancement and trimming need to be repeated. This process continues until the final speech data meets the above conditions before speech recognition is performed to obtain the corresponding speech recognition result. Speech recognition can be based on speech recognition technology that converts speech signals into text or commands, such as Hidden Markov Models (HMMs) and deep learning models.

[0102] Among them, the number of frames corresponding to the speech endpoint is used to represent the number of frames corresponding to the speech signal in the speech data, and the total number of frames refers to the total number of frames in the entire speech data, that is, the ratio between the number of frames corresponding to the speech signal in the speech data and the total number of frames.

[0103] For example, please refer to Figure 6 , Figure 6 This is a flowchart illustrating a speech recognition method provided in an embodiment of this application, as shown below. Figure 6 As shown, the process includes: inputting speech data into a second model to obtain an output result, which may include signal-to-noise ratio and endpoints; performing speech enhancement and speech trimming based on the output result; and re-inputting the enhanced and trimmed speech data into the second model; then determining whether a second condition is met based on the current output result. If the second condition is met, speech recognition is performed to obtain a speech recognition result; if the second condition is not met, speech enhancement and speech trimming are performed again, and the resulting speech data is re-inputted into the second model. The second condition is then determined again based on the output result of the second model, and so on, until the second condition is met, at which point speech recognition is performed to obtain a speech recognition result.

[0104] In this application, the input third speech data is enhanced and trimmed based on the output of the second model. Then, if the second condition is met, speech recognition is performed on the fourth speech data after inputting the enhanced and trimmed fourth speech data into the second model, thus obtaining a speech recognition result. If the second condition is not met, speech enhancement and trimming are performed again until the second condition is satisfied. This improves the accuracy of subsequent speech recognition.

[0105] As can be seen in this embodiment, regarding the model training process, when the second speech data used for training corresponds to the first speech data, and the signal-to-noise ratio (SNR) of the second speech data is lower than that of the first speech data, the first hidden layer feature vector obtained during model training using the first speech data is used to adjust the second hidden layer feature vector obtained during model training using the second speech data. Since training the model using the first speech data has less noise impact compared to training it using the second speech data, the resulting first hidden layer feature vector is more related to the speech signal. By aligning the second hidden layer feature vector with the first hidden layer feature vector, the output second hidden layer feature vector includes more feature vectors related to the speech signal, thus improving the accuracy of predicting endpoints based on the second hidden layer feature vector. Therefore, this application constructs a loss function based on the second and first hidden layer feature vectors, which can improve the accuracy of the second model obtained from model training in noisy scenarios for speech endpoint detection.

[0106] For embodiments consistent with those shown above, please refer to... Figure 7 , Figure 7 This is a functional unit block diagram of a model training device provided in an embodiment of this application. The model training device may be the aforementioned speech processing device or a part of a speech processing device, such as... Figure 7 As shown, the model training device 70 includes:

[0107] The training unit 701 is used to train the first model based on the first training data, and the determination unit 702 is used to determine the first hidden layer feature vector output by the hidden layer of the first model. The first training data includes the first speech data.

[0108] The training unit 701 is also used to train the second model based on the second training data and to construct a first loss function based on the first hidden layer feature vector and the second hidden layer feature vector. The second hidden layer feature vector is obtained from the hidden layer output of the second model. The second training data includes the second speech data. The second speech data has a corresponding relationship with the first speech data. The signal-to-noise ratio corresponding to the first speech data is higher than that corresponding to the second speech data.

[0109] Training unit 701 is also used to adjust the parameters of the second model based on the first loss function.

[0110] In one feasible embodiment, the second model is further used to detect the signal-to-noise ratio (SNR) of the speech data. The second training data also includes a preset SNR and preset endpoints. In terms of adjusting the parameters of the second model based on the first loss function, the training unit 701 is specifically used to: construct a second loss function based on the preset SNR and the predicted SNR; construct a third loss function based on the preset endpoints and the predicted endpoints, wherein the predicted SNR and the predicted endpoints are outputs of the second model during model training; construct a loss function based on the first loss function, the second loss function, and the third loss function; and adjust the parameters of the second model based on the loss function.

[0111] In a feasible embodiment, in terms of constructing a loss function based on a first loss function, a second loss function, and a third loss function, the training unit 701 is specifically used to: perform a weighted summation of the first loss function, the second loss function, and the third loss function according to the weights corresponding to the first loss function, the second loss function, and the third loss function, respectively, to obtain the loss function.

[0112] In a feasible embodiment, in terms of constructing a first loss function based on the first hidden layer feature vector and the second hidden layer feature vector, the training unit 701 is specifically used to: construct the first loss function based on the mean squared error loss between the first hidden layer feature vector and the second hidden layer feature vector.

[0113] In one feasible embodiment, the determining unit 702 is further configured to: determine the second speech data in the second training data based on noise, a preset signal-to-noise ratio, and the first speech data in the first training data.

[0114] It is understood that since the embodiments corresponding to the model training method and the embodiments corresponding to the model training device are different presentations of the same technical concept, the content of the embodiment section of the model training method in this application should be adapted to the embodiment section of the model training device, and will not be repeated here.

[0115] For embodiments consistent with those shown above, please refer to... Figure 8 , Figure 8 This is a functional unit block diagram of an endpoint detection device provided in an embodiment of this application. The endpoint detection device can also be the aforementioned speech processing device or a part of a speech processing device, such as... Figure 8 As shown, the endpoint detection device 80 includes:

[0116] Processing unit 801 is used to input third speech data into a second model to obtain a first output result, the first output result including a first endpoint, and the second model can be trained by model training device 70.

[0117] In a feasible embodiment, the first output result further includes a first signal-to-noise ratio (SNR). After obtaining the first output result, the processing unit 801 is further configured to: perform speech enhancement and speech trimming on the third speech data based on the first SNR and the first endpoint to obtain fourth speech data; input the fourth speech data into the second model to obtain a second SNR and a third endpoint; if the second SNR is greater than a first preset threshold and the first ratio is greater than the second preset threshold, then perform speech recognition based on the fourth speech data to obtain a speech recognition result; if the second SNR is not greater than the first preset threshold, or the first ratio is not greater than the second preset threshold, then perform speech enhancement and speech trimming on the fourth speech data based on the second SNR and the third endpoint to obtain fifth speech data, wherein the first ratio is the ratio between the number of frames corresponding to the third endpoint in the fourth speech data and the total number of frames.

[0118] It is understood that since the embodiments corresponding to the endpoint detection method and the embodiments corresponding to the endpoint detection device are different presentations of the same technical concept, the content of the endpoint detection method embodiment section in this application should be adapted to the endpoint detection device embodiment section, and will not be repeated here.

[0119] When using integrated units, such as Figure 9 As shown, Figure 9 This is a block diagram illustrating the functional units of a voice processing device provided in an embodiment of this application. Figure 9 In this document, the voice processing device 101 includes a processing module 912 and a communication module 911. The processing module 912 controls and manages the actions of the voice processing device 101, such as the steps of the training unit 701, the determination unit 702, and the processing unit 801, and / or other processes for executing the techniques described herein. The communication module 911 supports interaction between the voice processing device 101 and other devices. Figure 9 As shown, the voice processing device 101 may also include a storage module 913, which is used to store the program code and data of the voice processing device 101.

[0120] The processing module 912 can be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The communication module 911 can be a transceiver, RF circuitry, or a communication interface, etc. The storage module 913 can be a memory.

[0121] All relevant content in each scenario involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here. The above-mentioned voice processing device 101 can perform the above-mentioned... Figure 2 and / or Figure 5 The method shown.

[0122] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions according to the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. A computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. Semiconductor media can be solid-state drives.

[0123] Figure 10 This is a structural block diagram of an electronic device provided as an embodiment of this application. Figure 10As shown, the electronic device 1000 may include one or more of the following components: a processor 1001, a memory 1002, and a communication interface 1003. The processor 1001, the memory 1002, and the communication interface 1003 are interconnected and perform communication between them. The memory 1002 may store one or more computer programs, which may be configured to implement the methods described in the above embodiments when executed by one or more processors 1001.

[0124] Processor 1001 may include one or more processing cores. Processor 1001 connects to various parts within the electronic device 1000 using various interfaces and lines, and performs various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets, or instruction sets stored in memory 1002, and by calling data stored in memory 1002. Optionally, processor 1001 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 1001 may integrate one or more of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. It is understood that the aforementioned modem may also not be integrated into processor 1001, but may be implemented separately through a communication chip.

[0125] The memory 1002 may include random access memory (RAM) or read-only memory (ROM). The memory 1002 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1002 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described above. The data storage area may also store data created by the electronic device 1000 during use.

[0126] It is understood that the electronic device 1000 may include more or fewer structural elements than those shown in the above block diagram, such as a power module, physical buttons, a WiFi (Wireless Fidelity) module, a speaker, a Bluetooth module, sensors, etc., without limitation.

[0127] The aforementioned electronic device 1000 may be part of the voice endpoint detection system 100 (or a voice processing device or part of a voice processing device) or a device independent of the voice endpoint detection system 100.

[0128] This application provides a computer-readable storage medium storing program data, which, when executed by a processor, is used to perform some or all of the steps of any of the endpoint detection and model training methods described in the above method embodiments.

[0129] This application also provides a computer program product, which includes a computer program operable to cause a computer to perform some or all of the steps of any of the endpoint detection and model training methods described in the above method embodiments. The computer program product can be a software installation package.

[0130] It should be noted that, for the sake of simplicity, all the aforementioned endpoint detection and model training method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0131] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce a good effect.

[0132] Those skilled in the art will understand that all or part of the steps in the various methods of any of the above-described endpoint detection and model training method embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk, or an optical disk, etc.

[0133] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principle and implementation of an endpoint detection-related method and apparatus of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of ​​an endpoint detection-related method and apparatus of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0134] This application is described with reference to flowchart illustrations and / or block diagrams of methods, hardware products, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] It is understood that any product that is controlled or configured to perform the processing method of the flowchart described in the method embodiment of the endpoint detection and model training method of this application, such as the terminal and computer program product of the above flowchart, falls within the scope of the related products described in this application.

[0138] Obviously, those skilled in the art can make various modifications and variations to the endpoint detection method and apparatus provided in this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A model training method, characterized in that, The method includes: The first model is trained based on the first training data to determine the first hidden layer feature vector output by the hidden layer of the first model. The first training data includes the first speech data. The second model is trained based on the second training data, and a first loss function is constructed based on the first hidden layer feature vector and the second hidden layer feature vector. The second hidden layer feature vector is obtained from the hidden layer output of the second model. The second training data includes second speech data, which is generated based on the first speech data and noise. The signal-to-noise ratio corresponding to the first speech data is higher than that corresponding to the second speech data. The network structure of the first model and the second model is the same. The second training data also includes a preset signal-to-noise ratio and a preset endpoint. Based on the first loss function, the preset signal-to-noise ratio, and the preset endpoint, the parameters of the second model are adjusted. The second model is used to detect the speech endpoints and signal-to-noise ratio of the speech data.

2. The method according to claim 1, characterized in that, The step of adjusting the parameters of the second model based on the first loss function, the preset signal-to-noise ratio, and the preset endpoints includes: A second loss function is constructed based on the preset signal-to-noise ratio and the predicted signal-to-noise ratio; A third loss function is constructed based on the preset endpoint and the predicted endpoint, wherein the predicted signal-to-noise ratio and the predicted endpoint are respectively obtained from the output of the second model during model training; A loss function is constructed based on the first loss function, the second loss function, and the third loss function; The parameters of the second model are adjusted based on the loss function.

3. The method according to claim 2, characterized in that, The construction of the loss function based on the first loss function, the second loss function, and the third loss function includes: The first loss function, the second loss function, and the third loss function are weighted and summed according to their respective weights to obtain the loss function.

4. The method according to claim 2 or 3, characterized in that, A first loss function is constructed based on the first hidden layer feature vector and the second hidden layer feature vector, including: The first loss function is constructed based on the mean squared error loss between the first hidden layer feature vector and the second hidden layer feature vector.

5. The method according to claim 2 or 3, characterized in that, Before training the second model based on the second training data, the method further includes: The second speech data in the second training data is determined based on the noise, the preset signal-to-noise ratio, and the first speech data in the first training data.

6. An endpoint detection method, characterized in that, The method includes: The third speech data is input into the second model to obtain a first output result, the first output result including a first endpoint; the second model is trained by any one of the model training methods as claimed in claims 1 to 5.

7. The method according to claim 6, characterized in that, The first output result also includes a first signal-to-noise ratio. After obtaining the first output result, the method further includes: Based on the first signal-to-noise ratio and the first endpoint, the third speech data is enhanced and trimmed to obtain the fourth speech data. The fourth speech data is input into the second model to obtain the second signal-to-noise ratio and the third endpoint; If the second signal-to-noise ratio is greater than the first preset threshold and the first ratio is greater than the second preset threshold, then speech recognition is performed based on the third speech data to obtain a speech recognition result. The first ratio is the ratio between the number of frames corresponding to the third endpoint in the third speech data and the total number of frames. If the second signal-to-noise ratio is not greater than the first preset threshold, or the first ratio is not greater than the second preset threshold, then the third speech data is enhanced and truncated based on the second signal-to-noise ratio and the third endpoint to obtain the fifth speech data.

8. A model training device, characterized in that, The device includes: A training unit is used to train a first model based on first training data, wherein the first training data includes first speech data; A determining unit is used to determine the first hidden layer feature vector output by the hidden layer of the first model; The training unit is also used to train the second model based on the second training data and to construct a first loss function based on the first hidden layer feature vector and the second hidden layer feature vector. The second hidden layer feature vector is obtained from the hidden layer output of the second model. The second training data includes second speech data, which is generated based on the first speech data and noise. The signal-to-noise ratio corresponding to the first speech data is higher than that corresponding to the second speech data. The network structure of the first model and the second model is the same. The second training data also includes a preset signal-to-noise ratio and a preset endpoint. The training unit is further configured to adjust the parameters of the second model based on the first loss function, the preset signal-to-noise ratio, and the preset endpoint. The second model is used to detect the speech endpoints and signal-to-noise ratio of the speech data.

9. An endpoint detection device, characterized in that, The device includes: The processing unit is configured to input third speech data into the second model and obtain a first output result, wherein the first output result includes a first endpoint; The second model is obtained by training using the model training device as described in claim 8.

10. An electronic device, characterized in that, The apparatus includes a processor, a memory, and a computer program stored in the memory, characterized in that the processor is configured to retrieve the computer program stored in the memory to perform the method as described in any one of claims 1-7.

11. A computer-readable storage medium, characterized in that, A computer program for storing electronic data interchange, wherein the computer program causes a computer to perform the method as described in any one of claims 1-7.

Images