Audio frame loss detection method, frame loss detection model training method and device

Abstract

The embodiment of the application provides a kind of audio frame loss detection method, the training method and device of frame loss detection model, wherein frame loss detection method includes: obtaining target audio data to be detected, target audio data includes first mute segment;Target audio data is input into the audio feature extraction network of frame loss detection model, and target audio data is carried out audio feature extraction processing by audio feature extraction network, and target audio feature is obtained;Target audio feature is classified by the classification network of frame loss detection model, and frame loss detection result is obtained;Frame loss detection result characterizes whether there is frame loss in first mute segment. Through the embodiment of the application, accurate, efficient detection of audio frame loss is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, and in particular to an audio frame loss detection method, a frame loss detection model training method and device. BACKGROUND

[0002] Speech frame loss refers to incomplete speech data caused by missing of a meaningful part of a speech signal. It may not affect semantics, but such a speech signal will make the listener feel that the relevant part of the speech is suddenly disconnected, or that there is a burst noise, and the natural and complete speech information cannot be heard. For TTS (Text to Speech) speech synthesis technology, if speech frame loss occurs in the training sample, especially speech frame loss corresponding to a key phoneme, the acoustic model will learn incorrect mapping, which will eventually lead to synthesis errors. Therefore, how to detect speech frame loss and thus improve the quality of TTS training samples is crucial. SUMMARY

[0003] The present application provides an audio frame loss detection method, a frame loss detection model training method and device to realize accurate and efficient detection of audio frame loss.

[0004] In a first aspect, an embodiment of the present application provides an audio frame loss detection method, comprising:

[0005] obtaining target audio data to be detected; the target audio data comprising a first silent segment;

[0006] inputting the target audio data into an audio feature extraction network of a frame loss detection model, performing audio feature extraction processing on the target audio data through the audio feature extraction network, and obtaining target audio features;

[0007] performing classification processing on the target audio features through a classification network of the frame loss detection model, and obtaining a frame loss detection result; the frame loss detection result representing whether the first silent segment has frame loss.

[0008] As can be seen, in the embodiment of the present application, when the target audio data containing the first silent segment is obtained, the target audio data is input into the audio feature extraction network of the frame loss detection model for audio feature extraction processing, and the classification network of the frame loss detection model is used to perform classification processing on the extracted target audio features, to obtain the frame loss detection result representing whether the first silent segment has frame loss. In this process, since the frame loss detection processing is based on the frame loss detection model, and the frame loss detection model has the characteristics of high reliability, fast detection speed and low cost, accurate, efficient and low-cost detection of audio frame loss is realized, which can provide effective data basis for subsequent related audio processing.

[0009] In a second aspect, the embodiments of the present application provide a training method of a frame loss detection model, comprising:

[0010] obtaining a training sample set in a preset manner; the training sample set comprises positive samples with frame loss and negative samples without frame loss;

[0011] performing iterative training processing on a network to be trained by using the training sample set, to obtain a frame loss detection model;

[0012] The frame loss detection model comprises an audio feature extraction network and a classification network; the audio feature extraction network is configured to perform audio feature extraction processing on input target audio data, to obtain target audio features; and the classification network is configured to perform classification processing on the target audio features, to obtain a frame loss detection result.

[0013] It can be seen that, in the embodiments of the present application, a training sample set is obtained in a preset manner, and a network to be trained is iteratively trained by using the training sample set, to obtain a frame loss detection model for performing frame loss detection processing on target audio data. Since the frame loss detection model has the characteristics of high reliability, fast detection speed and low cost, training the frame loss detection model can realize accurate, efficient and low-cost detection of audio frame loss, and can provide effective data basis for subsequent related audio processing.

[0014] In a third aspect, the embodiments of the present application provide an audio frame loss detection device, comprising:

[0015] an obtaining module configured to obtain target audio data to be detected; the target audio data comprises a first silent segment;

[0016] an extracting module configured to input the target audio data into an audio feature extraction network of a frame loss detection model, perform audio feature extraction processing on the target audio data by using the audio feature extraction network, and obtain target audio features;

[0017] a classification module configured to perform classification processing on the target audio features by using a classification network of the frame loss detection model, and obtain a frame loss detection result; the frame loss detection result represents whether the first silent segment has frame loss.

[0018] In a fourth aspect, the embodiments of the present application provide a training device of a frame loss detection model, comprising:

[0019] an obtaining module configured to obtain a training sample set; the training sample set comprises positive samples with frame loss and negative samples without frame loss;

[0020] a training module configured to perform training processing on a network to be trained by using the training sample set, to obtain a frame loss detection model;

[0021] The frame loss detection model includes an audio feature extraction network and a classification network; the audio feature extraction network is configured to perform audio feature extraction on the input target audio data to obtain target audio features; and the classification network is configured to perform classification on the target audio features to obtain a frame loss detection result.

[0022] In a fifth aspect, an electronic device is provided, including:

[0023] a processor; and a memory arranged to store computer executable instructions configured to be executed by the processor, the executable instructions including steps for performing the audio frame loss detection method provided in the first aspect, or the executable instructions including steps for performing the training method of the frame loss detection model provided in the second aspect.

[0024] In a sixth aspect, a storage medium is provided, the storage medium being configured to store computer executable instructions, the executable instructions causing a computer to perform steps in the audio frame loss detection method provided in the first aspect, or the executable instructions causing a computer to perform steps in the training method of the frame loss detection model provided in the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of one or more embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0026] Figure 1 A first flowchart of an audio frame loss detection method provided by an embodiment of the present application;

[0027] Figure 2 A schematic diagram of normal speech pause and frame loss provided by an embodiment of the present application;

[0028] Figure 3 A structural schematic diagram of a frame loss detection model provided by an embodiment of the present application;

[0029] Figure 4 A second flowchart of an audio frame loss detection method provided by an embodiment of the present application;

[0030] Figure 5 A first flowchart of a training method of a frame loss detection model provided by an embodiment of the present application;

[0031] Figure 6 A second flow chart of a method for training a frame loss detection model provided in an embodiment of the present application;

[0032] Figure 7 A schematic diagram of the module composition of an audio frame loss detection device provided in an embodiment of the present application;

[0033] Figure 8 A schematic diagram of the module composition of a training device for a frame loss detection model provided in an embodiment of the present application;

[0034] Figure 9 A schematic diagram of the structure of an electronic device provided in one or more embodiments of the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present application, the technical solutions in one or more embodiments of the present application will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on one or more embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of this document.

[0036] The embodiment of the present application provides a kind of audio frame loss detection method, the training method and device of frame loss detection model.The artificial intelligence is more and more widely into people's life in recent years, TTS (Text to Speech) speech synthesis is an important link in artificial intelligence.Following, based on speech synthesis model carries out speech synthesis processing also becomes current research direction.In the training process of speech synthesis model, if the speech training sample used is obtained by voice recording, then the requirement of recording equipment, site, personnel and the professional of data annotation is high, and the cost is expensive.Therefore, using low-quality speech training sample to train speech synthesis model becomes the first choice.The recording voice of telephone channel is typical low-quality voice.But the recording voice of telephone channel, possibly because the environment of speaker is noisy, communication channel transmission, recording equipment is unstable and other factors, cause recording voice to exist frame loss problem.And based on the recording voice of existing frame loss problem carries out the training of speech synthesis model, can lead to model to learn the wrong mapping, finally lead to speech synthesis error.Therefore, accurately, efficiently carry out frame loss detection, to find the speech of existing frame loss problem is crucial.A kind of processing mode for carrying out frame loss detection in prior art is to detect whether there is frame loss in speech by artificial listening, which not only takes a long time, but also requires high professional degree of artificial.Another processing mode in prior art is to judge whether there is frame loss based on time domain features, and the threshold parameters involved in the judgment in this mode are often from artificial experience, or are based on statistics, so the reliability is poor, there are more misjudgment and missed judgment, and the generalization ability is poor, the threshold parameters for user A can not be suitable for user B.A kind of processing mode in prior art is to introduce new telephone speech encoder, to increase the robustness to telephone channel frame loss problem, but it does not really solve the problem of frame loss detection for speech, and the new encoder also increases the detection cost.Based on this, a kind of audio frame loss detection method is provided in the embodiment of the present application, by pre-training frame loss detection model, so that when target audio data containing first silence segment is acquired, the target audio data is input into frame loss detection model for frame loss detection processing, to obtain the detection result indicating whether the first silence segment exists frame loss.Because the frame loss detection processing in the present application is based on frame loss detection model, and frame loss detection model has the characteristics of high reliability, fast detection speed and low cost, accurate, efficient and low-cost detection of audio frame loss is realized, which can provide effective data basis for subsequent related audio processing.

[0037] Specifically, Figure 1 A flowchart of an audio frame loss detection method provided for one or more embodiments of the present application, Figure 1The method in the method can be executed by an audio frame loss detection device, which can be arranged in a terminal device or a server. The terminal device can be a mobile phone, a tablet computer, a desktop computer, a portable notebook computer, etc. The server can be a stand-alone server or a server cluster composed of multiple servers. As shown in Figure 1 The method comprises the following steps:

[0038] In step S102, target audio data to be detected is obtained; the target audio data comprises a first mute segment;

[0039] The target audio data can be telephone recording audio data, conference recording audio data or other forms of audio data, which are not limited in the embodiments of the present application. Taking the telephone recording audio data as an example, considering that people often have normal pauses when speaking, and the pauses and frame loss are both in the form of silence, as shown in Figure 2 The left box corresponds to a normal pause, and the right box corresponds to audio frame loss. Therefore, in order to distinguish whether the silence is caused by the normal pause of the speaker or the frame loss, frame loss detection is needed, so as to determine whether the target audio data is applied to other audio processing processes, such as a speech synthesis model training sample, etc.

[0040] It should be noted that, considering the particularity of audio data, there is a large difference between audio data of different sampling rates. Therefore, in order to improve the accuracy of frame loss detection, preferably, the sampling rate of the target audio data is consistent with the sampling rate corresponding to the frame loss detection model. For example, the training sample of the frame loss detection model is audio with a sampling rate of 16000 Hz, that is, the sampling rate corresponding to the frame loss detection model is 16000 Hz, and preferably the sampling rate of the target audio data is also 16000 Hz.

[0041] In step S104, the target audio data is input into an audio feature extraction network of the frame loss detection model, and the target audio data is subjected to audio feature extraction processing by the audio feature extraction network to obtain target audio features;

[0042] In order to realize accurate, efficient and low-cost detection of audio frame loss, the frame loss detection model is pre-trained in the embodiments of the present application, and the frame loss detection model is used to detect the target audio data. Specifically, as shown in Figure 3As shown, the frame loss detection model includes an audio feature extraction network and a classification network. Accordingly, in the process of using the frame loss detection model to perform frame loss detection on the target audio data, first, the target audio data is input into the audio feature extraction network of the frame loss detection model, and the target audio data is subjected to audio feature extraction processing by the audio feature extraction network to obtain target audio features. Then, the target audio features are subjected to classification processing by the classification network of the frame loss detection model to obtain a frame loss detection result, that is, step S106 is executed. The training process of the frame loss detection model can be referred to the relevant description hereinafter.

[0043] In step S106, the target audio features are subjected to classification processing by the classification network of the frame loss detection model to obtain a frame loss detection result. The frame loss detection result indicates whether the first silent segment has frame loss.

[0044] In one or more embodiments of the present application, when the target audio data containing the first silent segment is obtained, the target audio data is input into the audio feature extraction network of the frame loss detection model for audio feature extraction processing, and the extracted target audio features are subjected to classification processing by the classification network of the frame loss detection model to obtain a frame loss detection result indicating whether the first silent segment has frame loss. In this process, since the frame loss detection processing is based on the frame loss detection model, and the frame loss detection model has the characteristics of high reliability, fast detection speed, and low cost, accurate, efficient, and low-cost detection of audio frame loss is achieved, which can provide effective data basis for subsequent related audio processing.

[0045] Taking the target audio data as telephone recording audio data as an example, considering that the telephone recording audio data is usually long, frame loss detection based on the long audio data will reduce the detection efficiency. Based on this, in one or more embodiments of the present application, the target audio data to be detected is obtained from the long first initial audio data. Specifically, as shown in Figure 4 As shown, step S102 can include the following steps S102-2 to S102-6:

[0046] In step S102-2, the first initial audio data is obtained. The duration of the first initial audio data is greater than a preset duration.

[0047] The first initial audio data can be audio data of a telephone recording, audio data of a conference recording, or other forms of audio data. The first initial audio data has a time length greater than a preset time length of the target audio data, and the first initial audio data includes at least one first mute segment. Each first mute segment can be a normal speech pause or can have frame loss. The preset time length can be set as needed in actual application. In an embodiment, the first initial audio data is 16000Hz, and the preset time length is 188 milliseconds, i.e., 188*16=3008 samples.

[0048] In step S102-4, voice activity detection is performed on the first initial audio data to obtain at least one first mute segment in the first initial audio data. Each first mute segment has a time length less than the preset time length.

[0049] The voice activity detection (VAD) can also be referred to as mute detection, which can detect mute segments in audio data and start and end positions of the mute segments. For details of the voice activity detection process, refer to the prior art, which will not be described in detail herein.

[0050] In particular, when the first initial audio data includes audio data of multiple speakers, audio data of a target speaker can also be extracted from the first initial audio data according to a preset manner, and the audio data of the target speaker is subjected to voice activity detection.

[0051] It should be noted that the frame loss in the present application refers to incomplete audio data caused by missing of meaningful audio signal parts, which often does not affect the semantics, but such audio signal can make the listener feel that the relevant part of the audio is suddenly disconnected, or there is a loud noise, and the listener cannot hear the natural and complete audio information. The time length of the frame loss is usually any time length between 10 milliseconds and 100 milliseconds. In the field of audio technology, there is also a phenomenon of a large number of continuous frame losses in audio data. The time length of the continuous frame loss is usually greater than 100 milliseconds. Since this phenomenon can cause missing of words and phrases, it is difficult for the listener to hear the complete semantics, and therefore, this phenomenon can be referred to as audio damage. It can be understood that the audio damage is completely different from the frame loss in the present application.

[0052] In step S102-6, for each first mute segment, the first mute segment is continuously extended forward and / or backward in the first initial audio data in time from front to back to obtain target audio data with a preset time length.

[0053] Specifically, for each first silent segment, it is determined whether the first initial audio data can obtain the target audio data of the preset time length by extending the first silent segment forward in time from front to back; if not, it is determined whether the first initial audio data can obtain the target audio data of the preset time length by extending the first silent segment backward; if not, the first initial audio data is extended forward and backward at the same time to obtain the target audio data of the preset time length. When the first initial audio data is extended forward, or backward, or forward and backward at the same time, the first silent segment can obtain the target audio data of the preset time length, one of the extension methods can be randomly selected to extend the first silent segment in the first initial audio data.

[0054] As an example, the length of the first initial audio is 1000 milliseconds, the preset time length is 188 milliseconds, the starting time of a certain first silent segment is 135 milliseconds, and the ending time is 150 milliseconds. The first silent segment can be extended forward to 100 milliseconds and backward to 288 seconds in the first initial audio to obtain target audio data with a starting time of 100 milliseconds, an ending time of 288 milliseconds, and a length of 188 milliseconds; or, the first silent segment can be extended backward to 323 milliseconds in the first initial audio to obtain target audio data with a starting time of 135 milliseconds, an ending time of 323 milliseconds, and a length of 188 milliseconds.

[0055] Therefore, by performing speech activity detection processing on the first initial audio data, at least one first silent segment is obtained, and the first silent segment is extended in the first initial audio data to obtain target audio data of a preset time length, so that the frame loss detection processing is performed based on the target audio data. Not only can the frame loss detection be targeted and the detection efficiency be improved, but also the data length requirement of the frame loss detection model can be met.

[0056] In order to realize effective frame loss detection, after obtaining the target audio data, the target audio data is first processed by the audio feature extraction network of the frame loss detection model. The audio feature extraction network can be a pre-trained CPC model. Specifically, as shown in Figure 3 The audio feature extraction network includes a first stride convolution layer, a second stride convolution layer, a third stride convolution layer, a fourth stride convolution layer, and a fifth stride convolution layer. Correspondingly, the target audio data is processed by the audio feature extraction network in step S104 to obtain the target audio feature, which can include the following steps S104-2 to S104-10:

[0057] Step S104-2, performing first audio feature extraction processing on each audio frame of the target audio data through the first stride convolution layer to obtain a first audio feature of each audio frame;

[0058] Step S104-4, performing second audio feature extraction processing on each first audio feature through the second stride convolution layer to obtain a second audio feature of each audio frame;

[0059] Step S104-6, performing third audio feature extraction processing on each second audio feature through the third stride convolution layer to obtain a third audio feature of each audio frame;

[0060] Step S104-8, performing fourth audio feature extraction processing on each third audio feature through the fourth stride convolution layer to obtain a fourth audio feature of each audio frame;

[0061] Step S104-10, performing fifth audio feature extraction processing on each fourth audio feature through the fifth stride convolution layer to obtain a target audio feature of each audio frame.

[0062] The specific processes of the first audio feature extraction processing, the second audio feature extraction processing, the third audio feature extraction processing, the fourth audio feature extraction processing, and the fifth audio feature extraction processing, and the network parameters of the feature extraction network can be set as needed in actual applications. In an implementation, the convolution kernel of the first stride convolution layer is 10, and the convolution stride length is 5; the convolution kernel of the second stride convolution layer is 8, and the convolution stride length is 4; the convolution kernel of the third stride convolution layer is 4, and the convolution stride length is 2; the convolution kernel of the fourth stride convolution layer is 4, and the convolution stride length is 2; the convolution kernel of the fifth stride convolution layer is 4, and the convolution stride length is 2; that is, the network parameters of the audio feature extraction network are the convolution kernel [10, 8, 4, 4, 4] and the stride length [5, 4, 2, 2, 2]; the number of hidden layer nodes is 512, and the activation function can use the ReLU activation function.

[0063] Further, taking the target audio data including 7 audio frames as an example, it can be understood that 7 audio features are obtained by each stride convolution layer, and the target audio feature can be represented as (z1, z2, z3, z4, z5, z6, z7), where z1 is the target audio feature of the first audio frame of the target audio data, z2 is the target audio feature of the second audio frame of the target audio data, and so on, and z7 is the target audio feature of the seventh audio frame of the target audio data.

[0064] Therefore, the audio feature extraction network of the frame loss detection model performs multi-layer feature extraction processing on the target audio data, which guarantees the accuracy of the obtained target audio feature.

[0065] After obtaining the target audio feature of the target audio data, the target audio feature can be classified by the classification network of the frame loss detection model to obtain a frame loss detection result. Specifically, as shown in FIG. 11, the classification network of the frame loss detection model can include a bidirectional long short-term memory network, an average pooling layer, and a fully connected layer. Correspondingly, the classification of the target audio feature by the classification network of the frame loss detection model in step S106 to obtain the frame loss detection result can include the following steps S106-2 to S106-8: Figure 3

[0066] In step S106-2, the time sequence feature of the target audio feature is extracted by the bidirectional long short-term memory network.

[0067] In step S106-4, the time sequence feature is processed by the average pooling layer to obtain an average pooling feature.

[0068] In step S106-6, the average pooling feature is converted to obtain a converted feature.

[0069] In step S106-8, the converted feature is classified by the fully connected layer to obtain the frame loss detection result.

[0070] The conversion can be transposition. In an embodiment, the parameters of the bidirectional long short-term memory network (Bi-LSTM) can include an input dimension of 512, a hidden layer node number of 256, and an output dimension of 256; the parameters of the average pooling layer (Average Pooling) can include a convolution kernel of 7, an input dimension of 256, and an output dimension of 256; and the parameters of the fully connected layer (FC Layer) can include an input parameter dimension of 256 and an output dimension of 2. The output dimension of 2 of the fully connected layer means that the frame loss detection result includes two dimensions, one of which represents the probability of the existence of frame loss, and the other represents the probability of the non-existence of frame loss. Correspondingly, after obtaining the frame loss detection result of the target audio data, the two probabilities included in the frame loss detection result are compared to obtain a larger probability, and whether the first silent segment in the target audio data has frame loss is determined according to the larger probability.

[0071] It can be understood that when multiple target audio data are obtained from the first initial audio data, each target audio data is sequentially input into the frame loss detection model for frame loss detection processing to obtain the frame loss detection result of each target audio data.

[0072] ​Thus, after the audio feature extraction network of the frame loss detection model performs multi-layer feature extraction processing on the target audio data, the classification network of the frame loss detection model performs classification processing on the extracted target audio features, and a frame loss detection result is obtained. Based on the high reliability, fast detection speed, and low cost of the frame loss detection model, accurate, efficient, and low-cost detection of audio frame loss can be achieved, and effective data basis is provided for subsequent related audio processing.

[0073] Further, when the frame loss detection result indicates that the first silent segment has frame loss, the frame loss position can also be determined according to the first silent segment. Specifically, the start and end positions of the first silent segment are determined as the frame loss position.

[0074] In one or more embodiments of the present application, when the target audio data containing the first silent segment is obtained, the target audio data is input into the audio feature extraction network of the frame loss detection model for audio feature extraction processing, and the classification network of the frame loss detection model is used to classify the extracted target audio features, and a frame loss detection result indicating whether the first silent segment has frame loss is obtained. In this process, since the frame loss detection processing is based on the frame loss detection model, and the frame loss detection model has the characteristics of high reliability, fast detection speed, and low cost, accurate, efficient, and low-cost detection of audio frame loss is achieved, and effective data basis is provided for subsequent related audio processing.

[0075] Corresponding to the above-described audio frame loss detection method, based on the same technical concept, one or more embodiments of the present application also provide a frame loss detection model training method, Figure 5 A flowchart of a frame loss detection model training method provided by one or more embodiments of the present application is shown in Figure 5 The method in the above embodiment can be executed by a frame loss detection model training device. The frame loss detection model training device can be arranged in a terminal device or a server. The terminal device can be a mobile phone, a tablet computer, a desktop computer, a portable notebook computer, etc. The server can be a stand-alone server or a server cluster composed of multiple servers. As shown in Figure 5 The method includes the following steps:

[0076] In step S202, a training sample set is obtained in a predetermined manner. The training sample set includes positive samples with frame loss and negative samples without frame loss.

[0077] Considering that in actual applications, there are more audio data without frame loss and less audio data with frame loss. In order to better train the frame loss detection model, in the present embodiment, based on the second initial audio data without frame loss, a plurality of positive samples with frame loss and a plurality of negative samples without frame loss are generated, so that the generated samples are used for training processing. Specifically, as shown inFigure 6 As shown, step S202 can include steps S202-2 to S202-10.

[0078] Step S202-2, performing voice activity detection processing on the second initial audio data to obtain at least one second silent segment in the second initial audio data; the second initial audio data is audio data without frame loss, and the length of the second silent segment is within a preset length range;

[0079] The number of the second initial audio data can be one or multiple, and the length of the second initial audio data is much greater than the preset length. In an embodiment, the preset length is 188 milliseconds.

[0080] Further, as described above, the length of the frame loss audio in the present application is generally between 10 milliseconds and 100 milliseconds. The audio with a length greater than 100 milliseconds is audio damage and does not belong to the data processing range of the present application. Therefore, the length of each second silent segment obtained by performing voice activity detection processing on the second initial audio data is between 10 milliseconds and 100 milliseconds.

[0081] Step S202-4, determining target continuous audio data with a length greater than the preset length in the second initial audio data according to the second silent segment;

[0082] Specifically, determine whether the number of second silent segments is greater than one; if the number of second silent segments is not greater than one, that is, the number of second silent segments is one, determine whether the duration of the audio data before the start time of the second silent segment in the second initial audio data is greater than a preset duration, and determine whether the duration of the audio data after the end time of the second silent segment in the second initial audio data is greater than a preset duration; and determine the audio data with a duration greater than the preset duration as the target continuous audio data. If the number of second silent segments is greater than one, then determine each second silent segment as the current silent segment in chronological order from front to back, and determine whether the current silent segment is the first silent segment or the last silent segment; if the current silent segment is neither the first silent segment nor the last silent segment, determine the candidate duration between the end time of the current silent segment and the start time of the next silent segment after the current silent segment, determine whether the candidate duration is greater than the preset duration, and if so, determine the audio data between the end time of the current silent segment and the start time of the next silent segment after the current silent segment as the target audio data. If the current silent segment is the first silent segment, it is determined whether the duration of the audio data before the start time of the current silent segment in the second initial audio data is greater than a preset duration. If so, the audio data before the start time of the current silent segment is determined as the target audio data. If the current silent segment is the last silent segment, it is determined whether the duration of the audio data after the end time of the current silent segment in the second initial audio data is greater than a preset duration. If so, the audio data after the end time of the current silent segment is determined as the target audio data.

[0083] It is understandable that the target audio data is data that does not contain silence, and the duration of the target audio data is greater than the preset duration.

[0084] Step S202-6, generating positive samples based on the target continuous audio data;

[0085] Specifically, multiple candidate audio data with a preset duration are randomly extracted from each target continuous audio; the target duration is randomly selected from the preset duration range to generate dropped-frame audio data of the target duration; for each candidate audio data, the target position in the candidate audio data for replacing the dropped-frame audio data is determined; the audio data at the target position is replaced with the dropped-frame audio data to obtain a positive sample.

[0086] Among them, generating the lost-frame audio data of the target duration may include: generating a silent sequence of all zero values ​​of the target duration, and determining the generated silent sequence as the lost-frame audio data; or generating a low-energy Gaussian white noise sequence, and determining the Gaussian white noise sequence as the lost-frame audio data.

[0087] Further, determining the target position in the continuous audio data for replacing the lost-frame audio data can comprise: calculating a difference value between the preset time length and the target time length, determining a start position of the target position according to the difference value; and determining an end position of the target position according to the start position of the target position and the target time length. As an example, the preset time length is 188 milliseconds, the preset time length range is 10 milliseconds to 100 milliseconds, and the selected target time length is 15 milliseconds. Then, the difference value between the preset time length and the target time length is calculated as 188-25=163, any position between the 0th second and the 163rd second of the candidate audio data is determined as the start position of the target position, for example, the start position of the target position is the 10th second, and the end position of the target position can be determined as the 25th second.

[0088] It should be noted that for different candidate audio data, the target time length can be the same or different, and the target position for replacing the lost-frame audio data can be the same or different.

[0089] It can be understood that each candidate audio data is unlost-frame audio data, and after part of the data in the candidate audio data is replaced by the generated lost-frame audio data, the candidate audio data becomes a positive sample containing lost-frame audio data.

[0090] Step S202-8, generating a negative sample based on the second silence segment and the second initial audio data;

[0091] Specifically, in the second initial audio data, the second silence segment is continuously extended forward and / or backward in time from front to back to obtain audio data of a preset time length; and the audio data of the preset time length is determined as the negative sample. Wherein, the process of continuously extending the second silence segment forward and / or backward in the second initial audio data is the same as the process of continuously extending the first silence segment forward and / or backward in the first initial audio data, which can be referred to the foregoing description, and the repeated parts will not be described here.

[0092] Step S202-10, generating a training sample set according to the positive sample and the negative sample.

[0093] Specifically, each positive sample and each negative sample is labeled with a label, and the positive sample with the label and the negative sample with the label are combined as the training sample set. Wherein, the label is used to represent whether the corresponding sample has lost frames.

[0094] Step S204, iteratively training the to-be-trained network using the training sample set to obtain a lost-frame detection model.

[0095] The frame loss detection model includes an audio feature extraction network and a classification network. The audio feature extraction network is configured to perform audio feature extraction processing on the input target audio data to obtain target audio features. The classification network is configured to perform classification processing on the target audio features to obtain a frame loss detection result. Correspondingly, the step S204 of performing iterative training processing on the to-be-trained network using the training sample set to obtain the frame loss detection model can include the following steps:

[0096] The training sample in the training sample set is input into the audio feature extraction network, and the audio feature extraction network performs audio feature extraction processing on the training sample to obtain target audio features. The classification network performs classification processing on the target audio features to obtain a frame loss detection result.

[0097] Further, the specific process of performing iterative training processing on the to-be-trained network using the training sample set can be set by the user as needed in actual application, and the present application does not make specific limitations thereon. In an embodiment, a predetermined number of training samples can be randomly selected from the training sample set one by one as the current training sample of the current training step number. The current training sample is input into the current to-be-trained network for training processing. If it is determined that the stop training condition is met, the current to-be-trained network is determined as the frame loss detection model. The determination that the stop training condition is met can be that the training step number reaches a predetermined step number, and the determination that the stop training condition is met. Alternatively, based on the frame loss detection result obtained by the current training step number and the label of the current training sample, when the loss value calculated according to the preset loss function meets the preset condition, it is determined that the stop training condition is met. The preset loss function can be a binary cross-entropy loss function.

[0098] In the embodiments of the present application, the training sample set is obtained in a predetermined manner, and the to-be-trained network is iteratively trained using the training sample set to obtain a frame loss detection model for frame loss detection processing of target audio data. Since the frame loss detection model has the characteristics of high reliability, fast detection speed, and low cost, training the frame loss detection model can realize accurate, efficient, and low-cost detection of audio frame loss, and can provide effective data basis for subsequent related audio processing.

[0099] Corresponding to the audio frame loss detection method described above, based on the same technical concept, one or more embodiments of the present application also provide an audio frame loss detection device. Figure 7 A module composition schematic diagram of an audio frame loss detection device provided by one or more embodiments of the present application is shown in FIG. 3, which includes the following modules. Figure 7 As shown in FIG. 3, the device includes the following modules.

[0100] The acquisition module 301 is configured to acquire target audio data to be detected. The target audio data includes a first silent segment.

[0101] The extraction module 302 is configured to input the target audio data into an audio feature extraction network of the frame loss detection model, perform audio feature extraction processing on the target audio data through the audio feature extraction network, and obtain target audio features.

[0102] The classification module 303 is configured to perform classification processing on the target audio features through a classification network of the frame loss detection model, and obtain a frame loss detection result. The frame loss detection result indicates whether the first silent segment has frame loss.

[0103] The audio frame loss detection device provided in the embodiments of the present application can perform audio feature extraction processing on the target audio data containing the first silent segment by inputting the target audio data into the audio feature extraction network of the frame loss detection model, and perform classification processing on the extracted target audio features through the classification network of the frame loss detection model, and obtain the frame loss detection result indicating whether the first silent segment has frame loss. In this process, since the frame loss detection processing is based on the frame loss detection model, and the frame loss detection model has the characteristics of high reliability, fast detection speed, and low cost, the accurate, efficient, and low-cost detection of the audio frame loss is realized, and effective data basis can be provided for subsequent related audio processing.

[0104] It should be noted that the embodiments of the audio frame loss detection device in the present application and the embodiments of the audio frame loss detection method in the present application are based on the same inventive concept, and therefore the specific implementation of the embodiments can be referred to the foregoing implementation of the corresponding audio frame loss detection method, and the repeated parts will not be described herein.

[0105] Further, corresponding to the training method of the frame loss detection model described above, based on the same technical concept, one or more embodiments of the present application also provide a training device of a frame loss detection model. Figure 8 A module composition schematic diagram of a training device of a frame loss detection model provided by one or more embodiments of the present application is shown in Figure 8 The device comprises:

[0106] The acquisition module 401 is configured to acquire a training sample set. The training sample set comprises positive samples with frame loss and negative samples without frame loss.

[0107] The training module 402 is configured to perform training processing on the to-be-trained network by using the training sample set, and obtain a frame loss detection model.

[0108] The frame loss detection model comprises an audio feature extraction network and a classification network. The audio feature extraction network is configured to perform audio feature extraction processing on input target audio data, and obtain target audio features. The classification network is configured to perform classification processing on the target audio features, and obtain a frame loss detection result.

[0109] The training device of the frame loss detection model provided in the embodiments of the present application obtains a training sample set in a preset manner, and iteratively trains a network to be trained using the training sample set to obtain a frame loss detection model for performing frame loss detection processing on target audio data. Since the frame loss detection model has the characteristics of high reliability, fast detection speed, and low cost, training the frame loss detection model can achieve accurate, efficient, and low-cost detection of audio frame loss, and can provide effective data basis for subsequent related audio processing.

[0110] It should be noted that the embodiments of the training device of the frame loss detection model in the present application and the embodiments of the training method of the frame loss detection model in the present application are based on the same inventive concept, so the specific implementation of the embodiments can be referred to the foregoing implementation of the corresponding training method of the frame loss detection model, and the repeated parts will not be described herein.

[0111] Further, corresponding to the audio frame loss detection method and the training method of the frame loss detection model described above, based on the same technical concept, one or more embodiments of the present application also provide an electronic device, Figure 9 A structural schematic diagram of an electronic device provided by one or more embodiments of the present application.

[0112] As Figure 9 shown, the electronic device can have a large difference due to different configurations or performances, and can include one or more processors 501 and memories 502, and the memories 502 can store one or more storage applications or data. Among them, the memory 502 can be temporary storage or persistent storage. The application stored in the memory 502 can include one or more modules (not shown in the figure), and each module can include a series of computer executable instructions in the electronic device. Further, the processor 501 can be configured to communicate with the memory 502 and execute a series of computer executable instructions in the memory 502 on the electronic device. The electronic device can also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input and output interfaces 505, one or more keyboards 506, and the like.

[0113] In one specific embodiment, the electronic device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and the one or more programs can include one or more modules, and each module can include a series of computer executable instructions in the electronic device, and the one or more processors are configured to execute the one or more programs including the following computer executable instructions:

[0114] obtaining target audio data to be detected; the target audio data includes a first mute segment;

[0115] inputting the target audio data into an audio feature extraction network of a frame loss detection model, performing audio feature extraction processing on the target audio data through the audio feature extraction network, and obtaining target audio features;

[0116] performing classification processing on the target audio features through a classification network of the frame loss detection model, and obtaining a frame loss detection result; the frame loss detection result represents whether the first mute segment has frame loss.

[0117] The electronic device provided by one or more embodiments of the present application, when obtaining target audio data containing a first mute segment, inputs the target audio data into an audio feature extraction network of a frame loss detection model for audio feature extraction processing, and performs classification processing on the extracted target audio features through a classification network of the frame loss detection model to obtain a frame loss detection result representing whether the first mute segment has frame loss. In this process, since the frame loss detection processing is based on the frame loss detection model, and the frame loss detection model has the characteristics of high reliability, fast detection speed, and low cost, accurate, efficient, and low-cost detection of audio frame loss is achieved, which can provide effective data basis for subsequent related audio processing.

[0118] In another specific embodiment, the electronic device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and the one or more programs can include one or more modules, and each module can include a series of computer executable instructions in the electronic device, and is configured to be executed by one or more processors. The one or more programs include computer executable instructions for performing the following:

[0119] obtaining a training sample set in a preset manner; the training sample set includes positive samples with frame loss and negative samples without frame loss;

[0120] performing iterative training processing on the training sample set to obtain a frame loss detection model;

[0121] The frame loss detection model includes an audio feature extraction network and a classification network; the audio feature extraction network is used for performing audio feature extraction processing on the input target audio data to obtain target audio features; and the classification network is used for performing classification processing on the target audio features to obtain a frame loss detection result.

[0122] The electronic device provided in the embodiments of the present application obtains a training sample set in a preset manner, and iteratively trains a network to be trained by using the training sample set to obtain a frame loss detection model for frame loss detection processing of target audio data. Since the frame loss detection model has the characteristics of high reliability, fast detection speed, and low cost, accurate, efficient, and low-cost detection of audio frame loss can be achieved by training the frame loss detection model, and effective data basis can be provided for subsequent related audio processing.

[0123] It should be noted that the embodiments of the electronic device in the present application and the embodiments of the audio frame loss detection method and the training method of the frame loss detection model in the present application are based on the same inventive concept, and therefore the specific implementation of the embodiments can be referred to the foregoing implementation of the corresponding audio frame loss detection method and the training method of the frame loss detection model, and the repeated parts will not be described herein.

[0124] Further, corresponding to the audio frame loss detection method and the training method of the frame loss detection model described above, based on the same technical concept, one or more embodiments of the present application also provide a storage medium for storing computer executable instructions, in one specific embodiment, the storage medium can be a U disk, an optical disk, a hard disk, etc., and the computer executable instructions stored in the storage medium can implement the following process when executed by a processor:

[0125] obtaining target audio data to be detected; the target audio data includes a first silent segment;

[0126] inputting the target audio data into an audio feature extraction network of the frame loss detection model, performing audio feature extraction processing on the target audio data through the audio feature extraction network to obtain target audio features;

[0127] performing classification processing on the target audio features through a classification network of the frame loss detection model to obtain a frame loss detection result; the frame loss detection result represents whether the first silent segment has frame loss.

[0128] The computer executable instructions stored in the storage medium provided in one or more embodiments of the present application, when executed by a processor, when obtaining target audio data containing a first silent segment, input the target audio data into an audio feature extraction network of a frame loss detection model for audio feature extraction processing, and perform classification processing on the extracted target audio features through a classification network of the frame loss detection model to obtain a frame loss detection result representing whether the first silent segment has frame loss. In this process, since the frame loss detection processing is based on the frame loss detection model, and the frame loss detection model has the characteristics of high reliability, fast detection speed, and low cost, accurate, efficient, and low-cost detection of audio frame loss is achieved, and effective data basis can be provided for subsequent related audio processing.

[0129] In another specific embodiment, the storage medium can be a U disk, an optical disk, a hard disk, etc., and the computer executable instructions stored in the storage medium can implement the following process when executed by a processor:

[0130] obtaining a training sample set in a preset manner; the training sample set includes positive samples with lost frames and negative samples without lost frames;

[0131] performing iterative training processing on the network to be trained by using the training sample set to obtain a lost frame detection model;

[0132] The lost frame detection model includes an audio feature extraction network and a classification network; the audio feature extraction network is configured to perform audio feature extraction processing on input target audio data to obtain target audio features; and the classification network is configured to perform classification processing on the target audio features to obtain a lost frame detection result.

[0133] The storage medium stores computer executable instructions which, when executed by a processor, obtain a training sample set in a preset manner, and perform iterative training processing on a network to be trained by using the training sample set to obtain a lost frame detection model for performing lost frame detection processing on target audio data. Since the lost frame detection model has the characteristics of high reliability, fast detection speed, and low cost, training the lost frame detection model can realize accurate, efficient, and low-cost detection of audio lost frames, and can provide effective data basis for subsequent related audio processing.

[0134] It should be noted that the embodiments of the storage medium in the present application and the embodiments of the audio lost frame detection method and the training method of the lost frame detection model in the present application are based on the same inventive concept, and therefore the specific implementation of this embodiment can be referred to the foregoing corresponding audio lost frame detection method and the training method of the lost frame detection model. The repeated parts will not be described again.

[0135] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous or necessary.

[0136] In the 1990s, it was relatively easy to distinguish whether an improvement in a technology was a hardware improvement (e.g., an improvement in the circuit structure of a diode, transistor, switch, etc.) or a software improvement (an improvement in a method flow). However, as technology has evolved, many improvements in method flows today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flows into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using hardware entity modules. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming the PLD, rather than by ordering a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating integrated circuit chips, this programming is now mostly implemented using "logic compiler" software, which is similar to software compilers used in program development, and the original code to be compiled is written in a specific programming language, which is called a hardware description language (HDL), and there are many such languages, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit implementing the logical method flow can be easily obtained.

[0137] The controller can be implemented in any suitable way, e.g. the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, e.g. software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of controllers include but are not limited to the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to being implemented in pure computer readable program code form, the controller can perfectly well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered a hardware component, and the means comprised therein for performing the various functions can be considered structures within the hardware component. Alternatively, or even, the means for performing the various functions can be considered both a software module implementing the method and a structure within the hardware component.

[0138] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0139] For the sake of brevity, the above apparatuses are described in functional form in various units. Of course, the functions of each unit can be implemented in either software and / or hardware in the same or more software and / or hardware.

[0140] Those skilled in the art will appreciate that one or more embodiments of the application can provide a method, a system or a computer program product. Accordingly, one or more embodiments of the application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage devices, etc.) embodying computer readable program code thereon for use by or in connection with an instruction execution system.

[0141] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0142] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0143] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0144] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0145] The memory can include non-persistent memory and / or persistent memory, such as flash memory, read-only memory (ROM), and / or volatile or non-volatile random access memory (RAM), among others. The memory is an example of computer readable media.

[0146] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0147] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0148] One or more embodiments of the application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. One or more embodiments of the application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0149] Various embodiments in the present application are described in progressive manner, and the same or similar parts between various embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0150] The above merely provides the example of the present document and is not intended to limit the present document. For those skilled in the art, the present document can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present document shall be included in the scope of claims of the present document.

Claims

1. An audio frame loss detection method, characterized by, The method comprises: obtaining target audio data to be detected; the target audio data comprises a first silent segment; inputting the target audio data into an audio feature extraction network of a frame loss detection model, performing audio feature extraction processing on the target audio data through the audio feature extraction network, and obtaining target audio features; the target audio features are obtained based on multi-layer feature extraction processing of the target audio data by the audio feature extraction network; performing classification processing on the target audio features through a classification network of the frame loss detection model, and obtaining a frame loss detection result; the frame loss detection result represents whether the first silent segment has frame loss.

2. The method of claim 1, wherein, The duration of the first silent segment is less than a preset duration, and the obtaining of the target audio data to be detected comprises: obtaining first initial audio data; the duration of the first initial audio data is greater than the preset duration; performing voice activity detection processing on the first initial audio data, and obtaining at least one first silent segment in the first initial audio data; for each first silent segment, sequentially performing continuous extension of the first silent segment forward and / or backward in the first initial audio data according to time from front to back, and obtaining target audio data of the preset duration.

3. The method of claim 1, wherein, The audio feature extraction network comprises a first stride convolution layer, a second stride convolution layer, a third stride convolution layer, a fourth stride convolution layer, and a fifth stride convolution layer; the obtaining of the target audio features through the audio feature extraction network comprises: performing first audio feature extraction processing on each audio frame of the target audio data through the first stride convolution layer, and obtaining first audio features of each audio frame; performing second audio feature extraction processing on each first audio feature through the second stride convolution layer, and obtaining second audio features of each audio frame; performing third audio feature extraction processing on each second audio feature through the third stride convolution layer, and obtaining third audio features of each audio frame; performing fourth audio feature extraction processing on each third audio feature through the fourth stride convolution layer, and obtaining fourth audio features of each audio frame; performing fifth audio feature extraction processing on each fourth audio feature through the fifth stride convolution layer, and obtaining target audio features of each audio frame.

4. The method of claim 1, wherein, The classification network comprises a bidirectional long short-term memory network, an average pooling layer, and a full connection layer; the obtaining of the frame loss detection result through the classification network of the frame loss detection model comprises: extracting time sequence features of the target audio features through the bidirectional long short-term memory network; performing average pooling processing on the time sequence features through the average pooling layer, and obtaining average pooling features; performing conversion processing on the average pooling features, and obtaining converted features; performing classification processing on the converted features through the full connection layer, and obtaining the frame loss detection result.

5. The method of claim 1, wherein, The frame loss detection result represents that the first silent segment has frame loss, and the method further comprises: The start and end positions of the first mute segment are determined as the frame loss positions.

6. A method for training a frame loss detection model, the method comprising: The method comprises the steps of: acquiring a training sample set in a preset manner; the training sample set comprises positive samples with frame loss and negative samples without frame loss; the acquiring of the training sample set in the preset manner comprises: generating a plurality of positive samples with frame loss and a plurality of negative samples without frame loss based on the second initial audio data without frame loss; and generating the training sample set according to the positive samples and the negative samples; iteratively training a network to be trained by using the training sample set to obtain a frame loss detection model; the frame loss detection model comprises an audio feature extraction network and a classification network; the audio feature extraction network is used for audio feature extraction processing of input target audio data to obtain target audio features; and the classification network is used for classification processing of the target audio features to obtain frame loss detection results.

7. The method of claim 6, wherein, the acquiring of the training sample set in the preset manner comprises: performing speech activity detection processing on the second initial audio data to obtain at least one second mute segment in the second initial audio data; the second initial audio data is audio data without frame loss, and the duration of the second mute segment is within a preset duration range; determining target continuous audio data with a duration greater than a preset duration in the second initial audio data according to the second mute segment; generating a plurality of positive samples based on the target continuous audio data; generating a plurality of negative samples based on the second mute segment and the second initial audio data; generating a training sample set according to the positive samples and the negative samples.

8. The method of claim 7, wherein, the generating of the positive samples based on the target continuous audio data comprises: randomly extracting a plurality of candidate audio data with a preset duration from the target continuous audio data; randomly selecting a target duration from the preset duration range to generate frame loss audio data with the target duration; determining a target position in each candidate audio data for replacing the frame loss audio data; replacing the audio data at the target position with the frame loss audio data to obtain the positive sample.

9. The method of claim 7, wherein, the generating of the negative samples based on the second mute segment and the second initial audio data comprises: continuously extending the second mute segment forward and / or backward in the second initial audio data in a time sequence from front to back to obtain audio data with the preset duration; determining the audio data with the preset duration as the negative sample.

10. An audio frame loss concealment apparatus characterized by comprising: The method comprises the steps of: an acquisition module is configured to acquire target audio data to be detected; the target audio data comprises a first mute segment; an extraction module is configured to input the target audio data into an audio feature extraction network of a frame loss detection model, perform audio feature extraction processing of the target audio data by using the audio feature extraction network to obtain target audio features, and obtain the target audio features based on multi-layer feature extraction processing of the target audio data by using the audio feature extraction network. The classification module is configured to perform classification processing on the target audio feature by using a classification network of the frame loss detection model, to obtain a frame loss detection result. The frame loss detection result indicates whether the first silence segment has frame loss.

11. An electronic device, comprising: The method comprises: a processor; and a memory arranged to store computer executable instructions configured to be executed by the processor, the executable instructions comprising instructions for performing the steps of the method of any one of claims 1-5, or the executable instructions comprising instructions for performing the steps of the training method of any one of claims 6-9.

12. A storage medium, characterized by The storage medium is configured to store computer executable instructions, the executable instructions causing a computer to perform the method of any one of claims 1-5, or the executable instructions causing a computer to perform the training method of any one of claims 6-9.

Images