Audio endpoint detection method and device

Abstract

The purpose of the present application is to provide an endpoint detection method and device for audio, which obtains the target audio to be processed and its corresponding total duration; based on the total duration of the target audio, sequentially performs frequency domain transformation, feature extraction and effective feature filtering on the target audio to obtain the effective feature vector and sampling sequence number corresponding to each sampling moment in the target audio, and the absolute time difference between two adjacent sampling moments; based on the effective feature vector corresponding to each sampling moment, respectively predicts the category of each sampling moment and the relative deviation between each sampling moment and the moment of the true endpoint of the target audio; based on the absolute time difference and the sampling sequence number, category and relative deviation between the moment and the true endpoint of the target audio corresponding to each sampling moment, determines the moment of the starting endpoint and the moment of the ending endpoint of the target audio, thereby realizing accurate endpoint detection of audio with strong noise interference, with low computational complexity and high speed.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an audio endpoint detection method and device. Background Art

[0002] With the continuous development and advancement of technology, people's living standards are constantly improving, and the online world is greatly facilitating and serving our lives. In life, people have gradually switched from communicating through text input to communicating through voice input. However, due to the different environments in which people are in, the process of sending voice messages inevitably results in unclear voice, loud noise, or even no sound. Therefore, it is necessary to detect valid voice segments from the continuous voice stream after obtaining the voice segments.

[0003] For example, in the existing technology, valid speech segments can be detected from a continuous speech stream through speech endpoint detection. The speech endpoint detection includes two aspects: on the one hand, the starting point of the valid speech, that is, the front end point, is detected; on the other hand, the ending point of the valid speech, that is, the back end point, is detected. The purpose of the speech endpoint detection is to identify and eliminate long periods of silence or useless sound signals from the sound signal stream.

[0004] In the prior art, traditional speech endpoint detection methods can also be used. These methods specifically include speech signal preprocessing and endpoint detection. During speech signal preprocessing, the speech signal is a one-dimensional continuous function with time as the independent variable. The speech data processed by the computer is a time-ordered sequence of speech signal sample values. The magnitude of these sample values ​​also represents the energy of the speech signal at that sampling point. The energy of a sampling point is typically calculated as the square of the sample value. The energy of a speech segment containing N sampling points can be defined as the sum of the squares of each sample value. Thus, the energy of a speech segment is related to both the sample value and the number of sample points. To examine changes in speech energy, the speech signal is first segmented into fixed durations (e.g., 20 milliseconds). Each segment is called a frame, and each frame contains the same number of sample points. The energy of each frame is calculated as the sum of the energies of all corresponding sample points. During endpoint detection, if the energy values ​​of several consecutive frames in the first part of the audio are below a pre-specified energy threshold, and the energy values ​​of the following consecutive frames are greater than the threshold, the point where the speech energy value increases is the leading endpoint of the speech. Similarly, if the energy values ​​of several consecutive speech frames are high, followed by several frames with lower energy values ​​for a certain period of time, the point where the energy value decreases can be considered the end point of the speech. However, traditional speech endpoint detection methods often have a certain degree of noise in real-world scenarios, but the energy value is clearly not zero. Especially when there is background music, the basic energy value may be close to or even greater than that of human voices. However, it is desirable to filter out clips containing only background music. As a result, traditional speech endpoint detection methods cannot accurately detect endpoints in audio with strong noise interference.

[0005] Existing technologies also employ speech endpoint detection methods based on DNN (Deep Neural Network)-LSTM (Long Short-Term Memory). This DNN-LSTM-based method combines the DNN's expertise in nonlinear data transformation with the LSTM's expertise in time series analysis to perform binary classification (endpoint or non-endpoint) on audio samples to determine endpoint locations. However, because the audio sampling frequency is 200Hz, the LSTM model requires 200 convolution calculations per second, resulting in a significant computational burden and making it unsuitable for engineering applications.

[0006] Therefore, traditional existing detection methods cannot accurately detect endpoints of audio with strong noise interference; furthermore, the DNN-LSTM-based speech endpoint detection method has too much computational complexity and is slow. Summary of the Invention

[0007] One purpose of the present application is to provide an audio endpoint detection method and device, which solves the problem encountered by traditional voice endpoint detection when there is a lot of background noise in the audio, and cannot accurately detect the endpoint while reducing the amount of calculation to increase the speed.

[0008] According to one aspect of the present application, a method for detecting an audio endpoint is provided, wherein the method includes:

[0009] Get the target audio to be processed and its corresponding total duration;

[0010] Based on the total duration of the target audio, perform frequency domain transformation, feature extraction, and effective feature filtering on the target audio in sequence to obtain a valid feature vector and sampling sequence number corresponding to each sampling moment in the target audio, and an absolute time difference between two adjacent sampling moments;

[0011] Based on the valid feature vector corresponding to each sampling moment, respectively predicting the category of each sampling moment and the relative deviation between each sampling moment and the true endpoint of the target audio;

[0012] Based on the absolute time difference and the sampling sequence number, category and relative deviation between each sampling moment and the real endpoint moment of the target audio, the starting endpoint moment and the ending endpoint moment of the target audio are determined.

[0013] Furthermore, in the above method, based on the total duration of the target audio, the target audio is sequentially subjected to frequency domain transformation and feature extraction to obtain a feature vector and sampling sequence number corresponding to each sampling moment in the target audio, and an absolute time difference between two adjacent sampling moments, including:

[0014] Performing frequency domain transformation on the target audio to obtain transformed target audio;

[0015] Based on the total duration of the target audio, feature extraction is performed on the transformed target audio to obtain an initial feature vector and a sampling sequence number corresponding to each sampling moment in the target audio, and an absolute time difference between two adjacent sampling moments;

[0016] Effective feature filtering is performed on the initial feature vector corresponding to each sampling moment to obtain a valid feature vector corresponding to each sampling moment.

[0017] Furthermore, in the above method, the feature extraction of the transformed target audio based on the total duration of the target audio to obtain the initial feature vector and sampling sequence number corresponding to each sampling moment in the target audio, and the absolute time difference between two adjacent sampling moments, includes:

[0018] Based on the total duration of the target audio, input the transformed target audio into an encoder for feature extraction to obtain an initial feature vector and a sampling sequence number corresponding to each sampling moment in the target audio, and an absolute time difference between two adjacent sampling moments;

[0019] The encoder is composed of three convolutional neural networks and two pooling layers alternatingly.

[0020] Furthermore, in the above method, performing effective feature filtering on the initial feature vector corresponding to each sampling moment to obtain the effective feature vector corresponding to each sampling moment includes:

[0021] The initial feature vector corresponding to each sampling moment is input into the long short-term memory network for effective feature filtering to obtain the effective feature vector corresponding to each sampling moment.

[0022] Furthermore, in the above method, the predicting, based on the valid feature vector corresponding to each sampling moment, the category of each sampling moment and the relative deviation between each sampling moment and the true endpoint of the target audio, respectively, includes:

[0023] The valid feature vector corresponding to each sampling moment is input into the softmax logistic regression function to predict the category of each sampling moment respectively; at the same time,

[0024] The effective feature vector corresponding to each sampling moment is input into the prediction function for predicting the relative deviation between the current moment and the moment of the real endpoint, and the relative deviation between each sampling moment and the moment of the real endpoint of the target audio is predicted respectively.

[0025] Furthermore, in the above method, determining the start endpoint moment and the end endpoint moment of the target audio based on the absolute time difference and the sampling sequence number and category corresponding to each sampling moment and the relative deviation from the real endpoint moment of the target audio includes:

[0026] Obtaining a true deviation between each of the sampling moments and the moment of the true endpoint of the target audio based on the absolute time difference and the relative deviation between each of the sampling moments and the moment of the true endpoint of the target audio;

[0027] Based on the absolute time difference, the sampling sequence number corresponding to each sampling moment, the category and the true deviation, the starting endpoint moment and the ending endpoint moment of the target audio are determined from all sampling moments in the target audio.

[0028] Furthermore, in the above method, the categories of the sampling moments include: non-endpoints, starting endpoints, and ending endpoints.

[0029] According to another aspect of the present application, a non-volatile storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the processor implements the above-mentioned audio endpoint detection method.

[0030] According to another aspect of the present application, a device for an audio endpoint detection method is also provided, wherein the device includes:

[0031] one or more processors;

[0032] a computer-readable medium for storing one or more computer-readable instructions,

[0033] When the one or more computer-readable instructions are executed by the one or more processors, the one or more processors implement the above-mentioned audio endpoint detection method.

[0034] Compared with the prior art, the present application obtains the target audio to be processed and its corresponding total duration; based on the total duration of the target audio, performs frequency domain transformation, feature extraction and effective feature filtering on the target audio in sequence to obtain the effective feature vector and sampling sequence number corresponding to each sampling moment in the target audio, and the absolute time difference between two adjacent sampling moments; based on the effective feature vector corresponding to each of the sampling moments, predicts the category of each of the sampling moments and the relative deviation between each of the sampling moments and the moment of the real endpoint of the target audio; based on the absolute time difference and the sampling sequence number, category and relative deviation between the moment of the real endpoint of the target audio, determines the moment of the starting endpoint and the moment of the ending endpoint of the target audio, which not only achieves accurate endpoint detection of audio with strong noise interference, but also reduces the amount of calculation in the detection process, making the detection speed faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0036] Figure 1 A schematic diagram showing a flow chart of an audio endpoint detection method according to the present application;

[0037] Figure 2 A practical flow chart showing an audio endpoint detection method according to one aspect of the present application;

[0038] Figure 3A schematic diagram illustrating a flow chart of an encoder in an audio endpoint detection method according to one aspect of the present application in an actual application scenario;

[0039] Figure 4 A schematic diagram shows a flow chart of predicting effective feature vectors in an audio endpoint detection method according to one aspect of the present application in an actual application scenario.

[0040] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0041] The present application is described in further detail below with reference to the accompanying drawings.

[0042] In a typical configuration of the present application, the terminal, the device of the service network and the trusted party all include one or more processors (CPUs), input / output interfaces, network interfaces and memories.

[0043] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0044] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0045] like Figure 1 As shown, one aspect of the present application provides a flowchart of an audio endpoint detection method, wherein the method includes steps S11, S12, S13, and S14, specifically including the following steps:

[0046] Step S11 , obtaining target audio to be processed and its corresponding total duration. Here, the target audio may include but is not limited to voice, video, and a video containing voice and video.

[0047] Step S12: Based on the total duration of the target audio, the target audio is sequentially subjected to frequency domain transformation, feature extraction, and effective feature filtering to obtain the effective feature vector and sampling sequence number corresponding to each sampling moment in the target audio, and the absolute time difference between two adjacent sampling moments.

[0048] It should be noted that the sampling sequence number corresponding to each sampling moment is a unique identifier that represents the sampling sequence corresponding to different sampling moments when sampling the entire target audio. It has a temporal nature. For example, the sequence number of the first sampling moment is 1. When sampling is performed according to the timing of the target audio, the sequence number of the second sampling moment is 2, and so on, thereby determining the sampling sequence number corresponding to each sampling moment in the target audio. In a preferred embodiment of the present application, the sampling method is preferably equal sampling. Then, based on the total duration of the target audio and the sampling moments in the target audio, the absolute time difference between each two adjacent sampling moments can be determined.

[0049] For example, in a preferred embodiment of the present application, if the total duration of the target audio is 30 seconds, which is 0 seconds to 30 seconds respectively, after obtaining the target audio 1 with a total duration of 30 seconds, the target audio 1 is first subjected to frequency domain transformation, feature extraction and effective feature filtering in sequence to obtain 300 sampling moments, and 10 effective feature vectors per second, so the target audio 1 has a total of 300 effective feature vectors and corresponding 300 sampling numbers, that is, sampling moment: 0.1 second corresponding to effective feature vector 1 and sampling number 1, sampling moment: 0.2 second corresponding to effective feature vector 2 and sampling number 2, sampling moment: 0.3 second corresponding to effective feature vector 3 and sampling number 4. Sample number 3, ..., sampling time: valid feature vector 299 and sampling number 299 corresponding to 29.9 seconds and sampling time: valid feature vector 300 and sampling number 300 corresponding to 30 seconds; the valid feature vector corresponding to each sampling time corresponds to a unique sampling number with a time sequence relationship. Since in the preferred embodiment, the sampling method is preferably equal sampling, the absolute time difference between each adjacent two sampling times is 100 milliseconds, that is, 0.1 seconds, thereby realizing the quantization processing of the target audio and subdividing it into target audio information per millisecond, and the present application performs feature extraction and filtering on the overall target audio to ensure that the result is more complete.

[0050] Step S13 predicts the category of each sampling moment and the relative deviation between each sampling moment and the true endpoint of the target audio based on the valid feature vector corresponding to each sampling moment. The categories of the sampling moments include: non-endpoint, starting endpoint, and ending endpoint. In the embodiment of the present application, the category of the sampling moment can be accurately determined without further determination after determining whether it is an endpoint, thus breaking through the bottleneck of the prior art that can only determine whether it is an endpoint.

[0051] It should be noted that the relative deviation between each predicted sampling moment and the moment of the real endpoint of the target audio ranges from -1 to 1, and the real endpoint of the target audio is the real starting endpoint and the real ending endpoint of the target audio segment. For example, in a preferred embodiment of the present application, a 30-second target audio segment starting from 0 seconds to 30 seconds, the real starting endpoint is the 0th second, and the real ending endpoint is the 30th second. However, in different audios, the beginning of the audio segment is not necessarily the beginning of the audio segment. For example, in a video audio segment, there is no voice and video in the first 10 seconds, which is an invalid blank. Voice and video only start to enter after the 10th second. Then, in this video audio segment, although the real starting endpoint is the 0th second, the starting endpoint of this video audio segment is the 10th second. In order to avoid similar situations where invalid endpoints are found, it is ensured that the endpoints found are not the start and end moments of the video audio, but the start and end endpoints containing valid information about the video. In the present application, in step S14, the start endpoint moment and the end endpoint moment of the target audio are determined based on the absolute time difference and the sampling sequence number and category corresponding to each sampling moment and the relative deviation between each sampling moment and the moment of the real endpoint of the target audio.

[0052] Through the above steps S11 to S14, accurate and complete extraction of effective information of the target audio is achieved, and the effective feature vector corresponding to each sampling moment is predicted and calculated, so as to achieve accurate endpoint detection of audio with strong noise interference, while the calculation amount is low and simple, and the running speed is fast.

[0053] In a preferred embodiment of the present application, if the total duration of the target audio target audio is 30 seconds, which is 0 seconds to 30 seconds respectively, after obtaining the target audio 1 with a total duration of 30 seconds, the target audio 1 is first subjected to frequency domain change, feature extraction and effective feature filtering in sequence to obtain 300 sampling moments, and 10 effective feature vectors per second, so the target audio 1 has a total of 300 effective feature vectors and corresponding 300 sampling numbers, that is, sampling moment: effective feature vector 1 and sampling number 1 corresponding to 0.1 second, sampling moment: effective feature vector 2 and sampling number 2 corresponding to 0.2 second, sampling moment: effective feature vector 3 and sampling number 3 corresponding to 0.3 second, ..., sampling moment: effective feature vector 299 and sampling number 299 corresponding to 29.9 seconds and sampling moment: effective feature vector 300 and sampling number 300 corresponding to 30 seconds; the effective feature vector corresponding to each sampling moment has a time sequence relationship and Corresponding to a unique sampling serial number, since in a preferred embodiment, the sampling method is preferably equal sampling, the absolute time difference between each two adjacent sampling moments is 100 milliseconds, that is, 0.1 seconds, thereby realizing quantization processing of the target audio and subdividing it into target audio information per millisecond; then, the effective feature vectors corresponding to the 300 sampling moments are predicted respectively, and the predicted categories are extracted as non-endpoints, starting endpoints, and ending endpoints. If the 33rd sampling moment obtained is the starting endpoint of the target audio 1 and the 287th sampling moment is the ending endpoint of the target audio 1, then the starting endpoint moment and the ending endpoint moment of the target audio target audio can be determined based on the absolute time difference between the two adjacent sampling moments and the sampling serial number and category corresponding to each of the sampling moments and the relative deviation between each of the sampling moments and the moment of the real endpoint of the target audio, thereby realizing the detection of the endpoint of the target audio target audio.

[0054] Following the above embodiment of the present application, based on the total duration of the target audio, the frequency domain transformation and feature extraction of the target audio are performed in sequence to obtain the feature vector and sampling sequence number corresponding to each sampling moment in the target audio, and the absolute time difference between two adjacent sampling moments, including:

[0055] Performing frequency domain transformation on the target audio to obtain transformed target audio;

[0056] Based on the total duration of the target audio, feature extraction is performed on the transformed target audio to obtain an initial feature vector and a sampling sequence number corresponding to each sampling moment in the target audio, and an absolute time difference between two adjacent sampling moments;

[0057] Effective feature filtering is performed on the initial feature vector corresponding to each sampling moment to obtain a valid feature vector corresponding to each sampling moment.

[0058] It should be noted that the time domain and frequency domain are the basic properties of signals. Signals can be analyzed in a variety of ways, each providing a different perspective. The time domain describes the relationship between mathematical functions or physical signals and time; the frequency domain is a coordinate system used to describe the characteristics of signals in terms of frequency.

[0059] For example, in a preferred embodiment of the present application, if the total duration of the target audio is 60 seconds, which is 0 seconds to 60 seconds respectively, after obtaining the target audio 1 with a total duration of 60 seconds, the target audio 1 is sequentially subjected to frequency domain transformation and feature extraction to obtain 1200 sampling moments, and 20 initial feature vectors per second, so the target audio 1 has a total of 1200 initial feature vectors and corresponding 1200 sampling numbers, that is, sampling moment: initial feature vector 1 and sampling number 1 corresponding to 0.05 seconds, sampling moment: initial feature vector 2 and sampling number 2 corresponding to 0.1 seconds, sampling moment: initial feature vector 3 and sampling number 3 corresponding to 0.15 seconds, ..., sampling moment: initial feature vector 1199 and sampling number 1199 corresponding to 59.95 seconds and sampling moment: initial feature vector 1200 and sampling number 1200 corresponding to 60 seconds, and then the 1200 initial feature vectors are respectively subjected to effective feature extraction and invalid information filtering. As a result, 1200 valid feature vectors corresponding to the sampling moments are obtained, namely, valid feature vector 1 and sampling number 1 corresponding to sampling moment 0.05 seconds, valid feature vector 2 and sampling number 2 corresponding to sampling moment 0.1 seconds, valid feature vector 3 and sampling number 3 corresponding to sampling moment 0.15 seconds, ..., valid feature vector 1199 and sampling number 1199 corresponding to sampling moment 59.95 seconds and valid feature vector 1200 and sampling number 1200 corresponding to sampling moment 60 seconds. The valid feature vector corresponding to each sampling moment corresponds to a unique sampling number with a time sequence relationship. Since in the preferred embodiment, the sampling method is preferably equal sampling, the absolute time difference between each two adjacent sampling moments is 50 milliseconds, that is, 0.05 seconds, which realizes advanced processing of the target audio with a large background of strong noise, presents clearer and more accurate target audio information, and provides a clear and complete basis for subsequent operations.

[0060] Following the above embodiment of the present application, the feature extraction of the transformed target audio based on the total duration of the target audio is performed to obtain the initial feature vector and sampling sequence number corresponding to each sampling moment in the target audio, and the absolute time difference between two adjacent sampling moments, including:

[0061] Based on the total duration of the target audio, the transformed target audio is input into an encoder for feature extraction to obtain an initial feature vector and sampling sequence number corresponding to each sampling moment in the target audio, and the absolute time difference between two adjacent sampling moments; wherein the encoder is composed of three convolutional neural networks and two pooling layers alternatingly.

[0062] It should be noted that if Figure 3 The encoder consists of three CNN (Convolutional Neural Networks) convolution modules and two Pooling (pooling layer) pooling modules alternating with each other. The specific parameters of each module are as follows:

[0063] CNN1: 3*3 convolution kernel, stride 1, 32 output channels, ReLU activation function, no padding.

[0064] Pooling1: Use the max pooling kernel with a stride of 2.

[0065] CNN2: 3*3 convolution kernel, stride 1, 64 output channels, ReLU activation function, no padding.

[0066] Pooling1: Use the max pooling kernel with a stride of 2.

[0067] CNN1: 5*5 convolution kernel, stride 1, 128 output channels, ReLU activation function, no padding.

[0068] In a preferred embodiment of the present application, the target audio 1 is transformed in the frequency domain and the feature number is 40. 30 milliseconds is used for framing, and the frame shift is 10 milliseconds. Therefore, the number of samples per second is 100. After two Poolings and three CNNs, the output shape per second is [20, 5, 128]. The reshape function is then used to merge the last two dimensions into one channel to obtain an output with a shape of [20, 640], thereby obtaining 20 feature vectors per second, each vector consisting of 640 eigenvalues. An encoder composed of three convolutional neural networks and two pooling layers is used to extract the features of the target audio. The target audio in a strong noise environment is processed to generate the corresponding feature vector at each moment, thereby realizing the conversion of the target audio into a feature vector.

[0069] Following the above embodiment of the present application, the effective feature filtering is performed on the initial feature vector corresponding to each sampling moment to obtain the effective feature vector corresponding to each sampling moment, including:

[0070] The initial feature vector corresponding to each sampling moment is input into the long short-term memory network for effective feature filtering to obtain the effective feature vector corresponding to each sampling moment.

[0071] Here, after the initial feature vector is input into the long short-term memory network, effective features are extracted and invalid information is deleted, so as to obtain a valid feature vector that can represent the corresponding sampling moment.

[0072] For example, for target audio under a strong background of strong noise, after frequency domain transformation and feature extraction, several initial feature vectors are obtained. At this time, each initial feature vector still contains information interfered by strong noise. At this time, the initial feature vector of each sampling moment is input into the long short-term memory network to extract the effective information of each sampling moment and delete the invalid noise, so as to obtain the effective feature vector corresponding to each sampling moment. Further filtering is performed on the initial feature vector of the original target audio, so that the effective feature vector corresponding to each sampling moment is more credible and effective, so that the category and relative deviation corresponding to each sampling moment predicted based on the effective feature vector corresponding to each sampling moment are more accurate.

[0073] Following the above embodiment of the present application, the method of predicting the category of each sampling moment and the relative deviation between each sampling moment and the true endpoint of the target audio based on the valid feature vector corresponding to each sampling moment includes:

[0074] Inputting the valid feature vector corresponding to each sampling moment into the softmax logistic regression function to predict the category of each sampling moment respectively;

[0075] At the same time, the effective feature vector corresponding to each sampling moment is input into the prediction function for predicting the relative deviation between the current moment and the moment of the real endpoint, and the relative deviation between each sampling moment and the moment of the real endpoint of the target audio is predicted respectively.

[0076] like Figure 4 As shown in Figure 2, the effective feature vector output by the long short-term memory network module is set to Y(t), and the effective feature vectors at all sampling moments, that is, Y(t), are used as input. The model is divided into two branches, one branch predicts the category of the current frame, and the other branch predicts the deviation process between the current frame and the true endpoint. Both branches are composed of fully connected layers with the following parameters:

[0077] FC1: The number of output channels is 3, and the activation function is the softmax function, which is used to predict the category of each sampling moment.

[0078] FC2: The number of output channels is 1, and there is no activation function. It is used to predict the relative deviation between each sampling moment and the moment of the true endpoint of the target audio.

[0079] For example, the target audio 1 is transformed in the frequency domain, features are extracted, and effective features are filtered to generate 100 sampling moments, where each sampling moment corresponds to a valid feature vector and a sampling sequence number, and the effective feature vectors of the 100 sampling moments are predicted, that is, as Y(t), where t1, 2, 3, ..., 99 and 100 are input, and the category of each of the sampling moments is predicted through the softmax logistic regression function; at the same time, a prediction function is used to predict the relative deviation between the current moment and the moment of the real endpoint, and the relative deviation between each sampling moment and the moment of the real endpoint of the target audio is predicted. The relative deviation between them is obtained, and the category of the first sampling moment is: non-endpoint, and the relative deviation is: 0; the category of the second sampling moment is: non-endpoint, and the relative deviation is: 0, ..., the category of the 100th sampling moment is: termination endpoint, and the relative deviation is -0.3. Obviously, it is easy to find the predicted category as the starting endpoint or the termination endpoint, as well as the relative deviation corresponding to the starting endpoint or the relative deviation corresponding to the termination endpoint from the 100 sampling moments, which realizes the classification of the prediction results, and easily filters out the endpoints of the target audio. At the same time, the relative deviation data of the endpoints of the target audio are obtained, which greatly speeds up the output of the prediction results.

[0080] Following the above embodiment of the present application, the determining of the start endpoint and the end endpoint of the target audio based on the absolute time difference and the sampling sequence number and category corresponding to each sampling moment and the relative deviation from the real endpoint of the target audio includes:

[0081] Based on the absolute time difference and the relative deviation between each sampling moment and the moment of the real endpoint of the target audio, the real deviation between each sampling moment and the moment of the real endpoint of the target audio is obtained; based on the absolute time difference, the sampling sequence number, category and the real deviation corresponding to each sampling moment, the starting endpoint moment and the ending endpoint moment of the target audio are determined from all sampling moments in the target audio.

[0082] It should be noted that the true deviation = absolute time difference * relative deviation between each sampling moment and the true endpoint of the target audio.

[0083] In a preferred embodiment of the present application, if the total duration of the target audio 3 is 60 seconds, then after frequency domain transformation, feature extraction, and effective feature filtering of the acquired target audio 3 with a total duration of 60 seconds, 1200 sampling moments are obtained. 20 effective feature vectors are output per second, i.e., there are 20 sampling points per second, and the absolute time difference between adjacent sampling points is 50 milliseconds. The effective feature vectors in the 1200 sampling moments are predicted. If the predicted category of the 33rd sampling point is the starting endpoint, the relative deviation of the prediction is 0. 5, so the true deviation is 50*0.5=25 milliseconds, and the starting endpoint time is 33*50+25=1675 milliseconds; the predicted category of the 1160th point in the sampling moment is the ending endpoint, and the predicted relative deviation is 0.75, so the true deviation is 50*0.75=37.5 milliseconds, and the ending endpoint time is 1160*50+37.5=58037.5 milliseconds. It can be seen that the starting endpoint of the target audio 3 with a total length of 60 seconds is at 1675 milliseconds, and the ending endpoint is at 58037.5 milliseconds.

[0084] Following the above embodiment of the present application, the categories of the sampling moments include: non-endpoints, starting endpoints, and ending endpoints.

[0085] According to another aspect of the present application, a non-volatile storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the processor implements the above-mentioned audio endpoint detection method.

[0086] According to another aspect of the present application, a device for an audio endpoint detection method is also provided, wherein the device includes:

[0087] one or more processors;

[0088] a computer-readable medium for storing one or more computer-readable instructions,

[0089] When the one or more computer-readable instructions are executed by the one or more processors, the one or more processors implement an audio endpoint detection method of the device as described above.

[0090] Here, for the detailed contents of each embodiment of the device of the audio endpoint detection method, please refer to the corresponding part of the embodiment of the audio endpoint detection method of the above-mentioned device, which will not be repeated here.

[0091] In summary, the present application obtains the target audio to be processed and its corresponding total duration; based on the total duration of the target audio, performs frequency domain transformation, feature extraction and effective feature filtering on the target audio in sequence to obtain the effective feature vector and sampling sequence number corresponding to each sampling moment in the target audio, and the absolute time difference between two adjacent sampling moments; based on the effective feature vector corresponding to each of the sampling moments, predicts the category of each of the sampling moments and the relative deviation between each of the sampling moments and the moment of the real endpoint of the target audio; based on the absolute time difference and the sampling sequence number, category and relative deviation between the moment of the real endpoint of the target audio corresponding to each of the sampling moments, determines the moment of the starting endpoint and the moment of the ending endpoint of the target audio, which not only realizes accurate endpoint detection of audio with strong noise interference, but also reduces the amount of calculation in the detection process, making the detection speed faster.

[0092] It should be noted that the application can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the application can be executed by a processor to realize the steps or functions described above. Similarly, the software program of the application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the application can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0093] In addition, a part of the present application may be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. The program instructions for calling the method of the present application may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-carrying medium, and / or stored in a working memory of a computer device that runs according to the program instructions. Here, according to an embodiment of the present application, a device is included, which includes a memory for storing computer program instructions and a processor for executing program instructions, wherein, when the computer program instructions are executed by the processor, the device is triggered to run the method and / or technical solution based on the aforementioned multiple embodiments of the present application.

[0094] It is obvious to those skilled in the art that the present application is not limited to the details of the above-mentioned exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or basic characteristics of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. Words such as first and second are used to indicate names and do not indicate any particular order.

Claims

1. A method for detecting an endpoint of an audio signal, wherein: The method comprises: Get the target audio to be processed and its corresponding total duration; Based on the total duration of the target audio, the target audio is sequentially subjected to frequency domain transformation, feature extraction, and effective feature filtering to obtain an effective feature vector and sampling sequence number corresponding to each sampling moment in the target audio, and the absolute time difference between two adjacent sampling moments; based on the effective feature vector corresponding to each sampling moment, the category of each sampling moment and the relative deviation between each sampling moment and the true endpoint of the target audio are predicted respectively; Determine the start endpoint and the end endpoint of the target audio based on the absolute time difference and the sampling sequence number and category corresponding to each sampling moment and the relative deviation from the real endpoint of the target audio; Among them, based on the effective feature vector corresponding to each sampling moment, the category of each sampling moment and the relative deviation between each sampling moment and the moment of the real endpoint of the target audio are predicted respectively, including: inputting the effective feature vector corresponding to each sampling moment into the softmax logistic regression function, and predicting the category of each sampling moment respectively; at the same time, inputting the effective feature vector corresponding to each sampling moment into the prediction function for predicting the relative deviation between the current moment and the moment of the real endpoint, and predicting the relative deviation between each sampling moment and the moment of the real endpoint of the target audio respectively.

2. The method according to claim 1, wherein The method of performing frequency domain transformation and feature extraction on the target audio in sequence based on the total duration of the target audio to obtain a feature vector and a sampling sequence number corresponding to each sampling moment in the target audio, and an absolute time difference between two adjacent sampling moments, includes: Performing frequency domain transformation on the target audio to obtain transformed target audio; Based on the total duration of the target audio, feature extraction is performed on the transformed target audio to obtain an initial feature vector and a sampling sequence number corresponding to each sampling moment in the target audio, and an absolute time difference between two adjacent sampling moments; Effective feature filtering is performed on the initial feature vector corresponding to each sampling moment to obtain a valid feature vector corresponding to each sampling moment.

3. The method according to claim 2, wherein: The method of performing feature extraction on the transformed target audio based on the total duration of the target audio to obtain an initial feature vector and a sampling sequence number corresponding to each sampling moment in the target audio, and an absolute time difference between two adjacent sampling moments, includes: Based on the total duration of the target audio, input the transformed target audio into an encoder for feature extraction to obtain an initial feature vector and a sampling sequence number corresponding to each sampling moment in the target audio, and an absolute time difference between two adjacent sampling moments; The encoder is composed of three convolutional neural networks and two pooling layers alternatingly.

4. The method according to claim 2, wherein: The performing effective feature filtering on the initial feature vector corresponding to each sampling moment to obtain the effective feature vector corresponding to each sampling moment includes: The initial feature vector corresponding to each sampling moment is input into the long short-term memory network for effective feature filtering to obtain the effective feature vector corresponding to each sampling moment.

5. The method according to claim 1, wherein determining the start endpoint time and the end endpoint time of the target audio based on the absolute time difference and the sampling sequence number and category corresponding to each sampling moment and the relative deviation from the real endpoint time of the target audio comprises: Obtaining a true deviation between each of the sampling moments and the true endpoint of the target audio based on the absolute time difference and the relative deviation between each of the sampling moments and the true endpoint of the target audio; Based on the absolute time difference, the sampling sequence number corresponding to each sampling moment, the category and the true deviation, the starting endpoint moment and the ending endpoint moment of the target audio are determined from all sampling moments in the target audio.

6. The method according to any one of claims 1 to 5, wherein The categories of the sampling moments include: non-endpoint, starting endpoint and ending endpoint. 7 . A non-volatile storage medium having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executable by a processor, the processor is caused to implement the method according to any one of claims 1 to 6.

8. An audio endpoint detection device, wherein: The device includes: one or more processors; a computer-readable medium for storing one or more computer-readable instructions, When the one or more computer-readable instructions are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

Images