Speech punctuation detection method, apparatus, device, storage medium, and program product

Abstract

The application discloses a speech punctuation detection method, device, equipment, storage medium and program product. The method comprises the following steps: obtaining first speech basic data of a first speech and second speech basic data of a second speech; determining a second silence duration threshold according to the first speech basic data and the second speech basic data; determining a first punctuation detection result of the second speech according to a silence duration of the second speech and the second silence duration threshold, wherein the first punctuation detection result is used for indicating whether the second speech needs to be punctuated. The second speech basic data of the speech that needs to be punctuated and the first speech basic data of the speech that has been punctuated are used to dynamically adjust the silence duration threshold according to the needs, and then the speech punctuation detection is performed according to the second speech basic data and the adjusted silence duration threshold, so that the speech punctuation detection result is obtained, and the accuracy of the speech punctuation detection is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sentence detection, and in particular to a speech sentence detection method, device, equipment, storage medium and program product. BACKGROUND

[0002] In the process of recognizing user speech, a traditional sentence algorithm judges speech sentence by setting a fixed silence duration threshold. When the duration of the pause of the speech is greater than the silence duration threshold, it is judged as a sentence, and when it is less than the silence duration threshold, it is not judged as a sentence. It is found through research that the speech speed of different users is different when they speak, which is related to the environment, emotional state and content of the user, etc. The fixed silence duration threshold does not consider the speech speed and speaking duration of the speaker, which will lead to the inadaptability of the fixed silence duration threshold in certain situations, for example, in some cases less than the silence duration threshold, the sentence should be broken but is not broken, which may cause the accuracy of speech sentence detection to decrease. SUMMARY

[0003] The embodiments of the present application provide a speech sentence detection method, device, equipment, storage medium and program product, which dynamically adjusts the silence duration threshold according to the speech speed data and duration data of the speech to be detected and the speech speed data and duration data of the speech that has been detected, and the first silence duration threshold, obtains a second silence duration threshold, and then obtains a speech sentence detection result according to the silence duration of the speech to be detected and the second silence duration threshold, which is beneficial to improve the accuracy of speech sentence detection.

[0004] In a first aspect, the embodiments of the present application provide a speech sentence detection method, comprising:

[0005] obtaining first speech basic data of a first speech and second speech basic data of a second speech, wherein the first speech basic data comprises first speech duration data, first speech speed data and a first silence duration threshold, and the second speech basic data comprises second speech duration data and second speech speed data;

[0006] determining a second silence duration threshold according to the first speech duration data, the first speech speed data, the first silence duration threshold, the second speech duration data and the second speech speed data;

[0007] determining a first sentence detection result of the second speech according to the silence duration of the second speech and the second silence duration threshold, wherein the first sentence detection result is used to indicate whether the second speech needs to be broken.

[0008] In a second aspect, the embodiments of the present application provide a speech sentence detection device, comprising:

[0009] The acquisition module is configured to acquire first speech basic data of the first speech and second speech basic data of the second speech.

[0010] The processing module is configured to determine a second silence duration threshold according to the first speech basic data and the second speech basic data, and determine a first punctuation detection result of the second speech according to a silence duration of the second speech and the second silence duration threshold, where the first punctuation detection result is used to indicate whether the second speech needs to be punctuated.

[0011] In a third aspect, an embodiment of the present application provides a speech punctuation detection device, which comprises:

[0012] The memory, the processor, and a speech punctuation detection program stored in the memory and executable on the processor, where the speech punctuation detection program is configured to implement part or all of the steps described in any of the methods of the first aspect.

[0013] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, where the computer readable storage medium stores a speech punctuation detection program, and the speech punctuation detection program, when executed by a processor, implements part or all of the steps described in any of the methods of the first aspect.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, where the computer program product comprises a computer program, and the computer program is operable to cause a computer to perform part or all of the steps described in any of the methods of the first aspect of the present application. The computer program product can be a software installation package.

[0015] By implementing the embodiments of the present application, the server of the speech punctuation detection system first acquires first speech basic data of the first speech and second speech basic data of the second speech, the first speech basic data comprises first speech duration data, first speech speed data, and a first silence duration threshold, the second speech basic data comprises second speech duration data and second speech speed data, then determines a second silence duration threshold according to the first speech duration data, the first speech speed data, the first silence duration threshold, the second speech duration data, and the second speech speed data, and finally determines a first punctuation detection result of the second speech according to a silence duration of the second speech and the second silence duration threshold, where the first punctuation detection result is used to indicate whether the second speech needs to be punctuated. By dynamically adjusting the silence duration threshold according to the speed data and the duration data of the speech to be punctuated and the speed data and the duration data of the speech that has been punctuated and the first silence duration threshold, the second silence duration threshold is obtained, and then the speech punctuation detection result is obtained according to the silence duration of the speech to be punctuated and the second silence duration threshold, which is beneficial to improving the accuracy of speech punctuation detection. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background art, the accompanying drawings needed to be used in the embodiments of the present application or the background art will be described below.

[0017] Figure 1 is a schematic diagram of an architecture of a speech punctuation detection system provided by an embodiment of the present application;

[0018] Figure 2 is a flowchart of a speech punctuation detection method provided by an embodiment of the present application;

[0019] Figure 3 is a flowchart of another speech punctuation detection method provided by an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of a structure of a speech punctuation detection device provided by an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of a structure of a speech punctuation detection device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the personnel in the technical field better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. According to the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0023] The terms "first", "second", and "third" and the like in the specification of the present application and claims and drawings are used to distinguish different objects, and are not used to describe a particular order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0024] In this document, the term "embodiment" means that the specific features, structures or characteristics described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor is it independent or alternative to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0025] In the process of recognizing user speech, a conventional sentence breaking algorithm judges speech sentence breaking by setting a fixed silence duration threshold. When the duration of speech pause is greater than the silence duration threshold, it is judged as sentence breaking, and when it is less than the silence duration threshold, it is not sentence breaking. It is found through research that the speech speed of different users when speaking is different, which is related to the environment, emotional state and content of the user, and the fixed silence duration threshold does not consider the speech speed and speaking duration of the speaker, which will lead to the inapplicability of the fixed silence duration threshold in certain situations, for example, in some cases less than the silence duration threshold, it should be sentence breaking but is not, which may cause the accuracy of speech sentence breaking detection to decrease.

[0026] There are mainly the following methods for speech sentence breaking detection:

[0027] Silence duration counting: during the speech input process, the length of time of continuous silence is counted, and compared with a pre-set fixed threshold, when the continuous silence exceeds the pre-set fixed threshold, it is judged as speech sentence breaking.

[0028] Energy or power-based detection: by analyzing the energy or power change of the speech signal to judge whether to perform speech sentence breaking, when the energy or power drops below a pre-set fixed threshold for a period of time, it is considered that speech sentence breaking is needed.

[0029] Silence to noise ratio (SNR) detection: use the signal to noise ratio between environmental noise and speech signal to judge sentence breaking. When the signal to noise ratio is below a pre-set signal to noise ratio threshold for a period of time, it is considered that speech sentence breaking is needed.

[0030] It can be seen that the current methods for speech sentence breaking detection all need to compare the speech data to be detected with a pre-set threshold to obtain the speech sentence breaking detection result, but the threshold used in the current detection is a pre-set fixed threshold, which cannot be adjusted according to the actual application scenario (such as the speech speed and speaking duration of different speakers), leading to the decrease of the accuracy of speech sentence breaking detection.

[0031] To solve the above problems, the embodiment of the present application provides a speech punctuation detection method, device, equipment, storage medium and program product. A server of a speech punctuation detection system first acquires first speech basic data of a first speech and second speech basic data of a second speech. The first speech basic data includes first speech duration data, first speech speed data and a first silence duration threshold. The second speech basic data includes second speech duration data and second speech speed data. Then, the second silence duration threshold is determined according to the first speech duration data, the first speech speed data, the first silence duration threshold, the second speech duration data and the second speech speed data. Finally, the first punctuation detection result of the second speech is determined according to the silence duration of the second speech and the second silence duration threshold. The first punctuation detection result is used to indicate whether the second speech needs to be punctuated. By dynamically adjusting the silence duration threshold according to the speed data and duration data of the speech to be punctuated and the speed data and duration data of the speech that has been punctuated and the first silence duration threshold, the second silence duration threshold is obtained. Then, the speech punctuation detection result is obtained according to the silence duration of the speech to be punctuated and the second silence duration threshold, which is beneficial to improve the accuracy of speech punctuation detection.

[0032] The speech punctuation detection method, device, equipment, storage medium and program product provided by the embodiment of the present application can be applied to the speech punctuation detection system as shown in Figure 1 Figure 1 Figure 1 is an architecture schematic diagram of a speech punctuation detection system provided by the embodiment of the present application. The speech punctuation detection system 100 includes a terminal 101 and a server 102. The terminal 101 can communicate with the server 102 through a network. The terminal 101 refers to a device used by a user, such as a smart phone, a computer and the like. In the present scheme, the terminal 101 is mainly responsible for collecting an audio data set to be punctuated and sending it to the server 102 for processing. The user can interact with the system through the terminal 101 to set an initial silence duration threshold, upload a speech, select a text to be punctuated and the like. The server 102 refers to a remote computer used for processing a large amount of computing tasks and storing data. In the present scheme, the server 102 receives the audio data set from the terminal and calculates the second silence duration threshold in real time according to the speech basic data in the audio data set. Then, the speech punctuation detection is performed on the speech to be detected in the audio data set to obtain the speech punctuation detection result, and the speech punctuation detection result is returned to the terminal 101.

[0033] Based on this, the present application provides a speech punctuation detection method, device, equipment, storage medium and program product, which will be described in detail below with reference to the accompanying drawings.

[0034] Figure 2 Figure 2 ​​​​is a flowchart of a voice punctuation detection method provided by an embodiment of the present application, as shown in Figure 2 The method comprises the following steps:

[0035] S201, obtaining first voice basic data of a first voice and second voice basic data of a second voice.

[0036] The first voice basic data comprises first voice duration data, first voice speed data and a first silence duration threshold, and the second voice basic data comprises second voice duration data and second voice speed data.

[0037] The first voice duration data and the second voice duration data both represent the duration of a voice, the unit of the first voice duration data and the second voice duration data is second (s), the first voice speed data and the second voice speed data both represent the speed of a voice, the unit of the first voice speed data and the second voice speed data is word per second (word / s), for example, the first voice duration data is 2s, the second voice duration data is 4s, the first voice speed data is 2.5 word / s, and the second voice speed data is 3.25 word / s.

[0038] The first voice basic data of the first voice can be obtained from the last voice of the second voice, for example, the duration of the last voice of the second voice is 3s and the speed of the voice is 3 word / s, so the first voice duration data is 3s and the first voice speed data is 3 word / s, or the first voice basic data of the first voice can be obtained from the voice basic data of multiple voices before the second voice, for example, there are 10 voices before the second voice that have been punctuated, so the voice basic data of the 10 voices, including the voice duration data, the voice speed data and the silence duration threshold, are obtained, and then the first voice basic data is obtained by averaging all the data.

[0039] The silence duration threshold can be set by a user on the terminal 101 when the voice punctuation detection is first performed, for example, the user directly sets the silence duration threshold to 200ms on the terminal 101, or the silence duration threshold can be the average pause duration of the voice punctuation according to a large amount of experimental data, which is the silence duration threshold of the first sentence, and the adaptive silence duration thresholds of the remaining subsequent sentences are calculated according to the silence duration threshold of the first sentence.

[0040] S202, determining a second silence duration threshold according to the first voice duration data, the first voice speed data, the first silence duration threshold, the second voice duration data and the second voice speed data.

[0041] The second silence duration threshold can be determined by at least two of the first speech duration data, the first speech speed data, the second speech duration data, and the second speech speed data, and the first silence duration threshold.

[0042] The second silence duration threshold can be determined by the first speech duration data, the second speech duration data, and the first silence duration threshold, when only the influence of speech duration on speech punctuation detection is considered. Specifically, the calculation formula of the second silence duration threshold is as follows:

[0043]

[0044] Wherein, T1 is the second silence duration threshold, T0 is the first silence duration threshold, a is the weight factor, t1 is the second speech duration data, t0 is the first speech duration data.

[0045] The second silence duration threshold can be determined by the first speech speed data, the second speech speed data, and the first silence duration threshold, when only the influence of speech speed on speech punctuation detection is considered. Specifically, the calculation formula of the second silence duration threshold is as follows:

[0046]

[0047] Wherein, T1 is the second silence duration threshold, T0 is the first silence duration threshold, β is the weight factor, v1 is the second speech speed data, and v0 is the first speech speed data.

[0048] In one possible implementation, the second silence duration threshold is determined according to the first speech duration data, the first speech speed data, the first silence duration threshold, the second speech duration data, and the second speech speed data, comprising:

[0049] The first weight is determined according to the first weight balance factor, the second speech speed data, and the first speech speed data; the second weight is determined according to the first weight balance factor, the second speech duration data, and the first speech duration data; the target weight is determined according to the second weight balance factor, the first weight, and the second weight; and the second silence duration threshold is determined according to the target weight and the first silence duration threshold.

[0050] The first weight balance factor and the second weight balance factor are both set by the user on the terminal 101, and are constants. The specific values are not limited here.

[0051] The second silence duration threshold value can be determined by the first silence duration threshold value, the first speech duration data, the first speech speed data, the second speech duration data, and the second speech speed data.

[0052] Specifically, the first weight can be calculated by the first weight balance factor, the second speech speed data, and the first speech speed data through simple operation (combination of addition, subtraction, multiplication, and division) or linear weighting, exponential weighting, etc. The second weight can be calculated by the first weight balance factor, the second speech duration data, and the first speech duration data through simple operation or linear weighting, exponential weighting, etc. The target weight can be the sum of the second weight balance factor, the first weight, and the second weight. The second silence duration threshold value can be the product of the target weight and the first silence duration threshold value.

[0053] In another example, the first weight is determined by the first weight balance factor, the second speech speed data, and the first speech speed data. The second weight is determined by the second weight balance factor, the second speech duration data, and the first speech duration data. The target weight is determined according to the third weight balance factor, the first weight, and the second weight.

[0054] The first weight balance factor, the second weight balance factor, and the third weight balance factor can be set by a user on a terminal. The values of the first weight balance factor, the second weight balance factor, and the third weight balance factor are not limited herein.

[0055] It can be seen that in the present example, by setting multiple balance factors, the duration data and speed data of the speech that has undergone sentence breaking detection and the speech to be subjected to sentence breaking detection can be more accurately combined to determine the target weight. Then, the second silence duration threshold value is obtained by combining the target weight and the first silence duration threshold value, which is beneficial to provide a more accurate silence duration threshold value for subsequent speech sentence breaking detection.

[0056] In a possible implementation, the first weight is determined according to the first weight balance factor, the second speech speed data, and the first speech speed data, including:

[0057] The first difference value of the first speech speed data and the second speech speed data is obtained. The first quotient value of the first difference value and the first speech speed data is obtained. The second product of the first quotient value and the first weight balance factor is obtained. The second product is the first weight.

[0058] Specifically, the calculation formula of the first weight is as follows:

[0059]

[0060] Wherein, a is a first weight balance factor, v0 is first speech speed data, v1 is second speech speed data, and γ1 is a first weight.

[0061] It is found through research that the speech speed of different users is different when they speak. Whether a sentence with a pause length should be punctuated has a great relationship with the speech speed of the user narrating the sentence and the length of the sentence. The faster the speech speed and the longer the length, the higher the possibility of punctuating at the pause.

[0062] It can be seen that in the present example, by obtaining a first difference value between the first speech speed data and the second speech speed data, and combining the first difference value, the first speech speed data and the first weight balance factor, a dynamically changing first weight can be obtained. This is conducive to automatically adjusting the weight according to the actual speech situation to adapt to different speech punctuating detection scenarios.

[0063] In a possible implementation, determining a second weight according to the first weight balance factor, the second speech length data and the first speech length data comprises:

[0064] obtaining a second difference value of the first speech length data and the second speech length data; obtaining a second quotient value of the second difference value and the first speech length data; obtaining a third difference value of a first constant and the first weight balance factor; and obtaining a third product of the second quotient value and the third difference value, the third product being a second weight.

[0065] The first constant can be set by a user on a terminal, and the first constant can be 1. The value of the first constant is not limited herein.

[0066] Specifically, the calculation formula of the first weight is as follows:

[0067]

[0068] Wherein, a is a first weight balance factor, t0 is first speech length data, t1 is second speech length data, γ2 is a first weight, and C is a first constant.

[0069] It is found through research that the speech speed of different users is different when they speak. Whether a sentence with a pause length should be punctuated has a great relationship with the speech speed of the user narrating the sentence and the length of the sentence. The faster the speech speed and the longer the length, the higher the possibility of punctuating at the pause.

[0070] As can be seen in this example, by obtaining the first difference between the first speech rate data and the second speech rate data, and combining the first difference, the first speech rate data, and the first weight balancing factor, a dynamically changing first weight can be obtained. This is beneficial for automatically adjusting the weight according to the actual speech situation to adapt to different speech segmentation detection scenarios.

[0071] S203, determine the first sentence segmentation detection result of the second speech based on the silence duration of the second speech and the second silence duration threshold.

[0072] Specifically, the first sentence segmentation detection result of the second speech is determined by judging the relationship between the silence duration of the second speech and the second silence duration threshold.

[0073] In one possible implementation, determining the first sentence segmentation detection result of the second speech based on the silence duration of the second speech and the second silence duration threshold includes:

[0074] If the silence duration of the second speech is detected to be greater than the second silence duration threshold, the first sentence segmentation detection result of the second speech is determined to be that the second speech needs to be segmented.

[0075] If the silence duration of the second speech is not greater than the second silence duration threshold, the first sentence segmentation detection result of the second speech is determined to be that the second speech does not need to be segmented.

[0076] For example, the second silence duration threshold for a second speech segment, "The weather is nice today (pause), let's go for a spring outing," is set to 1 second. The system determines whether punctuation is needed based on the actual detected silence duration. If the detected silence duration (2 seconds) exceeds the target threshold (1 second), the first punctuation detection result for the second speech segment is determined to be "punctuation is needed." The system will then punctuate this segment into two parts: "The weather is nice today." (pause) "Let's go to the park."

[0077] As can be seen, in this example, sentence segmentation is performed by judging the silence duration and the silence duration threshold. The silence duration threshold can be dynamically adjusted according to different scenarios or user needs, which helps to improve the accuracy of speech segmentation detection.

[0078] In one possible implementation, after determining that the first sentence segmentation detection result of the second speech is that the second speech is not segmented, the method further includes:

[0079] According to a target number of speech frames before the pause of the second speech, a first speech frame set is obtained; according to a sum of powers of each speech frame in the first speech frame set, a first total power is obtained; according to a target number of speech frames after the pause of the second speech, a second speech frame set is obtained; according to a sum of powers of each speech frame in the second speech frame set, a second total power is obtained; according to the first total power, the second total power, and a power difference threshold, a second sentence break detection result is obtained, which is used to indicate whether the second speech needs to be subjected to a second sentence break detection.

[0080] Specifically, in determining the pause of the second speech, the following methods can be adopted:

[0081] Energy threshold detection: the speech signal is divided into frames, the energy of each frame is calculated, and if the energy of a certain continuous frame is lower than a set threshold, it can be considered as a potential pause point.

[0082] Short-time zero-crossing rate detection: the zero-crossing rate represents the number of times the signal crosses the 0 point from positive or negative, and if the zero-crossing rate between continuous frames is high and the duration is long, there may be a pause.

[0083] Short-time average amplitude difference (SAD) detection: whether there is a pause is determined by calculating the average value of the amplitude difference between adjacent speech frames, and if the amplitude difference exceeds a certain threshold and the duration is long, there may be a pause.

[0084] Speech activity detection based on an acoustic model: an acoustic model is constructed by using machine learning and deep learning technologies, different speech segments are classified and distinguished, and then speech activity and non-speech activity (silence, noise, etc.) are identified, and whether there is a pause is determined based on the acoustic model.

[0085] The target number can be set by a user on a terminal, and the target number is not limited herein, wherein the number of frames of the target number is not greater than the number of speech frames of the smallest number of the first speech before the pause of the second speech and the second speech after the pause of the second speech, for example, the number of speech frames of the first speech before the pause of the second speech is 20 frames, and the number of speech frames of the second speech after the pause of the second speech is 14 frames, and the target number is not greater than 14 frames.

[0086] When the first total power and the second total power are compared, the first total power can be the sum of powers of the target number of speech frames of the first speech, or can be the average value of powers of the target number of speech frames of the first speech, and the second total power can be the sum of powers of the target number of speech frames of the second speech, or can be the average value of powers of the target number of speech frames of the second speech, and the calculation method of the first total power and the second total power is not limited herein.

[0087] Specifically, the second sentence breaking detection result is obtained according to the first total power, the second total power and a power difference threshold, including: obtaining an absolute value of a fourth difference value of the first total power and the second total power; detecting that the absolute value of the fourth difference value is not greater than the power difference threshold, and determining that the second sentence breaking detection result is that the second speech does not need to be detected by second sentence breaking; detecting that the absolute value of the fourth difference value is greater than the power difference threshold, and determining that the second sentence breaking detection result is that the second speech needs to be detected by second sentence breaking.

[0088] The power difference threshold can be set by a user on a terminal, and is not limited herein.

[0089] In another example, the second sentence breaking detection result is obtained according to the first total power, the second total power and a power difference threshold, including: obtaining an absolute value of a fourth difference value of the first total power and the second total power; obtaining a first ratio of the absolute value of the fourth difference value and the first total power, detecting that the first ratio is not greater than a ratio threshold, and determining that the second sentence breaking detection result is that the second speech does not need to be detected by second sentence breaking; and detecting that the first ratio is greater than the ratio threshold, and determining that the second sentence breaking detection result is that the second speech needs to be detected by second sentence breaking.

[0090] The ratio threshold can be 5%, 10% or 15%, and the ratio threshold can be set by a user on a terminal, and is not limited herein.

[0091] For example, the second speech contains multiple pauses, and whether the second speech needs to be detected by second sentence breaking is determined according to an energy difference of speech frames before and after the pause. It is assumed that the target number is 5 frames, and the power difference threshold is 10 units. At a specific pause, 5 frames of the first speech before the pause and 5 frames of the second speech after the pause are selected; the first total power (before the pause) is obtained: the energies of the 5 frames are [15, 12, 14, 13, 16], and the first total power calculation result is (15+12+14+13+16) / 5=14; the second total power (after the pause) is obtained: the energies of the 5 frames are [20, 18, 22, 19, 21], and the second total power calculation result is (20+18+22+19+21) / 5=20; the energy difference is calculated: energy difference=second total power-first total power=20-14=6. The energy difference is less than the power difference threshold, and it is determined that the second speech does not need to be detected by second sentence breaking.

[0092] It can be seen that, in the example, by analyzing the energy difference of the speech frames before and after the pause, it can be more accurately determined whether further sentence breaking operation needs to be performed, thereby improving the sentence breaking accuracy.

[0093] In a possible implementation, after determining that the second punctuation detection result indicates that the second speech needs to be detected for the second time, the method further includes:

[0094] performing speech recognition on the second speech to obtain speech text data of the second speech; determining that the speech text data is located in a target text set, and determining that the second punctuation detection result of the second speech indicates that the second speech needs to be punctuated; determining that the speech text data is not located in the target text set, and determining that the second punctuation detection result of the second speech indicates that the second speech does not need to be punctuated.

[0095] In a possible implementation, after determining that the second punctuation detection result indicates that the second speech needs to be detected for the second time, the method for performing speech recognition on the second speech can be a traditional speech recognition method including a hidden Markov model (HMM) and dynamic time warping (DTW), a feature-based method including a mel-frequency cepstral coefficient (MFCC) and linear predictive coding (LPC), or a deep learning method including a convolutional neural network (CNN) and a recurrent neural network (RNN).

[0096] In a possible implementation, the target text set is set by a user on a terminal, and the target text set can include common mood words and summary words, for example, the target text set can be [in summary, as mentioned above, therefore].

[0097] In another example, the method for obtaining the target text set is one of the following methods:

[0098] Construction based on a corpus: analyzing an existing text corpus (such as books, articles, dialogue data, etc.) to extract common words and phrases, and generating a target text set suitable for a specific scenario or field.

[0099] Machine learning model training: using a machine learning algorithm to perform speech recognition on a large number of speeches that have been punctuated, and training according to obtained text data to identify important keywords and phrases in a specific field or theme, and automatically generating a target text set.

[0100] As can be seen, in this example, whether punctuation is needed is determined according to text obtained through speech recognition and a target text set defined by a user, which can avoid arbitrary judgment of speech punctuation only by using silence duration, and improve the accuracy of speech punctuation detection.

[0101] In a possible implementation, please refer to Figure 3 , Figure 3 is a flowchart of another speech punctuation detection method provided by an embodiment of the present application, as shown in Figure 3 The method includes the following steps:

[0102] S301, obtain a target voice.

[0103] In the process of obtaining the target voice, voice basic data of the target voice is obtained, and the voice basic data comprises voice duration data and voice speed data.

[0104] The voice duration data indicates the duration of the voice, and the unit of the voice duration data is second (s). The voice speed data indicates the speed of the voice, and the unit of the voice speed data is word per second (word / s). For example, the voice duration data is 4 s, and the voice speed data is 3.25 word / s.

[0105] S302, determine an adaptive silence duration threshold c according to the target voice, the speed of the previous sentence voice, and the voice duration.

[0106] The calculation formula of the silence duration threshold c is as follows:

[0107]

[0108] In the formula, T0 is the silence duration threshold of the previous sentence, v is the speed of the current sentence, v' is the speed of the previous sentence, t is the voice duration of the current sentence, t' is the voice duration of the previous sentence, and a is a weight balance factor in the range of [0, 1], which is used to adjust the weight between the speed and the duration. The value of a can be selected according to the experimental results according to the specific scene. For example, for the speaker whose speed is fast and slow, the value of a should not be too large to reduce the influence of the uneven speed on the result.

[0109] In the process of determining the punctuation of the first sentence, the self-defined silence duration threshold is used for determination. For example, the threshold can take the average pause duration of the voice punctuation as the silence duration threshold of the first sentence, that is, the punctuation of the first sentence is determined by judging whether the silence duration at the pause is greater than c. The adaptive silence duration thresholds of the remaining subsequent sentences are obtained according to the calculation formula of the silence duration threshold c.

[0110] For example, the speed of the first sentence is 2.5 word / s, the voice duration is 2 s, the silence duration threshold is 200 ms, the speed of the second sentence is 3.25 word / s, the voice duration is 4 s, and the parameter a is 0.9. Then, the silence duration threshold of the second sentence is 120 ms.

[0111] S303, compare the silence duration t of the target voice with the adaptive silence duration threshold c.

[0112] If the silence duration t of the target voice is greater than or equal to c, step S307 is performed. If the silence duration t of the target voice is less than c, step S304 is performed.

[0113] Exemplarily, the silence duration threshold is 120 ms, the silence duration t of the target voice is 200 ms, t > c, and then step S307 is performed.

[0114] S304, comparing the energy a of the first number of voice frames of the target voice with the energy b of the first number of voice frames of the next sentence voice.

[0115] Wherein, the energy a of the first number of voice frames before the pause of the target voice is compared with the energy b of the first number of voice frames after the pause of the target voice, when the absolute value of the difference between the energy a and the energy b is greater than the power difference threshold, step S305 is performed, and when the absolute value of the difference between the energy a and the energy b is not greater than the power difference threshold, step S308 is performed.

[0116] Wherein, the first number can be set by the user on the terminal, and the target number is not limited herein, wherein the frame number of the first number is not greater than the number of the smallest voice frames in the two sentences of voice before and after the pause of the target voice, for example, the voice frames of the first voice before the pause of the target voice are 20 frames, and the voice frames of the second voice after the pause of the target voice are 14 frames, then the first number is not greater than 14 frames.

[0117] Wherein, when the first total power and the second total power are compared, the first total power can be the sum of the power of the first number of voice frames of the first voice, or can be the average value of the power of the first number of voice frames of the first voice, the second total power can be the sum of the power of the first number of voice frames of the second voice, or can be the average value of the power of the first number of voice frames of the second voice, and the calculation method of the first total power and the second total power is not limited herein.

[0118] Exemplarily, 20 frames of voice signals of the voice before and after the pause of the target voice are taken for energy calculation, and it is assumed that the energy a of the previous sentence is 120 and the energy b of the next sentence is 130 after calculation.

[0119] S305, performing voice recognition on the target voice to obtain text data.

[0120] Wherein, the voice recognition method can be a traditional voice recognition method including hidden Markov model (HMM) and dynamic time warping (DTW), a feature-based method including mel frequency cepstral coefficient (MFCC) and linear predictive coding (LPC), or a deep learning method including convolutional neural network (CNN) and recurrent neural network (RNN).

[0121] S306, judging whether the text data is special text.

[0122] If the text data is detected as a special text, step S307 is performed; otherwise, step S308 is performed.

[0123] If the text data is detected as a special text, step S307 is performed; otherwise, step S308 is performed.

[0124] S307, punctuating the target speech.

[0125] S308, not punctuating, merging the target speech with the next speech to be detected, and re-performing speech punctuating detection.

[0126] It can be seen that in the present example, the silence duration threshold is dynamically adjusted according to the speech speed data and the duration data of the speech to be punctuated, the speech speed data and the duration data of the previous speech, and the silence duration threshold of the previous sentence, to obtain an adaptive silence duration threshold, and then the first speech punctuating detection result is obtained according to the silence duration of the speech to be punctuated and the adaptive silence duration threshold, and when the first speech punctuating detection result is not punctuated, the second speech punctuating detection is continued according to the energy of the speech frame and the text recognition result, which is beneficial to improve the accuracy of speech punctuating detection.

[0127] Please refer to Figure 4 , Figure 4 is a structural schematic diagram of a speech punctuating detection device provided by the present application, as shown in Figure 4 , the speech punctuating detection device 400 comprises:

[0128] The acquisition module 401 is configured to acquire first speech basic data of a first speech and second speech basic data of a second speech; the processing module 402 is configured to determine a second silence duration threshold according to the first speech basic data and the second speech basic data, and determine a first punctuating detection result of the second speech according to the silence duration of the second speech and the second silence duration threshold, wherein the first punctuating detection result is used to indicate whether the second speech needs to be punctuated.

[0129] In a possible implementation, in determining the second silence duration threshold according to the first speech duration data, the first speech speed data, the first silence duration threshold, the second speech duration data and the second speech speed data, the processing module 402 is specifically configured to: determine a first weight according to a first weight balance factor, the second speech speed data and the first speech speed data; determine a second weight according to the first weight balance factor, the second speech duration data and the first speech duration data; determine a target weight according to a second weight balance factor, the first weight and the second weight; and determine the second silence duration threshold according to the target weight and the first silence duration threshold, the second silence duration threshold being a first product of the target weight and the first silence duration threshold.

[0130] In a possible implementation, in determining the first weight according to a first weight balance factor, the second speech speed data and the first speech speed data, the processing module 402 is specifically configured to: obtain a first difference value of the first speech speed data and the second speech speed data; obtain a first quotient value of the first difference value and the first speech speed data; and obtain a second product of the first quotient value and the first weight balance factor, the second product being the first weight.

[0131] In a possible implementation, in determining the second weight according to the first weight balance factor, the second speech duration data and the first speech duration data, the processing module 402 is specifically configured to: obtain a second difference value of the first speech duration data and the second speech duration data; obtain a second quotient value of the second difference value and the first speech duration data; obtain a third difference value of a first constant and the first weight balance factor; and obtain a third product of the second quotient value and the third difference value, the third product being the second weight.

[0132] In a possible implementation, in determining the first punctuation detection result of the second speech according to the silence duration of the second speech and the second silence duration threshold, the processing module 402 is specifically configured to: in response to detecting that the silence duration of the second speech is greater than the second silence duration threshold, determine that the first punctuation detection result of the second speech is that the second speech needs to be punctuated; and in response to detecting that the silence duration of the second speech is not greater than the second silence duration threshold, determine that the first punctuation detection result of the second speech is that the second speech does not need to be punctuated.

[0133] In a possible implementation, after determining that the first punctuation detection result of the second speech is that the second speech is not punctuated, the processing module 402 is further configured to: obtain a first speech frame set according to a front target number of speech frames at the pause of the second speech; obtain a first total power according to a sum of powers of each speech frame in the first speech frame set; obtain a second speech frame set according to a rear target number of speech frames at the pause of the second speech; obtain a second total power according to a sum of powers of each speech frame in the second speech frame set; and obtain a second punctuation detection result according to the first total power, the second total power, and a power difference threshold, where the second punctuation detection result is used to indicate whether the second speech needs to be punctuated for a second time.

[0134] In a possible implementation, in the obtaining of the second punctuation detection result according to the first total power, the second total power, and the power difference threshold, the processing module 402 is specifically configured to: obtain an absolute value of a fourth difference value of the first total power and the second total power; determine, when the absolute value of the fourth difference value is not greater than the power difference threshold, that the second punctuation detection result is that the second speech does not need to be punctuated for a second time; and determine, when the absolute value of the fourth difference value is greater than the power difference threshold, that the second punctuation detection result is that the second speech needs to be punctuated for a second time.

[0135] In a possible implementation, after determining that the second punctuation detection result is that the second speech needs to be punctuated for a second time, the processing module is further configured to: perform character recognition on the second speech to obtain speech character data of the second speech; determine, when the speech character data is located in a target character set, that a second punctuation detection result of the second speech is that the second speech needs to be punctuated; and determine, when the speech character data is not located in the target character set, that the second punctuation detection result of the second speech is that the second speech does not need to be punctuated.

[0136] It is worth noting that specific functional implementations of the speech punctuation detection apparatus 400 are described above. Figure 2The description of the illustrated voice punctuation detection method, such as the acquisition module 401 is used to implement the related content of performing S201, the processing module 402 is used to implement the related content of performing S202-S203. The various units or modules in the voice punctuation detection device 400 can be respectively or all combined into one or several other units or modules to constitute, or some of the units or modules therein can also be further split into a plurality of units or modules with smaller functions to constitute, which can realize the same operation without affecting the implementation of the technical effects of the embodiments of the present application. The above-mentioned units or modules are divided according to the logical function, in actual application, the function of one unit (or module) is realized by a plurality of units (or modules), or the function of a plurality of units (or modules) is realized by one unit (or module).

[0137] According to the description of the above method embodiments and related device embodiments, please refer to Figure 5 , Figure 5 is a structural schematic diagram of a voice punctuation detection device provided by an embodiment of the present application, Figure 5 The voice punctuation detection device 500 illustrated includes a processor 501, a memory 502, a communication interface 503, and a bus 504. Among them, the processor 501, the memory 502, the communication interface 503 are communicated with each other through the bus 504.

[0138] Optionally, the memory 502 is a ROM, a static storage device, a dynamic storage device or a RAM.

[0139] The memory 502 can store executable program codes, when the executable program codes stored in the memory 502 are executed by the processor 501, the processor 501 and the communication interface 503 are used to execute Figure 2 The steps of the voice punctuation detection method of the illustrated embodiment.

[0140] The processor 501 adopts a general-purpose CPU, a microprocessor, an application-specific integrated circuit ASIC, a GPU or one or more integrated circuits, which is used to execute related programs to execute the voice punctuation detection method of the method embodiments of the present application.

[0141] The processor 501 can also be an integrated circuit chip having a processing capability of signals. In implementation, each step of the voice sentence breaking detection method of the present application can be completed by integrated logic circuits of hardware in the processor 501 or instructions in the form of software. Alternatively, the processor 501 is a general processor, a DSP, an ASIC, an FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor is a microprocessor or the processor is any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware coding of the processor for execution, or be executed by a combination of hardware and software modules in the coding processor. The optional software modules are located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage mediums in the art. The storage medium is located in the memory 502, and the processor 501 reads information in the memory 502, and combines hardware thereof to complete the functions required by the modules included in the voice sentence breaking detection device 400 in the embodiments of the present application, or execute the voice sentence breaking detection method in the method embodiments of the present application.

[0142] The communication interface 503 uses a transceiving related device such as but not limited to a transceiver.

[0143] The bus 504 can include a path for transmitting information between various components (for example, the memory 502, the processor 501, the communication interface 503) of the voice sentence breaking detection device 500.

[0144] It should be noted that although Figure 5 The voice sentence breaking detection device 500 shown only shows the memory, the processor, the communication interface, but in the specific implementation, those skilled in the art should understand that the voice sentence breaking detection device 500 also includes other devices necessary for normal operation. At the same time, according to the specific needs, those skilled in the art should understand that the voice sentence breaking detection device 500 can also include hardware devices for realizing other additional functions. In addition, those skilled in the art should understand that the voice sentence breaking detection device 500 can also only include devices necessary for the embodiments of the present application, and does not have to include all the devices shown in the voice sentence breaking detection device 500. Figure 5 The voice sentence breaking detection device 500 shown only shows the memory, the processor, the communication interface, but in the specific implementation, those skilled in the art should understand that the voice sentence breaking detection device 500 also includes other devices necessary for normal operation. At the same time, according to the specific needs, those skilled in the art should understand that the voice sentence breaking detection device 500 can also include hardware devices for realizing other additional functions. In addition, those skilled in the art should understand that the voice sentence breaking detection device 500 can also only include devices necessary for the embodiments of the present application, and does not have to include all the devices shown in the voice sentence breaking detection device 500.

[0145] The embodiments of the present application provide a computer readable storage medium, and the computer readable storage medium stores a computer program for electronic data exchange. The computer program includes execution instructions for executing part or all steps of any one of the voice sentence breaking detection methods as described in the voice sentence breaking method embodiments above. The computer includes an electronic terminal device.

[0146] The embodiment of the present application provides a computer program product, wherein the computer program product comprises a computer program, the computer program is operable to cause a computer to perform part or all of steps of any one of the voice punctuation detection methods described in the method embodiment, and the computer program product can be a software installation package.

[0147] It should be noted that, for any one of the foregoing voice punctuation detection method embodiments, in order to simply describe, the voice punctuation detection method is described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the present application.

[0148] The above describes the embodiments of the present application in detail, and the principles and implementation manners of the voice punctuation detection method, device, equipment, storage medium and program product are described by applying specific examples in this paper, and the above embodiment description is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the voice punctuation detection method, device, equipment, storage medium and program product, the specific implementation manner and application range will be changed, and according to the above, the content of the specification should not be understood as a limitation of the present application.

[0149] The present application is described with reference to the flowcharts and / or block diagrams of the method, hardware product and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device for realizing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The device for realizing the functions specified in one block or multiple blocks.

[0150] These computer program instructions can also be stored in a computer readable storage medium to cause the computer to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a manufactured product including instruction devices, which realize the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 Figure 1The memory can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0151] Although the present application is described herein in conjunction with various embodiments, those of ordinary skill in the art will appreciate that other variations of the disclosed embodiments are possible without departing from the spirit and scope of the application. In the claims, the word "comprising" does not exclude other elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The use of the word "including" as well as "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" does not exclude a plurality of elements or steps. The implementation of a having related elements in the independent claims enumerates possibilities where the claims are not limited to one of these only but are open to the implementation of a plurality of these in an interdependent manner.

[0152] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-described method embodiments of the voice punctuation detection method can be completed by programs instructing relevant hardware, and the programs can be stored in a computer-readable memory, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0153] It can be understood that products controlled or configured to execute the processing method of the flowchart described in the method embodiments of the voice punctuation detection method, such as the above-mentioned device and computer program product, all belong to the scope of the related products described in the present application.

[0154] Obviously, those of ordinary skill in the art can make various modifications and variations to the method, device, equipment, storage medium and program product of the voice punctuation detection method provided by the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A method of detecting a pause in speech, the method comprising: The method comprises: obtaining first speech basic data of a first speech and second speech basic data of a second speech, the first speech basic data comprising first speech duration data, first speech speed data and a first silence duration threshold, and the second speech basic data comprising second speech duration data and second speech speed data; determining a second silence duration threshold according to the first speech duration data, the first speech speed data, the first silence duration threshold, the second speech duration data and the second speech speed data; determining a first sentence breaking detection result of the second speech according to a silence duration of the second speech and the second silence duration threshold, the first sentence breaking detection result being used to indicate whether the second speech needs to be broken up.

2. The method of claim 1, wherein, The determining of the second silence duration threshold according to the first speech duration data, the first speech speed data, the first silence duration threshold, the second speech duration data and the second speech speed data comprises: determining a first weight according to a first weight balance factor, the second speech speed data and the first speech speed data; determining a second weight according to the first weight balance factor, the second speech duration data and the first speech duration data; determining a target weight according to a second weight balance factor, the first weight and the second weight; determining the second silence duration threshold according to the target weight and the first silence duration threshold, the second silence duration threshold being a first product of the target weight and the first silence duration threshold.

3. The method of claim 2, wherein, The determining of the first weight according to the first weight balance factor, the second speech speed data and the first speech speed data comprises: obtaining a first difference value of the first speech speed data and the second speech speed data; obtaining a first quotient value of the first difference value and the first speech speed data; obtaining a second product of the first quotient value and the first weight balance factor, the second product being the first weight.

4. The method of claim 2, wherein, The determining of the second weight according to the first weight balance factor, the second speech duration data and the first speech duration data comprises: obtaining a second difference value of the first speech duration data and the second speech duration data; obtaining a second quotient value of the second difference value and the first speech duration data; obtaining a third difference value of a first constant and the first weight balance factor; obtaining a third product of the second quotient value and the third difference value, the third product being the second weight.

5. The method of claim 1, wherein, The determining of the first sentence breaking detection result of the second speech according to the silence duration of the second speech and the second silence duration threshold comprises: detecting that the silence duration of the second speech is greater than the second silence duration threshold, and determining that the first sentence breaking detection result of the second speech is that the second speech needs to be broken up; detecting that the silence duration of the second speech is not greater than the second silence duration threshold, and determining that the first sentence breaking detection result of the second speech is that the second speech does not need to be broken up.

6. The method according to any one of claims 1 to 5, wherein, After the determining of the first sentence breaking detection result of the second speech as the second speech not being broken up, the method further comprises: obtaining a first speech frame set according to a front target number of speech frames at a pause of the second speech; obtaining a first total power according to a sum of powers of each speech frame in the first speech frame set; obtaining a second speech frame set according to a rear target number of speech frames at the pause of the second speech; obtaining a second total power according to a sum of powers of each speech frame in the second speech frame set; obtaining a second sentence detection result according to the first total power, the second total power and a power difference threshold, the second sentence detection result being used to indicate whether the second speech needs second sentence detection.

7. The method of claim 6, wherein, The obtaining a second sentence detection result according to the first total power, the second total power and a power difference threshold comprises: obtaining a fourth difference value of the first total power and the second total power; detecting that the fourth difference value is not greater than the power difference threshold, and determining that the second sentence detection result is that the second speech does not need second sentence detection; detecting that the fourth difference value is greater than the power difference threshold, and determining that the second sentence detection result is that the second speech needs second sentence detection.

8. The method of claim 7, wherein, After the determining that the second sentence detection result is that the second speech needs second sentence detection, the method further comprises: performing speech recognition on the second speech to obtain speech text data of the second speech; detecting that the speech text data is in a target text set, and determining that a second sentence detection result of the second speech is that the second speech needs sentence breaking; detecting that the speech text data is not in the target text set, and determining that the second sentence detection result of the second speech is that the second speech does not need sentence breaking.

9. A speech punctuation detection apparatus, characterized by The apparatus comprises: an obtaining module, configured to obtain first speech basic data of a first speech and second speech basic data of a second speech, the first speech basic data comprising first speech duration data, first speech speed data and a first silence duration threshold, and the second speech basic data comprising second speech duration data and second speech speed data; a processing module, configured to determine a second silence duration threshold according to the first speech duration data, the first speech speed data, the first silence duration threshold, the second speech duration data and the second speech speed data, and determine a first sentence detection result of the second speech according to a silence duration of the second speech and the second silence duration threshold, the first sentence detection result being used to indicate whether the second speech needs sentence breaking.

10. A speech punctuation detection device, characterized by The device comprises: a memory, a processor and executable program code stored on the memory and executable on the processor, the processor executing the executable program code to perform the steps of the speech sentence breaking detection method according to any one of claims 1-8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores executable program code, the executable program code comprising execution instructions for performing the steps of the speech sentence breaking detection method according to any one of claims 1-8.

12. A computer program product, characterised in that, The computer program product comprises a computer program for causing a computer to execute the steps of the voice sentence breaking detection method according to any one of claims 1-8.

Images