A method for regulatory speech segmentation based on speech recognition and end-point detection

Abstract

The application provides a regulation voice segmentation method based on speech recognition and endpoint detection, which is applied to air traffic control voice audio stream segmentation, and comprises the following steps: step 1, constructing a punctuation model based on speech recognition and a speech endpoint detection model; step 2, using the punctuation model based on speech recognition to recognize the audio data stream of the regulation voice, and outputting the corresponding text and sentence end identifier of the audio data stream; step 3, using the speech endpoint detection model to judge the speech starting point and ending point contained in the audio data stream of the regulation voice; step 4, segmenting the audio data stream of the regulation voice into audio segments; and step 5, applying the audio segments as data materials to the speech recognition process of the air traffic control system. Through the combination of speech recognition and endpoint detection, the application realizes the automatic segmentation of the air traffic control voice audio stream, and improves the accuracy and efficiency of the segmentation.

Description

Technical Field

[0001] This invention relates to a method for controlling speech segmentation, and more particularly to a method for controlling speech segmentation based on speech recognition and endpoint detection. Background Technology

[0002] According to IATA forecasts, flight volume will increase significantly over the next decade, placing higher demands on the safety and efficiency of air traffic management. To address aviation safety incidents caused by controller error, current methods primarily focus on enhancing surface monitoring equipment, using devices such as surface surveillance radar and multi-point positioning system sensors to prevent conflicts. Simultaneously, more advanced solutions based on artificial intelligence technology have been proposed. For example, speech recognition technology can be used to identify the audio stream of air-to-ground communications, convert it into textual control instructions, and then extract information from the text, transforming it into a computer-readable structured form to achieve an understanding of control semantics. This approach can improve the accuracy and efficiency of controller work, providing a more reliable solution for air traffic management.

[0003] To improve the accuracy of speech recognition algorithms and provide reliable contextual information for speech recognition models, speech segmentation algorithms are needed to divide the continuous audio stream into speech segments before speech recognition. Traditional speech segmentation methods use three types of features: 1. Energy features; 2. Frequency domain features; 3. Harmonic features. The most commonly used method is the dual-threshold method based on short-time energy and short-time zero-crossing rate. The principle is that vowels in Mandarin have higher energy, so vowels can be found from short-time energy. Consonants are consonants with higher frequencies and correspondingly higher short-time zero-crossing rates. Therefore, finding consonants and vowels using these two characteristics is equivalent to finding complete Mandarin syllables. However, traditional methods only use the low-level sound features of the audio stream, which can easily segment a complete sentence into multiple segments due to short pauses during speech, reducing the accuracy of speech segmentation and affecting the accuracy of subsequent speech recognition and control intent analysis results. Summary of the Invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide a regulated speech segmentation method based on speech recognition and endpoint detection, which addresses the shortcomings of the existing technology.

[0005] To address the aforementioned technical problems, this invention discloses a method for regulated speech segmentation based on speech recognition and endpoint detection, comprising the following steps:

[0006] Step 1: Construct a sentence segmentation model and a speech endpoint detection model based on speech recognition;

[0007] Step 2: Using a speech recognition-based sentence segmentation model, identify the audio data stream of the controlled speech and output the text and sentence end identifier corresponding to the audio data stream;

[0008] Step 3: Use the speech endpoint detection model to determine the speech start and end points contained in the audio data stream of the regulated speech;

[0009] Step 4: Based on the results returned in Step 2 and Step 3, the audio data stream of the controlled speech is segmented into audio segments;

[0010] Step 5: Use the above audio clips as data material in the speech recognition process of the air traffic control system.

[0011] Furthermore, the construction of the speech recognition-based sentence segmentation model and endpoint detection model described in step 1 includes the following steps:

[0012] Step 1-1: Design a speech recognition neural network;

[0013] Step 1-2: Construct a set of regulated speech samples, denoted as training set X = {X1, X2, ..., X...} N}, where N is the total number of samples, X1, X2, ..., X... N These are the 1st, 2nd, ..., Nth training samples, i.e., the Nth regulated speech samples;

[0014] Steps 1-3: Manually annotate the regulated speech sample set from Steps 1-2 to obtain the regulated speech annotation set for training, denoted as the first annotation set Y = {Y1, Y2, ..., Y}. N}, where Y1, Y2, ..., Y N These are training samples X1, X2, ..., X from the set of controlled speech samples. N Corresponding regulated speech text annotations;

[0015] Steps 1-4: Modify the first annotation set Y by adding a sentence end identifier at the end of each regulated speech-script annotation, resulting in the second annotation set Y′ = {Y′1, Y′2, ..., Y′}. N};

[0016] Steps 1-5: Train the speech recognition neural network designed in Step 1-1 using the training set X and the second annotation set Y′. Use the training set as the input sequence, compare the output of the speech recognition neural network with the manual annotation, and optimize the parameters of the speech recognition neural network according to the connection time classification loss function to obtain the sentence segmentation model based on speech recognition, that is, the optimized speech recognition neural network.

[0017] Steps 1-6: Constructing an endpoint detection model: Calculate the short-time energy curve and short-time zero-crossing rate curve of the audio data stream of the controlled speech. Set judgment thresholds T1 and T2 based on the short-time energy and short-time zero-crossing rate of the non-speech phase in the audio data stream. When the short-time energy of the non-speech phase is greater than the threshold T1, it is determined that the current moment is the beginning of the speech segment. When the short-time zero-crossing rate is less than the threshold T2, it is determined that the current moment is the end of the speech segment.

[0018] Furthermore, the speech recognition neural network described in step 1-1 specifically includes the following structure:

[0019] The speech recognition neural network consists of a speech feature coding layer and a recognition result output layer. The speech feature coding layer contains 3 layers of convolutional neural network and 12 layers of Transformer neural network, and the recognition result output layer contains 1 layer of fully connected neural network.

[0020] The speech recognition neural network converts the audio data stream into a text data stream by recognizing and outputting a text according to the ratio of audio data with a preset frame length.

[0021] Furthermore, the addition of a sentence-end identifier at the end of each regulated speech-script annotation, as described in steps 1-4, specifically includes:

[0022] For the i-th controlled speech text annotation Y i ="y1y2...y k ", where k represents the length of the annotation, y k Adding a sentence-ending identifier # will change the annotation to Y. i ′=″y1y2...y k #″.

[0023] Furthermore, the short-time energy in the audio data stream without speech phases mentioned in steps 1-6 is calculated using the following method:

[0024]

[0025] Among them, E j Let x represent the short-time energy of the j-th audio frame in the audio data stream, M represent the number of audio samples contained within an audio frame of a preset frame length, and x represent the short-time energy of the j-th audio frame in the audio data stream. n This represents the waveform value of the nth audio sampling point.

[0026] Furthermore, the short-time zero-crossing rate mentioned in steps 1-6 is calculated using the following method:

[0027]

[0028] Among them, Z pThe zero-crossing rate of the p-th audio frame in the audio data stream is represented by x, where M represents the number of audio samples contained within an audio frame of a preset frame length. n Let sgn[x] represent the waveform value of the nth audio sampling point, where sgn[x] is the sign function, as follows:

[0029]

[0030] Furthermore, the short-time energy curves described in steps 1-6 are as follows:

[0031] Set a window with a size equal to the preset frame length and a step size smaller than the preset frame length. Use this window to slide along the audio data stream and calculate the short-time energy within each window to obtain the short-time energy curve with a sampling interval of the step size.

[0032] Furthermore, the short-time zero-crossing rate curves mentioned in steps 1-6 are as follows:

[0033] Set a window with a size equal to the preset frame length and a step size smaller than the preset frame length. Use this window to slide along the audio data stream and calculate the short-time zero-crossing rate within each window to obtain a short-time zero-crossing rate curve with a sampling interval of the step size.

[0034] Furthermore, the judgment thresholds T1 and T2 mentioned in steps 1-6 are as follows:

[0035] Take the initial segment of the audio data stream of a preset length, and calculate the average short-time energy E based on the short-time energy curve data. avg :

[0036]

[0037] Among them, E m To represent the short-time energy of the m-th audio frame in the audio data stream;

[0038] Calculate the average short-time zero-crossing rate Z based on the short-time zero-crossing rate curve. avg :

[0039]

[0040] Among them, Z m Let represent the zero-crossing rate of the m-th audio frame in the audio data stream;

[0041] Set T1 = 2 × E avg T2 = 2 × Z avg .

[0042] Furthermore, step 4, which involves segmenting the audio data stream of the controlled speech into audio segments, includes the following steps:

[0043] Step 4-1: Search forward from the speech frame corresponding to the text obtained in Step 2 to find the moment t0 when the first endpoint detection model judges it as the start of the speech segment;

[0044] Step 4-2: Search backward from the speech frame corresponding to the sentence end identifier obtained in Step 2 to find the moment t1 when the first endpoint detection model judges it as the end of the speech segment;

[0045] Step 4-3: Extract the audio data stream from time t0 to time t1, form a speech segment, and continue processing from time t1 onwards;

[0046] Step 4-4: Repeat steps 4-1 to 4-3 until the audio data stream ends;

[0047] Steps 4-5: Output all audio segments.

[0048] Beneficial effects:

[0049] 1. Construct a sentence segmentation model based on speech recognition, implement a semantic-based audio segmentation algorithm, reduce speech segmentation errors caused by short pauses by the speaker, and improve the semantic integrity of the speech segmentation results.

[0050] 2. A speech segmentation method based on speech recognition and endpoint detection is constructed. The speech segmentation algorithm is implemented from two dimensions: high-level semantic features and low-level sound features. This improves the accuracy of speech segmentation and facilitates the subsequent structured extraction algorithm to accurately extract information such as aircraft call sign, aircraft type, and status. Attached Figure Description

[0051] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0052] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0053] Figure 2 This is a schematic diagram of a 3-layer convolutional neural network.

[0054] Figure 3 This is a schematic diagram of a single-layer Transformer neural network.

[0055] Figure 4 This is a schematic diagram of how the audio stream is segmented into audio segments based on the results returned by the sentence segmentation model and endpoint detection according to the present invention. Detailed Implementation

[0056] This invention discloses a regulated speech segmentation method based on speech recognition and endpoint detection. First, a sentence segmentation model based on speech recognition is designed, along with a training set construction method. Then, an improvement is made to the traditional dual-threshold method based on short-time energy and short-time zero-crossing rate. This improvement considers not only low-level sound information but also high-level semantic information during speech segmentation, thereby enhancing the accuracy of regulated speech segmentation. The specific steps of the above technical solution are as follows:

[0057] Step 1: Construct a sentence segmentation model and a speech endpoint detection model based on speech recognition;

[0058] The construction of the speech recognition-based sentence segmentation model and endpoint detection model described in step 1 includes the following steps:

[0059] Step 1-1: Design a speech recognition neural network;

[0060] Steps 1-2: Export a set of air traffic control speech samples with a sampling rate of 16000Hz from the voice recorder of the air traffic control automation system, denoted as the training set X = {X1, X2, ..., X...}. N}, where N is the total number of samples, X1, X2, ..., X... N These are the 1st, 2nd, ..., Nth training samples, respectively.

[0061] Steps 1-3: Manually annotate the regulated speech sample set from Steps 1-2 to obtain the regulated speech annotation set for training, denoted as annotation set Y = {Y1, Y2, ..., Y...} N}, where Y1, Y2, ..., Y N The voice samples are X1, X2, ..., X, respectively. N Corresponding regulated speech text annotations;

[0062] Steps 1-4: Modify the annotation set Y by adding a sentence end identifier at the end of each annotation, resulting in the second annotation set Y′ = {Y′1, Y′2, ..., Y′}. N};

[0063] Steps 1-5: Train the speech recognition neural network using the training set X and the second annotation set Y′. Use the training data as the input sequence, compare the model output with the annotation, and optimize the neural network parameters according to the connection time sequence classification loss function to obtain a sentence segmentation model based on speech recognition.

[0064] Steps 1-6: Calculate the short-time energy curve and the average zero-crossing rate curve of the audio stream. Set judgment thresholds T1 and T2 based on the short-time energy and short-time zero-crossing rate of the audio stream in the speechless phase. When the short-time energy is greater than the threshold T1, the current moment is considered to be the beginning of the speech segment. When the short-time zero-crossing rate is less than the threshold T2, it is considered to be the end of the speech segment. Construct an endpoint detection model.

[0065] The speech recognition neural network structure described in step 1-1 specifically includes:

[0066] It consists of a speech feature coding layer and a recognition result output layer. The speech feature coding layer contains 3 layers of convolutional neural network and 12 layers of Transformer neural network, and the recognition result output layer contains 1 layer of fully connected neural network. It converts the audio stream into a text stream at a ratio of one text output every 50ms of audio.

[0067] The definition in steps 1-4, which involves adding a sentence-ending identifier at the end of each tag, specifically includes:

[0068] For the label Y i ="y1y2...y k ", where k represents the length of the annotation, y k Adding a sentence-ending identifier, such as "#", will then change the annotation to Y. i ′=″y1y2...y k #″. For example, if there is a label that is “Dongfang 3984 down to 12 keep”, then the modified label is “Dongfang 3984 down to 12 keep #”.

[0069] The short-time energy mentioned in steps 1-6 is as follows:

[0070]

[0071] Where E i Let x represent the short-time energy of the i-th audio frame with a length of 50ms, where N represents the number of audio samples contained within the 50ms audio frame, and x represents the short-time energy of the i-th audio frame with a length of 50ms. n This represents the waveform value of the nth audio sampling point.

[0072] The short-time zero-crossing rate mentioned in steps 1-6 is as follows:

[0073]

[0074] Z i This represents the zero-crossing rate of the i-th audio frame with a length of 50ms, where N represents the number of audio samples contained within the 50ms audio frame, and x... n Let sgn[x] represent the waveform value of the nth audio sample point, where sgn[x] is the sign function.

[0075]

[0076] The short-time energy curves mentioned in steps 1-6 are as follows:

[0077] Set a window with a size of 50ms and a step size of 10ms. Use this window to slide along the audio stream and calculate the short-time energy within each window to obtain a short-time energy curve with a sampling interval of 10ms.

[0078] The short-time zero-crossing rate curves mentioned in steps 1-6 are as follows:

[0079] Set a window with a size of 50ms and a step size of 10ms. Use this window to slide along the audio stream and calculate the short-time zero-crossing rate within each window to obtain the short-time zero-crossing rate curve with a sampling interval of 10ms.

[0080] The judgment thresholds T1 and T2 mentioned in steps 1-6 are as follows:

[0081] Take the first 100ms segment of the audio stream and calculate the average short-time energy based on the short-time energy curve data:

[0082]

[0083] Calculate the average short-time zero-crossing rate based on the short-time zero-crossing rate curve:

[0084]

[0085] Set T1 = 2 × E avg T2 = 2 × Z avg .

[0086] The importance of step 1 lies in the following: 1. General speech recognition models only output text and spaces, lacking sentence end identifiers, and speech recognition training samples also lack relevant identifiers, requiring reconstruction; 2. This step obtains a sentence segmentation model based on speech recognition by modifying the training dataset and training the speech recognition model; 3. This step proposes an endpoint detection model based on low-level features, and uses statistical information to adaptively set the start and end thresholds, improving the robustness of the model to audio streams with different signal-to-noise ratios.

[0087] Step 2: Use a speech recognition-based sentence segmentation model to identify the text and sentence end markers corresponding to the audio stream;

[0088] Step 3: Use an endpoint detection model to determine the start and end points of speech in the audio stream;

[0089] Step 4: Segment the audio stream into audio segments based on the results returned by the sentence segmentation model and endpoint detection.

[0090] Step 4, which involves segmenting the audio stream into audio segments based on the results returned by the sentence segmentation model and endpoint detection, includes the following steps:

[0091] Step 4-1: Search forward from the speech frame corresponding to the Chinese character output by the sentence segmentation model to find the moment t0 when the first endpoint detection model judges it as the start of the speech segment;

[0092] Step 4-2: Search backward from the speech frame corresponding to the sentence end identifier output by the sentence segmentation model to find the moment t1 when the first endpoint detection model judges it as the end of the speech segment;

[0093] Step 4-3: Extract the audio stream from time t0 to time t1, form a speech segment, and continue processing from time t1 onwards;

[0094] Step 4-4: Repeat steps 4-1 to 4-3 until the audio stream ends;

[0095] Steps 4-5: Output all audio segments;

[0096] The importance of step 4 lies in the following: 1. General speech segmentation models only utilize the low-level sound features of the audio stream, which can easily split a complete sentence into multiple segments due to short pauses during speech, thus reducing the accuracy of speech segmentation. This algorithm uses speech recognition technology and endpoint detection technology to implement the speech segmentation algorithm from two dimensions: high-level semantic features and low-level sound features, thereby improving the accuracy of speech segmentation.

[0097] Example:

[0098] This embodiment provides a speech segmentation method for air traffic control based on speech recognition and endpoint detection, which can be applied to the speech segmentation of air-to-ground communication audio streams in air traffic control systems.

[0099] The specific implementation process and steps are as follows, and the flow is as follows: Figure 1 As shown.

[0100] Step 1: Construct a sentence segmentation model and a speech endpoint detection model based on speech recognition;

[0101] The construction of the speech recognition-based sentence segmentation model and endpoint detection model described in step 1 includes the following steps:

[0102] Step 1-1: Design a speech recognition neural network; the structure of the neural network consists of a speech feature encoding layer and a recognition result output layer. The speech feature encoding layer contains 3 layers of convolutional neural networks and 12 layers of Transformer neural networks. The recognition result output layer contains 1 layer of fully connected neural networks. The audio stream is converted into a text stream at a ratio of one text per 50ms (i.e., one text is output every 800 sampling points).

[0103] The structure of a 3-layer convolutional neural network is as follows: Figure 2As shown, the input is an audio waveform data stream with a sampling rate of 16000Hz. The value of each sampling point is a floating-point number between [-1, 1]. The output ratio is to output a 512-dimensional feature every 50ms (i.e., output a 512-dimensional feature every 800 sampling points).

[0104] The structure of a 1-layer Transformer neural network is as follows: Figure 3 As shown, it includes an attention layer and a mapping layer. The input to the attention layer is calculated using Equations 1 to 3:

[0105] Q = x × W Q (1)

[0106] K = x × W K (2)

[0107] V = x × W V (3)

[0108] Where Q represents the query vector, K represents the index vector, V represents the value vector, and W represents the value vector. Q W K W V These are the transformation matrices for the corresponding vectors. The output of the attention layer is calculated using equations 4 to 6:

[0109] X=X+Muiltihead(Q,K,V) (4)

[0110] Muiltihead(Q,K,V)=Concat(head1,head2,...,head n W O (5)

[0111]

[0112] The softmax function is:

[0113]

[0114] Where e represents the base of the natural logarithm, z j Let K represent the j-th value of the input feature vector z, and K be the number of values ​​in the feature vector. T d represents the transpose of vector K. k Indicates the dimension of the vector.

[0115] Calculate the output of the mapping layer using Formula 7:

[0116] FFN(X)=X+max(0,XW1+b1)W2+b2 (7)

[0117] Among them, W1 and W2 are the transformation matrices of the mapping layer, and b1 and b2 are the biases of the mapping layer.

[0118] Stack 12 layers of Transformer neural networks, where the output of each layer is the input of the next layer, and the dimensions of the input and output matrices remain unchanged.

[0119] Finally, use Equation 8 to calculate the output of the 1-layer fully connected neural network:

[0120] FC(X) = max(0, XW Fc + b Fc ) (8)

[0121] Among them, W Fc is the transformation matrix of the fully connected layer, and b FC is the bias of the fully connected layer, and the dimensions of the input and output matrices remain unchanged.

[0122] Step 1-2: Export the set of air traffic control voice samples with a sampling rate of 16000 Hz that can be used for training from the voice recorder of the air traffic control automation system, denoted as the training set X = {X1, X2,..., X N}, where N is the total number of samples, and X1, X2,..., X N are the 1st, 2nd,..., Nth training samples respectively;

[0123] Step 1-3: Manually annotate the set of air traffic control voice samples in Step 1-2 to obtain the set of air traffic control voice annotations for training, denoted as the annotation set Y = {Y1, Y2,..., Y N}, where Y1, Y2,..., Y N are the corresponding air traffic control voice text annotations for the air traffic control voice samples X1, X2,..., X N respectively;

[0124] Step 1-4: Modify the annotation set Y by adding a sentence end identifier at the end of each annotation to obtain the second annotation set Y' = {Y'1, Y'2,..., Y' N}, for example, for the following annotation "Dongfang Sanjiuba Sidao Yaoliang Baochi", its corresponding second annotation is "Dongfang Sanjiuba Sidao Yaoliang Baochi#";

[0125] Step 1-5: Use the training set X and the second annotation set Y' to train the speech recognition neural network. Take the training data as the input sequence, compare the output of the neural network with the annotation, and optimize the neural network parameters according to the connection time series classification loss function to obtain a sentence segmentation model based on speech recognition;

[0126] Steps 1-6: Calculate the short-time energy curve and short-time zero-crossing rate curve of the audio stream. Set judgment thresholds T1 and T2 based on the short-time energy and short-time zero-crossing rate of the audio stream in the speechless phase. When the short-time energy is greater than the threshold T1, the current moment is considered to be the beginning of the speech segment. When the short-time zero-crossing rate is less than the threshold T2, it is considered to be the end of the speech segment. Construct an endpoint detection model.

[0127] The aforementioned short-term energy is as follows:

[0128]

[0129] Where E j Let x represent the short-time energy of the j-th audio frame with a length of 50ms, M represent the number of audio samples contained within the 50ms audio frame, and x represent the short-time energy of the j-th audio frame with a length of 50ms. n This represents the waveform value of the nth audio sampling point.

[0130] The short-time zero-crossing rate is as follows:

[0131]

[0132] Z k This represents the short-time zero-crossing rate of the k-th audio frame with a length of 50ms, where M represents the number of audio samples contained within the 50ms audio frame, and x represents the short-time zero-crossing rate of the k-th audio frame with a length of 50ms. n Let sgn[x] represent the waveform value of the nth audio sample point, where sgn[x] is the sign function.

[0133]

[0134] The aforementioned short-time energy curve is as follows:

[0135] Set a window with a size of 50ms and a step size of 10ms. Use this window to slide along the audio stream and calculate the short-time energy within each window to obtain a short-time energy curve with a sampling interval of 10ms.

[0136] The short-time zero-crossing rate curve is as follows:

[0137] Set a window with a size of 50ms and a step size of 10ms. Use this window to slide along the audio stream and calculate the short-time zero-crossing rate within each window to obtain the short-time zero-crossing rate curve with a sampling interval of 10ms.

[0138] The specific judgment thresholds T1 and T2 are as follows:

[0139] Take the first 100ms segment of the audio stream and calculate the average short-time energy based on the short-time energy curve data:

[0140]

[0141] Calculate the average short-time zero-crossing rate based on the short-time zero-crossing rate curve:

[0142]

[0143] Set T1 = 2 × E avg T2 = 2 × Z avg .

[0144] Step 2: Use a speech recognition-based sentence segmentation model to identify the text and sentence end markers corresponding to the audio stream;

[0145] Step 3: Use an endpoint detection model to determine the start and end points of speech in the audio stream;

[0146] Step 4: Segment the audio stream into audio segments based on the results returned by the sentence segmentation model and endpoint detection.

[0147] Step 4 describes segmenting the audio stream into audio segments based on the results returned by the sentence segmentation model and endpoint detection, such as... Figure 4 As shown, it includes the following steps:

[0148] Step 4-1: Search forward from the speech frame corresponding to the Chinese character output by the sentence segmentation model to find the moment t0 when the first endpoint detection model judges it as the start of the speech segment;

[0149] Step 4-2: Search backward from the speech frame corresponding to the sentence end identifier output by the sentence segmentation model to find the moment t1 when the first endpoint detection model judges it as the end of the speech segment;

[0150] Step 4-3: Extract the audio stream from time t0 to time t1, form a speech segment, and continue processing from time t1 onwards;

[0151] Step 4-4: Repeat steps 4-1 to 4-3 until the audio stream ends;

[0152] Steps 4-5: Output all audio segments;

[0153] Step 5: Use the above audio clips as data material in the speech recognition process of the air traffic control system.

[0154] The voice recognition process of the air traffic control system described in step 5 includes the following steps:

[0155] Step 5-1: Input the segmented audio fragment set into the speech recognition neural network in sequence to obtain the recognition text corresponding to each audio fragment;

[0156] Step 5-2: Based on the time position of the audio segment in the original audio stream, add the corresponding time tag to the text corresponding to it to form the recognized text with time tag information corresponding to the original audio stream, i.e., the subtitle;

[0157] Step 5-3: Output the original audio stream and its subtitles for controllers to view historical control instructions.

[0158] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a controlled speech segmentation method based on speech recognition and endpoint detection, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0159] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0160] This invention provides a concept and method for regulated speech segmentation based on speech recognition and endpoint detection. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for controlling speech segmentation based on speech recognition and endpoint detection, characterized in that, Includes the following steps: Step 1: Construct a sentence segmentation model and a speech endpoint detection model based on speech recognition; Step 2: Using a speech recognition-based sentence segmentation model, identify the audio data stream of the controlled speech and output the text and sentence end identifier corresponding to the audio data stream; Step 3: Use the speech endpoint detection model to determine the speech start and end points contained in the audio data stream of the regulated speech; Step 4: Based on the results returned in Step 2 and Step 3, the audio data stream of the controlled speech is segmented into audio segments; Step 5: Apply the above audio clips as data material to the speech recognition process of the air traffic control system; Step 4, which involves segmenting the audio data stream of the controlled speech into audio segments, includes the following steps: Step 4-1: Search forward from the speech frame corresponding to the text obtained in Step 2 to find the moment when the endpoint detection model first judges it as the start of a speech segment. ; Step 4-2: Search backwards from the speech frame corresponding to the sentence end identifier obtained in Step 2 to find the moment when the endpoint detection model first judges it as the end of the speech segment. ; Step 4-3: Capture Time At the time The audio data stream is used to form a speech segment, starting from time... Continue processing; Step 4-4: Repeat steps 4-1 to 4-3 until the audio data stream ends; Steps 4-5: Output all audio segments.

2. The regulated speech segmentation method based on speech recognition and endpoint detection according to claim 1, characterized in that, The construction of the speech recognition-based sentence segmentation model and endpoint detection model described in step 1 includes the following steps: Step 1-1: Design a speech recognition neural network; Step 1-2: Construct a set of regulated speech samples, denoted as the training set. Where N is the total number of samples, The first training samples The aforementioned controlled voice; Steps 1-3: Manually annotate the regulated speech sample set from Steps 1-2 to obtain the regulated speech annotation set used for training, denoted as the first annotation set. ,in These are training samples from the set of regulated speech samples. Corresponding regulated speech text annotations; Steps 1-4: Modify the first annotation set Y by adding a sentence end identifier at the end of each regulated speech-text annotation to obtain the second annotation set. ; Steps 1-5: Using the training set Second annotation set The speech recognition neural network designed in step 1-1 is trained by taking the training set as the input sequence, comparing the output of the speech recognition neural network with the manual annotation, and optimizing the parameters of the speech recognition neural network according to the connection time classification loss function to obtain a sentence segmentation model based on speech recognition, that is, the optimized speech recognition neural network. Steps 1-6: Constructing the endpoint detection model: Calculate the short-time energy curve and short-time zero-crossing rate curve of the audio data stream of the controlled speech, and set a judgment threshold based on the short-time energy and short-time zero-crossing rate of the non-speech phase in the audio data stream. and When the short-term energy of the speechless phase is greater than the threshold The system determines the start of a speech segment based on the current moment, when the short-time zero-crossing rate is less than a threshold. Then, the current moment is determined to be the end of the speech segment.

3. The regulated speech segmentation method based on speech recognition and endpoint detection according to claim 2, characterized in that, The speech recognition neural network described in step 1-1 has the following specific structure: The speech recognition neural network consists of a speech feature coding layer and a recognition result output layer. The speech feature coding layer contains 3 layers of convolutional neural network and 12 layers of Transformer neural network, and the recognition result output layer contains 1 layer of fully connected neural network. The speech recognition neural network converts the audio data stream into a text data stream by recognizing and outputting a text according to the ratio of audio data with a preset frame length.

4. The regulated speech segmentation method based on speech recognition and endpoint detection according to claim 2, characterized in that, The step 1-4, which involves adding a sentence-end identifier at the end of each regulated speech-script annotation, specifically includes: For the One control voice text label ,in Indicates the length of the annotation. Adding a sentence-ending identifier # will result in the modified annotation being... .

5. The regulated speech segmentation method based on speech recognition and endpoint detection according to claim 2, characterized in that, The short-time energy of the audio data stream without speech phase mentioned in steps 1-6 is calculated using the following method: ； in, Indicates the first in the audio data stream The short-time energy of an audio frame, where M represents the number of audio samples contained within an audio frame of a preset frame length. Indicates the first Waveform values ​​of each audio sampling point.

6. The regulated speech segmentation method based on speech recognition and endpoint detection according to claim 2, characterized in that, The short-time zero-crossing rate mentioned in steps 1-6 is calculated using the following method: ； in, Indicates the first in the audio data stream The zero-crossing rate of an audio frame, where M represents the number of audio samples contained within an audio frame of a preset frame length. Indicates the first Waveform values ​​of each audio sampling point It is a symbolic function, as detailed below: 。 7. The regulated speech segmentation method based on speech recognition and endpoint detection according to claim 2, characterized in that, The short-time energy curves mentioned in steps 1-6 are as follows: Set a window with a size equal to the preset frame length and a step size smaller than the preset frame length. Use this window to slide along the audio data stream and calculate the short-time energy within each window to obtain the short-time energy curve with a sampling interval of the step size.

8. The regulated speech segmentation method based on speech recognition and endpoint detection according to claim 2, characterized in that, The short-time zero-crossing rate curves mentioned in steps 1-6 are as follows: Set a window with a size equal to the preset frame length and a step size smaller than the preset frame length. Use this window to slide along the audio data stream and calculate the short-time zero-crossing rate within each window to obtain a short-time zero-crossing rate curve with a sampling interval of the step size.

9. A method for controlling speech segmentation based on speech recognition and endpoint detection according to claim 2, characterized in that, The judgment thresholds described in steps 1-6 and The details are as follows: Take the initial segment of the audio data stream of a preset length, and calculate the average short-time energy based on the short-time energy curve data. : ； in, To represent the first in the audio data stream The short-time energy of an audio frame; Calculate the average short-time zero-crossing rate based on the short-time zero-crossing rate curve. : ； in, To represent the first in the audio data stream Zero-crossing rate of each audio frame; set up , .

Images