Intelligent sentence segmentation activity voice detection method and device based on multi-state time sequence modeling

Abstract

The invention discloses an intelligent sentence segmentation activity voice detection method and device based on multi-state time sequence modeling. The method comprises the following steps: receiving an audio signal of at least one channel; extracting an acoustic feature sequence of the audio signal by adopting a target speech recognition model corresponding to the channel number; probability distribution that each voice frame corresponding to the acoustic feature sequence belongs to different voice activity states is determined, a state sequence corresponding to each channel is obtained, and the voice activity states comprise at least one of the following states: an initial mute state, a voice state, an in-speech-round pause mute state and a speech round discontinuous sentence mute state; and determining the sentence segmentation time in the audio signal according to the state sequence. According to the method and the device, the technical problem that wrong sentence segmentation exists in voice activity detection based on a fixed silence threshold in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of signal processing, and in particular to a method and device for intelligent sentence segmentation and active speech detection based on multi-state temporal modeling. Background Art

[0002] Voice activity detection (VAD), a fundamental step in speech signal processing, aims to distinguish valid speech segments from non-speech segments. Traditional VAD methods, such as those based on energy thresholds or silence duration thresholds, are simple and intuitive, but they can suffer from incorrect segmentation.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a method and device for intelligent sentence segmentation and active speech detection based on multi-state time series modeling, so as to at least solve the technical problem of incorrect sentence segmentation in speech activity detection based on a fixed silence threshold in the related art.

[0005] According to one aspect of an embodiment of the present application, a method for intelligent sentence-breaking active speech detection based on multi-state time series modeling is provided, comprising: receiving an audio signal of at least one channel; extracting an acoustic feature sequence of the audio signal using a target speech recognition model corresponding to the number of channels; determining the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to a different speech activity state, and obtaining a state sequence corresponding to each channel, wherein the speech activity state includes at least one of the following: a starting silence state, a speech state, a pause silence state within a turn, and a sentence-breaking silence state between turns; and determining the time when a sentence break occurs in the audio signal based on the state sequence.

[0006] Optionally, when the number of channels is single, the target speech recognition model is a first speech recognition model, and the first speech recognition model is trained in the following manner: obtaining a first speech data set corresponding to a single channel, and determining a first speech activity state to which each speech frame in the first speech data set belongs; training a first initial speech recognition model based on the first speech data set, and determining a first output result of the first initial speech recognition model, wherein the first initial speech recognition model includes an encoder model, a long short-term memory network layer, and a linear classification layer; determining a cross-entropy loss based on the first output result and the first speech activity state; when the cross-entropy loss does not meet a first preset condition, updating the parameters of the first initial speech recognition model, and determining an updated cross-entropy loss corresponding to the updated first output result, and stopping training until the updated cross-entropy loss meets the first preset condition to obtain the first speech recognition model.

[0007] Optionally, when the number of channels is dual, the target speech recognition model is a second speech recognition model, and the second speech recognition model is trained in the following manner: obtaining a second speech data set corresponding to the dual channels, and determining the second speech activity state to which each speech frame in the second speech data set belongs, wherein the second speech activity state includes a real turn-taking time point; training a second initial speech recognition model based on the second speech data set, and determining the second output result of the second initial speech recognition model, wherein the second initial speech recognition model includes two branches, each branch contains a shared encoder model, and the outputs of the two branches are connected to a cross-attention module for feature fusion, and the cross-attention module is then connected to a long short-term memory network and a linear classification layer; determining the state classification loss based on the second output result and the second speech activity state; when the state classification loss does not meet the second preset condition, updating the parameters of the second initial speech recognition model, and determining the updated state classification loss corresponding to the updated second output result, until the updated state classification loss meets the second preset condition, stopping training, and obtaining the second speech recognition model.

[0008] Optionally, determining the second output result of the second initial speech recognition model includes: determining the first feature corresponding to the first channel in the dual channels and determining the second feature corresponding to the second channel in the dual channels through the encoder model in the second initial speech recognition model; determining the first vector based on the second feature and the query weight matrix; determining the second vector based on the first feature and the key weight matrix; determining the third vector based on the first feature and the value weight matrix; determining the fused feature based on the dimensions of the first vector, the second vector, the third vector, and the second vector; inputting the fused feature into the long short-term memory network and the linear classification layer for analysis to obtain the second output result.

[0009] Optionally, the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states is determined to obtain a state sequence corresponding to each channel, including: determining the maximum value of the probability distribution corresponding to each speech frame; determining the label corresponding to the maximum value as the state label to which the corresponding speech frame belongs, wherein each state label corresponds to a speech activity state; and obtaining the state label to which each speech frame belongs in sequence according to the order of the audio signal to obtain a state sequence.

[0010] Optionally, before determining the time of dynamic segmentation based on the state sequence, the method also includes: detecting whether the state sequence contains a preset sequence to obtain a detection result, wherein the preset sequence represents a transition from an initial silent state to a voice state and then to the initial silent state, and the duration of the voice state in the preset sequence is less than a preset duration; when the detection result indicates that the state sequence contains the preset sequence, determining that invalid speech exists in the audio signal, and filtering the preset sequence from the state sequence; when the detection result indicates that the state sequence does not contain the preset sequence, determining that invalid speech does not exist in the audio signal.

[0011] Optionally, determining the time of dynamic punctuation based on the state sequence includes: determining the first consecutive frame number of the silent state of the inter-turn punctuation in the state sequence; when the first consecutive frame number is greater than or equal to a first threshold, determining the time corresponding to the last frame in the first consecutive frame number as the time of dynamic punctuation; determining the second consecutive frame number of the silent state of the pause within the turn in the state sequence; when the second consecutive frame number is greater than or equal to a second threshold, determining the time corresponding to the last frame in the second consecutive frame number as the time of dynamic punctuation.

[0012] According to another aspect of an embodiment of the present application, an intelligent sentence-breaking active speech detection device based on multi-state time series modeling is also provided, including: a receiving module for receiving an audio signal of at least one channel; an extraction module for extracting an acoustic feature sequence of the audio signal using a target speech recognition model corresponding to the number of channels; a first determination module for determining the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, and obtaining a state sequence corresponding to each channel, wherein the speech activity state includes at least one of the following: an initial silence state, a speech state, a pause silence state within a turn, and a sentence-breaking silence state between turns; a second determination module for determining the time when a sentence break occurs in the audio signal based on the state sequence.

[0013] According to another aspect of the embodiments of the present application, an electronic device is also provided, including: a memory for storing program instructions; a processor, connected to the memory, for executing program instructions to implement the following functions: receiving an audio signal of at least one channel; extracting an acoustic feature sequence of the audio signal using a target speech recognition model corresponding to the number of channels; determining the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, and obtaining a state sequence corresponding to each channel, wherein the speech activity state includes at least one of the following: an initial silence state, a speech state, a silence state for pauses within a turn, and a silence state for breaks between turns; and determining the time when a break occurs in the audio signal based on the state sequence.

[0014] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned intelligent sentence segmentation active speech detection method based on multi-state time series modeling by running the computer program.

[0015] According to another aspect of the embodiments of the present application, a computer program product is also provided, including computer instructions, which, when executed by a processor, implement the above-mentioned intelligent sentence segmentation active speech detection method based on multi-state temporal modeling.

[0016] In an embodiment of the present application, an audio signal of at least one channel is received; an acoustic feature sequence of the audio signal is extracted using a target speech recognition model corresponding to the number of channels; the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to a different speech activity state is determined, and a state sequence corresponding to each channel is obtained, wherein the speech activity state includes at least one of the following: an initial silence state, a speech state, a silence state for pauses within a turn, and a silence state for breaks between turns; based on the state sequence, the time when a break occurs in the audio signal is determined, thereby achieving the purpose of refined dynamic breakup, thereby realizing the technical effect of improving the accuracy and robustness of speech activity detection, and further solving the technical problem of incorrect breakup in speech activity detection based on a fixed silence threshold in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing an intelligent sentence segmentation active speech detection method based on multi-state temporal modeling according to an embodiment of the present application;

[0019] Figure 2 This is a flow chart of a method for intelligent sentence segmentation active speech detection based on multi-state temporal modeling according to an embodiment of the present application;

[0020] Figure 3 This is a state transition diagram according to an embodiment of the present application;

[0021] Figure 4 is a structural diagram of a dual-channel model according to an embodiment of the present application;

[0022] Figure 5 This is a structural diagram of an intelligent sentence segmentation active speech detection device based on multi-state time series modeling according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or reject the automated decision results; if the user chooses to reject, the expert decision-making process will be entered.

[0026] First, some nouns or terms that appear in the process of explaining the embodiments of this application are subject to the following explanations:

[0027] Dynamic Sentence Segmentation (Dynamic Utterance Detection): This determines where a speaker's current sentence or segment ends, specifically where a sentence boundary is located. This segmentation is primarily performed within a single speaker's speech stream. This approach relies not only on long periods of silence but also incorporates clues such as acoustic features (such as prosody and intonation), semantic information, and possible grammatical structure to determine whether a segment should end.

[0028] Turn-Taking (Speaker Turn Detection): This detects when a speaker switches during a conversation, i.e., when one speaker switches to another. This involves the interaction between multiple speakers. This typically involves one speaker stopping and another starting.

[0029] Traditional voice activity detection (VAD) algorithms are typically based on energy thresholds or silence duration thresholds. While their logical rules are simple and intuitive, they face numerous challenges in practical application. In real-world conversations, especially in complex acoustic environments and natural dialogue, energy-based VAD methods exhibit significant shortcomings. Static thresholds struggle to adapt to varying speech rates, volumes, and pause patterns, and pauses of varying lengths (e.g., pauses for thought or interjections) can occur within a single turn. These issues make it difficult for fixed silence thresholds to accurately distinguish between pauses within a turn and the end of a turn between turns, leading to frequent mis-segmentation or omission of sentences. Furthermore, traditional VAD methods are susceptible to background noise, easily misidentifying speech as valid or masking the end of actual speech, affecting accurate segmentation. In conversational systems requiring rapid response, turn-change mechanisms based on a posteriori silence detection are slow to respond and cannot meet the demands of real-time interaction.

[0030] In order to solve the problems existing in the related art, the embodiment of the present application provides an intelligent segmentation active speech detection method based on multi-state temporal modeling. Based on the dynamic segmentation of multi-state temporal modeling and cross-channel attention interaction, the acoustic features of speech (such as rhythm, intonation) and implicit semantic information are divided into four states. By judging the duration and state transition of the four states, dynamic segmentation and background voice filtering are achieved. In the dual-channel scenario, the Cross Attention mechanism is introduced to perform inter-channel information fusion to further improve the background voice filtering capability. This method can be run on Figure 1 Among the computer terminals shown, the computer terminal will be described below.

[0031] The embodiment of the intelligent sentence segmentation active voice detection method based on multi-state time series modeling provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal for implementing an intelligent sentence segmentation active speech detection method based on multi-state temporal modeling is shown. Figure 1As shown, the computer terminal 10 may include one or more processors (illustrated as 102a, 102b, ..., 102n in the figure) (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions via a wired and / or wireless network connection. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0032] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be fully or partially integrated into any of the other components of the computer terminal 10. As discussed in the embodiments of the present application, the data processing circuitry functions as a processor control (e.g., the selection of a variable resistor terminal path connected to an interface).

[0033] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the intelligent sentence segmentation active voice detection method based on multi-state temporal modeling in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned intelligent sentence segmentation active voice detection method based on multi-state temporal modeling. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0034] The transmission module 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission module 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission module 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0035] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0036] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0037] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of an intelligent sentence segmentation active speech detection method based on multi-state timing modeling. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] Figure 2 Flowchart of a method for intelligent sentence segmentation and active speech detection based on multi-state temporal modeling according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0039] Step S202: Receive an audio signal of at least one channel.

[0040] In the above step S202, in voice signal processing and communication technology, audio signals can be transmitted through different channels, each of which can represent an independent sound source or spatial location. The channel here refers to an independent path for the audio data stream, which can be physical (for example, different microphones, different speakers) or logical (for example, the audio streams corresponding to the two parties in a telephone call). The audio signal of at least one channel includes two situations: one is a single-channel audio signal, and the other is a multi-channel audio signal. A single-channel audio signal means that all audio signals are mixed together and input into the system as a single data stream. For example, a recorded music file, a recording of a person speaking alone, etc. In a single-channel scenario, a unified audio signal is received. A multi-channel audio signal refers to an audio signal received from multiple sources or devices, each signal has its own independent channel, and the multi-channel can be dual-channel. For example, in a meeting record, a telephone conference, or a multi-person call scenario, the voice of each participant can be picked up by their respective microphones to form an independent audio channel.

[0041] Step S204: extracting an acoustic feature sequence of the audio signal using a target speech recognition model corresponding to the number of channels.

[0042] In step S204, the target speech recognition model is selected based on the number of channels in the received audio signal. If the received audio signal is single-channel, a model optimized for that single channel is used, namely the first speech recognition model described below. If the audio signal is multi-channel, a multi-channel model is used, namely the second speech recognition model described below. The target speech recognition model uses a pre-trained Zipformer encoder to extract acoustic features from the input audio signal, including implicit speech rhythm, intonation, and other features that aid in voice activity detection, to produce an acoustic feature sequence.

[0043] Step S206, determining the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, and obtaining a state sequence corresponding to each channel, wherein the speech activity state includes at least one of the following: initial silence state, speech state, silence state during pauses within a turn, and silence state during breaks between turns.

[0044] In step S206, the target speech recognition model's temporal modeling module (e.g., an LSTM network) analyzes the acoustic feature sequence of each channel to predict the speech activity state of each frame of audio (typically, audio is divided into multiple time segments, each corresponding to a frame). In this embodiment, the speech activity states are categorized as follows:

[0045] 1. Initial silence state (e.g., represented by 0): The silence before the start of the speech segment, usually used to filter out the non-speech part at the beginning of the recording;

[0046] 2. Speech state (e.g., represented by 1): the state during active speech activity, including the actual speech content of the speaker;

[0047] 3. Intra-turn pause silence (e.g., 2): A short silence or pause that occurs within the same speaker's turn, such as a pause when the speaker is thinking or a natural pause in a sentence, but the speaker does not end the current turn;

[0048] 4. Inter-turn silence (e.g., 3): The silence between turns of a speaker (or two different speakers) usually indicates that a complete sentence or turn has ended and another is about to begin.

[0049] For each speech frame, the target speech recognition model outputs a probability distribution indicating the likelihood that the frame belongs to one of the four states described above. For example, for a given frame, if the target speech recognition model outputs: initial silence state 0.1, speech state 0.6, intra-turn pause silence state 0.2, and inter-turn pause silence state 0.1, this indicates that the frame is more likely to be in the speech state. By concatenating the predicted states for each frame, the model generates a state sequence for each channel, which records the evolution of speech activity across the entire audio signal. For example, the state sequence might be 11121130, indicating that the audio signal for that channel begins with continuous speech (multiple 1 states), followed by a short pause (state 2), continued speech, and finally the end of a complete sentence followed by silence (state 3), followed by initial silence (state 0).

[0050] This approach enables a refined understanding of the speech activity in each channel, distinguishing between pauses within a turn, silence at the end of a turn, and silence at the beginning of a turn, thereby enabling accurate dynamic segmentation and turn-transition prediction.

[0051] Step S208: determining the time when a sentence break occurs in the audio signal according to the state sequence.

[0052] In the above step S208, the time when the sentence break occurs can be determined by analyzing the state sequence obtained.

[0053] Through steps S202 to S208 above, the goal of fine-tuning dynamic sentence segmentation is achieved, thereby achieving the technical effect of improving the accuracy and robustness of voice activity detection, and further solving the technical problem of incorrect sentence segmentation in voice activity detection based on a fixed silence threshold in related technologies. This is explained below.

[0054] In the above-mentioned intelligent sentence segmentation active speech detection method based on multi-state time series modeling, when the number of channels is single, the target speech recognition model is the first speech recognition model, and the first speech recognition model is trained in the following manner: obtaining a first speech data set corresponding to a single channel, and determining the first speech activity state to which each speech frame in the first speech data set belongs; training a first initial speech recognition model based on the first speech data set, and determining the first output result of the first initial speech recognition model, wherein the first initial speech recognition model includes an encoder model, a long short-term memory network layer, and a linear classification layer; determining the cross-entropy loss based on the first output result and the first speech activity state; when the cross-entropy loss does not meet the first preset condition, updating the parameters of the first initial speech recognition model, and determining the updated cross-entropy loss corresponding to the updated first output result, until the updated cross-entropy loss meets the first preset condition, stopping the training, and obtaining the first speech recognition model.

[0055] In some embodiments of the present application, when the number of channels is single, the target speech recognition model is specifically referred to as the first speech recognition model. When training the first initial speech recognition model, the audio data collected by a single sound source or a single microphone is determined as the first speech data set, wherein each speech frame in the first speech data set corresponds to a first speech activity state, including: starting silence state (0), speech state (1), pause silence state within a speech turn (2), and silence state between speech turns (3). The first initial speech recognition model is trained by the first speech data set, and the first initial speech recognition model includes an encoder model (such as a Zipformer encoder), a long short-term memory network layer (LSTM), and a linear classification layer. The encoder model (referred to as the encoder) is used to extract acoustic features, the LSTM is used to capture temporal dependencies, and the linear classification layer is used to output the state category probability distribution of each speech frame. During training, the speech activity state of each frame is predicted through forward propagation, resulting in a first output result (i.e., the prediction result). This first output result is then compared with the labeled true state (i.e., the corresponding first speech activity state) to calculate the cross-entropy loss. The cross-entropy loss measures the difference between the model's predicted probability distribution and the true state label. If the cross-entropy loss does not meet the preset convergence condition (i.e., the first preset condition, which means the loss value falls below a certain threshold or the rate of loss decrease slows to a certain level), backpropagation is used to adjust the model parameters and update the model state. After multiple iterations and parameter adjustments, when the updated cross-entropy loss meets the first preset condition, the predetermined convergence criterion is reached, and the training process stops. The resulting model is then the first speech recognition model.

[0056] In the above-mentioned intelligent sentence segmentation active speech detection method based on multi-state time series modeling, when the number of channels is dual, the target speech recognition model is the second speech recognition model, and the second speech recognition model is trained in the following manner: obtaining a second speech data set corresponding to the dual channels, and determining the second speech activity state to which each speech frame in the second speech data set belongs, wherein the second speech activity state includes a real turn-taking time point; training a second initial speech recognition model based on the second speech data set, and determining the second output result of the second initial speech recognition model, wherein the second initial speech recognition model includes two branches, each branch contains a shared encoder model, and the outputs of the two branches are connected to a cross-attention module for feature fusion, and the cross-attention module is then connected to a long short-term memory network and a linear classification layer; determining the state classification loss based on the second output result and the second speech activity state; when the state classification loss does not meet the second preset condition, updating the parameters of the second initial speech recognition model, and determining the updated state classification loss corresponding to the updated second output result, until the updated state classification loss meets the second preset condition, stopping training, and obtaining the second speech recognition model.

[0057] In some embodiments of the present application, in a dual-channel scenario, the target speech recognition model serves as the second speech recognition model. When training the second speech recognition model, the data in the second speech dataset is collected from two independent audio channels, such as recordings of telephone conferences, video conferences, or multi-person conversations. Each speech frame in the second speech dataset corresponds to a second speech activity state (initial silence, speech, silence during turn pauses, and silence during turn breaks). The second speech dataset also includes actual turn-transition times, i.e., the moments when the speaker switches during the conversation. The second initial speech recognition model comprises two branches, each responsible for processing the audio signal of a channel. The front end of each branch is a shared encoder model (such as a Zipformer encoder) for extracting acoustic features. Using a shared encoder across both branches reduces the number of model parameters, improving training efficiency and model generalization. The outputs of the two branches are connected to a cross-attention module, which allows information from the two channels to interact and fuse, thereby generating a more comprehensive and robust feature representation. The feature sequence after cross-attention is fed into an LSTM network for time series modeling, and finally a linear classification layer outputs the state category of each speech frame. Like the single-channel model, the second speech recognition model also classifies the four states. However, it also considers the timing of turn-taking, enabling the model to more accurately identify speaker transitions when processing multi-channel audio.

[0058] When training the second initial speech recognition model using the second speech dataset, the model predicts the state category of each speech frame and compares it with the true annotated state of the second speech activity to calculate the state classification loss (e.g., cross-entropy loss). If the state classification loss does not meet the preset convergence condition (i.e., the second preset condition, which means the loss value falls below a certain threshold or reaches a stable minimum), the model parameters are adjusted and the model weights are optimized through backpropagation to reduce the state classification loss. The model is continuously updated until the state classification loss meets the second preset condition, meaning that the model's prediction accuracy meets the requirement and training is complete, resulting in the second speech recognition model.

[0059] In the above steps, determining the second output result of the second initial speech recognition model includes: determining the first feature corresponding to the first channel in the dual channels and determining the second feature corresponding to the second channel in the dual channels through the encoder model in the second initial speech recognition model; determining the first vector based on the second feature and the query weight matrix; determining the second vector based on the first feature and the key weight matrix; determining the third vector based on the first feature and the value weight matrix; determining the fusion feature based on the dimensions of the first vector, the second vector, the third vector, and the second vector; inputting the fusion feature into the long short-term memory network and the linear classification layer for analysis to obtain the second output result.

[0060] In some embodiments of the present application, the encoder model (such as Zipformer encoder) in the second initial speech recognition model processes the audio signals of the first channel and the second channel in the dual channel respectively, and extracts the first feature F1 and the second feature F2 corresponding to each. In the cross-channel attention interaction, the feature of the second channel is used as the query in the first channel, and the feature F1 is extracted by comparing it with the query weight matrix. Multiply them together to get the first vector, which is At the same time, the features of the first channel are used as the key and value, respectively, with the key weight matrix Sum weight matrix Multiply to get the second vector and the third vector The attention calculation is based on the dot product between the first vector (query) and the second vector (key), and the attention weight is obtained after normalization by the Softmax function. The attention weight is then multiplied by the third vector (value) to obtain the fused feature. The fused feature reflects the information dependency and interaction between the two channels. The calculation of the fused feature obtained by Cross Attention can be expressed as:

[0061]

[0062] in, yes The dimension of , that is, the dimension of the second vector mentioned above.

[0063] The generated fused features are fed into a long short-term memory (LSTM) network for temporal modeling. LSTM can capture long-term dependencies between features over time. The output of the LSTM network is further processed by a linear classification layer, which converts the LSTM output into a probability distribution corresponding to four speech activity states (initial silence, speech, silence during turn pauses, and silence during turn breaks), forming the second output.

[0064] In step S206 of the above-mentioned intelligent sentence segmentation active speech detection method based on multi-state time series modeling, the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states is determined to obtain a state sequence corresponding to each channel, including: determining the maximum value of the probability distribution corresponding to each speech frame; determining the label corresponding to the maximum value as the state label to which the corresponding speech frame belongs, wherein each state label corresponds to a speech activity state; and obtaining the state label to which each speech frame belongs in sequence according to the order of the audio signal to obtain a state sequence.

[0065] In some embodiments of the present application, when the target speech recognition model predicts each speech frame, it outputs a probability distribution, which indicates the probability that the frame belongs to different speech activity states (initial silence, speech, silence during a turn, and silence during a turn break). For example, for a certain frame, the model may output the following probability distribution: initial silence state (0): 0.1, speech state (1): 0.6, silence during a turn break (2): 0.2, and silence during a turn break (3): 0.1. In the above example, the probability value of speech state (1) is the highest, which is 0.6. Therefore, the state label of the speech frame is determined to be speech state (1). Repeat the above steps for each speech frame, that is, obtain the state label corresponding to the maximum value of its probability distribution. Then, according to the time sequence of the audio signal, the state labels of all speech frames are connected in series to form a state sequence, which records the dynamic changes of the speech activity state in the entire audio stream.

[0066] In step S208 of the above-mentioned intelligent sentence segmentation active voice detection method based on multi-state time series modeling, before determining the time of dynamic segmentation according to the state sequence, the method also includes: detecting whether the state sequence contains a preset sequence to obtain a detection result, wherein the preset sequence represents a transition from an initial silent state to a voice state and then to an initial silent state, and the duration of the voice state in the preset sequence is less than the preset duration; when the detection result indicates that the state sequence contains the preset sequence, determining that invalid voice exists in the audio signal, and filtering the preset sequence from the state sequence; when the detection result indicates that the state sequence does not contain the preset sequence, determining that invalid voice does not exist in the audio signal.

[0067] In some embodiments of the present application, before determining the timing of dynamic segmentation based on the state sequence, invalid speech detection and filtering are required to further improve the accuracy and robustness of voice activity detection. Invalid speech is short, non-semantic, or voice activity that does not contain important information, such as human voices in background noise, short coughs, breathing sounds, keystrokes, etc.

[0068] In some embodiments of the present application, the preset sequence refers to a continuous state transition from an initial silent state (state 0) to a speech state (state 1), and then back to the initial silent state (state 0). This transition pattern represents invalid speech in voice activity detection, especially when the duration of the speech state is very short, that is, less than a preset duration. The preset duration can be set according to the specific application scenario and requirements. For example, it can be set to 0.5 seconds or 1 second to filter out short-term interfering sounds.

[0069] Traverse the entire state sequence and check whether there is a pattern of transitioning from state 0 to state 1 and then back to state 0, and the duration of state 1 is less than the preset duration. If a preset sequence that meets the conditions is detected in the state sequence, the detection result will indicate the presence of invalid speech; conversely, if no such preset sequence is detected, the detection result will indicate that there is no invalid speech in the audio signal. In the case where the detection result indicates the presence of invalid speech, the preset sequence needs to be filtered out from the state sequence. That is, for the detected preset sequence, its corresponding audio segment will be considered invalid, will not be counted in any sentence or turn, and will not be used as the basis for dynamic segmentation.

[0070] In step S208 of the above-mentioned intelligent segmentation active speech detection method based on multi-state time series modeling, the time of dynamic segmentation is determined according to the state sequence, including: determining the first consecutive frame number of the silent state of the inter-turn segmentation in the state sequence; when the first consecutive frame number is greater than or equal to the first threshold, determining the time corresponding to the last frame in the first consecutive frame number as the time of dynamic segmentation; determining the second consecutive frame number of the silent state of the pause within the turn in the state sequence; when the second consecutive frame number is greater than or equal to the second threshold, determining the time corresponding to the last frame in the second consecutive frame number as the time of dynamic segmentation.

[0071] In some embodiments of the present application, dynamic segmentation is used in voice activity detection (VAD) to identify segment boundaries or the end points of speaker turns in a speech stream. In state-sequence-based dynamic segmentation methods, segmentation timing is determined by monitoring the number of consecutive frames in which a specific silence state occurs, namely, the number of consecutive occurrences of the inter-turn segmentation silence state (state 3) and the intra-turn pause silence state (state 2) in the state sequence.

[0072] Specifically, the state sequence is traversed to find all consecutive frames of state 3 (silent state during turns) to obtain the first consecutive frame count, and all consecutive frames of state 2 (silent state during pauses within turns) to obtain the second consecutive frame count. If the first consecutive frame count is greater than or equal to the first threshold, the current turn is considered to have ended, and the time of dynamic punctuation is the time corresponding to the last frame in the first consecutive frame count. If the second consecutive frame count is greater than or equal to the second threshold, punctuation is also triggered. This mechanism is designed to handle pauses in speech flow caused by prolonged thinking or hesitation, avoiding long system wait times. The last frame of the second consecutive frame count is used as the punctuation point. The first and second thresholds are fixed values ​​that can be preset or adaptively adjusted based on the application scenario and data characteristics.

[0073] The intelligent segmentation active speech detection method based on multi-state temporal modeling provided in the embodiment of the present application can more accurately identify the real sentence boundaries in the speech flow and significantly reduce the segmentation errors caused by pauses within the turn by introducing fine-grained speech state division and segmentation rules based on state duration. The background human voice filtering mechanism based on state transition avoids the effective speech loss that may be caused by traditional noise suppression methods and improves the robustness of the system in complex acoustic environments. In addition, in the dual-channel scenario, by integrating the Cross Attention mechanism to fuse information between channels, the model can effectively utilize multi-channel voice information and assist each other, thereby more accurately segmenting and more quickly predicting turn transitions, thereby improving the interactive experience of the dialogue system.

[0074] In order to better understand the solution provided by the embodiment of the present application, the following Figure 3 and Figure 4 Explain the construction of the model network and the structure of the dual-channel model. Figure 3 is a state transition diagram according to an embodiment of the present application, Figure 4 This is a structural diagram of a dual-channel model according to an embodiment of the present application. The following is a description:

[0075] In constructing the model network, an encoder based on a pre-trained speech recognition model is used as a feature extractor for the input speech signal. Time series modeling is performed using an LSTM network. A linear classification layer is introduced to model the state of each frame in the speech signal. There are four states: 0 (starting silence), 1 (speech), 2 (inter-turn pause silence), and 3 (inter-turn pause silence). Figure 3As shown, when the number of consecutive frames detected in state 3 exceeds the sentence break threshold (the first threshold mentioned above), a sentence break is determined. When the number of consecutive frames detected in state 2 exceeds the strong sentence break threshold (the second threshold mentioned above), forced sentence break is performed. Background vocalization is detected by detecting whether there is a 0->1->0 transition pattern in the state sequence. Because short-term background vocal interference is filtered out during state transitions, the loss of effective speech that can occur with traditional noise suppression methods is avoided, improving the model's robustness in complex acoustic environments.

[0076] In the dual-channel scenario, the Cross Attention mechanism is introduced to fuse information between channels, such as Figure 4 As shown in the figure, the model takes two-channel audio (wav1 and wav2) as input and outputs active speech detection results for the corresponding two channels (out1 and out2). Model parameters, including encoder and LSTM network parameters, are shared. Information exchange between different channels enhances robustness to background speech in the own channel. While simultaneously perceiving its own channel, the model also focuses on and perceives speech activity in other channels, enabling it to better identify background vocal interference in its own channel and more effectively suppress it, thereby improving the accuracy of active speech detection in its own channel.

[0077] The solution in the embodiment of the present application can more accurately detect the boundaries of real speech pauses and ends by introducing fine-grained speech state division and segmentation rules based on state duration, significantly reducing segmentation errors caused by short pauses, thereby greatly reducing speech recognition insertion errors. Experiments have shown that the ability to filter background human voices is significantly improved, and the proportion of correct pause judgments is significantly increased (73.39%->86.07%). At the same time, the delay in judging pauses is significantly reduced, and the 90% percentile delay is reduced from 1374 milliseconds to 770 milliseconds, a relative reduction of 44%. Therefore, the intelligent segmentation active speech detection method based on multi-state time series modeling in the embodiment of the present application has a higher detection accuracy than the traditional VAD algorithm.

[0078] Figure 5 is a structural diagram of an intelligent sentence segmentation active speech detection device based on multi-state temporal modeling according to an embodiment of the present application, such as Figure 5 As shown, the device includes:

[0079] A receiving module 30, configured to receive an audio signal of at least one channel;

[0080] An extraction module 32, configured to extract an acoustic feature sequence of the audio signal using a target speech recognition model corresponding to the number of channels;

[0081] A first determination module 34 is configured to determine a probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, thereby obtaining a state sequence corresponding to each channel, wherein the speech activity state includes at least one of the following: an initial silence state, a speech state, a pause silence state within a turn, and a pause silence state between turns;

[0082] The second determining module 36 is configured to determine the time when a sentence break occurs in the audio signal according to the state sequence.

[0083] In the receiving module, extraction module, first determination module and second determination module in the above-mentioned intelligent sentence segmentation activity voice detection device based on multi-state time series modeling, the purpose of refined dynamic sentence segmentation is achieved, thereby realizing the technical effect of improving the accuracy and robustness of voice activity detection, and further solving the technical problem of incorrect sentence segmentation in voice activity detection based on a fixed silence threshold in related technologies.

[0084] The above-mentioned intelligent sentence segmentation active speech detection device based on multi-state time series modeling also includes a training module 38. When the number of channels is single, the target speech recognition model is the first speech recognition model. The training module is used to train the first speech recognition model. Specifically, the first speech recognition model is trained in the following manner: obtaining a first speech data set corresponding to a single channel, and determining the first speech activity state to which each speech frame in the first speech data set belongs; training a first initial speech recognition model based on the first speech data set, and determining the first output result of the first initial speech recognition model, wherein the first initial speech recognition model includes an encoder model, a long short-term memory network layer and a linear classification layer; determining the cross-entropy loss based on the first output result and the first speech activity state; when the cross-entropy loss does not meet the first preset condition, updating the parameters of the first initial speech recognition model, and determining the updated cross-entropy loss corresponding to the updated first output result, until the updated cross-entropy loss meets the first preset condition, stopping the training to obtain the first speech recognition model.

[0085] In the above-mentioned intelligent sentence segmentation active speech detection device based on multi-state time series modeling, when the number of channels is dual, the target speech recognition model is the second speech recognition model, and the training module is used to train the second speech recognition model. Specifically, the second speech recognition model is trained in the following manner: obtaining a second speech data set corresponding to the dual channels, and determining the second speech activity state to which each speech frame in the second speech data set belongs, wherein the second speech activity state includes a real turn-taking time point; training a second initial speech recognition model based on the second speech data set, and determining a second output result of the second initial speech recognition model, wherein the second initial speech recognition model includes two branches, each branch includes a shared encoder model, and the outputs of the two branches are connected to a cross-attention module for feature fusion, and the cross-attention module is then connected to a long short-term memory network and a linear classification layer; determining a state classification loss based on the second output result and the second speech activity state; when the state classification loss does not meet the second preset condition, updating the parameters of the second initial speech recognition model, and determining the updated state classification loss corresponding to the updated second output result, until the updated state classification loss meets the second preset condition, stopping training, and obtaining the second speech recognition model.

[0086] In the training module of the above-mentioned intelligent sentence segmentation active speech detection device based on multi-state time series modeling, the training module is also used to determine the first feature corresponding to the first channel in the dual channels and the second feature corresponding to the second channel in the dual channels through the encoder model in the second initial speech recognition model; determine the first vector based on the second feature and the query weight matrix; determine the second vector based on the first feature and the key weight matrix; determine the third vector based on the first feature and the value weight matrix; determine the fusion feature based on the dimensions of the first vector, the second vector, the third vector, and the second vector; input the fusion feature into the long short-term memory network and the linear classification layer for analysis to obtain the second output result.

[0087] In the first determination module of the above-mentioned intelligent sentence segmentation active speech detection device based on multi-state time series modeling, the first determination module is also used to determine the maximum value of the probability distribution corresponding to each speech frame; determine the label corresponding to the maximum value as the state label to which the corresponding speech frame belongs, wherein each state label corresponds to a speech activity state; obtain the state label to which each speech frame belongs in sequence according to the order of the audio signal to obtain a state sequence.

[0088] In the second determination module in the above-mentioned intelligent sentence segmentation active voice detection device based on multi-state time series modeling, the second determination module is also used to detect whether a preset sequence is included in the state sequence to obtain a detection result, wherein the preset sequence represents a transition from an initial silent state to a voice state and then to an initial silent state, and the duration of the voice state in the preset sequence is less than the preset duration; when the detection result indicates that the preset sequence is included in the state sequence, it is determined that invalid voice exists in the audio signal, and the preset sequence is filtered from the state sequence; when the detection result indicates that the preset sequence is not included in the state sequence, it is determined that invalid voice does not exist in the audio signal.

[0089] In the second determination module in the above-mentioned intelligent sentence-segmentation active speech detection device based on multi-state time series modeling, the second determination module is also used to determine the first consecutive frame number of the silent state of the inter-turn sentence segmentation in the state sequence; when the first consecutive frame number is greater than or equal to the first threshold, the time corresponding to the last frame in the first consecutive frame number is determined as the time of dynamic sentence segmentation; determine the second consecutive frame number of the silent state of the pause within the turn in the state sequence; when the second consecutive frame number is greater than or equal to the second threshold, the time corresponding to the last frame in the second consecutive frame number is determined as the time of dynamic sentence segmentation.

[0090] It should be noted that Figure 5 The intelligent sentence segmentation active speech detection device based on multi-state temporal modeling is used to perform Figure 2 The intelligent sentence segmentation active voice detection method based on multi-state temporal modeling is shown, so the relevant explanations in the above-mentioned intelligent sentence segmentation active voice detection method based on multi-state temporal modeling are also applicable to the intelligent sentence segmentation active voice detection device based on multi-state temporal modeling, and will not be repeated here.

[0091] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute program instructions to implement the following functions: receiving an audio signal of at least one channel; using a target speech recognition model corresponding to the number of channels to extract an acoustic feature sequence of the audio signal; determining the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, and obtaining a state sequence corresponding to each channel, wherein the speech activity state includes at least one of the following: an initial silence state, a speech state, a silence state for pauses within a turn, and a silence state for breaks between turns; and determining the time when a break occurs in the audio signal based on the state sequence.

[0092] It should be noted that the above electronic equipment is used to perform Figure 2The intelligent sentence segmentation active voice detection method based on multi-state temporal modeling is shown, so the relevant explanations in the above-mentioned intelligent sentence segmentation active voice detection method based on multi-state temporal modeling are also applicable to this electronic device and will not be repeated here.

[0093] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following intelligent sentence-segmentation active speech detection method based on multi-state time series modeling by running the computer program: receiving an audio signal of at least one channel; using a target speech recognition model corresponding to the number of channels to extract an acoustic feature sequence of the audio signal; determining the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to a different speech activity state, and obtaining a state sequence corresponding to each channel, wherein the speech activity state includes at least one of the following: an initial silence state, a speech state, a pause silence state within a turn, and a sentence-segmentation silence state between turns; and determining the time when a sentence segmentation occurs in the audio signal based on the state sequence.

[0094] It should be noted that the above non-volatile storage medium is used to execute Figure 2 The intelligent sentence segmentation active voice detection method based on multi-state temporal modeling is shown, so the relevant explanations in the above-mentioned intelligent sentence segmentation active voice detection method based on multi-state temporal modeling are also applicable to the non-volatile storage medium, and will not be repeated here.

[0095] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the intelligent sentence segmentation active speech detection method based on multi-state temporal modeling in each embodiment of the present application.

[0096] An embodiment of the present application also provides a computer program, which, when executed by a processor, implements the steps of the intelligent sentence segmentation active speech detection method based on multi-state temporal modeling in each embodiment of the present application.

[0097] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0098] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0100] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0101] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program code.

[0103] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An intelligent sentence segmentation active speech detection method based on multi-state temporal modeling, characterized in that: include: receiving an audio signal of at least one channel; Extracting an acoustic feature sequence of the audio signal using a target speech recognition model corresponding to the number of channels; Determining the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, and obtaining a state sequence corresponding to each channel, wherein the speech activity state includes at least one of the following: an initial silence state, a speech state, a pause silence state within a turn, and a pause silence state between turns; The time when a sentence break occurs in the audio signal is determined according to the state sequence.

2. The method according to claim 1, characterized in that In the case where the number of channels is single, the target speech recognition model is a first speech recognition model, and the first speech recognition model is trained in the following manner: Acquire a first speech data set corresponding to a single channel, and determine a first speech activity state to which each speech frame in the first speech data set belongs; Training a first initial speech recognition model based on the first speech data set, and determining a first output result of the first initial speech recognition model, wherein the first initial speech recognition model includes an encoder model, a long short-term memory network layer, and a linear classification layer; determining a cross entropy loss based on the first output result and the first voice activity state; When the cross entropy loss does not meet the first preset condition, the parameters of the first initial speech recognition model are updated, and the updated cross entropy loss corresponding to the updated first output result is determined, and the training is stopped until the updated cross entropy loss meets the first preset condition to obtain the first speech recognition model.

3. The method according to claim 1, characterized in that In the case where the number of channels is two, the target speech recognition model is a second speech recognition model, and the second speech recognition model is trained in the following manner: Acquire a second speech data set corresponding to the dual channels, and determine a second speech activity state to which each speech frame in the second speech data set belongs, wherein the second speech activity state includes a real turn-taking time point; Training a second initial speech recognition model based on the second speech dataset, and determining a second output result of the second initial speech recognition model, wherein the second initial speech recognition model includes two branches, each branch includes a shared encoder model, and the outputs of the two branches are connected to a cross-attention module for feature fusion, and the cross-attention module is then connected to a long short-term memory network and a linear classification layer; determining a state classification loss according to the second output result and the second voice activity state; When the state classification loss does not meet the second preset condition, the parameters of the second initial speech recognition model are updated, and the updated state classification loss corresponding to the updated second output result is determined. Training is stopped until the updated state classification loss meets the second preset condition to obtain the second speech recognition model.

4. The method according to claim 3, characterized in that Determining a second output result of the second initial speech recognition model includes: Determining, by means of an encoder model in the second initial speech recognition model, a first feature corresponding to a first channel in the dual channels, and determining a second feature corresponding to a second channel in the dual channels; determining a first vector based on the second feature and a query weight matrix; determining a second vector based on the first feature and the key weight matrix; Determining a third vector based on the first feature and the value weight matrix; determining a fusion feature according to the dimensions of the first vector, the second vector, the third vector, and the second vector; The fusion feature is input into the long short-term memory network and the linear classification layer for analysis to obtain the second output result.

5. The method according to claim 1, wherein Determining the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, and obtaining a state sequence corresponding to each channel, including: Determine the maximum value of the probability distribution corresponding to each speech frame; Determining the label corresponding to the maximum value as the state label to which the corresponding speech frame belongs, wherein each state label corresponds to a speech activity state; The state label of each speech frame is acquired in sequence according to the order of the audio signal to obtain the state sequence.

6. The method according to claim 1, characterized in that Before determining the time for dynamic segmentation based on the state sequence, the method further includes: detecting whether the state sequence includes a preset sequence, and obtaining a detection result, wherein the preset sequence represents a transition from an initial silent state to a speech state and then to the initial silent state, and a duration of the speech state in the preset sequence is less than a preset duration; If the detection result indicates that the state sequence includes the preset sequence, determining that invalid speech exists in the audio signal, and filtering the preset sequence from the state sequence; When the detection result indicates that the state sequence does not include the preset sequence, it is determined that the invalid speech does not exist in the audio signal.

7. The method according to claim 1, characterized in that Determining the time of dynamic segmentation based on the state sequence includes: Determine the first consecutive frame number of the inter-turn silence state in the state sequence; When the first number of consecutive frames is greater than or equal to a first threshold, determining a time corresponding to a last frame in the first number of consecutive frames as a time of dynamic sentence segmentation; Determining a second consecutive number of frames of a pause-silent state within a turn in the state sequence; When the second number of consecutive frames is greater than or equal to a second threshold, the time corresponding to the last frame in the second number of consecutive frames is determined as the time of dynamic sentence segmentation.

8. An intelligent sentence segmentation active speech detection device based on multi-state temporal modeling, characterized in that: include: A receiving module, configured to receive an audio signal of at least one channel; an extraction module, configured to extract an acoustic feature sequence of the audio signal using a target speech recognition model corresponding to the number of channels; a first determination module, configured to determine a probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, and obtain a state sequence corresponding to each channel, wherein the speech activity state includes at least one of the following: an initial silence state, a speech state, a pause silence state within a turn, and a pause silence state between turns; The second determining module is configured to determine a time when a sentence break occurs in the audio signal according to the state sequence.

9. An electronic device, characterized in that: include: a memory for storing program instructions; The processor is connected to the memory and is used to execute program instructions that implement the following functions: receiving an audio signal from at least one channel; extracting an acoustic feature sequence from the audio signal using a target speech recognition model corresponding to the number of channels; determining a probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, and obtaining a state sequence corresponding to each channel, wherein the speech activity state includes at least one of the following: an initial silence state, a speech state, a silence state for a pause within a turn, and a silence state for a pause between turns; and determining, based on the state sequence, a time when a pause occurs in the audio signal.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the intelligent sentence segmentation active speech detection method based on multi-state temporal modeling as described in any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the intelligent sentence segmentation active speech detection method based on multi-state temporal modeling described in any one of claims 1 to 7 is implemented.

Images