Intelligent sentence segmentation active speech detection method and device based on multi-state temporal modeling

Abstract

This application discloses an intelligent method and apparatus for detecting active speech with sentence segmentation based on multi-state temporal modeling. The method includes: receiving audio signals from at least one channel; extracting acoustic feature sequences from the audio signals using a target speech recognition model corresponding to the number of channels; determining the probability distribution of each speech frame corresponding to the acoustic feature sequences belonging to different speech activity states, obtaining a state sequence corresponding to each channel, wherein the speech activity state includes at least one of the following: initial silence state, speech state, intra-turn pause silence state, and inter-turn sentence segmentation silence state; and determining the time of sentence segmentation in the audio signal based on the state sequence. This application solves the technical problem of erroneous sentence segmentation in speech activity detection based on a fixed silence threshold in related technologies.

Description

Technical Field

[0001] This application relates to the field of signal processing, and more specifically, to an intelligent method and apparatus for detecting active speech based on multi-state temporal modeling. Background Technology

[0002] Speech activity detection (VAD), a fundamental step in speech signal processing, aims to distinguish between effective speech segments and non-speech segments in a speech signal. Traditional VAD methods, such as strategies based on energy thresholds or silence duration thresholds, are simple and intuitive, but suffer from the problem of incorrect sentence segmentation.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides an intelligent sentence segmentation activity speech detection method and apparatus based on multi-state temporal modeling, so as to at least solve the technical problem of erroneous sentence segmentation in speech activity detection based on a fixed silence threshold in related technologies.

[0005] According to one aspect of the embodiments of this application, an intelligent speech activity detection method based on multi-state temporal modeling is provided, comprising: receiving audio signals from at least one channel; extracting acoustic feature sequences of the audio signals using a target speech recognition model corresponding to the number of channels; determining the probability distribution of each speech frame corresponding to the acoustic feature sequences belonging to different speech activity states, thereby obtaining a state sequence corresponding to each channel, wherein the speech activity states include at least one of the following: initial silence state, speech state, pause silence state within a turn, and pause silence state between turns; and determining the time when a sentence breaks occur in the audio signal based on the state sequence.

[0006] Optionally, when the number of channels is single-channel, the target speech recognition model is a first speech recognition model, which is trained as follows: First speech dataset corresponding to a single channel is obtained, and the first speech activity state to which each speech frame in the first speech dataset belongs is determined; a first initial speech recognition model is trained based on the first speech dataset, and the first output result of the first initial speech recognition model is determined, wherein the first initial speech recognition model includes an encoder model, a long short-term memory network layer, and a linear classification layer; cross-entropy loss is determined based on the first output result and the first speech activity state; when the cross-entropy loss does not meet a first preset condition, the parameters of the first initial speech recognition model are updated, and the updated cross-entropy loss corresponding to the updated first output result is determined, until the updated cross-entropy loss meets the first preset condition, training stops, and the first speech recognition model is obtained.

[0007] Optionally, when the number of channels is dual-channel, the target speech recognition model is a second speech recognition model, which is trained as follows: The second speech dataset corresponding to the dual channels is obtained, and the second speech activity state to which each speech frame in the second speech dataset belongs is determined, wherein the second speech activity state includes the actual turn-taking time points; a second initial speech recognition model is trained based on the second speech dataset, and the second output result of the second initial speech recognition model is determined, wherein the second initial speech recognition model includes two branches, each branch containing a shared encoder model, and the outputs of the two branches are connected to a cross-attention module for feature fusion, after which a long short-term memory network and a linear classification layer are connected; a state classification loss is determined based on the second output result and the second speech activity state; when the state classification loss does not meet the second preset condition, the parameters of the second initial speech recognition model are updated, and the updated state classification loss corresponding to the updated second output result is determined, until the updated state classification loss meets the second preset condition, training stops, and the second speech recognition model is obtained.

[0008] Optionally, determining the second output result of the second initial speech recognition model includes: determining the first feature corresponding to the first channel in the dual-channel model and the second feature corresponding to the second channel in the dual-channel model using the encoder model in the second initial speech recognition model; determining the first vector based on the second feature and the query weight matrix; determining the second vector based on the first feature and the key weight matrix; determining the third vector based on the first feature and the value weight matrix; determining the fusion feature based on the dimensions of the first vector, the second vector, the third vector, and the second vector; and inputting the fusion feature into a long short-term memory network and a linear classification layer for analysis to obtain the second output result.

[0009] Optionally, the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states is determined to obtain the state sequence corresponding to each channel, including: determining the maximum value of the probability distribution corresponding to each speech frame; determining the label corresponding to the maximum value as the state label of the corresponding speech frame, wherein each state label corresponds to a speech activity state; and sequentially obtaining the state label of each speech frame according to the order of the audio signal to obtain the state sequence.

[0010] Optionally, before determining the time of dynamic sentence segmentation based on the state sequence, the method further includes: detecting whether the state sequence contains a preset sequence to obtain a detection result, wherein the preset sequence represents the transition from an initial silence state to a speech state and back to an initial silence state, and the duration of the speech state in the preset sequence is less than a preset duration; if the detection result indicates that the state sequence contains a preset sequence, determining that there is invalid speech in the audio signal and filtering the preset sequence from the state sequence; if the detection result indicates that the state sequence does not contain a preset sequence, determining that there is no invalid speech in the audio signal.

[0011] Optionally, determining the time of dynamic sentence interruption based on the state sequence includes: determining the first consecutive frame number of the inter-turn pause silence state in the state sequence; if the first consecutive frame number is greater than or equal to a first threshold, determining the time corresponding to the last frame in the first consecutive frame number as the time of dynamic sentence interruption; determining the second consecutive frame number of the inter-turn pause silence state in the state sequence; if the second consecutive frame number is greater than or equal to a second threshold, determining the time corresponding to the last frame in the second consecutive frame number as the time of dynamic sentence interruption.

[0012] According to another aspect of the embodiments of this application, an intelligent speech activity detection device based on multi-state temporal modeling is also provided, comprising: a receiving module for receiving audio signals from at least one channel; an extraction module for extracting acoustic feature sequences of the audio signals using a target speech recognition model corresponding to the number of channels; a first determining module for determining the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, thereby obtaining a state sequence corresponding to each channel, wherein the speech activity state includes at least one of the following: initial silence state, speech state, pause silence state within a turn, and sentence interruption silence state between turns; and a second determining module for determining the time when sentence interruption occurs in the audio signal based on the state sequence.

[0013] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory for storing program instructions; and a processor connected to the memory for executing program instructions to perform the following functions: receiving audio signals from at least one channel; extracting acoustic feature sequences of the audio signals using a target speech recognition model corresponding to the number of channels; determining the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, thereby obtaining a state sequence corresponding to each channel, wherein the speech activity state includes at least one of the following: initial silence state, speech state, pause silence state within a turn, and pause silence state between turns; and determining the time when a pause occurs in the audio signal based on the state sequence.

[0014] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-described intelligent sentence segmentation active speech detection method based on multi-state temporal modeling by running the computer program.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the above-described intelligent sentence segmentation active speech detection method based on multi-state temporal modeling.

[0016] In this embodiment, audio signals from at least one channel are received; an acoustic feature sequence of the audio signal is extracted using a target speech recognition model corresponding to the number of channels; the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states is determined, resulting in a state sequence corresponding to each channel. The speech activity state includes at least one of the following: initial silence state, speech state, pause silence state within a turn, and pause silence state between turns. Based on the state sequence, the time of sentence segmentation in the audio signal is determined, achieving the purpose of refined dynamic sentence segmentation. This improves the accuracy and robustness of speech activity detection, thereby solving the technical problem of erroneous sentence segmentation in speech activity detection based on a fixed silence threshold in related technologies. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing an intelligent sentence segmentation active speech detection method based on multi-state temporal modeling, according to an embodiment of this application.

[0019] Figure 2 This is a flowchart of an intelligent sentence segmentation active speech detection method based on multi-state temporal modeling according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of a state transition according to an embodiment of this application;

[0021] Figure 4 This is a structural diagram of a dual-channel model according to an embodiment of this application;

[0022] Figure 5 This is a structural diagram of an intelligent sentence segmentation active speech detection device based on multi-state temporal modeling according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] The information collected in this application embodiment is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant regions, and necessary confidentiality measures have been taken. It does not violate public order and good morals, and provides corresponding operation entry points for users to choose to authorize or reject the automated decision results. If the user chooses to reject, the process will proceed to the expert decision-making process.

[0026] First, some nouns or terms that appear in the explanation of the embodiments of this application shall be interpreted as follows:

[0027] Dynamic Sentence Segmentation (also known as Dynamic Utterance Detection): This technique determines where a speaker's current sentence or segment ends, essentially establishing sentence boundaries. It primarily segments the speech of a single speaker. Instead of relying solely on prolonged silences, it combines acoustic features (such as prosody and intonation), semantic information, and clues about possible grammatical structures to determine whether a segment should end.

[0028] Turn-Taking (or Speaker Turn Detection): This refers to the detection of a switch in the speaking process during a conversation, where one person begins speaking while another begins speaking. It involves interaction between multiple speakers. Typically, it involves one speaker stopping and another speaker starting.

[0029] Traditional Voice Activity Detection (VAD) algorithms typically rely on energy thresholds or silence duration thresholds. While the logical rules are simple and intuitive, they face numerous challenges in practical applications. In real-world call scenarios, especially in complex acoustic environments and natural dialogue scenarios, energy-based VAD methods exhibit significant shortcomings. Static thresholds struggle to adapt to varying speech rates, volumes, and pause habits, and pauses of varying lengths (such as thinking or interjections) may occur within the same turn. These issues make it difficult for fixed silence thresholds to accurately distinguish between pauses within a turn and the end of a turn, often resulting in incorrect sentence segmentation or omissions. Furthermore, traditional VAD methods are susceptible to background noise, easily misjudged as valid speech or masking the end of genuine speech, affecting the accuracy of sentence segmentation. In dialogue systems requiring rapid responses, turn-switching mechanisms based on posterior silence detection are slow and cannot meet the demands of real-time interaction.

[0030] To address the problems existing in related technologies, this application provides an intelligent sentence segmentation method for active speech detection based on multi-state temporal modeling. This method utilizes multi-state temporal modeling and cross-channel attention interaction for dynamic sentence segmentation, dividing the acoustic features (such as prosody and intonation) and implicit semantic information of speech into four states. Dynamic sentence segmentation and background voice filtering are achieved by judging the duration and state transitions of these four states. Furthermore, in a dual-channel scenario, a Cross Attention mechanism is introduced to fuse information between channels, further enhancing the background voice filtering capability. This method can run on... Figure 1 The computer terminal shown is described below.

[0031] The intelligent sentence segmentation active speech detection method based on multi-state temporal modeling provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing an intelligent sentence-segmentation active speech detection method based on multi-state temporal modeling is shown. Figure 1As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0032] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0033] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the intelligent sentence segmentation active speech detection method based on multi-state temporal modeling in this embodiment of the application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned intelligent sentence segmentation active speech detection method based on multi-state temporal modeling. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0034] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a radio frequency (RF) module, used for wireless communication with the Internet.

[0035] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0036] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0037] In the above operating environment, this application provides an embodiment of an intelligent sentence segmentation active speech detection method based on multi-state temporal modeling. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.

[0038] Figure 2 This is a flowchart of an intelligent sentence segmentation active speech detection method based on multi-state temporal modeling according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0039] Step S202: Receive audio signals from at least one channel.

[0040] In step S202 above, in speech signal processing and communication technology, audio signals can be transmitted through different channels, each channel representing an independent sound source or spatial location. Here, a channel refers to an independent path for the audio data stream, which can be physical (e.g., different microphones, different speakers) or logical (e.g., the respective audio streams of the two parties in a telephone call). At least one channel of audio signal includes two cases: single-channel audio signal and multi-channel audio signal. A single-channel audio signal refers to all audio signals being mixed together and input into the system as a single data stream. Examples include a recorded music file or a recording of a person speaking alone. In a single-channel scenario, a unified audio signal is received. A multi-channel audio signal refers to audio signals received from multiple sources or devices, each signal having its own independent channel; multi-channel can be dual-channel. For example, in meeting recordings, teleconferences, or multi-person calls, each participant's voice can be picked up by their respective microphone, forming an independent audio channel.

[0041] Step S204: Use a target speech recognition model corresponding to the number of channels to extract the acoustic feature sequence of the audio signal.

[0042] In step S204 above, the target speech recognition model is selected based on the number of channels of the received audio signal. If the received audio signal is a single-channel signal, a model optimized for single-channel audio is used, i.e., the first speech recognition model described below; if the audio signal is multi-channel, a multi-channel model is required, i.e., the second speech recognition model described below. The target speech recognition model uses a pre-trained Zipformer encoder to extract acoustic features from the input audio signal, including implicit speech rhythm, intonation, and other feature information that helps in speech activity detection, to obtain an acoustic feature sequence.

[0043] Step S206: Determine the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, and obtain the state sequence corresponding to each channel. The speech activity state includes at least one of the following: initial silence state, speech state, pause silence state within a turn, and pause silence state between turns.

[0044] In step S206 above, the acoustic feature sequence of each channel is analyzed by the temporal modeling module (such as an LSTM network) in the target speech recognition model to predict the speech activity state of each frame of audio (usually audio is divided into multiple small time segments, each time segment corresponding to a frame). In this embodiment, the speech activity state is divided into the following types:

[0045] 1. Initial silence state (e.g., represented by 0): The silence phase before the start of a speech segment, usually used to filter out non-speech parts at the beginning of the recording;

[0046] 2. Speech state (e.g., represented by 1): The state during effective speech activity, including the actual content of the speaker's speech;

[0047] 3. Intra-turn pauses and silences (e.g., represented by 2): brief silences or pauses that occur within the same speaker's turn, such as pauses when the speaker is thinking, or natural pauses within a sentence, but the speaker has not ended the current turn;

[0048] 4. Silence during turn-taking (e.g., represented by 3): The silence between turns by a speaker (or two different speakers) usually indicates that a complete statement or turn has ended and another is about to begin.

[0049] For each speech frame, the target speech recognition model outputs a probability distribution representing the likelihood that the frame belongs to one of the four states mentioned above. For example, if the target speech recognition model outputs: initial silence state 0.1, speech state 0.6, pause silence state within a turn 0.2, and pause silence state between turns 0.1, then the frame is more likely to be in the speech state. By concatenating the predicted states of each frame, the model generates a state sequence for each channel, which records the changes in the speech activity state throughout the audio signal. For example, the state sequence could be 11121130, indicating that the audio signal of this channel first consists of a continuous speech (multiple states 1), followed by a brief pause (state 2), then the speech continues, finally a complete sentence ends and silence occurs (state 3), followed by initial silence (state 0).

[0050] The above methods enable a refined understanding of speech activity in each channel, distinguishing between pauses within a turn, silences at the end of a turn, and initial silences, thereby allowing for accurate dynamic sentence segmentation and turn-taking prediction.

[0051] Step S208: Determine the time when a sentence breaks in the audio signal based on the state sequence.

[0052] In step S208 above, the time when the sentence breaks can be determined by parsing the obtained state sequence.

[0053] Through steps S202 to S208, the goal of refined dynamic sentence segmentation is achieved, thereby improving the accuracy and robustness of speech activity detection. This solves the technical problem of erroneous sentence segmentation in speech activity detection based on a fixed silence threshold in related technologies. The following is a further explanation.

[0054] In the above-mentioned intelligent sentence segmentation activity speech detection method based on multi-state temporal modeling, when the number of channels is single, the target speech recognition model is a first speech recognition model. The first speech recognition model is trained in the following way: acquiring the first speech dataset corresponding to the single channel and determining the first speech activity state to which each speech frame in the first speech dataset belongs; training a first initial speech recognition model based on the first speech dataset and determining the first output result of the first initial speech recognition model, wherein the first initial speech recognition model includes an encoder model, a long short-term memory network layer and a linear classification layer; determining the cross-entropy loss based on the first output result and the first speech activity state; when the cross-entropy loss does not meet the first preset condition, updating the parameters of the first initial speech recognition model and determining the updated cross-entropy loss corresponding to the updated first output result, until the updated cross-entropy loss meets the first preset condition and training stops, thus obtaining the first speech recognition model.

[0055] In some embodiments of this application, when the number of channels is single, the target speech recognition model specifically refers to the first speech recognition model. When training the first initial speech recognition model, the audio data collected by a single sound source or a single microphone is determined as the first speech dataset. Each speech frame in the first speech dataset corresponds to a first speech activity state, including: initial silence state (0), speech state (1), pause silence state within a turn (2), and pause silence state between turns (3). The first initial speech recognition model is trained using the first speech dataset. The first initial speech recognition model includes an encoder model (such as a Zipformer encoder), a long short-term memory network layer (LSTM), and a linear classification layer. The encoder model (referred to as the encoder) is used to extract acoustic features, the LSTM is used to capture temporal dependencies, and the linear classification layer is used to output the state category probability distribution of each speech frame. During training, the speech activity state of each frame is predicted through forward propagation to obtain the first output result (i.e., the prediction result). The first output result is then compared with the labeled true state (i.e., the corresponding first speech activity state) to calculate the cross-entropy loss. The cross-entropy loss measures the difference between the model's predicted probability distribution and the true state label. If the cross-entropy loss does not meet the preset convergence condition (i.e., the first preset condition, meaning the loss value is below a certain threshold or the loss decreases at a certain rate), backpropagation is required to adjust the model parameters and update the model state. With multiple iterations and parameter adjustments, when the updated cross-entropy loss meets the first preset condition, i.e., the predetermined convergence criterion is reached, the training process stops, and the model obtained at this point is the first speech recognition model.

[0056] In the above-mentioned intelligent sentence segmentation activity speech detection method based on multi-state temporal modeling, when the number of channels is two, the target speech recognition model is a second speech recognition model. The second speech recognition model is trained in the following way: First, the second speech dataset corresponding to the two channels is obtained, and the second speech activity state to which each speech frame in the second speech dataset belongs is determined. The second speech activity state includes the actual turn-taking time points. Second, a second initial speech recognition model is trained based on the second speech dataset, and the second output result of the second initial speech recognition model is determined. The second initial speech recognition model includes two branches, each containing a shared encoder model. The outputs of the two branches are connected to a cross-attention module for feature fusion. After the cross-attention module, a long short-term memory network and a linear classification layer are connected. The state classification loss is determined based on the second output result and the second speech activity state. When the state classification loss does not meet the second preset condition, the parameters of the second initial speech recognition model are updated, and the updated state classification loss corresponding to the updated second output result is determined. Training stops when the updated state classification loss meets the second preset condition, thus obtaining the second speech recognition model.

[0057] In some embodiments of this application, in a dual-channel scenario, the target speech recognition model serves as the second speech recognition model. When training the second speech recognition model, the data in the second speech dataset is collected from two independent audio channels, such as recordings of telephone conferences, video conferences, or multi-person conversations. Each speech frame in the second speech dataset corresponds to a second speech activity state (initial silence, speech, pause silence within a turn, pause silence between turns). The second speech dataset also includes actual turn-switching times, i.e., the moments when speakers switch during a conversation. The second initial speech recognition model comprises two branches, each responsible for processing the audio signal of one channel. The front end of each branch is a shared encoder model (such as a Zipformer encoder) used to extract acoustic features. Using a shared encoder for both branches reduces the number of model parameters, improves training efficiency, and enhances the model's generalization ability. The outputs of the two branches are connected to a cross-attention module, which allows information interaction and fusion between the two channels, thereby generating a more comprehensive and robust feature representation. The feature sequences after cross-attention are fed into an LSTM network for temporal modeling, and finally, a linear classification layer outputs the state category of each speech frame. Similar to the single-channel model, the second speech recognition model also performs four-state classification. However, in addition to this, the timing of turn-taking needs to be considered to enable the model to more accurately identify the timing of speaker transitions when processing multi-channel audio.

[0058] When training the second initial speech recognition model using the second speech dataset, the second initial speech recognition model predicts the state category of each speech frame and compares it with the truth-labeled second speech activity states to calculate the state classification loss (such as cross-entropy loss). If the state classification loss does not meet the preset convergence condition (i.e., the second preset condition, meaning the loss value is below a certain threshold or reaches a stable minimum value), the model parameters need to be adjusted through backpropagation to optimize the model weights and reduce the state classification loss. The model is continuously updated until the state classification loss meets the second preset condition, that is, the model's prediction accuracy reaches the requirement, training is complete, and the second speech recognition model is obtained.

[0059] In the above steps, determining the second output result of the second initial speech recognition model includes: determining the first feature corresponding to the first channel in the dual-channel system and the second feature corresponding to the second channel in the dual-channel system using the encoder model in the second initial speech recognition model; determining the first vector based on the second feature and the query weight matrix; determining the second vector based on the first feature and the key weight matrix; determining the third vector based on the first feature and the value weight matrix; determining the fusion feature based on the dimensions of the first vector, the second vector, the third vector, and the second vector; and inputting the fusion feature into a long short-term memory network and a linear classification layer for analysis to obtain the second output result.

[0060] In some embodiments of this application, the audio signals of the first and second channels in the dual-channel system are processed by an encoder model (such as a Zipformer encoder) in the second initial speech recognition model to extract their respective first feature F1 and second feature F2. In cross-channel attention interaction, the features of the second channel are used as a query in the first channel, and then compared with the query weight matrix. Multiplying them together yields the first vector, i.e. Simultaneously, the features of the first channel are used as keys and values, respectively, and correlated with the key weight matrix. Sum weight matrix Multiplying them together yields the second vector. and the third vector Attention calculation is based on the dot product between the first vector (query) and the second vector (key), which is then normalized using the Softmax function to obtain attention weights. These attention weights are then multiplied by the third vector (value) to obtain the fused features, which reflect the information dependencies and interactions between the two channels. The calculation of the fused features obtained through Cross Attention can be expressed as:

[0061]

[0062] in, yes The dimension of the second vector mentioned above is the same as the dimension of the vector.

[0063] The generated fused features are fed into a Long Short-Term Memory (LSTM) network for temporal modeling. LSTM can capture long-term dependencies of features over time. The output of the LSTM network is further processed through a linear classification layer, which converts the LSTM output into probability distributions corresponding to four speech activity states (initial silence, speech, pause silence within a turn, and pause silence between turns), forming the second output result.

[0064] In step S206 of the above-mentioned intelligent sentence segmentation active speech detection method based on multi-state temporal modeling, determining the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, and obtaining the state sequence corresponding to each channel, includes: determining the maximum value of the probability distribution corresponding to each speech frame; determining the label corresponding to the maximum value as the state label of the corresponding speech frame, wherein each state label corresponds to a speech activity state; and sequentially obtaining the state label of each speech frame according to the order of the audio signal to obtain the state sequence.

[0065] In some embodiments of this application, when the target speech recognition model predicts each speech frame, it outputs a probability distribution representing the probability that the frame belongs to different speech activity states (initial silence, speech, pause silence within a turn, and pause silence between turns). For example, for a certain frame, the model may output the following probability distribution: initial silence state (0): 0.1, speech state (1): 0.6, pause silence within a turn state (2): 0.2, pause silence between turns state (3): 0.1. In the above example, the probability value of speech state (1) is the highest, which is 0.6. Therefore, the state label of this speech frame is determined to be speech state (1). The above steps are repeated for each speech frame to obtain the state label corresponding to the maximum value of its probability distribution. Then, according to the time order of the audio signal, the state labels of all speech frames are concatenated to form a state sequence, which records the dynamic changes of speech activity states in the entire audio stream.

[0066] In step S208 of the above-mentioned intelligent sentence segmentation active speech detection method based on multi-state temporal modeling, before determining the dynamic sentence segmentation time according to the state sequence, the method further includes: detecting whether the state sequence contains a preset sequence to obtain a detection result, wherein the preset sequence represents the transition from the initial silence state to the speech state and back to the initial silence state, and the duration of the speech state in the preset sequence is less than the preset duration; if the detection result indicates that the state sequence contains the preset sequence, it is determined that there is invalid speech in the audio signal, and the preset sequence is filtered out from the state sequence; if the detection result indicates that the state sequence does not contain the preset sequence, it is determined that there is no invalid speech in the audio signal.

[0067] In some embodiments of this application, before determining the time of dynamic sentence segmentation based on the state sequence, invalid speech needs to be detected and filtered to further improve the accuracy and robustness of speech activity detection. Invalid speech refers to brief, non-semantic speech activities that do not contain important information, such as human voices in background noise, brief coughs, breathing sounds, key presses, etc.

[0068] In some embodiments of this application, the preset sequence refers to a continuous state transition from an initial silence state (state 0) to a speech state (state 1), and then back to the initial silence state (state 0). This transition pattern represents invalid speech in speech activity detection, especially when the duration of the speech state is very short, i.e., less than the preset duration. The preset duration can be set according to specific application scenarios and requirements; for example, it can be set to 0.5 seconds or 1 second to filter out brief interfering sounds.

[0069] The entire state sequence is traversed to check for a pattern where a voice transitions from state 0 to state 1 and back to state 0, and the duration of state 1 is less than a preset duration. If a preset sequence matching the conditions is detected in the state sequence, the detection result indicates the presence of invalid speech; otherwise, if no such preset sequence is detected, the detection result indicates that there is no invalid speech in the audio signal. If the detection result indicates the presence of invalid speech, the preset sequence needs to be filtered out of the state sequence. That is, for the detected preset sequence, its corresponding audio segment will be considered invalid, will not be counted in any statement or turn, and will not be used as the basis for dynamic sentence segmentation.

[0070] In step S208 of the above-mentioned intelligent sentence segmentation active speech detection method based on multi-state temporal modeling, determining the dynamic sentence segmentation time according to the state sequence includes: determining the first consecutive frame number of the inter-turn pause silence state in the state sequence; if the first consecutive frame number is greater than or equal to a first threshold, determining the time corresponding to the last frame in the first consecutive frame number as the dynamic sentence segmentation time; determining the second consecutive frame number of the inter-turn pause silence state in the state sequence; if the second consecutive frame number is greater than or equal to a second threshold, determining the time corresponding to the last frame in the second consecutive frame number as the dynamic sentence segmentation time.

[0071] In some embodiments of this application, dynamic sentence segmentation is used in speech activity detection (VAD) to identify sentence boundaries or the end points of speaker turns in a speech stream. In the state sequence-based dynamic sentence segmentation method, the determination of the segmentation time depends on the number of consecutive frames of a specific silence state, i.e., the number of consecutive occurrences of the inter-turn pause silence state (state 3) and the intra-turn pause silence state (state 2) in the state sequence.

[0072] Specifically, the system iterates through the state sequence, finding the number of consecutive frames for all consecutive occurrences of state 3 (inter-turn pause silence state) to obtain the first consecutive frame count, and the number of consecutive occurrences of state 2 (inter-turn pause silence state) to obtain the second consecutive frame count. If the first consecutive frame count is greater than or equal to the first threshold, the current turn can be considered to have ended, and the dynamic sentence break point is the time corresponding to the last frame in the first consecutive frame count. If the second consecutive frame count is greater than or equal to the second threshold, sentence break is also triggered. This mechanism aims to handle speech pauses caused by prolonged thinking or hesitation, avoiding long system wait times, and uses the time of the last frame in the second consecutive frame count as the sentence break point. The first and second thresholds are fixed values ​​or adaptively adjusted values ​​preset according to the application scenario and data characteristics.

[0073] The intelligent sentence-segmentation active speech detection method based on multi-state temporal modeling provided in this application introduces fine-grained speech state segmentation and sentence segmentation rules based on state duration. This enables more accurate identification of real sentence boundaries in the speech stream and significantly reduces sentence segmentation errors caused by pauses within turn-taking. The background human voice filtering mechanism based on state transitions avoids the effective speech loss that may be caused by traditional noise suppression methods, improving the system's robustness in complex acoustic environments. Furthermore, in dual-channel scenarios, the integration of a Cross Attention mechanism for inter-channel information fusion allows the model to effectively utilize multi-channel speech information, mutually assisting each other, thereby achieving more accurate sentence segmentation and faster turn-taking prediction, enhancing the interactive experience of the dialogue system.

[0074] To better understand the solutions provided in the embodiments of this application, the following is combined with... Figure 3 and Figure 4 The construction of the model network and the structure of the dual-channel model are explained. Figure 3 This is a schematic diagram of a state transition according to an embodiment of this application. Figure 4 This is a structural diagram of a dual-channel model according to an embodiment of this application. The explanation is as follows:

[0075] In constructing the model network, an encoder based on a pre-trained speech recognition model is used as a feature extractor for the input speech signal, and temporal modeling is performed using an LSTM network. A linear classification layer is introduced to model the state of each frame in the speech signal. There are four states: 0 (initial silence), 1 (speech), 2 (silence within a turn), and 3 (silence between turns / sentence breaks); for example... Figure 3As shown, when the number of consecutively detected frames in state 3 exceeds the segmentation threshold (i.e., the first threshold mentioned above), a segmentation point is determined; when the number of consecutively detected frames in state 2 exceeds the strong segmentation threshold (i.e., the second threshold mentioned above), forced segmentation is performed; background human voices are determined by detecting whether there is a 0->1->0 transition pattern in the state sequence. Since the interference of short-term background human voices is filtered out during state transitions, the effective speech loss that may be caused by traditional noise suppression methods is avoided, and the robustness of the model in complex acoustic environments is improved.

[0076] In dual-channel scenarios, a Cross Attention mechanism is introduced to fuse information between channels, such as... Figure 4 As shown, the model takes dual-channel audio (wav1 and wav2) as input and outputs the corresponding active speech detection results for both channels (out1 and out2). The parameters in the model, including the encoder and LSTM network parameters, are shared. Information sharing between different channels enhances the robustness of the channel to background human voice interference. While perceiving its own channel, the model also pays attention to and perceives speech activity in other channels, helping it to better identify and more effectively suppress background human voice interference in its own channel, thereby improving the accuracy of effective speech detection in its own channel.

[0077] The scheme in this application, by introducing fine-grained speech state segmentation and sentence segmentation rules based on state duration, can more accurately detect the boundaries of real speech pauses and endings, significantly reducing sentence segmentation errors caused by short pauses, thereby greatly reducing speech recognition insertion errors. Experiments show that the ability to filter background human voices is significantly improved, and the proportion of correct pause detection increases significantly (73.39%->86.07%). At the same time, the delay in judging pauses is significantly reduced, with the 90th percentile delay decreasing from 1374 milliseconds to 770 milliseconds, a relative reduction of 44%. Therefore, the intelligent sentence segmentation active speech detection method based on multi-state temporal modeling in this application has a higher detection accuracy than the traditional VAD algorithm.

[0078] Figure 5 This is a structural diagram of an intelligent sentence segmentation active speech detection device based on multi-state temporal modeling according to an embodiment of this application, as shown below. Figure 5 As shown, the device includes:

[0079] The receiving module 30 is used to receive audio signals from at least one channel;

[0080] Extraction module 32 is used to extract the acoustic feature sequence of the audio signal using a target speech recognition model corresponding to the number of channels;

[0081] The first determining module 34 is used to determine the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, and to obtain the state sequence corresponding to each channel. The speech activity states include at least one of the following: initial silence state, speech state, pause silence state within a turn, and pause silence state between turns.

[0082] The second determining module 36 is used to determine the time when a sentence breaks in the audio signal based on the state sequence.

[0083] In the above-mentioned intelligent sentence segmentation activity speech detection device based on multi-state temporal modeling, the receiving module, extraction module, first determination module and second determination module achieve the goal of refined dynamic sentence segmentation, thereby improving the accuracy and robustness of speech activity detection and solving the technical problem of erroneous sentence segmentation in speech activity detection based on a fixed silence threshold in related technologies.

[0084] The aforementioned intelligent speech detection device based on multi-state temporal modeling further includes a training module 38. When the number of channels is single, the target speech recognition model is a first speech recognition model. This training module is used to train the first speech recognition model. Specifically, the first speech recognition model is trained as follows: First speech dataset corresponding to a single channel is acquired, and the first speech activity state of each speech frame in the first speech dataset is determined; a first initial speech recognition model is trained based on the first speech dataset, and the first output result of the first initial speech recognition model is determined. The first initial speech recognition model includes an encoder model, a long short-term memory network layer, and a linear classification layer; cross-entropy loss is determined based on the first output result and the first speech activity state; when the cross-entropy loss does not meet a first preset condition, the parameters of the first initial speech recognition model are updated, and the updated cross-entropy loss corresponding to the updated first output result is determined. Training stops when the updated cross-entropy loss meets the first preset condition, thus obtaining the first speech recognition model.

[0085] In the aforementioned intelligent speech detection device based on multi-state temporal modeling, when the number of channels is dual-channel, the target speech recognition model is a second speech recognition model. This training module is used to train the second speech recognition model. Specifically, the second speech recognition model is trained in the following way: acquiring the second speech dataset corresponding to the dual channels and determining the second speech activity state to which each speech frame in the second speech dataset belongs, wherein the second speech activity state includes the actual turn-taking time point; training a second initial speech recognition model based on the second speech dataset and determining the second output result of the second initial speech recognition model, wherein the second initial speech recognition model includes two branches, each branch containing a shared encoder model, the outputs of the two branches are connected to a cross-attention module for feature fusion, and the cross-attention module is then connected to a long short-term memory network and a linear classification layer; determining the state classification loss based on the second output result and the second speech activity state; when the state classification loss does not meet the second preset condition, updating the parameters of the second initial speech recognition model and determining the updated state classification loss corresponding to the updated second output result, until the updated state classification loss meets the second preset condition and training stops, thus obtaining the second speech recognition model.

[0086] In the training module of the aforementioned intelligent sentence segmentation active speech detection device based on multi-state temporal modeling, the training module is further used to determine the first feature corresponding to the first channel in the dual-channel system and the second feature corresponding to the second channel in the dual-channel system through the encoder model in the second initial speech recognition model; determine the first vector based on the second feature and the query weight matrix; determine the second vector based on the first feature and the key weight matrix; determine the third vector based on the first feature and the value weight matrix; determine the fusion feature based on the dimensions of the first vector, the second vector, the third vector, and the second vector; and input the fusion feature into the long short-term memory network and the linear classification layer for analysis to obtain the second output result.

[0087] In the first determining module of the above-mentioned intelligent sentence segmentation activity speech detection device based on multi-state temporal modeling, the first determining module is further used to determine the maximum value of the probability distribution corresponding to each speech frame; determine the label corresponding to the maximum value as the state label of the corresponding speech frame, wherein each state label corresponds to a speech activity state; and obtain the state label of each speech frame in the order of the audio signal to obtain the state sequence.

[0088] In the second determining module of the aforementioned intelligent sentence segmentation active speech detection device based on multi-state temporal modeling, the second determining module is further used to detect whether the state sequence contains a preset sequence to obtain a detection result. The preset sequence represents the transition from the initial silence state to the speech state and back to the initial silence state, and the duration of the speech state in the preset sequence is less than the preset duration. If the detection result indicates that the state sequence contains the preset sequence, it is determined that there is invalid speech in the audio signal, and the preset sequence is filtered out from the state sequence. If the detection result indicates that the state sequence does not contain the preset sequence, it is determined that there is no invalid speech in the audio signal.

[0089] In the second determining module of the intelligent sentence segmentation active speech detection device based on multi-state temporal modeling, the second determining module is further used to determine the first consecutive frame number of the turn-to-turn pause silence state in the state sequence; if the first consecutive frame number is greater than or equal to the first threshold, the time corresponding to the last frame in the first consecutive frame number is determined as the dynamic sentence segmentation time; determine the second consecutive frame number of the turn-to-turn pause silence state in the state sequence; if the second consecutive frame number is greater than or equal to the second threshold, the time corresponding to the last frame in the second consecutive frame number is determined as the dynamic sentence segmentation time.

[0090] It should be noted that, Figure 5 The intelligent sentence segmentation active speech detection device shown is based on multi-state temporal modeling and is used to perform... Figure 2 The intelligent sentence segmentation active speech detection method based on multi-state temporal modeling shown above is also applicable to this intelligent sentence segmentation active speech detection device based on multi-state temporal modeling. Therefore, the relevant explanations in the above intelligent sentence segmentation active speech detection method based on multi-state temporal modeling are also applicable to this device.

[0091] This application also provides an electronic device, which includes a memory and a processor. The memory stores program instructions, and the processor is connected to the memory to execute program instructions that perform the following functions: receiving audio signals from at least one channel; extracting acoustic feature sequences of the audio signals using a target speech recognition model corresponding to the number of channels; determining the probability distribution of each speech frame corresponding to the acoustic feature sequences belonging to different speech activity states, thereby obtaining a state sequence corresponding to each channel, wherein the speech activity states include at least one of the following: initial silence state, speech state, pause silence state within a turn, and pause silence state between turns; and determining the time when a pause occurs in the audio signal based on the state sequence.

[0092] It should be noted that the aforementioned electronic equipment is used to perform Figure 2The intelligent sentence segmentation active speech detection method based on multi-state temporal modeling shown above is also applicable to this electronic device, and will not be repeated here.

[0093] This application embodiment also provides a non-volatile storage medium, which includes a stored computer program. The device containing the non-volatile storage medium executes the following intelligent sentence-segmentation activity speech detection method based on multi-state temporal modeling by running the computer program: receiving audio signals from at least one channel; extracting acoustic feature sequences of the audio signals using a target speech recognition model corresponding to the number of channels; determining the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, obtaining a state sequence corresponding to each channel, wherein the speech activity state includes at least one of the following: initial silence state, speech state, pause silence state within a turn, and sentence-segmentation silence state between turns; and determining the time of sentence segmentation in the audio signal based on the state sequence.

[0094] It should be noted that the aforementioned non-volatile storage media is used for execution. Figure 2 The intelligent sentence segmentation active speech detection method based on multi-state temporal modeling shown above is also applicable to this non-volatile storage medium, and will not be repeated here.

[0095] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the intelligent sentence segmentation active speech detection method based on multi-state temporal modeling in various embodiments of this application.

[0096] This application also provides a computer program that, when executed by a processor, implements the steps of the intelligent sentence segmentation active speech detection method based on multi-state temporal modeling in various embodiments of this application.

[0097] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0098] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0100] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0101] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0102] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0103] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for intelligent sentence segmentation and active speech detection based on multi-state temporal modeling, characterized in that, include: Receive audio signals from at least one channel; An acoustic feature sequence of the audio signal is extracted using a target speech recognition model corresponding to the number of channels. When the number of channels is dual-channel, the target speech recognition model is a second speech recognition model, which is trained as follows: A second speech dataset corresponding to the dual channels is acquired, and the second speech activity state to which each speech frame in the second speech dataset belongs is determined. The second speech activity state includes the actual turn-taking time points. A second initial speech recognition model is trained based on the second speech dataset, and a second output result of the second initial speech recognition model is determined. The second initial speech recognition model includes two branches, each containing a shared encoder model. The outputs of the two branches are connected to a cross-attention module for feature fusion. The cross-attention module is then connected to a long short-term memory network and a linear classification layer. A state classification loss is determined based on the second output result and the second speech activity state. If the state classification loss does not meet a second preset condition... When the conditions are met, the parameters of the second initial speech recognition model are updated, and the updated state classification loss corresponding to the updated second output result is determined. Training stops when the updated state classification loss meets the second preset condition, and the second speech recognition model is obtained. The second speech dataset includes the time when the speaker switches during the dialogue. The first feature corresponding to the first channel in the dual channels and the second feature corresponding to the second channel in the dual channels are determined through the encoder model in the second initial speech recognition model. The second feature is used as a query in the first channel to obtain a first vector, which is used to determine the fusion feature. When the number of channels is single channel, the target speech recognition model is the first speech recognition model. The first speech recognition model is trained based on the first speech dataset. Each speech frame in the first speech dataset corresponds to a first speech activity state, including: initial silence state, speech state, pause silence state within a turn, and pause silence state between turns. Determine the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states to obtain the state sequence corresponding to each channel. The speech activity state includes at least one of the following: initial silence state, speech state, pause silence state within a turn, and pause silence state between turns. The initial silence state is used to filter out the non-speech part at the beginning of the recording. Based on the state sequence, the time when a sentence breaks occur in the audio signal is determined.

2. The method according to claim 1, characterized in that, The first speech recognition model was trained in the following way: Obtain the first speech dataset corresponding to a single channel, and determine the first speech activity state to which each speech frame in the first speech dataset belongs; A first initial speech recognition model is trained based on the first speech dataset, and a first output result of the first initial speech recognition model is determined. The first initial speech recognition model includes an encoder model, a long short-term memory network layer, and a linear classification layer. The cross-entropy loss is determined based on the first output result and the first voice activity state; When the cross-entropy loss does not meet the first preset condition, the parameters of the first initial speech recognition model are updated, and the updated cross-entropy loss corresponding to the updated first output result is determined. Training is stopped when the updated cross-entropy loss meets the first preset condition, and the first speech recognition model is obtained.

3. The method according to claim 1, characterized in that, Determining the second output of the second initial speech recognition model includes: Based on the second feature and the query weight matrix, the first vector is determined; Based on the first feature and the key weight matrix, determine the second vector; The third vector is determined based on the first feature and the value weight matrix; Based on the dimensions of the first vector, the second vector, the third vector, and the second vector, the fusion features are determined; The fused features are input into the long short-term memory network and the linear classification layer for analysis to obtain the second output result.

4. The method according to claim 1, characterized in that, Determine the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states to obtain the state sequence corresponding to each channel, including: Determine the maximum value of the probability distribution corresponding to each speech frame; The label corresponding to the maximum value is determined as the state label of the corresponding voice frame, wherein each state label corresponds to a voice activity state; The state label of each voice frame is obtained sequentially according to the order of the audio signal to obtain the state sequence.

5. The method according to claim 1, characterized in that, Before determining the time for dynamic sentence segmentation based on the state sequence, the method further includes: The detection result is obtained by detecting whether the state sequence contains a preset sequence, wherein the preset sequence represents the transition from the initial silence state to the voice state and back to the initial silence state, and the duration of the voice state in the preset sequence is less than the preset duration. If the detection result indicates that the state sequence contains the preset sequence, it is determined that there is invalid speech in the audio signal, and the preset sequence is filtered out from the state sequence; If the detection result indicates that the preset sequence is not included in the state sequence, it is determined that the invalid speech does not exist in the audio signal.

6. The method according to claim 1, characterized in that, Based on the state sequence, the timing of dynamic sentence segmentation is determined, including: Determine the first consecutive frame number of the turn-interrupted silence state in the state sequence; If the first consecutive frame count is greater than or equal to the first threshold, the time corresponding to the last frame in the first consecutive frame count is determined as the time of dynamic sentence segmentation. Determine the second consecutive frame number of the pause / mute state within the talk turn in the state sequence; If the second consecutive frame number is greater than or equal to the second threshold, the time corresponding to the last frame in the second consecutive frame number is determined as the time of dynamic sentence segmentation.

7. An intelligent sentence-segmentation active speech detection device based on multi-state temporal modeling, characterized in that, include: A receiving module for receiving audio signals from at least one channel; An extraction module is used to extract the acoustic feature sequence of the audio signal using a target speech recognition model corresponding to the number of channels. When the number of channels is dual-channel, the target speech recognition model is a second speech recognition model, which is trained as follows: acquiring a second speech dataset corresponding to the dual channels and determining the second speech activity state to which each speech frame in the second speech dataset belongs, wherein the second speech activity state includes the actual turn-taking time points; training a second initial speech recognition model based on the second speech dataset and determining the second output result of the second initial speech recognition model, wherein the second initial speech recognition model includes two branches, each branch containing a shared encoder model, and the outputs of the two branches are connected to a cross-attention module for feature fusion. The cross-attention module is then connected to a long short-term memory network and a linear classification layer; determining a state classification loss based on the second output result and the second speech activity state; and determining a state classification loss when the state classification loss does not satisfy... Under the second preset condition, the parameters of the second initial speech recognition model are updated, and the updated state classification loss corresponding to the updated second output result is determined. Training stops when the updated state classification loss satisfies the second preset condition, thus obtaining the second speech recognition model. The second speech dataset includes the moment when the speaker switches during the dialogue. The encoder model in the second initial speech recognition model is used to determine the first feature corresponding to the first channel in the dual channels, and the second feature corresponding to the second channel in the dual channels. The second feature is used as a query in the first channel to obtain a first vector, which is used to determine the fusion feature. When the number of channels is single-channel, the target speech recognition model is the first speech recognition model. The first speech recognition model is trained based on the first speech dataset. Each speech frame in the first speech dataset corresponds to a first speech activity state, including: initial silence state, speech state, pause silence state within a turn, and pause silence state between turns. The first determining module is used to determine the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states, and to obtain the state sequence corresponding to each channel. The speech activity states include at least one of the following: initial silence state, speech state, pause silence state within a turn, and pause silence state between turns. The initial silence state is used to filter out the non-speech part at the beginning of the recording. The second determining module is used to determine the time when a sentence breaks occur in the audio signal based on the state sequence.

8. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor, connected to the memory, is configured to execute program instructions to perform the following functions: receiving audio signals from at least one channel; extracting acoustic feature sequences of the audio signals using a target speech recognition model corresponding to the number of channels, wherein, when the number of channels is dual-channel, the target speech recognition model is a second speech recognition model, which is trained as follows: acquiring a second speech dataset corresponding to the dual channels and determining the second speech activity state to which each speech frame in the second speech dataset belongs, wherein the second speech activity state includes the actual turn-taking time points; training a second initial speech recognition model based on the second speech dataset and determining the second output result of the second initial speech recognition model, wherein the second initial speech recognition model includes two branches, each branch containing a shared encoder model, the outputs of the two branches are connected to a cross-attention module for feature fusion, the cross-attention module is then connected to a long short-term memory network and a linear classification layer; determining a state classification loss based on the second output result and the second speech activity state; when the state classification loss does not meet a second preset condition, updating the parameters of the second initial speech recognition model and determining the updated state classification loss corresponding to the updated second output result, until the updated state classification loss is obtained. Training stops when the new state classification loss meets the second preset condition, resulting in the second speech recognition model. The second speech dataset includes the moments when speakers switch during a dialogue. Using the encoder model in the second initial speech recognition model, the first feature corresponding to the first channel in the dual-channel system is determined, as well as the second feature corresponding to the second channel in the dual-channel system. The second feature is used as a query in the first channel to obtain a first vector, which is used to determine the fusion feature. When the number of channels is single-channel, the target speech recognition model is the first speech recognition model, which is based on the first speech... The dataset is trained such that each speech frame in the first speech dataset corresponds to a first speech activity state, including: initial silence state, speech state, pause silence state within a turn, and pause silence state between turns; the probability distribution of each speech frame corresponding to the acoustic feature sequence belonging to different speech activity states is determined to obtain the state sequence corresponding to each channel, wherein the speech activity state includes at least one of the following: initial silence state, speech state, pause silence state within a turn, and pause silence state between turns, wherein the initial silence state is used to filter out the non-speech part at the beginning of the recording; based on the state sequence, the time when the sentence breaks occur in the audio signal is determined.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the intelligent sentence segmentation active speech detection method based on multi-state temporal modeling as described in any one of claims 1 to 6 by running the computer program.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the intelligent sentence segmentation active speech detection method based on multi-state temporal modeling as described in any one of claims 1 to 6.

Images