Method for detecting verbal disfluency based on multimodal and window attention mechanisms

Abstract

This invention provides a method for detecting language disfluency based on multimodal and window attention mechanisms. [Solution] The method includes the steps of: converting input audio into text and timestamp information; extracting acoustic features of the input audio and semantic features of the text, converting the timestamp information into a pause vector; generating a text representation by concatenating the semantic features and the pause vector; obtaining fused features by embedding each token in the text representation into acoustic features corresponding to the same time step; constructing a detection model used to obtain the specific time and disfluency category of the linguistic disfluency of the input audio based on the fused features; training the detection model based on a predetermined dataset to obtain a trained detection model; and inputting the audio to be detected into the trained detection model to obtain the specific time and disfluency category of the linguistic disfluency of the audio to be detected.

Description

Technical Field

[0001] The present invention belongs to the field of speech processing technology, and more specifically, relates to a method for detecting speech disfluency based on a multimodal and window attention mechanism.

Background Art

[0002] Conventional oral expression analysis methods mainly identify the types of speech disfluency in speech based on single-modal features. Therefore, the output results tend to depend on a single modality and have certain limitations. For this reason, it is difficult to accurately capture multi-dimensional expressions in speech fluency such as speech pauses, repetitions, and stretching of phonemes, and the types of disfluency that can be identified are also limited. As a result, it is impossible to provide accurate feedback for different fluency problems. For example, it is impossible to accurately identify the specific time interval during which disfluency occurs in language expression or the type of speech disfluency, so there is a problem that the accuracy of speech disfluency detection decreases.

Summary of the Invention

[0003] In view of the above problems of the prior art, an object of the present invention is to provide a method for detecting speech disfluency based on a multimodal and window attention mechanism, and to solve the problem that the accuracy of speech disfluency detection decreases by identifying the category of speech disfluency in speech based on single-modal features.

[0004] To achieve the above object, in a first aspect, the present invention provides a method for detecting speech disfluency based on a multimodal and window attention mechanism, including the following steps 100, 110, and 120. In step 100, a detection model is constructed, and the detection model converts the input speech into text and timestamp information, extracts the acoustic features of the input speech and the semantic features of the text, and converts the timestamp information into a pose vector, The semantic features and the pose vector are concatenated to generate a text representation, each token in the text representation is embedded in an acoustic feature of the same time step, and a fused feature is obtained. Based on the aforementioned fusion features, it is used to obtain specific time and disfluency categories of the linguistic disfluency of the input speech. In step 110, the detection model is trained using a predetermined dataset, and the trained detection model is obtained. In step 120, the target speech is input to the trained detection model to obtain the specific time and disfluency category of the target speech. The specific time of the speech represents the specific time interval in which speech disfluency occurred in the input speech, and the disfluency category represents the specific type of disfluency that occurred in the input speech.

[0005] In this invention, a detection model is constructed, which acquires multimodal features including acoustic features, semantic features, and pause vectors by performing transcription and feature extraction on input speech. Here, semantic features are used by the model to identify redundancy and interruptions in linguistic meaning, acoustic features are used by the model to identify changes in acoustic features such as tone tension, hesitations, and drawn-out sounds, and pause vectors are used to capture the duration of each token based on a timestamp and the pause time after that token. As a result, the model can improve the accuracy of detecting the occurrence time of linguistic disfluency and the disfluency category. Furthermore, by concatenating semantic features and pause vectors to generate a text representation, and embedding each token in the text representation into an acoustic feature corresponding to the same time step, the completeness of the speech information can be preserved to the greatest extent possible. The model accurately captures the multidimensional representation of language fluency by performing language disfluency detection based on fused features. This overcomes the challenge of conventional techniques based on single-modal features, which have difficulty accurately capturing the multidimensional representation of language fluency, and improves the accuracy of detecting the specific time and type of language disfluency.

[0006] In the language disfluency detection method based on the multimodal and window attention mechanism according to the present invention, the extraction of acoustic features of input speech and semantic features of text is The steps include inputting audio into a whisker model and obtaining the acoustic representation vector output by the whisker model, The steps include: fusing the second layer expression, the fifth layer expression, and the eighth layer expression of the aforementioned acoustic representation vector to obtain a fused acoustic representation, which is used as the acoustic feature of the input sound; Includes.

[0007] In this invention, input speech is input to a whisker model, and the representations of the second, fifth, and eighth layers are extracted from the output result. The information in these layers includes each phoneme and prosodic feature, and by fusing these, more representative acoustic features can be obtained.

[0008] In the language disfluency detection method based on the multimodal and window attention mechanism according to the present invention, the extraction of acoustic features of input speech and semantic features of text is The steps include: aligning all sequences to the maximum text length in the current batch by padding the text sequences and generating corresponding padding masks; The steps include inputting the padded text sequence into a Bidirectional Encoder Representations from Transformers (BERT) model to obtain semantic features of the text, and Includes.

[0009] In this invention, padding is applied to text sequences, and all sequences are aligned to the maximum text length in the current batch, thereby ensuring tensor dimension consistency during batch computation and generating a corresponding padding mask. This padding mask is used in subsequent computations to obscure the padding positions and prevent invalid information from affecting the model's results.

[0010] In the language disfluency detection method based on the multimodal and window attention mechanism according to the present invention, the detection model further comprises: After extracting the acoustic features of the input audio, these acoustic features are input into a hierarchical multiscale window attention model for fine-tuning, and then used to obtain the fine-tuned acoustic features. The aforementioned hierarchical multiscale window attention model is used to perform interlayer multiscale modeling of acoustic features based on a 1D (one-dimensional) window attention mechanism.

[0011] In this invention, acoustic features are input into a hierarchical multiscale window attention model and fine-tuned to train the model according to desired requirements, thereby improving the detection accuracy of the model.

[0012] In the language disfluency detection method based on the multimodal and window attention mechanism according to the present invention, training the detection model using a predetermined dataset is: The learning process of the detection model includes the step of performing a first operation and / or a second operation on the training dataset, The first operation described above involves generating random noise having the same shape as the text vectors in the training dataset and adding the random noise to the corresponding positions of the text vectors. The second operation described above is to randomly set some of the vector elements in the training dataset to 0 based on a predetermined probability.

[0013] In this invention, by introducing a dual perturbation mechanism, the model can avoid relying on limited training data to memorize specific text patterns, thereby improving overall adaptability and generalization performance to different input texts.

[0014] In the language disfluency detection method based on the multimodal and window attention mechanism according to the present invention, training the detection model using a predetermined dataset is: This includes the step of training the detection model using a predetermined dataset and loss function, The aforementioned loss function is obtained by weighting the loss function of the Conditional Random Fields (CRF) and the Binary Cross Entropy (BCE) loss function.

[0015] In this invention, a loss function for model learning is constructed based on the CRF loss function and the BCE loss function. By using the inter-frame dependency modeling capability of the CRF loss function and the frame-by-frame training signal of the BCE loss function in combination, it is possible to weight different categories, reduce the effects of bias in the number of labels, and improve the ability to model the fine time-series structure in speech.

[0016] In a second embodiment, the present invention provides a language disfluency detection device based on a multimodal and window attention mechanism, comprising a construction module, a learning module, and a detection module. The aforementioned construction module is for constructing a detection model, and the detection model is Converts input audio into text and timestamp information. The acoustic features of the input audio and the semantic features of the text are extracted, and the timestamp information is converted into a pause vector. The semantic features and the pose vector are concatenated to generate a text representation, each token in the text representation is embedded in an acoustic feature of the same time step, and a fused feature is obtained. Based on the aforementioned fusion features, this is for obtaining the specific time and disfluency category of the linguistic disfluency of the input speech. The aforementioned learning module is for training the detection model using a predetermined dataset and obtaining the trained detection model. The detection module inputs the target speech to the trained detection model to obtain the specific time of occurrence and the disfluency category of the target speech. The specific time of the disfluency represents the specific time interval in which the disfluency occurred in the input speech, and the disfluency category represents the specific type of disfluency that occurred in the input speech.

[0017] As a third aspect, the present invention provides an electronic device. The electronic device includes at least one memory for storing a program and at least one processor for executing the program stored in the memory. When the program stored in the memory is executed, the processor executes a method for detecting language non-fluency based on a multimodal and window attention mechanism, as described in any possible embodiment of the first aspect or the first aspect.

[0018] As a fourth aspect, the present invention provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed on a processor, the processor is caused to execute a method for detecting language non-fluency based on a multimodal and window attention mechanism, as described in any possible embodiment of the first aspect or the first aspect.

[0019] As a fifth aspect, the present invention provides a computer program product. When the computer program product is executed on a processor, the processor is caused to execute a method for detecting language non-fluency based on a multimodal and window attention mechanism, as described in any possible embodiment of the first aspect or the first aspect.

[0020] Regarding the beneficial effects of the above second to fifth aspects, reference can be made to the relevant descriptions in the above first aspect, so duplicate explanations are omitted here.

[0021] The above technical means of the present invention have the following beneficial effects as compared with the prior art. In this invention, a detection model is constructed, and this detection model acquires multimodal features including acoustic features, semantic features, and pause vectors by performing transcription and feature extraction on input speech. Here, semantic features are used by the model to identify redundancy and interruptions in linguistic meaning, acoustic features are used by the model to identify changes in acoustic features such as tone tension, hesitations, and drawn-out sounds, and pause vectors are used to capture the duration of each token based on a timestamp and the pause time after that token. As a result, the model can improve the accuracy of detecting the occurrence time of linguistic disfluency and the disfluency category. Furthermore, by concatenating the semantic features and pause vectors to generate a text representation, and embedding each token in the text representation into the acoustic feature corresponding to the same time step, the completeness of the speech information can be preserved to the greatest extent possible. The model accurately captures the multidimensional representation of language fluency by performing language disfluency detection based on fused features. This overcomes the drawback of conventional techniques based on single-modal features, which have difficulty accurately capturing the multidimensional representation of language fluency, and improves the accuracy of detecting the specific time and type of language disfluency. [Brief explanation of the drawing]

[0022] To more clearly explain the technical solutions in the present invention or the prior art, the drawings used in the descriptions of the examples or the prior art are briefly described below. As will be clear, the drawings described below illustrate some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these without any creative effort.

[0023] [Figure 1] This is a flowchart of a language disfluency detection method based on a multimodal and window attention mechanism according to an embodiment of the present invention. [Figure 2] This is a schematic diagram of the structure of a detection module according to an embodiment of the present invention. [Figure 3]This is a schematic diagram of the structure of a language non-fluency detection device based on a multimodal and window attention mechanism according to an embodiment of the present invention. [Figure 4] This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. [Modes for carrying out the invention]

[0024] To further clarify the object, technical solution, and advantages of the present invention, the present invention will be described in more detail below with reference to the drawings and examples. The specific examples described herein are for illustrative purposes only and do not limit the present invention.

[0025] As used herein, the terms "and / or" represent a relationship between related objects. For example, A and / or B means that A exists alone, A and B exist simultaneously, and B exists alone. In this specification, the symbol " / " indicates that the related objects are in a selective relationship. For example, A / B means A or B.

[0026] In the embodiments of the present invention, terms such as “exemplary” or “for example” are used to indicate an example, illustration, or explanation. No embodiment or design solution described as “exemplary” or “for example” in the embodiments of the present invention should be construed as being preferable or superior to other embodiments or design solutions. That is, terms such as “exemplary” or “for example” are used to specifically illustrate the relevant concepts.

[0027] In the description of embodiments of the present invention, unless otherwise stated, "multiple" means two or more. For example, "multiple processing units" means two or more processing units, and "multiple elements" means two or more elements.

[0028] The following describes a method for detecting language disfluency based on a multimodal and window attention mechanism according to an embodiment of the present invention, with reference to Figures 1 and 2.

[0029] Figure 1 is a schematic flowchart of a language disfluency detection method based on a multimodal and window attention mechanism according to an embodiment of the present invention. As shown in Figure 1, the method includes the following steps. In step 100, a detection model is constructed, and the detection model is Converting input audio into text and timestamp information. Extract acoustic features of input audio and semantic features of text, and convert the timestamp information into a pause vector. The semantic features and the pose vector are concatenated to generate a text representation, and each token in the text representation is embedded in an acoustic feature corresponding to the same time step to obtain a fused feature. Based on the aforementioned fusion features, the specific time and disfluency category of the input speech's linguistic disfluency are obtained. It is used for this purpose. Optionally, after inputting the audio, the audio data can be aligned to a length of 30 seconds for subsequent batch processing calculations. During this alignment process, the padded audio sequence and the corresponding padding mask are output. In the forward propagation and normalization processes of the model, portions with a mask of 0 are occluded and do not participate in the calculations. Subsequently, root mean square (RMS) normalization is performed on the audio to reduce the impact of volume differences between different recordings.

[0030] Optionally, the detection model can use a paraformer to transcribe the audio into text in the language corresponding to the audio and a timestamp for each character.

[0031] Arbitrarily, acoustic features refer to a set of mathematical parameters that can quantitatively describe the auditory attributes and physical nature of speech, such as pitch and volume.

[0032] Optionally, semantic features of text refer to basic units or attributes extracted from text that reflect its deeper meaning and informational content, and include, for example, words, phrases, or sentences that reflect semantic information.

[0033] Optionally, the timestamp information may be time information corresponding to each token in the audio, and may include information such as the specific time and the pause time between tokens.

[0034] Optionally, the duration of each token in the timestamp and the pause time after each token can be generated as a pause vector through an embedding process.

[0035] By arbitrarily concatenating semantic features and pose vectors to generate a text representation, embedding each token in the text representation into an acoustic feature corresponding to the same time step, and then inputting the concatenated features into a Swin Transformer model to extract fused time-series features, a certain degree of continuity of output labels and agreement with the actual label distribution can be ensured.

[0036] Specifically, the `align_with_timestamps` function implements custom time-alignment logic that maps the timestamps of text tokens to audio frame-level features. PauseEmbedding independently encodes the duration and pause time of each token and concatenates them after the token. Then, by calculating the start and end time steps corresponding to the tokens, the text embeddings are aligned to the corresponding audio frames of the acoustic feature `audio_repr`. Furthermore, using a repeat strategy, the text token vectors are concatenated to all audio vectors at the time step corresponding to the tokens, and finally, the multimodal concatenated vectors are compressed and feature gaps are reduced through a two-stage dimensionality reduction (linear layer + ReLU + linear layer) using `self.fused_proj`.

[0037] In this invention, the reason the detection model fuses semantic features, acoustic features, and pause vectors is that some forms of linguistic disfluency, such as repetitive expressions, do not involve significant phonetic changes during speech and primarily manifest as redundancy in linguistic meaning. Semantic features are primarily used to detect such disfluency. On the other hand, disfluency expressions such as sudden tensions in tone, hesitations, drawn-out sounds, and phonetic filler words manifest as changes in acoustic features such as phonetic intonation, and may simultaneously be reflected in semantic modals as interruptions in meaning. Furthermore, some speakers frequently use words such as "um" and "ah" to adjust their speech rhythm. By combining semantic and acoustic features in this way, the model can better identify these forms of disfluency. Moreover, the temporal and pause information included in the pause features enhances the model's identification and detection effects, maximizing the accuracy of detecting the specific time and type of linguistic disfluency.

[0038] Optionally, the non-fluency category may include pauses, repetitions, and phoneme stretching.

[0039] Optionally, based on the fused feature representation, a fully connected classifier is used to obtain raw classification scores (logits) on a frame-by-frame basis. Then, the logits for each category are converted to binary classification radial scores, i.e., [-logit, +logit], which indicate whether the category is activated in the corresponding time frame. Next, Viterbi decoding is performed using the CRF corresponding to each category to obtain the optimal path sequence. By integrating the path results for all categories, a prediction tensor with shape [B,T,C] is finally generated, where the value at each position is 0 / 1, indicating whether the category is activated in the given frame.

[0040] In step 110, a detection model is trained based on a predetermined dataset, and the trained detection model is obtained. Optionally, audio data and corresponding text data can be collected, annotated for non-fluency events, and used as a predetermined dataset. The present invention is not particularly limited to methods for acquiring audio data and corresponding text data.

[0041] Optionally, 6,000 audio and corresponding text data points can be collected from the AS-70 dataset and a self-built dataset. The audio data can then be resampled to 16kHz and normalized, word segmentation can be performed on the text data, the audio and text can be aligned based on timestamps, and annotations can be added for non-fluent events to create a usable dataset.

[0042] The learning environment and settings used to train the detection model can be configured as needed, according to the actual requirements.

[0043] In one embodiment of the present invention, the learning environment is set up as follows. GPU environment: NVIDIA RTX 5090, CUDA 13.0; CPU environment: Intel(R) Xeon(R) Gold 6530, frequency 2.0GHz, 32 cores; Memory: 256GB; Operating system: Ubuntu 24.04.3 LTS.

[0044] In one embodiment of the present invention, the learning settings are set as follows. Optimizer: Adam; Initial learning rate: 1e-4; Batch size: 128; Number of training epochs: 100.

[0045] In step 120, the target speech is input into the trained detection model to obtain the specific time and disfluency category of the target speech. The specific time of the speech represents the specific time interval in which speech disfluency occurred in the input speech, and the disfluency category represents the specific type of disfluency that occurred in the input speech.

[0046] By inputting the target speech into a trained detection model, the detection results output by the detection model, namely the specific time and category of language disfluency, are obtained.

[0047] Conventional techniques identify types of linguistic disfluency in speech based on single modal features, resulting in output that tends to depend on a single modality. Furthermore, single modal features have limitations, making it difficult to accurately capture multidimensional representations of linguistic fluency, such as pauses, repetitions, and phoneme extensions. To overcome these shortcomings of conventional techniques, the present invention constructs a detection model that acquires multimodal features, including acoustic features, semantic features, and pause vectors, by performing transcription and feature extraction on the input speech. Here, semantic features are used by the model to identify redundancy and interruptions in linguistic meaning; acoustic features are used to identify changes in acoustic features such as tone tension, hesitations, and extended sounds; and pause vectors are used to capture the duration of each token based on a timestamp and the pause time after that token. Additionally, by concatenating semantic features and pause vectors to generate a text representation, and embedding each token in the text representation into the corresponding acoustic feature at the same time step, the completeness of the speech information can be preserved to the greatest extent possible. The model accurately captures the multidimensional representation of language fluency by performing language disfluency detection based on fused features. This overcomes the drawback of conventional techniques based on single-modal features, which have difficulty accurately capturing the multidimensional representation of language fluency, and improves the accuracy of detecting the specific time and type of language disfluency.

[0048] In some embodiments, the extraction of acoustic features of the input audio and semantic features of the text in step 100 includes the following steps. In step 1001, the input audio is input to the whisker model, and the acoustic representation vector output by the whisker model is obtained. In step 1002, the second layer expression, the fifth layer expression, and the eighth layer expression of the acoustic representation vector are fused to obtain the resulting acoustic representation, which is used as the acoustic feature of the input sound. Includes.

[0049] The whisper model is a high-performance automatic speech recognition model developed by OpenAI. Based on the Transformer architecture, it employs an end-to-end learning method that allows it to directly generate text output from speech input.

[0050] When input audio is fed into the Whisper model, the Whisper model outputs a time series, with each time step associated with a representation vector, and this representation vector has multiple layers. By extracting the representations from layers 2, 5, and 8 of these layers, the features of the lower layers become closer to the original audio signal, while the features of layers 2 through 8 become closer to the phoneme-level information. The information in these layers includes each phoneme and prosodic feature. Finally, by fusing the features of these three layers using a learnable weighted vector and using the fusing features as acoustic features, it is possible to obtain acoustic features that are more representative compared to using the representation vector directly as acoustic features.

[0051] In some embodiments, the extraction of acoustic features of the input audio and semantic features of the text in step 100 includes the following steps. Step 1003: Align all sequences to the maximum text length in the current batch by padding the text sequences and generating corresponding padding masks. Step 1004: Input the padded text sequence into the BERT model to obtain semantic features of the text.

[0052] In the preprocessing and feature extraction process for text modals, the input text sequence is first padded to align all sequences to the maximum text length in the current batch. This ensures tensor dimension matching during batch computation and generates a corresponding padding mask, compared to directly extracting semantic features using the text sequence. This padding mask is used in subsequent calculations to obscure the padding positions and prevent invalid information from affecting the model's results.

[0053] Next, the padded text is input into a pre-trained BERT model to obtain context-dependent vector representations corresponding to each token. The final output vectors include not only semantic information but also contextual information, and based on this information, the detection model can automatically learn information about the linguistic confusion dimension regarding unfluent expressions.

[0054] The BERT model is a pre-trained language model based on the Transformer architecture. Unlike conventional unidirectional language models, it employs a bidirectional Transformer encoder structure, allowing it to simultaneously consider contextual information on both the left and right sides of the input sequence, thereby generating more accurate and semantically richer lexical representations.

[0055] In some embodiments, the detection model further, After extracting the acoustic features of the input audio, these acoustic features are input into a hierarchical multiscale window attention model for fine-tuning, and the refined acoustic features are also used to obtain the refined acoustic features. The aforementioned hierarchical multiscale window attention model is used to perform interlayer multiscale modeling of acoustic features based on a 1D window attention mechanism.

[0056] Conventional Transformers apply global attention directly to the entire sequence, resulting in computationally intensive solutions and a tendency to lose local temporal features. This invention achieves a balance between performance and local time-series sensitivity by using a 1D sliding window attention mechanism, specifically as follows. 1a, 1D Window splitting and shifted window The time sequence [B,T,C] is divided into [B*nW,window_size,C], where nW is the number of windows and window_size is the window size. Computational complexity is reduced by performing attention calculations independently within each window. Additionally, "sliding window attention" is implemented using the shift_size parameter, allowing different layers to capture time-series dependencies across windows. 2a. Interlayer multiscale modeling The three-layer window transformer block has different window sizes and head counts. Through "multiscale local time-series modeling" across layers, the model can simultaneously perceive short-term pronunciation features and long-term rhythmic changes. It relies simultaneously on two time scales: "instantaneous discontinuity" and "overall fluency."

[0057] The model of the present invention implements not a primitive Swin Transformer, but a customized 1D window attention mechanism designed for audio time sequence processing tasks, extending the Swin Transformer from a two-dimensional (2D) image structure to a one-dimensional (1D) audio time sequence structure and adapting it to time-series signals.

[0058] The differences between the hierarchical multiscale window attention model used in this invention and the conventional Swin Transformer are mainly the following two points. 1b. While the Swin Transformer has a hierarchical pyramidal structure, the hierarchical multiscale window attention model used in the present invention also has a hierarchical structure, but does not involve an increase in feature dimensions and exhibits less reduction in temporal resolution and less block merging.

[0059] 2b. While the Swin Transformer is primarily suited for image processing and speech classification, the hierarchical multiscale window attention model used in the present invention processes one-dimensional sequence data while maintaining temporal resolution, making it more suitable for language disfluency detection tasks.

[0060] If it is difficult to determine language disfluency directly using the extracted acoustic features, the quality of the acoustic features can be improved by inputting the acoustic features into a hierarchical multiscale window attention model, fine-tuning them, and training the model based on the provided model parameters, thereby improving the detection accuracy of the detection model.

[0061] In one embodiment of the present invention, interlayer multiscale modeling of acoustic features is performed using three layers of WindowTransformerBlock, with window sizes of 64, 32, and 128, and attention head counts of 4, 4, and 8, respectively.

[0062] In one embodiment of the present invention, the acoustic feature vector output by the whisperer is obtained by passing it through a hierarchical multiscale window attention model to acquire a long sequence feature vector, audio_repr.

[0063] In some embodiments, step 110 specifically includes the following: Step 1101: During the training process of the detection model, perform the first and / or second operations on a predetermined training dataset. The first operation described above involves generating random noise having the same shape as the text vectors in the training dataset and adding the random noise to the corresponding positions of the text vectors. The second operation described above is to randomly set some of the vector elements in the training dataset to 0 based on a predetermined probability.

[0064] To enhance the robustness of the model and prevent overfitting, a further random perturbation mechanism is introduced to the encoded vectors. Specifically, this includes a first operation and / or a second operation, the first of which is Gaussian noise injection, which generates random noise (mean 0, following a Gaussian distribution with adjustable variance) having the same shape as the text vectors in the training dataset, and adds this random noise to the corresponding positions of the text vectors. This method can simulate the uncertainty of the input data during the learning process and improve the generalization ability of the model when faced with new samples. The second operation is a dropout operation, which randomly sets some of the vector elements in the training dataset to 0 based on a predetermined probability, which is equivalent to sub-network sampling for the model, effectively mitigating the neural network's over-reliance on specific features and further reducing the risk of overfitting.

[0065] The predetermined probability can be set arbitrarily according to the requirements or actual circumstances, for example, it can be set to 0.1.

[0066] The dual perturbation mechanism described above prevents the model from relying on limited training data to memorize specific text patterns, thereby improving overall adaptability and generalization performance to different input texts.

[0067] In some embodiments, step 110 specifically includes the following: Step 1102: This step involves training a detection model based on a predetermined dataset and loss function, the loss function being obtained by weighting and summing the CRF loss function and the BCE loss function.

[0068] The loss function can be arbitrarily set as a weighted sum of the CRF loss function and the BCE loss function, and its formula is as follows:

number

[0069] CRF Loss: For each category, the log-likelihood based on CRF is calculated individually and summed within the batch. The loss is then defined as the Negative Log-Likelihood (NLL), and normalized based on the number of valid frames.

[0070] BCE Loss: For valid frames specified by the mask, the frame-by-frame BCE loss is directly calculated for logits and multi-label one-hot vectors.

[0071] This design leverages the interframe dependency modeling capabilities of CRF and combines them with frame-by-frame training signals from BCE, allowing for weighting of different categories. This reduces the impact of label imbalances and improves the model's ability to represent subtle time-series structures within speech.

[0072] Figure 2 is a schematic diagram of the configuration of a detection model according to an embodiment of the present invention. As shown in Figure 2, in one embodiment of the present invention, the detection model includes the following components. The transcription module uses Paraformer to transcribe input audio into text and timestamp information. Text Encoding Module: This module uses the BERT model to extract semantic features from text, encodes each token into 768 dimensions, and then reduces the dimensions to 256. Pause Information Encoding Module: This module encodes the duration of each token and the pause time after each token, based on timestamps, into a pause vector using an embedding process, with a dimension of 256. The acoustic coding module extracts acoustic information using the Whisper model, obtains information from layers 2, 5, and 8, then performs softmax processing on the learnable tensors to obtain parameters corresponding to each layer, and integrates these to generate a 768-dimensional representation vector. Acoustic Information Modeling Module: This module is used to model acoustic information extracted using the Swin Transformer. It consists of a total of three layers, with window sizes of 64, 32, and 128 respectively, and attention head counts of 4, 4, and 8, respectively. Feature fusion layer: This layer is used to fuse semantic features, acoustic features, and pause vectors. Output layer: This layer is used to perform classification or prediction.

[0073] In one embodiment of the present invention, experiments were conducted on a uniquely constructed language disfluency dataset. This dataset includes annotation samples related to multiple language disfluency types such as pauses, repetitions, and phoneme stretching, and covers speech and text data from speakers of different age groups, genders, and fluency levels. Cross-validation was employed in the experiment, and precision, recall, and F1-score were used as the main evaluation metrics to assess the accuracy, completeness, and overall performance in disfluency type detection, respectively. As a result, the overall performance was 0.73 for precision, 0.52 for F1, and 0.4 for recall, fully demonstrating that the language disfluency detection method based on the multimodal and window attention mechanism according to the present invention effectively improves the accuracy of detecting the specific time and type of language disfluency.

[0074] The following describes a language disfluency detection device based on the multimodal and window attention mechanisms according to the present invention. The language disfluency detection device described below is mutually referential with the language disfluency detection method based on the multimodal and window attention mechanisms described above.

[0075] Figure 3 is a schematic diagram of the configuration of a language nonfluency detection device based on a multimodal and window attention mechanism according to an embodiment of the present invention. As shown in Figure 3, the device 300 includes the following: Construction module 310: This module is for constructing a detection model, and the detection model is Transcribing input audio into text and timestamp information. Extract acoustic features of input audio and semantic features of text, and convert the timestamp information into a pause vector. The semantic features and the pause vector are concatenated to generate a text representation, and each token in the text representation is populated with an acoustic feature corresponding to the same time step to obtain a fused feature. Based on the aforementioned fusion features, it is used to obtain the specific time and disfluency category of the linguistic disfluency of the input speech. Learning module 320: This module is used to train a detection model based on a given dataset and to obtain the trained detection model. Detection module 330: This module inputs the target speech to a trained detection model to obtain the specific time and disfluency category of the target speech. The specific time of the speech represents the specific time interval in which speech disfluency occurred in the input speech, and the disfluency category represents the specific type of disfluency that occurred in the input speech.

[0076] The above-mentioned apparatus is used to carry out the method according to the embodiment described above, and the implementation principles and technical effects of each program module in the apparatus are the same as those described in the method described above. Since the operating process of the apparatus can be described by referring to the corresponding process in the method described above, a redundant explanation is omitted here.

[0077] Based on the method according to the embodiment described above, Figure 4 is a schematic diagram showing an example of the physical configuration of an electronic device. As shown in Figure 4, the electronic device according to the embodiment of the present invention includes a processor 410, a communications interface 420, a memory 430, and a communications bus 440, and the processor 410, communications interface 420, and memory 430 communicate with each other via the communications bus 440. The processor 410 can execute the language disfluency detection method based on the multimodal and window attention mechanism according to the embodiment described above by calling logical instructions stored in the memory 430.

[0078] Furthermore, the logical instructions stored in the memory 430 can be stored on a computer-readable storage medium if they are implemented in the form of a software function unit and sold or used as an independent product. Based on this understanding, the technical solution of the present invention, i.e., the contribution to the prior art or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored on a storage medium and includes a plurality of instructions, which cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the language disfluency detection method based on the multimodal and window attention mechanism according to each embodiment of the present invention.

[0079] Based on the method described in the above-described embodiment, an embodiment of the present invention provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed on a processor, it causes the processor to execute the language disfluency detection method based on the multimodal and window attention mechanism described in the above-described embodiment.

[0080] Based on the method described in the above-described embodiment, an embodiment of the present invention provides a computer program product. When the computer program product is executed on a processor, the processor is made to execute the language disfluency detection method based on the multimodal and window attention mechanism described in the above-described embodiment.

[0081] The processor in the embodiments of the present invention may be a Central Processing Unit (CPU), or it may be another general-purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or another programmable logic device, a transistor logic device, a hardware component, or any combination thereof. The general-purpose processor may be a microprocessor, or any ordinary processor.

[0082] The method steps in embodiments of the present invention may be implemented by hardware or by a processor executing software instructions. The software instructions consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, external hard disks, CD-ROMs, or any other form of storage medium well known to those skilled in the art. For example, the storage medium may be coupled to a processor and configured to allow the processor to read and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and storage medium may also be located within an ASIC.

[0083] In the above embodiments, the whole or a part of the invention can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, the whole or a part of the invention can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded into a computer and executed, processing according to the flow or function of the embodiments of the invention is generated in whole or in part. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted between websites, computers, servers, or data centers by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, radio, microwave, etc.) means. The computer-readable storage medium may be any available medium accessible by a computer, or a data storage device such as a server or data center that includes one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0084] The various numerical designations used in the embodiments of the present invention are merely distinctions for the sake of explanation and do not limit the scope of the embodiments of the present invention.

[0085] As will be readily apparent to those skilled in the art, the above description is merely a preferred embodiment of the present invention and does not limit it. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall all be within the scope of protection of the present invention.

Claims

1. A method for detecting language disfluency based on a multimodal and window attention mechanism, comprising the following steps 100, 110, and 120, In step 100, a detection model is constructed, and the detection model is The input audio is converted into text and timestamp information containing time information corresponding to each token in the input audio. The acoustic features of the input audio and the semantic features of the text are extracted, and based on the time information contained in the timestamp information, the duration of each token and the pause time after each token are encoded as a pause vector by embedding. The semantic features and the pose vector are concatenated to generate a text representation, each token in the text representation is embedded in an acoustic feature of the same time step, and a fused feature is obtained. Based on the aforementioned fusion features, it is used to obtain specific time and disfluency categories of the linguistic disfluency of the input speech. In step 110, the detection model is trained using a predetermined dataset, and the trained detection model is obtained. In step 120, the target speech is input to the trained detection model to obtain the specific time and disfluency category of the target speech. The specific time of the speech represents the specific time interval in which speech disfluency occurred in the input speech, and the disfluency category represents the specific type of disfluency that occurred in the input speech. The aforementioned detection model further, A method for detecting language disfluency, characterized in that, after extracting acoustic features from input speech, the acoustic features are input into a hierarchical multiscale window attention model for fine-tuning, and the method is also used to obtain the fine-tuned acoustic features, the hierarchical multiscale window attention model is used to perform interlayer multiscale modeling of acoustic features based on a 1D window attention mechanism.

2. Extracting acoustic features from input audio and semantic features from text is possible. The steps include inputting the input sound into a whisker model and obtaining the sound representation vector output by the whisker model, The steps include: fusing the second layer expression, the fifth layer expression, and the eighth layer expression of the aforementioned acoustic representation vector to obtain a fused acoustic representation, which is used as the acoustic feature of the input sound; A method for detecting language disfluency according to claim 1, characterized by including the following:

3. Extracting acoustic features from input audio and semantic features from text is possible. The process involves padding the text sequences and generating corresponding padding masks to align all sequences to the maximum text length in the current batch, and The steps include inputting the padded text sequence into the BERT model to obtain semantic features of the text, A method for detecting language disfluency according to claim 1, characterized by including the following:

4. Training the detection model using a predetermined dataset is, The learning process of the detection model includes the step of performing a first operation and / or a second operation on a predetermined dataset, The first operation involves generating random noise having the same shape as the text vectors in a predetermined dataset, and adding the random noise to the corresponding positions of the text vectors. The method for detecting language disfluency according to claim 1, characterized in that the second operation is an operation to randomly set some vector elements in a predetermined dataset to 0 based on a predetermined probability.

5. Training the detection model using a predetermined dataset is, This includes the step of training the detection model using a predetermined dataset and loss function, The method for detecting language disfluency according to claim 1, characterized in that the loss function is obtained by weighted summing a CRF loss function and a BCE loss function.

6. It is an electronic device, At least one memory for storing computer programs, At least one processor for executing a program stored in the memory, Includes, An electronic device characterized in that, when a program stored in the memory is executed, the processor performs a language disfluency detection method based on a multimodal and window attention mechanism as described in any one of claims 1 to 5.

7. A computer-readable storage medium on which a computer program is stored, A computer-readable storage medium characterized in that, when the computer program is executed on the processor, the processor is caused to execute a language nonfluency detection method based on a multimodal and window attention mechanism as described in any one of claims 1 to 5.

8. A computer program, A computer program characterized in that, when the computer program is executed on the processor, it causes the processor to execute a language disfluency detection method based on a multimodal and window attention mechanism as described in any one of claims 1 to 5.

Images