Intelligent voice sentence segmentation method and system based on multi-modal fusion

Abstract

The invention discloses an intelligent voice sentence segmentation method and system based on multi-modal fusion. The method comprises the following steps: acquiring voice data to be processed; processing the voice data by using a pre-trained acoustic sentence segmentation model to obtain an acoustic time sequence feature vector; processing the voice data by using a pre-trained text semantic model to obtain a semantic feature vector; and performing cross attention fusion on the acoustic time sequence feature vector and the semantic feature vector to obtain a fusion result, and determining a sentence segmentation result corresponding to the voice data according to the fusion result. According to the method and the device, the technical problem that the sentence segmentation position cannot be accurately determined due to the fact that acoustic features and semantic information are not fully combined in a related sentence segmentation method is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent speech processing, in particular, to an intelligent speech punctuation method and system based on multi-modal fusion. BACKGROUND

[0002] In an intelligent speech interaction system, Voice Activity Detection (VAD) and sentence boundary division are the core links that determine the interaction experience. Traditional VAD technology has significant defects: it can only output a binary label of "speech / silence", cannot make fine-grained semantic distinctions in the silence context, and is difficult to identify the preparatory silence before the start of a sentence, the natural pause silence within a sentence, and the terminal silence after the end of a sentence, resulting in a disconnect between the punctuation logic and human conversation habits.

[0003] Related punctuation methods generally have the problem of modal unification, and fail to effectively combine acoustic features and semantic information. In a complex speech environment, it is difficult to accurately determine the punctuation position relying only on acoustic prosodic features. When background noise interferes with acoustic signals or users have a short pause of semantic continuity, simple acoustic analysis is prone to mispunctuation. If only relying on text semantic information, it is also limited by the accuracy of Automatic Speech Recognition (ASR) results, and recognition errors such as ASR word omission and word errors will directly lead to punctuation deviation.

[0004] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0005] The present application provides an intelligent speech punctuation method and system based on multi-modal fusion, to at least solve the technical problem of being unable to accurately determine the punctuation position due to the related punctuation methods failing to fully combine acoustic features and semantic information.

[0006] According to one aspect of the present application, an intelligent speech punctuation method based on multi-modal fusion is provided, comprising: obtaining speech data to be processed; processing the speech data using a pre-trained acoustic punctuation model to obtain an acoustic time sequence feature vector; processing the speech data using a pre-trained text semantic model to obtain a semantic feature vector; cross-attention fusion of the acoustic time sequence feature vector and the semantic feature vector to obtain a fusion result, and determining the punctuation result corresponding to the speech data according to the fusion result.

[0007] Optionally, the acoustic sentence breaking model comprises at least: an automatic speech recognition model and a sequence classification network, the sequence classification network comprises at least one recurrent neural network layer and one fully connected output layer; the acoustic sentence breaking model is obtained by training in the following manner: fixing all parameters in an encoder module of the automatic speech recognition model; obtaining a speech data set, wherein the speech data set comprises at least: speech data carrying a class label of a speech segment and a silence segment; extracting basic acoustic features of the speech data set by using the encoder module; performing time series modeling on the basic acoustic features by using the recurrent neural network layer, and mapping the time series modeling result to a class label space by using the fully connected output layer; optimizing and adjusting parameters of the sequence classification network with the class label as a training target, to obtain the acoustic sentence breaking model after training.

[0008] Optionally, the class label space comprises at least: a first label, a second label, a third label and a fourth label, wherein the first label is used to represent a silence segment before a sentence, the second label is used to represent an effective speech segment in a sentence, the third label is used to represent a silence segment of a pause in a sentence, and the fourth label is used to represent a silence segment after a sentence.

[0009] Optionally, the text semantic model is obtained by training in the following manner: obtaining an initial text semantic model, wherein the initial text semantic model comprises at least: a word embedding layer, a multi-layer recurrent neural network and a linear classification layer; obtaining a text data set, wherein the text data set comprises natural sentences; training the initial text semantic model based on the text data set to obtain a pre-trained text model; obtaining a speech-text alignment data set, wherein the speech-text alignment data set comprises at least: a speech signal and a time-aligned text sequence corresponding to the speech signal and generated by decoding of the automatic speech recognition model; using the time-aligned text sequence as input, using a sentence boundary label as a training target, and using a first loss function to optimize parameters of the pre-trained text model to obtain the text semantic model after training.

[0010] Optionally, the training of the initial text semantic model based on the text data set to obtain the pre-trained text model comprises: using the text data set as input, using forward language modeling or sentence end prediction as a training target, and optimizing parameters of the initial text semantic model by using a second loss function to obtain the pre-trained text model.

[0011] Optionally, the acoustic time sequence feature vector and the semantic feature vector are cross-attention fused to obtain a fusion result, including: inputting the acoustic time sequence feature vector into a first linear transformation layer to obtain a query vector sequence; inputting the semantic feature vector into a second linear transformation layer to obtain a key vector sequence, and inputting the semantic feature vector into a third linear transformation layer to obtain a value vector sequence; calculating an attention weight matrix between the query vector sequence and the key vector sequence, wherein the attention weight matrix is used to represent the correlation strength between the acoustic feature and the semantic feature; performing weighted summation on the value vector sequence by using the attention weight matrix to obtain an acoustic context vector weighted by semantic information; performing residual connection on the acoustic context vector and the original acoustic time sequence feature vector, and performing feature fusion on the residual connection result by using a fourth linear transformation layer to obtain the fusion result.

[0012] Optionally, the acoustic time sequence feature vector includes a time-frequency pattern, a prosody feature, and a speech segment time sequence dependency relationship; the text semantic model includes a long short-term memory network; and the semantic feature vector is a decoded text based on automatic speech recognition output by the long short-term memory network.

[0013] According to still another aspect of the present application, an intelligent speech punctuation system based on multi-modal fusion is also provided, including: an acquisition module configured to acquire speech data to be processed; a first processing module configured to process the speech data by using a pre-trained acoustic punctuation model to obtain an acoustic time sequence feature vector; a second processing module configured to process the speech data by using a pre-trained text semantic model to obtain a semantic feature vector; and a determination module configured to cross-attention fuse the acoustic time sequence feature vector and the semantic feature vector to obtain a fusion result, and determine a punctuation result corresponding to the speech data according to the fusion result.

[0014] According to still another aspect of the present application, a non-volatile storage medium is also provided, including a stored program, wherein the program, when executed, controls a device in which the storage medium is located to perform the above intelligent speech punctuation method based on multi-modal fusion.

[0015] According to still another aspect of the present application, an electronic device is also provided, including a memory and a processor, wherein the processor is configured to execute a program stored in the memory, and the program, when executed, performs the above intelligent speech punctuation method based on multi-modal fusion.

[0016] According to still another aspect of the present application, a computer program is also provided, wherein the computer program, when executed by a processor, implements the above intelligent speech punctuation method based on multi-modal fusion.

[0017] According to still another aspect of the present application, a computer program product is also provided, which comprises a non-volatile computer readable storage medium, wherein the non-volatile computer readable storage medium stores a computer program, which, when executed by a processor, implements the above intelligent speech punctuation method based on multi-modal fusion.

[0018] In the present application, the speech data to be processed is obtained, the pre-trained acoustic punctuation model is used to process the speech data to obtain an acoustic timing feature vector, the pre-trained text semantic model is used to process the speech data to obtain a semantic feature vector, the acoustic timing feature vector and the semantic feature vector are cross-attention fused to obtain a fusion result, and the punctuation result corresponding to the speech data is determined according to the fusion result. The purpose of effectively combining acoustic features and semantic information is achieved, thereby realizing the technical effect of accurately determining the punctuation position, and further solving the technical problem that the punctuation position cannot be accurately determined due to insufficient combination of acoustic features and semantic information in related punctuation methods. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate embodiments of the present application and its description, which serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0020] Figure 1 is a flowchart of an intelligent speech punctuation method based on multi-modal fusion according to an embodiment of the present application;

[0021] Figure 2 is a flowchart of another intelligent speech punctuation method based on multi-modal fusion according to an embodiment of the present application;

[0022] Figure 3 is a structural diagram of an intelligent speech punctuation system based on multi-modal fusion according to an embodiment of the present application;

[0023] Figure 4 is a hardware structure block diagram of a computer terminal of an intelligent speech punctuation method based on multi-modal fusion according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", and the like in the description and in the claims of the present application and above-described accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular sequential or chronological order. It should be understood that the data thus used can be interchanged, where appropriate, so that the embodiments of the present application described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a list of steps or units need not be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to such processes, methods, products, or apparatus.

[0026] According to an embodiment of the present application, a method embodiment of an intelligent speech punctuation method based on multi-modal fusion is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0027] Figure 1 is a flowchart of an intelligent speech punctuation method based on multi-modal fusion according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0028] Step S102, obtaining speech data to be processed.

[0029] Step S104, processing the speech data using a pre-trained acoustic punctuation model to obtain an acoustic timing feature vector.

[0030] According to some optional embodiments of the present application, the acoustic punctuation model at least includes an automatic speech recognition model and a sequence classification network, and the sequence classification network at least includes a recurrent neural network layer and a fully connected output layer. Further, the acoustic punctuation model is obtained by training in the following way: fixing all parameters in the encoder module of the automatic speech recognition model; obtaining a speech data set, wherein the speech data set at least includes speech data carrying class labels of speech segments and silence segments; extracting basic acoustic features of the speech data set using the encoder module; using the recurrent neural network layer to model the timing of the basic acoustic features, and using the fully connected output layer to map the timing modeling results to the class label space; optimizing and adjusting the parameters of the sequence classification network with the class labels as the training target, to obtain the trained acoustic punctuation model.

[0031] Specifically, the acoustic sentence breaking model comprises an automatic speech recognition (ASR) model and a sequence classification network, the sequence classification network comprising at least one recurrent neural network (RNN) layer and a fully connected output layer. The architecture of the acoustic sentence breaking model has the advantage of utilizing the rich acoustic information processing capability of the ASR model to further enhance the model's understanding and execution of the speech activity detection and sentence breaking tasks.

[0032] The process of training the acoustic sentence breaking model comprises the following key steps. First, all parameters of the encoder module in the ASR model are fixed, that is, this part of the network has been fully trained and can stably extract the underlying features of the speech signal, including time-frequency characteristics, intonation and prosody, and will not change in subsequent training.

[0033] Then, training samples are obtained from the speech data set, each sample in the speech data set is provided with detailed class labels of speech segments and silence segments, the class labels are used to indicate when speech activity should be determined and when silence segments should be regarded, including silence before a sentence, normal speech within a sentence, pause silence in between, and silence after a sentence.

[0034] During the training process, the encoder module is used to extract features from the original speech data to generate a set of basic acoustic features. The RNN is used to model the time sequence of the basic acoustic features, the RNN is particularly suitable for processing sequence data and can capture the dependencies between speech segments, thus effectively understanding prosody and sentence breaking. The features after time sequence modeling are converted to the class label space through the fully connected output layer, and the corresponding acoustic activity classification probability is output.

[0035] Finally, the class labels in the speech data set are used as the supervision target for training, and the backpropagation algorithm is used to optimize and adjust the parameters in the sequence classification network to minimize the difference between the predicted results and the true labels, thereby improving the sentence breaking accuracy and robustness of the model. Through iterative training, the acoustic sentence breaking model gradually learns to make fine distinctions between different types of silence and speech activity, and eventually reaches an ideal state, that is, it can accurately identify sentence boundaries, and even in the presence of background noise or unclear speech, it can maintain high sentence breaking performance.

[0036] Optionally, the class label space comprises at least a first label, a second label, a third label and a fourth label, wherein the first label is used to represent the silence segment before the start of a sentence, the second label is used to represent the valid speech segment within a sentence, the third label is used to represent the pause silence segment within a sentence, and the fourth label is used to represent the silence segment after the termination of a sentence.

[0037] The first label (e.g., 0) represents the silence segment before the start of a sentence, which is the silent time before the speaker starts to speak. The first label takes into account that in natural conversation, people often do not start speaking immediately, but have a short period of thinking or breathing adjustment before making a sound. The second label (e.g., 1) represents the valid speech segment within a sentence, which is the part where the speaker actually outputs meaningful information. The valid speech segment contains complete words and sentences, and is the data type that the intelligent speech interaction system needs to focus on and process most. The third label (e.g., 2) is used to identify the silent segment of the pause within a sentence, which is a short silence produced by the speaker for expressing emotion, logical pause or breathing in a sentence or paragraph. This type of silent segment is different from the silence at the beginning or end of a sentence, and it appears more in the middle of a sentence, reflecting the speaker's tone, speed change and emotional state. The fourth label (e.g., 3) represents the silence segment after the end of a sentence, which is a natural silence after the speaker completes a complete expression, and can be a sign of the speaker waiting for feedback, completing thinking or conversation pause. Correctly identifying the fourth label helps the system respond in time and avoid premature or late sentence breaks.

[0038] Through the above label definition, the voice data can be more accurately analyzed and classified, effectively distinguishing different types of silence and speech segments, and providing a solid foundation for intelligent voice assistants, voice-to-text services and language understanding applications.

[0039] In step S106, the pre-trained text semantic model is used to process the voice data to obtain a semantic feature vector.

[0040] According to some optional embodiments of the present application, the text semantic model is obtained by the following method: obtaining an initial text semantic model, wherein the initial text semantic model at least includes: a word embedding layer, a multi-layer recurrent neural network and a linear classification layer; obtaining a text data set, wherein the text data set includes natural sentences; based on the text data set, training the initial text semantic model to obtain a pre-trained text model; obtaining a voice-text alignment data set, wherein the voice-text alignment data set at least includes: a voice signal and a time-aligned text sequence corresponding to the voice signal, which is generated by decoding the voice signal through an automatic speech recognition model; using the time-aligned text sequence as input and the sentence boundary label as training target, using a first loss function to optimize the parameters of the pre-trained text model to obtain a trained text semantic model.

[0041] Specifically, to build a model that can effectively judge semantic punctuation, first, an initial text semantic model is obtained, and the basic structure of the initial text semantic model should at least include a word embedding layer, a multi-layer recurrent neural network, and a linear classification layer. The word embedding layer is used to vectorize the input text vocabulary, converting each word into a dense numerical vector. The purpose of this is to convert text information into a form that is easy for machine learning algorithms to process, while also preserving the semantic similarity between words. The multi-layer recurrent neural network is used to process the word embedding vectors. RNN can understand the sequential nature of text and capture the relevance of context. The linear classification layer is used to classify each text segment based on the output of the RNN, determine whether it is the boundary of a sentence, and output a probability distribution.

[0042] Secondly, the initial model is pre-trained using a text dataset. The text dataset includes a large number of natural sentences, covering different language styles and topics, to ensure that the model can generalize to various text punctuation tasks in various contexts. Through a large amount of text input, the model establishes a complex weight network between the word embedding layer, the recurrent neural network layer, and the linear classification layer, learns to distinguish the internal structure of the sentence and the punctuation rules, and thus obtains a pre-trained text model.

[0043] Thirdly, in order to make the text semantic model better serve the speech data, a speech-text alignment dataset is further introduced for fine-tuning. In the speech-text alignment dataset, each speech signal has a synchronized text sequence, i.e. a text description generated by an automatic speech recognition model and aligned with the speech signal timing. Using the speech-text pair, the model can be trained to more accurately identify the sentence boundaries that match the speech signal when processing the decoded text.

[0044] During the fine-tuning process, the time-aligned text sequence can be used as input, and the sentence boundary label (i.e. marking which text segment is the end of a sentence) can be used as a supervised target. A first loss function (such as a cross-entropy loss function) is used to measure the difference between the model's prediction and the true label. By continuously optimizing the model parameters and reducing the difference between the prediction and the true value, the model gradually learns the rules for finding sentence boundaries in decoded text, even in the face of recognition errors that may be caused by ASR. The trained text semantic model can provide effective punctuation guidance in the process of converting speech into text.

[0045] Optionally, based on the text dataset, the initial text semantic model is trained to obtain a pre-trained text model, which can be achieved by the following method: taking the text dataset as input, taking forward language modeling or sentence end prediction as training target, and optimizing the parameters of the initial text semantic model through a second loss function to obtain the pre-trained text model.

[0046] It can be understood that there are two goals for training the initial text semantic model, one is forward language modeling, the task of the model is to learn how to predict the next word, not only requires the model to understand the semantics of the current word, but also has the ability to perceive the structure of the entire sentence, to ensure the continuity and rationality of the prediction. The other is sentence end prediction, the task of the model is to judge whether the given text segment is the end of a complete sentence or sentence, which is directly related to the task of sentence breaking, which can train the model's sentence breaking recognition skills at the text semantic level. In the pre-training process, the second loss function is used to optimize the parameters of the initial text semantic model. The second loss function, for example, the cross-entropy loss function, can effectively measure the difference between the model's prediction results and the actual labels, prompting the model to adjust its internal parameters to minimize this difference. As the training progresses, the model gradually learns to predict the end of a sentence based on the semantic and structural features of the text, which not only improves the model's accuracy in text sentence breaking, but also enhances the model's adaptability to different text types and styles. Finally, through continuous iterative optimization, a pre-trained text model is obtained, which can effectively identify sentence boundaries based on a single text modality.

[0047] Optionally, the acoustic time sequence feature vector includes: a time-frequency pattern, a prosody feature, and a speech segment time sequence dependency relationship; the text semantic model includes: a long short-term memory network; and the semantic feature vector is automatic speech recognition-based decoded text output by the long short-term memory network.

[0048] Among them, the acoustic time sequence feature vector is a key information carrier in the intelligent speech sentence breaking model for representing speech signals, and its composition mainly includes three aspects: time-frequency pattern, prosody feature, and speech segment time sequence dependency relationship. The time-frequency pattern is a feature representation in frequency and time extracted from the speech signal, which can use Mel-frequency cepstral coefficients (MFCCs) or linear frequency cepstral coefficients (LFCCs), etc. These features can reflect the frequency spectrum information of the speech signal and its dynamic characteristics changing over time, and are the basis for identifying speech and silence. The prosody feature includes information such as the rhythm, stress, and intonation of speech, which can help the model understand the speaker's tone changes and sentence structure. In addition, the speech segment time sequence dependency relationship refers to the continuity and correlation between speech segments, and the model can capture deeper speech activity patterns by analyzing these dependencies, such as sentence run-on phenomena or changes in speech rate, which are important clues for determining sentence boundaries.

[0049] The text semantic model mainly adopts a long short-term memory network. The LSTM network is a special recurrent neural network that can effectively solve the long-term dependency problem, that is, the model can remember the information at a remote position in the sequence without losing details. The LSTM network receives a text sequence generated based on the automatic speech recognition model decoding, wherein the text sequence corresponds to the real-time speech stream one by one and can timely reflect the change of the speech content. The LSTM network can filter and remember important semantic information through an internal gating mechanism, ignore details that have little effect on the punctuation judgment, and output a highly condensed semantic feature vector to represent the semantic end tendency and sentence integrity of the decoded text.

[0050] In step S108, the acoustic time sequence feature vector and the semantic feature vector are cross-attention fused to obtain a fusion result, and the punctuation result corresponding to the speech data is determined according to the fusion result.

[0051] According to some optional embodiments of the present application, the acoustic time sequence feature vector and the semantic feature vector are cross-attention fused to obtain a fusion result, which can be realized by the following method: the acoustic time sequence feature vector is input into a first linear transformation layer to obtain a query vector sequence; the semantic feature vector is input into a second linear transformation layer to obtain a key vector sequence, and the semantic feature vector is input into a third linear transformation layer to obtain a value vector sequence; an attention weight matrix between the query vector sequence and the key vector sequence is calculated, wherein the attention weight matrix is used to represent the correlation strength between the acoustic feature and the semantic feature; the value vector sequence is weighted and summed by using the attention weight matrix to obtain an acoustic context vector weighted by semantic information; the acoustic context vector and the original acoustic time sequence feature vector are residual connected, and the residual connection result is subjected to feature fusion by a fourth linear transformation layer to obtain the fusion result.

[0052] In the present embodiment, the first linear transformation layer is first applied to the acoustic time sequence feature vector, which maps the acoustic feature vector to a new dimensional space to obtain a query vector sequence. The semantic feature vector is input into the second and third linear transformation layers to generate a key vector sequence and a value vector sequence, respectively. The key vector sequence is used to match the query vector sequence to find the correlation degree between the acoustic and semantic features; the value vector sequence includes rich information of the semantic feature and will be used as an object of weighted summation in the fusion process so as to integrate the semantic information into the analysis of the acoustic feature.

[0053] Secondly, the dot product between the query vector sequence and the key vector sequence is calculated to obtain an attention weight matrix. Each element in the attention weight matrix represents the correlation strength between a time sequence segment in the acoustic feature vector and the semantic feature vector, thereby quantifying the association between the two modalities. The attention weight matrix is normalized by the Softmax function, which ensures that each time sequence segment can obtain a weight value based on its association with the semantic feature.

[0054] Thirdly, the value vector sequence is weighted and summed using the above attention weight matrix, resulting in an acoustic context vector that includes semantic information. The acoustic context vector is obtained by multiplying each item in the value vector sequence of the semantic feature by its corresponding attention weight and then summing the results, which can concentrate the most relevant semantic information to the acoustic feature.

[0055] Finally, in order to further fuse the two features, the acoustic context vector is connected in residual with the original acoustic time sequence feature vector. The result of the residual connection is then subjected to feature fusion by a fourth linear transformation layer, which integrates the acoustic and semantic features into a unified representation space to obtain the fusion result. The fusion result is the direct output of the model's comprehensive analysis of the speech data, and it combines the time sequence information of the acoustic feature and the content understanding of the semantic feature to provide a more comprehensive and accurate basis for intelligent sentence segmentation.

[0056] The above steps S102 and S108 break through the limitations of traditional single-modal sentence segmentation and innovatively precisely fuse acoustic features (from the intermediate output of the ASR model Encoder layer) and text semantic features (ASR decoded text processed by the LSTM network) through cross-attention mechanism, using text semantic vectors as Query, acoustic features as Key and Value, to realize collaborative judgment of the two modalities and significantly improve the sentence segmentation accuracy in complex scenarios. On the one hand, the intermediate layer features of the ASR model are reused as the input of the acoustic sentence segmentation VAD model, avoiding the redundancy of additional feature extraction. On the other hand, the text semantic sentence segmentation model is decoupled from the ASR, pre-trained on a large amount of text and fine-tuned on aligned data, which not only utilizes the output resources of the ASR but also reduces the impact of ASR recognition errors on sentence segmentation, achieving deep and flexible integration with the ASR model.

[0057] The above steps S102 and S108 can realize multi-modal fusion of acoustic features and text semantic features, solving the problems of single modality, insufficient precision, and poor robustness of traditional sentence segmentation methods, improving the sentence segmentation accuracy and real-time performance in complex scenarios, and optimizing the intelligent voice interaction experience. In addition, by cross-attention fusion of acoustic time sequence features and text semantic features, the limitations of single modality can be solved, the sentence segmentation accuracy can be improved, the sentence segmentation logic can be made to conform to human conversation habits, the interaction can be reduced, and the naturalness of intelligent assistants and real-time transcription can be improved.

[0058] Figure 2 is a flowchart of another intelligent speech punctuation method based on multi-modal fusion according to an embodiment of the present application, as shown in the figure, the method comprises the following steps: Figure 2

[0059] Step S201, a pre-training acoustic punctuation VAD model is constructed.

[0060] Feature extraction: directly use the intermediate output of the Encoder layer of the ASR model as the basic acoustic feature. The basic acoustic feature is trained by a large amount of speech data, and at least includes a time-frequency pattern (such as a mel spectrum), a prosodic feature (a fundamental frequency, an energy change), and a speech segment timing dependency.

[0061] Model structure: the basic acoustic feature is input into a 2-layer LSTM network, each layer contains 256 hidden units, and the dropout value is set to 0.3. The LSTM network models the timing of acoustic information, maps the feature to a 4-dimensional space through a Linear layer, and outputs a probability distribution through a softmax activation: 0: start silence (silence segment before the start of a sentence); 1: speech (effective speech segment within a sentence); 2: intermediate pause silence (silence segment of pause within a sentence); 3: end silence (silence segment after the termination of a sentence).

[0062] Training method: during the training process, the ASR network parameters are frozen, only the LSTM layer and the Linear layer parameters are updated, and the cross-entropy loss function is used. When streaming inference, if 320ms audio is input, 8 frames of timing features (each frame corresponds to a 40ms speech segment) are output.

[0063] Step S202, a pre-training text semantic punctuation model is constructed.

[0064] Input processing: the text sequence of ASR is decoded in real time through a greedy search algorithm, and the sequence is strictly time-aligned with the current speech segment.

[0065] Model structure: the text sequence is converted into a 256-dimensional dense vector through Tokenizer, and is input into a 2-layer LSTM network (256 hidden units, dropout=0.2). The hidden vector corresponding to the last word of the text sequence is extracted as a semantic feature, which is mapped to a 2-dimensional space through a Linear layer, and a probability distribution is output through a softmax activation: end: the end of the text is the termination boundary of the sentence; not end: the end of the text is the intermediate continuation of the sentence.

[0066] Training advantage: the model is decoupled from the ASR, pre-trained on a large amount of text (dialogue, novel, etc.), fine-tuned with 10,000 “speech-text” alignment data, fully utilizes the large amount of text resources to improve performance, and the training can use the cross-entropy loss function. ​

[0067] Step S203, multi-modal fusion.

[0068] Input features: receive two types of features, one is the multi-frame acoustic timing feature vector (256 dimensions per frame) output by the acoustic sentence segmentation VAD model, including acoustic clues such as speech prosody and silence duration; the other is the single-frame semantic feature vector (256 dimensions) output by the text semantic model, containing semantic end words, sentence completion, and other text clues. If no text is decoded at present, use the semantic feature vector of the previous state.

[0069] Cross-attention calculation: single-round feature interaction, with the text semantic vector as Query (Q), and the acoustic feature as Key (K) and Value (V). The weight distribution calculation is as follows:

[0070]

[0071] where d k is the feature dimension. The single-frame semantic feature and multi-frame acoustic feature are weighted and fused by Attention(Q,K)·V, and finally 1 frame of fusion feature is output.

[0072] Output layer processing: the fusion feature is output through the Linear layer and the softmax activation (4 classification) to output the final probability. If the maximum probability is state 3, it indicates end silence, state 2 indicates pause silence, and state 1 indicates that it is speaking. After the pre-training acoustic VAD model and the pre-training text semantic sentence segmentation model are trained, the multi-modal fusion model is trained using the cross-entropy loss function.

[0073] Step S204, flow control.

[0074] Triggering logic: the acoustic sentence segmentation VAD monitors in real time with short speech segments (320ms), and when the "non-speech" state is detected, the ASR and the text semantic model are waiting; when the "speech" state is detected, the ASR real-time recognition is started.

[0075] Real-time inference: the acoustic sentence segmentation VAD and the text semantic model are processed in parallel, and the features are updated every 320ms. After the text semantic model obtains new text, it outputs semantic features, which are fused with acoustic features through cross-attention, and then processed through Linear and softmax layers to output the sentence segmentation result. If it is state 3, the sentence segmentation is triggered, state 2 is identified as an intermediate pause silence, and state 1 is in the process of speaking.

[0076] Figure 3 is a structural diagram of an intelligent speech sentence segmentation system based on multi-modal fusion according to an embodiment of the present application, as Figure 3 shown, the system comprises:

[0077] An acquisition module 32 is configured to acquire voice data to be processed.

[0078] A first processing module 34 is configured to process the voice data by using a pre-trained acoustic sentence breaking model to obtain an acoustic time sequence feature vector.

[0079] A second processing module 36 is configured to process the voice data by using a pre-trained text semantic model to obtain a semantic feature vector.

[0080] A determination module 38 is configured to cross-attentionally fuse the acoustic time sequence feature vector and the semantic feature vector to obtain a fusion result, and determine a sentence breaking result corresponding to the voice data according to the fusion result.

[0081] Optionally, the acoustic sentence breaking model at least includes an automatic speech recognition model and a sequence classification network, the sequence classification network at least includes one recurrent neural network layer and one fully connected output layer; the acoustic sentence breaking model is obtained by training in the following manner: fixing all parameters in an encoder module in the automatic speech recognition model; obtaining a voice data set, wherein the voice data set at least includes voice data carrying a class label of a voice segment and a silence segment; extracting basic acoustic features of the voice data set by using the encoder module; performing time sequence modeling on the basic acoustic features by using the recurrent neural network layer, and mapping a time sequence modeling result to a class label space by using the fully connected output layer; optimizing and adjusting parameters of the sequence classification network with the class label as a training target to obtain the acoustic sentence breaking model after completion of training.

[0082] Optionally, the class label space at least includes a first label, a second label, a third label and a fourth label, wherein the first label is used to represent a silence segment before a sentence, the second label is used to represent an effective voice segment within a sentence, the third label is used to represent a silence segment of a pause within a sentence, and the fourth label is used to represent a silence segment after termination of a sentence.

[0083] Optionally, the text semantic model is obtained by training in the following manner: obtaining an initial text semantic model, wherein the initial text semantic model at least includes a word embedding layer, a multi-layer recurrent neural network and a linear classification layer; obtaining a text data set, wherein the text data set includes natural sentences; training the initial text semantic model based on the text data set to obtain a pre-trained text model; obtaining a voice-text alignment data set, wherein the voice-text alignment data set at least includes a voice signal and a time sequence alignment text sequence corresponding to the voice signal and generated by decoding of the automatic speech recognition model; using the time sequence alignment text sequence as input, using a sentence boundary label as a training target, and using a first loss function to optimize parameters of the pre-trained text model to obtain the text semantic model after completion of training.

[0084] Optionally, based on the text data set, the initial text semantic model is trained to obtain a pre-trained text model, including: taking the text data set as input, taking forward language modeling or sentence end prediction as a training target, optimizing parameters of the initial text semantic model through a second loss function, and obtaining the pre-trained text model.

[0085] Optionally, the acoustic time sequence feature vector and the semantic feature vector are cross-attention fused to obtain a fusion result, including: inputting the acoustic time sequence feature vector into a first linear transformation layer to obtain a query vector sequence; inputting the semantic feature vector into a second linear transformation layer to obtain a key vector sequence, and inputting the semantic feature vector into a third linear transformation layer to obtain a value vector sequence; calculating an attention weight matrix between the query vector sequence and the key vector sequence, wherein the attention weight matrix is used to represent the correlation strength between the acoustic feature and the semantic feature; performing weighted summation on the value vector sequence by using the attention weight matrix to obtain an acoustic context vector weighted by semantic information; performing residual connection on the acoustic context vector and the original acoustic time sequence feature vector, performing feature fusion on the residual connection result by using a fourth linear transformation layer, and obtaining the fusion result.

[0086] Optionally, the acoustic time sequence feature vector includes: a time-frequency pattern, a prosodic feature, and a speech segment time sequence dependency relationship; the text semantic model includes: a long short-term memory network; and the semantic feature vector is automatic speech recognition based decoding text output by the long short-term memory network.

[0087] It should be noted that the above Figure 3 Each module in the above

[0088] It should be noted that Figure 3 The preferred embodiments of the embodiments shown in the above Figure 1 will not be described here.

[0089] Figure 4 A hardware structure block diagram of a computer terminal for implementing the intelligent speech punctuation method based on multi-modal fusion is shown. As Figure 4As shown, the computer terminal 40 may include one or more processors 402 (shown as 402a, 402b, ..., 402n in the figure) 402 (processor 402 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 404 for storing data, and a transmission module 406 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 4 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 40 may also include... Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown.

[0090] It should be noted that the aforementioned one or more processors 402 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 40. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0091] The memory 404 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the intelligent speech segmentation method based on multimodal fusion in this embodiment. The processor 402 executes various functional applications and data processing by running the software programs and modules stored in the memory 404, thereby realizing the aforementioned intelligent speech segmentation method based on multimodal fusion. The memory 404 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 404 may further include memory remotely located relative to the processor 402, and these remote memories can be connected to the computer terminal 40 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0092] The transmission module 406 is configured to receive or send data via a network. The network can include a wireless network provided by a communication provider of the computer terminal 40. In one example, the transmission module 406 includes a network interface controller (NIC) that can connect to other network devices through a base station to communicate with the Internet. In one example, the transmission module 406 can be a radio frequency (RF) module that is configured to communicate with the Internet wirelessly.

[0093] The display can be a liquid crystal display (LCD) that is touch screen, for example, which can enable a user to interact with a user interface of the computer terminal 40.

[0094] It should be noted that, in some optional embodiments, the above Figure 4 The computer terminal shown can include hardware elements (including circuitry), software elements (including computer code stored on a computer readable medium), or a combination of both hardware and software elements. It should be noted that, Figure 4 is merely one example of a particular implementation and is intended to illustrate the types of components that can be present in the computer terminal described above.

[0095] It should be noted that, Figure 4 The computer terminal shown is configured to perform Figure 1 The multi-modal fusion based intelligent speech punctuation method shown above, and the related explanations in the execution method of the above commands also apply to the electronic device, which will not be described here.

[0096] The embodiments of the present application also provide a non-volatile storage medium, which includes a stored program, wherein the program controls a device in which the storage medium is located to execute the above multi-modal fusion based intelligent speech punctuation method when the program is running.

[0097] The program of the non-volatile storage medium performs the following functions: obtaining speech data to be processed; processing the speech data using a pre-trained acoustic punctuation model to obtain an acoustic time sequence feature vector; processing the speech data using a pre-trained text semantic model to obtain a semantic feature vector; cross-attention fusion of the acoustic time sequence feature vector and the semantic feature vector to obtain a fusion result, and determining a punctuation result corresponding to the speech data according to the fusion result.

[0098] The embodiments of the present application also provide an electronic device, which includes a memory and a processor, and the processor is configured to run a program stored in the memory, wherein the program performs the above multi-modal fusion based intelligent speech punctuation method when the program is running.

[0099] The processor is configured to run a program to perform the following functions: obtaining voice data to be processed; processing the voice data by using a pre-trained acoustic sentence breaking model to obtain an acoustic time sequence feature vector; processing the voice data by using a pre-trained text semantic model to obtain a semantic feature vector; performing cross-attention fusion on the acoustic time sequence feature vector and the semantic feature vector to obtain a fusion result, and determining a sentence breaking result corresponding to the voice data according to the fusion result.

[0100] The sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0101] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0102] In the above embodiments of the present application, the collected information is information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary protection measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal.

[0103] In the several embodiments of the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the unit described as the division is only a description of logical function division. For example, there can be another division way for actual implementation, such as combination or integration of multiple units or components, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0104] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0105] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0106] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part that essentially contributes to the related art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0107] The above only describes the preferred embodiments of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A method for intelligent speech punctuation based on multi-modal fusion, characterized in that, The method comprises the following steps: acquiring voice data to be processed; processing the voice data by using a pre-trained acoustic sentence breaking model to obtain an acoustic time sequence feature vector; processing the voice data by using a pre-trained text semantic model to obtain a semantic feature vector; cross-attention fusing the acoustic time sequence feature vector and the semantic feature vector to obtain a fusion result, and determining a sentence breaking result corresponding to the voice data according to the fusion result.

2. The method of claim 1, wherein: the acoustic sentence breaking model comprises at least an automatic speech recognition model and a sequence classification network, and the sequence classification network comprises at least one recurrent neural network layer and one fully connected output layer; the acoustic sentence breaking model is obtained by the following method: fixing all parameters in an encoder module of the automatic speech recognition model; acquiring a voice data set, wherein the voice data set comprises at least voice data carrying a category label of a voice segment and a silence segment; extracting basic acoustic features of the voice data set by using the encoder module; performing time sequence modeling on the basic acoustic features by using the recurrent neural network layer, and mapping a time sequence modeling result to a category label space by using the fully connected output layer; optimizing and adjusting parameters of the sequence classification network to obtain a trained acoustic sentence breaking model.

3. The method of claim 2, wherein, the category label space comprises at least a first label, a second label, a third label and a fourth label, wherein the first label is used to represent a silence segment before a sentence, the second label is used to represent an effective voice segment in a sentence, the third label is used to represent a silence segment of a pause in a sentence, and the fourth label is used to represent a silence segment after a sentence.

4. The method of claim 1, wherein, the text semantic model is obtained by the following method: acquiring an initial text semantic model, wherein the initial text semantic model comprises at least a word embedding layer, a multi-layer recurrent neural network and a linear classification layer; acquiring a text data set, wherein the text data set comprises natural sentences; training the initial text semantic model based on the text data set to obtain a pre-trained text model; acquiring a voice-text alignment data set, wherein the voice-text alignment data set comprises at least a voice signal and a time sequence alignment text sequence corresponding to the voice signal and generated by an automatic speech recognition model; using a first loss function to optimize parameters of the pre-trained text model to obtain a trained text semantic model, with the time sequence alignment text sequence as input and a sentence boundary label as a training target.

5. The method of claim 4, wherein, training the initial text semantic model based on the text data set to obtain a pre-trained text model, comprising: using a second loss function to optimize parameters of the initial text semantic model to obtain the pre-trained text model, with the text data set as input and forward language modeling or sentence end prediction as a training target.

6. The method of claim 1, wherein, cross-attention fusing the acoustic time sequence feature vector and the semantic feature vector to obtain a fusion result, comprising: input the acoustic time sequence feature vector into a first linear transformation layer to obtain a query vector sequence; input the semantic feature vector into a second linear transformation layer to obtain a key vector sequence, and input the semantic feature vector into a third linear transformation layer to obtain a value vector sequence; calculate an attention weight matrix between the query vector sequence and the key vector sequence, wherein the attention weight matrix is used to represent the correlation strength between acoustic features and semantic features; perform weighted summation on the value vector sequence by using the attention weight matrix to obtain an acoustic context vector weighted by semantic information; perform residual connection on the acoustic context vector and the original acoustic time sequence feature vector, and perform feature fusion on the residual connection result by using a fourth linear transformation layer to obtain the fusion result.

7. The method of claim 1, wherein, The acoustic time sequence feature vector comprises: a time-frequency pattern, prosodic features, and speech segment time sequence dependency relationships. The text semantic model comprises: a long short-term memory network; and the semantic feature vector is a decoded text based on automatic speech recognition output by the long short-term memory network. The method comprises the steps of:

8. An intelligent speech punctuation system based on multi-modal fusion, characterized in that, The acquisition module is configured to acquire speech data to be processed. The first processing module is configured to process the speech data by using a pre-trained acoustic sentence breaking model to obtain an acoustic time sequence feature vector. The second processing module is configured to process the speech data by using a pre-trained text semantic model to obtain a semantic feature vector. The determination module is configured to perform cross-attention fusion on the acoustic time sequence feature vector and the semantic feature vector to obtain a fusion result, and determine a sentence breaking result corresponding to the speech data according to the fusion result. The non-volatile storage medium comprises a stored program, wherein the program controls a device in which the non-volatile storage medium is located to perform the intelligent speech sentence breaking method based on multi-modal fusion according to any one of claims 1 to 7 when the program is running.

9. A non-volatile storage medium, comprising: The method comprises the steps of:

10. An electronic device, comprising: The memory and the processor are configured to run a program stored in the memory, wherein the program performs the intelligent speech sentence breaking method based on multi-modal fusion according to any one of claims 1 to 7 when the program is running. The computer program is executed by the processor to implement the intelligent speech sentence breaking method based on multi-modal fusion according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that, ​

Images