Method and device for converting voice into text, equipment, storage medium and program product

Abstract

The invention provides a method and device for converting voice into characters, equipment, a storage medium and a program product, and relates to the technical field of information, and the method for converting voice into characters comprises the steps: carrying out the fusion processing of the acoustic features of to-be-converted voice frame data and the semantic context of the voice frame data, and obtaining the fusion features of different input channels; performing dimension reduction processing on the fusion features of the different input channels to obtain first features; according to the attention weight of the input channel, the voice frame data and the first feature, obtaining rhythm frame data in the voice frame data; and converting the voice frame data into target characters according to the rhythm frame data and the voice frame data. According to the invention, the accuracy and real-time performance of character conversion are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, and in particular to a method and device for converting speech into text, equipment, storage medium and program product. BACKGROUND

[0002] At present, the technology of converting speech into text is applied in various aspects, such as human-computer interaction in intelligent customer service systems, real-time subtitle generation of multimedia content, voice instruction recognition of intelligent devices, etc., and with the wide application of this technology, there are higher requirements in terms of accuracy, real-time performance and readability of conversion.

[0003] The existing technology of converting speech into text mainly adopts the following four typical architectures:

[0004] I. Traditional cascading speech processing system, such as including automatic speech recognition (ASR), large language model (LLM) and text to speech (TTS) architecture, which adopts modular pipeline design to perform three independent processing stages of speech recognition (ASR), language model optimization (LLM) and speech synthesis (TTS), and each module transmits data through a standard text interface, with the output of the previous stage as the input of the next stage. However, the traditional cascading speech processing system is prone to information loss in the modular pipeline design, especially the prosody, emotion and other paralinguistic information of the original speech cannot be effectively preserved, and the recognition errors of the previous stage modules will be accumulated, affecting the accuracy of the final output.

[0005] II. End-to-end speech language model, such as speech language model (SpeechLM), based on the unified encoder-decoder architecture of Transformer, directly establishes the mapping relationship between speech signal and text, models acoustic features and language features through multi-task learning, and realizes single-model end-to-end conversion. However, the end-to-end speech language model simplifies the processing flow, but requires high computing resources, making it difficult to deploy on edge devices, and the self-recursive generation mode leads to insufficient real-time performance.

[0006] III. General speech-to-text tool, such as Whisper, which adopts the architecture of large-scale pre-trained speech encoder and self-recursive text decoder, supports multi-language speech recognition, realizes nonlinear conversion of acoustic features to text tokens through attention mechanism, and outputs text with basic punctuation. However, the general speech-to-text tool lacks coherent modeling of long dialogue context, and the redundant word filtering relies only on rule-based post-processing, which is not effective, and the punctuation generation lacks semantic understanding, often causing improper use;

[0007] Four, the flow Transducer hybrid architecture, such as Transducer and attention based encoder-decoder (Transducer and Attention based Encoder-Decoder, TAED) structure, combines recurrent neural network transducer (RNN-T) streaming processing framework and dynamic block attention mechanism, realizes low-delay speech transcription, adopts time-synchronous decoding strategy, and incrementally outputs text while the voice input is being input. However, although the flow Transducer architecture optimizes real-time performance, the ability to capture complex semantics is weak, and deep fusion of voice and text features is not realized, resulting in limited semantic optimization effect.

[0008] In summary, the existing voice conversion text technology has the problems of low conversion accuracy and low real-time performance. SUMMARY

[0009] The present application provides a voice conversion text method, device, equipment, storage medium and program product, to solve the existing voice conversion text technology has the problems of low conversion accuracy and low real-time performance.

[0010] In order to solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0011] In a first aspect, the embodiments of the present application provide a voice conversion text method, which comprises:

[0012] Fusing the acoustic features of the voice frame data to be converted and the semantic context of the voice frame data to obtain fusion features of different input channels;

[0013] Dimensionally reducing the fusion features of the different input channels to obtain first features;

[0014] According to the attention weights of the input channels, the voice frame data and the first features, obtaining prosodic frame data in the voice frame data;

[0015] According to the prosodic frame data and the voice frame data, converting the voice frame data into target text.

[0016] Optionally, the fusion of the acoustic features of the voice frame data to be converted and the semantic context of the voice frame data to obtain fusion features of different input channels comprises:

[0017] Obtaining the voice frame data to be converted;

[0018] Performing feature extraction processing on the voice frame data to obtain acoustic features of different input channels;

[0019] predicting a semantic context of the speech frame data;

[0020] fusing the acoustic features of different input channels and the semantic context to obtain fused features of the response input channel.

[0021] Optionally, the obtaining of the speech frame data to be converted comprises:

[0022] performing frame processing on the speech to be converted to obtain initial frame data;

[0023] performing windowing processing on the initial frame data to obtain the speech frame data.

[0024] Optionally, the feature extraction processing on the speech frame data to obtain acoustic features of different input channels comprises:

[0025] converting the speech frame data into frequency domain features, wherein the frequency domain features comprise frequency domain features of different input channels;

[0026] processing the frequency domain features of each input channel respectively by using a channel self-attention mechanism to obtain second features of the corresponding input channel;

[0027] obtaining third features of each input channel by using a time sequence attention mechanism according to the frequency domain features of the input channel, the number of frames of the speech frame data and the dimension of the input channel;

[0028] performing compression and dimension reduction processing on the second features and the third features to obtain fourth features;

[0029] performing time sequence reservation processing on the fourth features to obtain fifth features;

[0030] performing frequency domain feature extraction on the fifth features, and performing time domain feature extraction on the fifth features to obtain the acoustic features.

[0031] Optionally, the processing of the frequency domain features of each input channel respectively by using a channel self-attention mechanism to obtain second features of the corresponding input channel comprises:

[0032] performing compression processing on the frequency domain features of each input channel respectively by using a channel self-attention mechanism according to the number of frames of the speech frame data to obtain first target features;

[0033] obtaining the first features of the corresponding input channel according to the attention weight of the input channel and the first target features.

[0034] Optionally, the utilizing the time sequence attention mechanism comprises: obtaining a third feature of each input channel according to the frequency domain feature of each input channel, the number of frames of the speech frame data, and the dimension of the input channel.

[0035] The time sequence attention mechanism is utilized to obtain a second target feature of each input channel according to the frequency domain feature of each input channel and the dimension of the input channel.

[0036] The second feature of each input channel is obtained according to the second target feature of each input channel and the number of frames of the speech frame data.

[0037] Optionally, the compressing and reducing dimension processing of the second feature and the third feature comprises:

[0038] The second feature and the third feature are fused according to the number of frames of the speech frame data, the number of input channels, and the number of frequency bands of the frequency domain feature to obtain a third target feature.

[0039] The third target feature is compressed and reduced in dimension according to a preset dimension number to obtain the third feature.

[0040] Optionally, the frequency domain feature extraction of the fifth feature and the time domain feature extraction of the fifth feature to obtain the acoustic feature comprise:

[0041] An LFMMI target function is constructed according to the fifth feature.

[0042] The frequency domain feature extraction of the fifth feature and the time domain feature extraction of the fifth feature are performed according to the LFMMI target function to obtain the acoustic feature.

[0043] Optionally, the fusion processing of the acoustic features of different input channels and the semantic context comprises:

[0044] The acoustic feature is projected to a target dimension by linear projection to obtain a fourth target feature, and the semantic context is projected to a target dimension by linear projection to obtain a target context.

[0045] The fusion feature is obtained according to the fourth target feature and the target context.

[0046] Optionally, the reducing dimension processing of the fusion features of different input channels comprises:

[0047] An attention weight of an input channel is obtained.

[0048] According to the attention weight, the fusion feature corresponding to the key input channel and the fusion feature corresponding to the non-key input channel in the input channel are obtained;

[0049] The fusion feature corresponding to the non-key input channel is rounded to obtain a sixth feature;

[0050] According to the attention score matrix, the fusion feature corresponding to the key input channel and the first feature corresponding to the weak dependence interval in the sixth feature are obtained.

[0051] Optionally, the obtaining of the fusion feature corresponding to the key input channel and the fusion feature corresponding to the non-key input channel in the input channel according to the attention weight comprises:

[0052] According to the normal distribution information of the attention weight, the key input channel and the non-key input channel in the input channel are determined;

[0053] The fusion feature corresponding to the key input channel and the fusion feature corresponding to the non-key input channel in the input channel are obtained.

[0054] Optionally, the obtaining of the fusion feature corresponding to the key input channel and the first feature corresponding to the weak dependence interval in the sixth feature according to the attention score matrix comprises:

[0055] According to the attention score matrix, the inter-frame dependence score in a preset division interval is obtained;

[0056] According to the inter-frame dependence score, the fusion feature corresponding to the key input channel and the weak dependence interval in the sixth feature are obtained.

[0057] The fifth target feature corresponding to the weak dependence interval is obtained;

[0058] The fifth target feature is decentered to obtain a sixth target feature;

[0059] According to the sixth target feature, a covariance matrix is constructed to obtain a seventh target feature;

[0060] According to the principal component matrix of the seventh target feature and the fifth target feature matrix, the first feature is obtained.

[0061] Optionally, the prosodic frame data comprises stress frame data and pause frame data;

[0062] According to the attention weight of the input channel, the speech frame data and the first feature, the prosodic frame data in the speech frame data is obtained, comprising:

[0063] According to the first feature and the attention weight, stress frame data in the speech frame data is obtained;

[0064] According to an energy difference between speech frame data of two adjacent frames in the speech frame data, pause frame data in the speech frame data is obtained.

[0065] Optionally, the obtaining of the pause frame data in the speech frame data according to the energy difference between the speech frame data of the two adjacent frames in the speech frame data comprises:

[0066] Frame energy of each speech frame data in the speech frame data is obtained.

[0067] In a case where frame energies corresponding to the first speech frame data, the second speech frame data and the third speech frame data are all less than a preset energy value, the first speech frame data, the second speech frame data and the third speech frame data are taken as the pause frame data.

[0068] The first speech frame data, the second speech frame data and the third speech frame data are three continuous speech frame data in the speech frame data.

[0069] Optionally, the converting of the speech frame data into the target text according to the prosody frame data and the speech frame data comprises:

[0070] According to the prosody frame data and the speech frame data in a preset sliding time window, the speech frame data in the preset sliding time window is converted into window text.

[0071] The window text corresponding to the preset sliding time window is subjected to splicing processing and de-overlapping processing, and the target text is obtained.

[0072] Optionally, the method further comprises:

[0073] A dynamic weight of the target text is obtained.

[0074] In a case where the dynamic weight is greater than a first dynamic weight, the target text is taken as a final text of the speech frame data.

[0075] In a case where the dynamic weight is less than a second dynamic weight, the target text is semantically corrected according to the acoustic feature and the target text, and a corrected text is obtained, and the corrected text is taken as the final text of the speech frame data.

[0076] in a case that the dynamic weight is less than or equal to a first dynamic weight and greater than or equal to a second dynamic weight, performing semantic correction on the target text according to the acoustic feature and the target text, obtaining a corrected text, and obtaining a final text of the speech frame data according to the corrected text and the target text;

[0077] wherein the first dynamic weight is greater than the second dynamic weight.

[0078] In a second aspect, an embodiment of the present application further provides a device for converting speech into text, the device comprising:

[0079] a first processing module configured to perform fusion processing on an acoustic feature of speech frame data to be converted and a semantic context of the speech frame data, to obtain fusion features of different input channels;

[0080] a second processing module configured to perform dimension reduction processing on the fusion features of the different input channels, to obtain first features;

[0081] a third processing module configured to obtain prosodic frame data in the speech frame data according to the attention weight of the input channel, the speech frame data and the first features;

[0082] a fourth processing module configured to convert the speech frame data into target text according to the prosodic frame data and the speech frame data.

[0083] In a third aspect, an embodiment of the present application further provides a device for converting speech into text, comprising a processor, a memory and a program stored in the memory and executable on the processor, the program being executed by the processor to implement the method for converting speech into text according to any one of the first aspect.

[0084] In a fourth aspect, an embodiment of the present application further provides a readable storage medium, the readable storage medium storing a program, the program being executed by a processor to implement the method for converting speech into text according to any one of the first aspect.

[0085] In a fifth aspect, an embodiment of the present application provides a computer program product comprising computer instructions, the computer instructions being executed by a processor to implement the method for converting speech into text according to any one of the first aspect.

[0086] The present application has the following beneficial effects:

[0087] The speech-to-text method provided by this invention fuses the acoustic features and semantic context of the speech frame data to be converted, obtaining fusion features from different input channels. These fusion features combine acoustic features with a language model, thereby improving the accuracy and real-time performance of subsequent text conversion based on these fusion features. Dimensionality reduction is performed on the fusion features of key input channels among the fusion features from different input channels to obtain a first feature. Text conversion based on this first feature improves the transmission effect during the text conversion process. Prosodic frame data is obtained from the input channel attention weights, the speech frame data, and the first feature. Based on the prosodic frame data and the speech frame data, the speech frame data is converted into target text. Obtaining the target text from the speech frame data based on the prosodic frame data reduces the loss of speech information in the speech frame data, further improving the conversion accuracy. Attached Figure Description

[0088] Figure 1 A flowchart illustrating the speech-to-text method provided in an embodiment of the present invention;

[0089] Figure 2 This diagram illustrates the architecture of the encoder model provided in this embodiment of the invention.

[0090] Figure 3 This is a schematic diagram illustrating the structure of the speech-to-text system provided in an embodiment of the present invention;

[0091] Figure 4 A schematic diagram showing the structure of the speech-to-text device provided in an embodiment of the present invention;

[0092] Figure 5 This is a schematic diagram illustrating the structure of the speech-to-text device provided in an embodiment of the present invention. Detailed Implementation

[0093] To make the technical problems, technical solutions, and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments. In the following description, specific details such as particular configurations and components are provided merely to aid in a comprehensive understanding of the embodiments of this application. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Furthermore, for clarity and brevity, descriptions of known functions and structures have been omitted.

[0094] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0095] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0096] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, without limiting the number of objects; for example, the first object can be one or more. Furthermore, "or" in this application indicates at least one of the connected objects. For example, "A or B" covers three scenarios: Scenario 1: including A but not B; Scenario 2: including B but not A; Scenario 3: including both A and B. The character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0097] The term "instruction" in this application can be either a direct instruction (or explicit instruction) or an indirect instruction (or implicit instruction). A direct instruction can be understood as one in which the sender explicitly informs the receiver of specific information, the operation to be performed, or the requested result, etc.; an indirect instruction can be understood as one in which the receiver determines the corresponding information based on the instruction sent by the sender, or makes a judgment and determines the operation to be performed or the requested result, etc., based on the judgment result.

[0098] To address the issue of client memory overflow easily triggered when uploading large files in chunks in existing technologies, embodiments of the present invention provide a file sending method, a file receiving method, an apparatus, a device, and a storage medium.

[0099] like Figure 1 As shown, this embodiment of the invention provides a method for speech-to-text conversion, the method comprising:

[0100] Step 101: The acoustic features of the speech frame data to be converted and the semantic context of the speech frame data are fused to obtain the fused features of different input channels.

[0101] In this step, feature extraction processing is performed on the speech frame data to obtain the acoustic features of the speech frame data.

[0102] In addition, in this step, the semantic context of the speech frame data is predicted.

[0103] The acoustic features include acoustic features from different input channels. The acoustic features from different input channels are fused with the semantic context to obtain the fused features of the corresponding input channels.

[0104] The fusion features obtained in this step combine acoustic features with the language model, improving the accuracy and real-time performance of subsequent text conversion based on these fusion features.

[0105] Step 102: Perform dimensionality reduction processing on the fusion features of key input channels among the fusion features of the different input channels to obtain the first feature.

[0106] In this step, one input channel corresponds to one attention weight. Based on the attention weight, key input channels and non-key input channels in different input channels are determined. The fusion feature of the key input channel is then subjected to dimensionality reduction processing to obtain the first feature.

[0107] Using the first feature obtained from the dimensionality reduction process in this step for text conversion can improve the transmission effect during the text conversion process.

[0108] Step 103: Obtain prosodic frame data from the speech frame data based on the attention weight of the input channel, the speech frame data, and the first feature.

[0109] Among them, prosodic frame data represents the prosodic information in speech frame data.

[0110] The prosodic frame data includes accent frame data and pause frame data.

[0111] This step, which performs text conversion based on the prosodic frame data, can reduce the loss of speech information in the speech frame data and further improve the accuracy of the conversion.

[0112] Step 104: Based on the prosodic frame data and the speech frame data, convert the speech frame data into target text.

[0113] In this step, based on the above steps, the speech frame data is converted based on the prosodic frame data and the speech frame data to obtain the target text. This can improve the accuracy and real-time performance of speech-to-text conversion, and also improve the contextual coherence of the converted text.

[0114] In some embodiments, the acoustic features of the speech frame data to be converted and the semantic context of the speech frame data are fused to obtain fused features from different input channels, including:

[0115] Obtain the audio frame data to be converted.

[0116] Among them, speech frame data is frame data (or frame sequence) that is obtained by processing the original speech stream and arranged in chronological order.

[0117] Feature extraction processing is performed on the speech frame data to obtain acoustic features of different input channels. Specifically, convolutional neural networks (CNNs) and time-delay neural networks (TDNNs) are used to perform feature extraction processing on the speech frame data to obtain the acoustic features of the speech frame data.

[0118] Predict the semantic context of the speech frame data.

[0119] Specifically, the semantic context of the speech frame data is predicted using a language model. In particular, the semantic context of the speech frame data is predicted using a Recurrent Neural Network-Transducer (RNN-T).

[0120] Specifically, the prediction of the semantic context can be performed in the decoder module, which is based on an RNN-T architecture and implements joint optimization of the acoustic-language model, and its architecture includes a prediction network.

[0121] The prediction network uses a 2-layer LSTM (256-dimensional hidden layers) and takes a sequence of historical prediction words as input. (i.e., word sequences in the speech frame data), the output vocabulary context vector is By limiting the decoding space using a finite state transition machine (FST), only word maps that match the context vectors with the current acoustic features are retained, which constitute the semantic context of the speech frame data.

[0122] The output formula for the i-th LSTM layer is as follows:

[0123]

[0124] in, Indicates the prediction term from the previous moment. This represents the previous hidden state. By modeling word order patterns, unreasonable outputs are suppressed, improving semantic rationality.

[0125] The acoustic features of different input channels and the semantic context are fused to obtain the fused features of the response input channels.

[0126] Specifically, the encoder module uses RNN-T to fuse acoustic features and semantic context of speech frame data to obtain fused features from different input channels.

[0127] The acoustic features include acoustic features from different input channels. The acoustic features from different input channels and the semantic context are fused using RNN-T to obtain the fused features of the corresponding input channels.

[0128] Optionally, obtaining the voice frame data to be converted includes:

[0129] The speech to be converted is segmented into frames to obtain initial frame data.

[0130] The speech to be converted can also be called the original speech stream or the speech input stream.

[0131] To acquire the speech to be converted, and to achieve real-time processing, the speech to be converted is divided into frames to obtain initial frame data. For example, the frame division parameters are 20ms per frame (corresponding to 320 sampling points) and 10ms frame shift (50% overlap) to ensure the short-term stability of the speech signal.

[0132] The initial frame data is windowed to obtain the speech frame data.

[0133] Specifically, a Hamming window is applied to the initial frame data of each frame to suppress spectral leakage. The Hamming window function is as follows:

[0134]

[0135] in, Indicates the Hamming window in the first The weighting coefficients for each sampling point This is the index of the sampling points within a single frame of audio data, with a value of 319 (each frame has 320 sampling points, with the index starting from 0).

[0136] Furthermore, after obtaining the voice frame data, hardware-level optimization is performed, and the audio hardware buffer is directly accessed through memory mapping to achieve zero-copy framing, with single-frame processing latency controlled within 2ms.

[0137] For example, the speech to be converted is segmented into a sequence of speech frame data in chronological order. The audio frame data of frame t is as follows: ,in Let t be the t-th sampling point, where t represents the frame index.

[0138] Optionally, the step of performing feature extraction processing on the speech frame data to obtain acoustic features of different input channels includes:

[0139] The speech frame data is converted into frequency domain features, wherein the frequency domain features include frequency domain features of different input channels.

[0140] Specifically, audio is framed using hierarchical residual blocks to obtain speech frame data. The time-domain signal of the speech frame data is then converted into multi-scale frequency-domain features. Specifically, a Res2Net model is used to perform hierarchical convolutions on each subset (speech frame data) to capture different frequency band features. The multi-scale feature vector (frequency-domain feature) of each speech frame data is represented as follows: Where T represents the number of frames in the speech frame data. This represents the feature vector (frequency domain feature) of the T-th frame.

[0141] By utilizing the channel self-attention mechanism, the frequency domain features of each input channel are processed separately to obtain the second feature of the corresponding input channel.

[0142] Specifically, through the channel self-attention mechanism, global average pooling is used to compress the frequency domain features of each input channel into a global feature, namely the second feature.

[0143] Using a temporal attention mechanism, a third feature of the corresponding input channel is obtained based on the frequency domain features of each input channel, the number of frames of the speech frame data, and the dimension of the input channel.

[0144] The third feature is a time-weighted feature.

[0145] The second and third features are compressed and reduced in dimensionality to obtain the fourth feature.

[0146] Specifically, the second and third features are reduced in dimensionality using n 1×1 convolution kernels to compress and obtain the fourth feature.

[0147] The fourth feature is subjected to time-preservation processing to obtain the fifth feature.

[0148] Specifically, to compensate for the lack of temporal modeling in the Res2Net model and capture long-term dependencies between speech frames, a lightweight temporal modeling temporal convolutional network (TCN) is introduced. This network expands the receptive field through an exponential dilation rate, captures context at different time scales, and outputs features that preserve temporal information, namely the fifth feature.

[0149] The fifth feature is subjected to frequency domain feature extraction, and the fifth feature is subjected to time domain feature extraction to obtain the acoustic feature.

[0150] Specifically, the frequency domain features and time domain features extracted from the fifth feature are fused to obtain the acoustic features.

[0151] As an optional approach, the method of utilizing channel self-attention mechanism to process the frequency domain features of each input channel separately to obtain the second feature of the corresponding input channel includes:

[0152] Using a channel self-attention mechanism, the frequency domain features of each input channel are compressed according to the number of frames in the speech frame data to obtain the first target feature.

[0153] Specifically, the channel attention mechanism first compresses the spatiotemporal information (i.e., frequency domain features) of each channel into a global feature vector (i.e., the first target feature) through global average pooling. ,in This represents the feature value (frequency domain feature) of the c-th channel, t-th frame, and f-th frequency band. F represents the number of frequency bands of the frequency domain feature, and T represents the number of frames of the speech frame data.

[0154] The first feature of the corresponding input channel is obtained based on the attention weight of the input channel and the first target feature.

[0155] The dependencies between channels are learned through a two-layer fully connected multilayer perceptron (MLP) network, resulting in the attention weights S for the input channels. c Finally, the channel attention weights are multiplied by the original features (i.e., the first target features) to obtain the first feature, denoted as: Highlight key channels.

[0156] As an optional approach, the use of a temporal attention mechanism to obtain a third feature for each input channel based on the frequency domain features of each input channel, the number of frames in the speech frame data, and the dimension of the input channel includes:

[0157] By utilizing a temporal attention mechanism, the second target feature of each input channel is obtained based on the frequency domain features of each input channel and the dimension of the input channel.

[0158] Specifically, by utilizing the temporal attention mechanism, three-dimensional features (i.e., frequency domain features) are reshaped into two-dimensional features, and the dimensions of frequency domain features and input channels are merged into a new feature dimension, namely the second target feature, which facilitates temporal modeling.

[0159] The third feature of the corresponding input channel is obtained based on the second target feature of each input channel and the number of frames of the speech frame data.

[0160] Specifically, for each input channel, the second target feature is used to generate a query vector (Query, Q), a key vector (Key, K), and a value vector (Value, V). The attention score matrix A is then calculated using the following formula: .

[0161] Where T represents the number of frames in the speech frame data and D represents the dimension of the query vector.

[0162] Next, time-weighted features, i.e., the second features, are generated based on the attention score matrix A and the value vector V, and are expressed as follows: .

[0163] As an optional method, the compression and dimensionality reduction processing of the second feature and the third feature to obtain the fourth feature includes:

[0164] The second feature and the third feature are fused based on the number of frames in the speech frame data, the number of input channels, and the number of frequency bands of the frequency domain feature to obtain the third target feature.

[0165] Specifically, the channel attention-weighted features (secondary features) Features weighted by temporal attention (third feature) splicing and merging into dimensions The third target feature. Where C represents the number of input channels, T represents the number of frames of speech frame data, and F represents the number of frequency bands of frequency domain features.

[0166] The third target feature is compressed and reduced in dimensionality according to a preset number of dimensions to obtain the third feature.

[0167] The preset number of dimensions can be referred to as the target dimension.

[0168] Specifically, the number of feature channels of the fused fourth target feature is reduced to the target dimension using n 1×1 convolutional kernels according to the following formula:

[0169]

[0170] in, Input the number of channels. The target dimension, i.e., the preset number of dimensions. The connection weights between the input k-th channel and the output c-th channel are... The input feature (i.e., the fourth target feature) contains the original feature values ​​of the k-th channel, t-th time frame, and f-th frequency band. The bias term for the c-th channel is used to adjust the numerical distribution of the output features.

[0171] Optionally, the step of extracting frequency domain features from the fifth feature and extracting time domain features from the fifth feature to obtain the acoustic feature includes:

[0172] Based on the fifth feature, construct the LFMMI objective function for maximum mutual information of the closed grid.

[0173] Specifically, the processed fifth feature is mapped to the phone posterior probability through a convolutional layer to construct a posterior probability matrix. This matrix is ​​then compiled using Kaldi's fstcompile tool, converting it into a denominator finite element state transformer (FST). To ensure compatibility of path probability calculations within blocks with the complete sentence, an intermediate transformer (i.e., a normalized FST) is created to normalize the initial and final state probabilities. Then, a numerator finite state transformer (numerator FST) is constructed for each utterance, thus restricting the model to focus only on paths matching real speech. Finally, the LFMMI objective function is constructed, as shown in the following formula:

[0174]

[0175] in, These are the network parameters of the CNN-TDNN encoder. Indicates the first One voice sample, The total number of samples, It is the first Speech observation features of a sample (encoder input, i.e., the fifth feature). Let be the real text sequence of the r-th sample (labeled answer), and W represent all possible text sequences (candidate paths).

[0176] Based on the LFMMI objective function, frequency domain features are extracted from the fifth feature, and time domain features are extracted from the fifth feature to obtain the acoustic feature.

[0177] Specifically, according to the LFMMI objective function, the frequency domain feature extraction of the fifth feature is performed using a CNN.

[0178] Based on the LFMMI objective function, temporal feature extraction is performed on the fifth feature to obtain the acoustic feature, which is obtained by using TDNN to extract the temporal feature of the fifth feature.

[0179] Specifically, the front-end CNN layer uses 2D convolution to capture frequency domain features. The input is the dimensionality-reduced time-frequency features (i.e., the fifth feature). Local frequency domain patterns are extracted through 3x3 convolutional kernels, and the output is activated by a Rectified Linear Unit (ReLU). The number of channels in the convolutional blocks is gradually increased, and the feature map size is reduced through pooling to preserve multi-scale frequency domain information. The back-end TDNN uses multi-branch dilated convolutions (dilation=1, 3, 5) to capture cross-frame temporal dependencies. The TDNN-F module uses TDNN combined with linear and affine layers to process current frame and cross-frame features respectively, outputting the phone posterior probability. Furthermore, the gradient of the probability difference between the numerator and denominator paths is calculated to update the CNN-TDNN parameters. The constructed LFMMI loss function is minimized for backpropagation, forcing the network's output feature vector to have stronger class discriminative power.

[0180] The following details the process of generating fused features:

[0181] The fusion feature generation process is performed through an encoder module, the architecture of which is as follows: Figure 2 As shown.

[0182] The encoder module adopts a CNN-TDNN hybrid architecture, combining the temporal local feature extraction capability of convolutional neural networks (CNN) with the long-term context modeling advantage of temporal delay neural networks (TDNN).

[0183] The system incorporates a parallel dual-channel mechanism module: a channel attention mechanism and a temporal attention mechanism, which operate on the frequency and temporal dimensions of features, respectively, resulting in comprehensive optimization of multi-scale features. The channel attention module identifies and enhances frequency bands relevant to phoneme classification (e.g., high-frequency unvoiced sounds, low-frequency voiced sounds) while suppressing noise channels. The temporal attention module captures long-term temporal dependencies and strengthens temporal structures in speech (e.g., word boundaries, prosodic features).

[0184] The ASR audio module takes input speech frame data; the Res2Net model converts the speech frame data into frequency domain features; a channel self-attention mechanism processes the frequency domain features of each input channel separately to obtain the second feature of the corresponding input channel; a temporal attention mechanism, based on the frequency domain features of each input channel, the number of frames in the speech frame data, and the dimension of the input channel, obtains the third feature of the corresponding input channel; feature reduction is performed on the second and third features to obtain the fourth feature; TCN temporal modeling captures the context at different time scales and outputs features that retain temporal information, i.e., the fifth feature (time-frequency feature); the Phone phoneme model maps the fifth feature to a phone number through a convolutional layer. The posterior probability matrix is ​​constructed using the posterior probability matrix, and then compiled using the open-source tool Kaldi's fstcompile to convert it into a denominator FST. To ensure compatibility between the path probability calculation within the block and the complete sentence, an intermediate transformer (i.e., normalized FST) is created to normalize the initial and final state probabilities. Then, a numerator FST is constructed for each utterance, and an LFMMI objective function is built. The front-end CNN layer of the CNN-TDNN module uses 2D convolutions to capture frequency domain features. The input is the reduced-dimensional time-frequency features (i.e., the fifth feature), and local frequency domain patterns are extracted using 3x3 convolution kernels. The output is activated by a Rectified Linear Unit (ReLU). The number of channels in the convolutional block is gradually increased, and pooling is used to reduce the feature map size, preserving multi-scale frequency domain information. The backend TDNN uses multi-branch dilated convolution (dilation=1, 3, 5) to capture cross-frame temporal dependencies. The TDNN-F module uses TDNN combined with linear and affine layers to process the features of the current frame and cross-frames respectively, outputs the phone posterior probability, calculates the gradient of the probability difference between the numerator and denominator paths, updates the CNN-TDNN parameters, minimizes the constructed LFMMI loss function for backpropagation, forces the feature vector output by the network to have stronger class discriminativeness, obtains fused features, and finally outputs the fused features by the encoder module.

[0185] In some embodiments, the fusion processing of the acoustic features of different input channels and the semantic context to obtain the fused features of the response input channels includes:

[0186] The acoustic features are linearly projected onto the target dimension to obtain the fourth target feature, and the semantic context is linearly projected onto the target dimension to obtain the target context.

[0187] Specifically, the encoder module also includes a joint network that integrates acoustic features and semantic context. Specifically, the joint network projects the high-dimensional acoustic features and semantic context to the target dimension through linear projection, thereby obtaining the fourth target feature and target context.

[0188] Optionally, the target dimension is 512.

[0189] The fusion feature is obtained based on the fourth target feature and the target context.

[0190] Specifically, the phoneme-level probability distribution, i.e., the fusion feature, is generated by adding the fourth target feature and the target context.

[0191] In summary, the above embodiments can be implemented using a lightweight front-end speech-to-text model. In this model, the speech signal input module receives the original speech stream, divides it into frames with a 20ms frame length and a 10ms frame shift, applies a Hamming window, and achieves zero-copy framing through memory mapping, with a single-frame processing latency of ≤2ms. The encoder module adopts a CNN-TDNN hybrid architecture, optimizes acoustic features through channel / temporal attention, maps temporal features to phone posterior probabilities, constructs an LF-MMI objective function, and updates parameters through backpropagation. The decoder module is based on an RNN-T architecture, with the prediction network outputting lexical context vectors through a two-layer LSTM, and the joint network fusing acoustic features and context to generate phoneme-level probability distributions.

[0192] In some embodiments, the dimensionality reduction processing of the fusion features of key input channels in the fusion features of the different input channels to obtain the first feature includes:

[0193] Obtain the attention weights of the input channels;

[0194] The attention weight of this input channel is the attention weight S of the input channel generated by the encoder module mentioned above. c .

[0195] Based on the attention weights, obtain the fusion features corresponding to key input channels and non-key input channels in the input channels.

[0196] It is understandable that the input channels include key input channels and non-key input channels, and attention weights are used to distinguish between key and non-key input channels.

[0197] To ensure that the amount of data can be reduced while avoiding the loss of key information, an attention-driven mechanism is introduced to retain the accuracy of high-weight channels and reduce the amount of data by quantizing low-weight channels.

[0198] The fusion features corresponding to the non-critical input channels are rounded down to obtain the sixth feature.

[0199] Specifically, the fusion feature X_critical of the critical input channel is kept in the original FP16 format, while the fusion feature X_non_critical of the non-critical input channel is converted to uint8 format, thus obtaining the sixth feature.

[0200] Based on the attention score matrix, obtain the first feature corresponding to the weakly dependent interval in the fusion features corresponding to the key input channel.

[0201] The generation process of the attention score matrix is ​​as follows: Using a temporal attention mechanism, the second target feature of each input channel is obtained based on its frequency domain features and dimension; the second target feature of each input channel is used to generate a query vector (Query, Q), a key vector (Key, K), and a value vector (Value, V); and the attention score matrix A is calculated according to the following formula:

[0202]

[0203] Where T represents the number of frames in the speech frame data. Indicates the dimension of the query vector.

[0204] As an optional approach, obtaining the fusion features corresponding to the key input channels in the input channels based on the attention weights includes:

[0205] Based on the normal distribution information of the attention weights, the key input channels in the input channels are determined.

[0206] Specifically, the attention weight represents the importance of the c-th channel, with a value range of [0,1]. This attention weight is generated by global average pooling and an MLP network.

[0207] Determine the input channel threshold, and statistically analyze the attention weights (e.g., 1000 attention weights). The results approximate a log-normal distribution, expressed as: Next, calculate the weights and proportions of the first k channels using the following formula:

[0208]

[0209] When τ=0.7, corresponding to k≈300 channels, the cumulative contribution reaches 87.2%, so the threshold is determined to be 0.7.

[0210] That is, the input channels corresponding to attention weights with a threshold exceeding 0.7 in the normal distribution are identified as key input channels.

[0211] Obtain the fusion features corresponding to the key input channels and the fusion features corresponding to the non-key input channels in the input channels.

[0212] Based on a feature compression strategy that weights information importance, channel features are classified according to a threshold. If the value is greater than the threshold, it is determined to be a key input channel; otherwise, it is a non-key input channel. The fusion features corresponding to key input channels and non-key input channels are then obtained.

[0213] Optionally, obtaining the fusion feature corresponding to the key input channel and the first feature corresponding to the weak dependency interval in the sixth feature based on the attention score matrix includes:

[0214] Based on the attention score matrix, obtain the inter-frame dependency score within the preset division interval.

[0215] Specifically, based on the attention score matrix A, the inter-frame dependency score A' is calculated, where A'[i,j] represents the degree of closeness between two frames, and A'[i,j]=A[i,j].

[0216] Based on the inter-frame dependency score, obtain the fusion features corresponding to the key input channels and the weak dependency intervals in the sixth feature.

[0217] Specifically, the fusion features corresponding to the key input channels and the sixth feature are divided into time intervals, with each interval divided into 50 frames. Then, based on the granularity of the time interval (50 frames), the inter-frame dependency scores within each interval are averaged using the following formula. Based on the average value, strong dependency intervals and weak dependency intervals are divided:

[0218]

[0219] For example, greater than or equal to this average value The interval is considered a strongly dependent interval, and will be less than this average value. The interval is taken as the strongly dependent interval.

[0220] Obtain the fifth target feature corresponding to the weak dependency interval.

[0221] Specifically, for each weakly dependent interval r_i, the fusion feature corresponding to the key input channel and the feature matrix corresponding to the sixth feature (i.e., the fifth target matrix) are extracted. ∈ R^{L×D}, where L=50 is the number of frames in the interval and D=512 is the original feature dimension.

[0222] The fifth target feature is then decentralized to obtain the sixth target feature.

[0223] Specifically, the mean of the fifth target feature is calculated according to the following formula:

[0224]

[0225] Next, to eliminate the overall shift in the feature matrix and focus on the relative differences between features, the fifth target feature is then centered to obtain the sixth target feature, as shown in the following formula:

[0226]

[0227] Based on the sixth target feature, a covariance matrix is ​​constructed to obtain the seventh target feature, and the specific formula is as follows:

[0228]

[0229] The principal component matrix of the seventh target feature is subjected to dimensionality reduction processing to obtain the first feature.

[0230] Specifically, the seventh target feature is decomposed into eigenvalues ​​to obtain eigenvalues, which are then arranged in descending order to obtain the eigenvalue sequence. The seventh target feature corresponding to the feature value is Principal components are selected based on the seventh target feature corresponding to the eigenvalues, and the cumulative variance contribution is calculated using the following formula:

[0231]

[0232] Where K=256.

[0233] Based on the screening results, a principal component matrix is ​​then constructed. .

[0234] The first feature is obtained based on the principal component matrix of the seventh target feature and the fifth target feature matrix.

[0235] Specifically, based on the following formula, the direction of maximum variance is captured by a linear combination of the sixth target features corresponding to each principal component matrix in the 256-dimensional principal component space, which is a 512-dimensional feature transformation of the weakly dependent intervals:

[0236]

[0237] The 256-dimensional features after dimensionality reduction are reconstructed to the original dimensions according to the following reconstruction formula to obtain the first feature. The similarity between the reconstructed feature and the original feature is compared to determine whether the dimensionality reduction has preserved the key information.

[0238]

[0239] in, Indicates the direction of maximum variance. Represents the principal component matrix. This represents the feature matrix of the fifth target. This indicates the first characteristic.

[0240] Optionally, whether the dimensionality reduction preserves key information can be determined by the error, which is calculated as follows:

[0241]

[0242] Where L represents the number of frames in the interval, and D represents the number of original dimensions. This represents the original dimension (i.e., the fifth target feature). To reconstruct the dimension (i.e., the first feature).

[0243] In some embodiments, the prosodic frame data includes accent frame data and pause frame data;

[0244] Specifically, prosodic frame data in the speech frame data is obtained based on the attention weights of the input channel, the speech frame data, and the first feature, including:

[0245] Based on the first feature and the attention weight, the accented frame data in the speech frame data is obtained.

[0246] It should be noted that this attention weight is the attention weight S of the input channel generated by the encoder module mentioned above. c .

[0247] Specifically, the attention weights of the first feature corresponding to each frame of speech data are summed:

[0248]

[0249] Where T represents the number of frames in the speech frame data.

[0250] And calculate the mean of the above sums. and standard deviation And determine the accented frame data; if the speech frame data satisfies Then it is the accent frame data.

[0251] Based on the energy difference between two adjacent audio frames in the audio frame data, pause frame data in the audio frame data is obtained.

[0252] The energy difference between two adjacent speech frames is represented as follows:

[0253]

[0254] Optionally, obtaining pause frame data in the speech frame data based on the energy difference between two adjacent speech frame data includes:

[0255] Obtain the frame energy of each voice frame in the voice frame data.

[0256] Specifically, the energy of the speech frame data is calculated using the sum of squared amplitudes of the short-time Fourier transform, as shown in the following formula:

[0257]

[0258] If the frame energy corresponding to the first voice frame data, the second voice frame data, and the third voice frame data is less than the preset energy value, the first voice frame data, the second voice frame data, and the third voice frame data are used as the pause frame data.

[0259] The first voice frame data, the second voice frame data, and the third voice frame data are three consecutive voice frame data in the voice frame data.

[0260] For example, the preset energy value is 0.1.

[0261] Right now If the value is less than 0.1 and lasts for three consecutive frames, it is determined to be an intra-sentence / inter-sentence pause, meaning that the three frames of speech data are pause frame data.

[0262] Finally, it is encapsulated into Protocol Buffers structured data, where the tag types are divided into frame start time, whether there is an accent, and pause type.

[0263] In some embodiments, converting the speech frame data into target text based on the prosodic frame data and the speech frame data includes:

[0264] According to the preset sliding time window, the prosody frame data and the speech frame data within the preset sliding time window are converted into window text.

[0265] The text corresponding to the preset sliding time window is concatenated and de-overlapped to obtain the target text. The preset sliding time window has a length of 30 seconds.

[0266] Specifically, a 30-second sliding time window is defined. For the utterance converted from prosodic frame data and speech frame data, a time range label containing its start and end times is generated, i.e., time_range = [t_start, t_end]. When a new utterance enters the sliding time window, if it overlaps with the time of a previous utterance within the window (e.g., t_new_start < t_old_end), the latest transcription result is retained after sorting by timestamp, and the old duplicate parts are discarded, i.e., de-overlap processing.

[0267] Concatenate all utterance text within the window in chronological order to form a long context text (i.e., the target text): C = concat (u_1.text, u_2.text, ..., u_N.text), and simultaneously record the corresponding time stamp array T = [(u_1.t_start, u_1.t_end), ..., (u_N.t_start, u_N.t_end)]. This ensures that context information is included.

[0268] In some embodiments, the method further includes:

[0269] Obtain the dynamic weight of the target text.

[0270] Specifically, a joint modeling approach based on confidence score and semantic entropy is constructed, and the dynamic weight generation function for the target text is as follows:

[0271]

[0272] in, This represents the Sigmoid function. =0.65, =0.35, normalization coefficient =0.25, The word-level confidence matrix (i.e., the confidence of the target text) output by the front-end small model. This represents the semantic entropy (i.e., the semantic entropy of the target text) calculated by the backend large model.

[0273] Optionally, a conflict detection mechanism can be introduced secondly, when And the confidence level of the large model is corrected. When the value is greater than 0.9, expert rule arbitration is triggered.

[0274] Next, an incremental dynamic time warping algorithm is used to align the cross-model output sequences.

[0275] Optionally, if the dynamic weight is greater than the first dynamic weight, the target text is taken as the final text of the speech frame data;

[0276] The first dynamic weight is 0.75.

[0277] That is, when the dynamic weight is greater than 0.75, the target text is directly used as the final text output.

[0278] When the dynamic weight is less than the second dynamic weight, the target text is semantically corrected according to the acoustic features and the target text to obtain the corrected text, and the corrected text is used as the final text of the speech frame data.

[0279] The first dynamic weight is greater than the second dynamic weight. The second dynamic weight is 0.4.

[0280] Specifically, the WordPiece segmenter of the Bidirectional Encoder Representations from Transformers (BERT) model converts the target text into a sequence of token IDs. The acoustic features are concatenated with the BERT token embeddings (i.e., the token ID sequence) and input into the cross-modal attention layer. For tokens with low confidence (score < 0.6) at the front end, a weight term is added to the BERT loss function, as shown below:

[0281]

[0282] in, Represents the total number of samples. Indicates the confidence level. Represents cross-entropy loss, The one-hot encoded vector representing the real label. This represents the probability distribution predicted by the model.

[0283] Then, a variable-length context cache (text from the last 10 seconds) is maintained. The semantic similarity between the current sentence and the context is calculated using the [CLS] vector of BERT. If the similarity is less than the threshold, the cache is reset.

[0284] The distributed inference engine concatenates the acoustic feature vector with BERT's token embedding, inputs it into the cross-modal attention layer, and uses a confidence-weighted loss function and dynamic context caching to perform semantic correction for speech perception, thus obtaining the corrected text.

[0285] When the dynamic weight is lower than the second dynamic weight, the correction of the large model is fully adopted, and the above-mentioned corrected text is used as the final text of the speech frame data.

[0286] When the dynamic weight is less than or equal to the first dynamic weight and the dynamic weight is greater than or equal to the second dynamic weight, the target text is semantically corrected according to the acoustic features and the target text to obtain the corrected text. The final text of the speech frame data is obtained according to the corrected text and the target text.

[0287] Specifically, if the dynamic weight is between 0.4 and 0.75, a linear interpolation fusion strategy is adopted to obtain the final text of the speech frame data based on the corrected text and the target text. The specific formula is as follows:

[0288]

[0289] in, Indicates the final text, Represents the target text. This indicates a correction to the text.

[0290] Finally, the punctuation layout was optimized based on the improved Knuth-Plass line break algorithm, and the "aesthetics" of the layout were minimized through dynamic programming.

[0291] The gated fusion module generates dynamic weights based on joint modeling of confidence and semantic entropy, and combines incremental time warping algorithm and improved Knuth-Plass punctuation optimization to generate the final text.

[0292] The following is combined with Figure 3 The following describes the speech-to-text method provided in the embodiments of the present invention:

[0293] The speech-to-text method provided in this invention is applied to speech-to-text systems, such as... Figure 3 As shown, the speech-to-text system includes: a lightweight front-end speech-to-text model, a bidirectional channel transmission module, and a distributed inference engine for the back-end semantic optimization module. The lightweight front-end speech-to-text model includes a speech signal input module, an encoder module, and a decoder module; the distributed inference engine for the back-end semantic optimization module includes a distributed inference engine and a gating fusion module.

[0294] The specific functions of each part are as follows:

[0295] In the lightweight speech-to-text model at the front end, the speech signal input module receives the raw speech stream, divides it into frames with a 20ms frame length and a 10ms frame shift, applies a Hamming window, and achieves zero-copy framing through memory mapping, with a single frame processing latency of ≤2ms. The encoder module adopts a CNN-TDNN hybrid architecture, optimizes acoustic features through channel / temporal attention, maps temporal features to phone posterior probabilities, constructs an LF-MMI objective function, and updates parameters through backpropagation. The decoder module is based on an RNN-T architecture, with the prediction network outputting lexical context vectors through a 2-layer LSTM, and the joint network fuses acoustic features and context to generate phoneme-level probability distributions.

[0296] The distributed inference engine of the backend semantic optimization module concatenates acoustic features with BERT's token embeddings, inputs them into the cross-modal attention layer, applies a weighted loss to low-confidence tokens, maintains a 10-second context cache, and resets it through semantic similarity. The gated fusion module generates dynamic weights based on confidence-semantic entropy, aligns the output sequence using an incremental DTW algorithm, fuses the text in three levels according to confidence, and optimizes punctuation layout using an improved Knuth-Plass algorithm.

[0297] In summary, this invention proposes a speech-to-text method aimed at solving the problems of redundant interjections, unreasonable punctuation, and poor contextual coherence caused by the insufficient capabilities of small models in traditional speech-to-text models. This application achieves collaborative processing of real-time speech transcription and semantic optimization by constructing a cascaded hybrid model architecture that deeply integrates a lightweight speech-to-text model (small model) with a high-performance language optimization model (large model).

[0298] The overall solution achieves deep collaboration between the lightweight front-end model and the high-performance back-end model through a cascaded hybrid model architecture, effectively solving problems such as information loss, error accumulation, insufficient real-time performance, and poor contextual coherence in speech-to-text technology.

[0299] In this application, the cascaded hybrid model architecture design includes a front-end lightweight speech-to-text model, which includes a speech signal input module to achieve zero-copy frame processing, a CNN-TDNN hybrid architecture encoder module to optimize acoustic features, and an RNN-T decoder module to fuse semantic context. This collaborative framework with the back-end semantic optimization module's distributed inference engine, combined with a bidirectional feature transmission channel and a dynamic gating fusion mechanism, enables joint optimization of acoustic features and language models, significantly improving accuracy, real-time performance, and readability.

[0300] The bidirectional feature transmission channel utilizes hierarchical feature compression and semantic annotation mechanisms, temporal attention-guided feature dimensionality reduction technology, and prosodic information transformation processes to construct a dynamic context window to maintain dialogue coherence, reduce information loss, and optimize transmission efficiency.

[0301] The dynamic gating fusion module generates dynamic weights based on joint modeling of confidence and semantic entropy, aligns the output sequence with the incremental dynamic time warping algorithm, and implements a three-level text fusion strategy: high confidence is directly output, low confidence adopts large model correction, and medium confidence is fused by linear interpolation to achieve adaptive text optimization.

[0302] By optimizing punctuation generation by minimizing typographical unappealingness through dynamic programming, compiling the phone posterior probability using a finite state converter, enhancing feature discriminativeness by updating parameters through loss function gradients, and designing a cross-modal attention layer, semantic coherence and output readability are improved.

[0303] Furthermore, this application constructs a bidirectional linkage architecture between a lightweight front-end speech-to-text model and a back-end semantic optimization module. This includes a lightweight front-end consisting of a speech signal input module, a CNN-TDNN encoder, and an RNN-T decoder, and a high-performance back-end consisting of a distributed inference engine and a gating fusion module. Real-time interaction between acoustic features and language model optimization data is achieved through a bidirectional feature transmission channel, forming an end-to-end processing closed loop. The speech signal input module uses a memory-mapped hardware buffer to implement zero-copy framing operations, segmenting the speech stream with a 20ms frame length and a 10ms frame shift, and applying a Hamming window function to suppress spectral leakage. The encoder module executes a channel attention mechanism in parallel to enhance key frequency band features, while simultaneously using a temporal attention matrix to capture long-term inter-frame dependencies. After dimensionality reduction via 1×1 convolution, optimized acoustic features are output to the decoder module. Based on channel attention weights, key and non-key channels are dynamically distinguished. A statistical threshold is set to retain FP16 precision for key channels, while non-key channels are converted to uint8 format for storage. A temporal dependency matrix is ​​used to divide the strong and weak dependency intervals. Principal component analysis is applied to reduce dimensionality in the weak dependency intervals, and core acoustic feature information is preserved through centering and covariance decomposition. Attention weights and stressed frames are calculated for each frame, and the pause frame type is continuously determined based on short-time energy differences. Prosodic information is encapsulated into structured Protocol Buffers tag data, including frame start time and type identifier. A 30-second sliding time window is constructed to manage the utterance sequence, and the context cache is dynamically refreshed by calculating the similarity of BERT's [CLS] vector. A confidence-semantic entropy joint weight generation function is designed to control the text fusion ratio, and a three-level threshold strategy is implemented to distinguish output selection in high and low confidence scenarios. An incremental dynamic time warping algorithm is used to align the output sequences of multiple models, and an improved Knuth-Plass dynamic programming algorithm is combined to optimize punctuation layout and line break logic.

[0304] This application has significant technical advantages over existing technologies: Addressing the information loss problem in traditional cascaded speech processing systems, this application utilizes a bidirectional feature transmission channel to enable real-time interaction between the acoustic feature vectors of the front-end small model and the semantic embeddings of the back-end large model. This preserves the prosodic information of the speech and the deep semantics of the text in the feature dimension, thus avoiding information attenuation caused by pure text transmission. To address the error accumulation defects of cascaded architectures, a dynamic weight adjustment mechanism is adopted. Based on the word-level confidence matrix output by the front end, the optimization intensity of the back-end large model for different segments is dynamically adjusted. When a low-confidence region is detected, the entire layer of the large model is activated for in-depth correction, while for high-confidence regions... The confidence region operates only with shallow networks, reducing error propagation and controlling computational overhead. To address the lack of contextual coherence in general speech tools, a cross-modal attention mechanism is introduced, jointly paying attention to acoustic features and text semantics. This enables large models to perceive paralinguistic features such as stress and pauses in speech signals, thereby generating punctuation that conforms to dialogue logic more accurately. To address the limitations of semantic understanding in streaming Transducer architectures, an incremental dynamic time warping algorithm is used to achieve streaming alignment between the outputs of small and large models, completing acoustic-semantic feature matching within a 200ms time window, which reduces memory usage compared to the traditional DTW algorithm.

[0305] This invention proposes a speech-to-text optimization method based on a cascaded hybrid model architecture. The innovative "cascaded hybrid model architecture" achieves deep joint optimization of acoustic features and language models through the collaboration of a lightweight front-end speech-to-text model and a back-end distributed semantic optimization engine, combined with a bidirectional feature transmission channel and a dynamic gating fusion module. This significantly improves the accuracy, real-time performance, and readability of speech-to-text conversion, and effectively solves problems such as poor contextual coherence and high computational resource consumption.

[0306] This application provides a core solution to address the problems of severe information loss, cascading errors, excessive computational resource consumption, poor contextual coherence, and insufficient real-time performance in existing speech-to-text technologies. It designs a cascaded hybrid model architecture and its bidirectional feature transmission and dynamic gating fusion mechanism. In existing high-concurrency customer service scenarios, adopting this technology significantly improves the accuracy and coherence of voice work order generation, reduces manual review costs, greatly alleviates the workload of frontline work order personnel, and improves work efficiency. This solution can be provided externally via API interfaces or embedded SDKs, adapting to devices such as smart set-top boxes and smart home terminals, making it applicable to a wide range of scenarios.

[0307] like Figure 4 As shown, embodiments of the present invention also provide a speech-to-text device, the device comprising:

[0308] The first processing module 401 is used to fuse the acoustic features of the speech frame data to be converted and the semantic context of the speech frame data to obtain the fusion features of different input channels.

[0309] The second processing module 402 is used to perform dimensionality reduction processing on the fusion features of the different input channels to obtain the first feature;

[0310] The third processing module 403 is used to obtain prosodic frame data in the speech frame data based on the attention weight of the input channel, the speech frame data, and the first feature.

[0311] The fourth processing module 404 is used to convert the speech frame data into target text based on the prosody frame data and the speech frame data.

[0312] Optionally, the first processing module 401 includes:

[0313] The first acquisition unit is used to acquire the voice frame data to be converted;

[0314] The first processing unit is used to perform feature extraction processing on the speech frame data to obtain acoustic features of different input channels;

[0315] The second processing unit is used to predict the semantic context of the speech frame data;

[0316] The third processing unit is used to fuse the acoustic features of different input channels with the semantic context to obtain the fused features of the response input channels.

[0317] Optionally, the first acquisition unit is specifically used for:

[0318] The speech to be converted is segmented into frames to obtain initial frame data;

[0319] The initial frame data is windowed to obtain the speech frame data.

[0320] Optionally, the first processing unit is specifically used for:

[0321] The speech frame data is converted into frequency domain features, wherein the frequency domain features include frequency domain features of different input channels;

[0322] By utilizing the channel self-attention mechanism, the frequency domain features of each input channel are processed separately to obtain the second feature of the corresponding input channel;

[0323] Using a temporal attention mechanism, a third feature of the corresponding input channel is obtained based on the frequency domain features of each input channel, the number of frames of the speech frame data, and the dimension of the input channel.

[0324] The second and third features are compressed and reduced in dimensionality to obtain the fourth feature;

[0325] The fourth feature is subjected to time-preservation processing to obtain the fifth feature;

[0326] The fifth feature is subjected to frequency domain feature extraction, and the fifth feature is subjected to time domain feature extraction to obtain the acoustic feature.

[0327] Optionally, the first processing unit is specifically used for:

[0328] Using a channel self-attention mechanism, the frequency domain features of each input channel are compressed according to the number of frames in the speech frame data to obtain the first target features;

[0329] The first feature of the corresponding input channel is obtained based on the attention weight of the input channel and the first target feature.

[0330] Optionally, the first processing unit is specifically used for:

[0331] Using a temporal attention mechanism, the second target feature of each input channel is obtained based on the frequency domain features of each input channel and the dimension of the input channel;

[0332] The second feature of the corresponding input channel is obtained based on the second target feature of each input channel and the number of frames of the speech frame data.

[0333] Optionally, the first processing unit is specifically used for:

[0334] The second feature and the third feature are fused based on the number of frames in the speech frame data, the number of input channels, and the number of frequency bands of the frequency domain feature to obtain the third target feature;

[0335] The third target feature is compressed and reduced in dimensionality according to a preset number of dimensions to obtain the third feature.

[0336] Optionally, the first processing unit is specifically used for:

[0337] Based on the fifth feature, construct the LFMMI objective function for maximum mutual information of the closed grid;

[0338] Based on the LFMMI objective function, frequency domain features are extracted from the fifth feature, and time domain features are extracted from the fifth feature to obtain the acoustic feature.

[0339] Optionally, the third processing unit is specifically used for:

[0340] The acoustic features are linearly projected onto the target dimension to obtain the fourth target feature, and the semantic context is linearly projected onto the target dimension to obtain the target context;

[0341] The fusion feature is obtained based on the fourth target feature and the target context.

[0342] Optionally, the second processing module 402 includes:

[0343] The second acquisition unit is used to acquire the attention weights of the input channels;

[0344] The fourth processing unit is used to obtain the fusion features corresponding to key input channels and non-key input channels in the input channels according to the attention weights.

[0345] The fifth processing unit is used to round down the fusion features corresponding to the non-critical input channels to obtain the sixth feature;

[0346] The sixth processing unit is used to obtain the fusion feature corresponding to the key input channel and the first feature corresponding to the weak dependency interval in the sixth feature based on the attention score matrix.

[0347] Optionally, the fourth processing unit is specifically used for:

[0348] Based on the normal distribution information of the attention weights, determine the key input channels and non-key input channels in the input channels;

[0349] Obtain the fusion features corresponding to the key input channels and the fusion features corresponding to the non-key input channels in the input channels.

[0350] Optionally, the sixth processing unit is specifically used for:

[0351] Based on the attention score matrix, obtain the inter-frame dependency score within the preset division interval;

[0352] Based on the inter-frame dependency score, obtain the fusion features corresponding to the key input channels and the weak dependency intervals in the sixth feature;

[0353] Obtain the fifth target feature corresponding to the weak dependency interval;

[0354] The fifth target feature is decentralized to obtain the sixth target feature;

[0355] Based on the sixth target feature, a covariance matrix is ​​constructed to obtain the seventh target feature;

[0356] The first feature is obtained based on the principal component matrix of the seventh target feature and the fifth target feature matrix.

[0357] Optionally, the prosodic frame data includes accent frame data and pause frame data;

[0358] The third processing module 403 is used for:

[0359] The seventh processing unit is used to obtain accented frame data in the speech frame data based on the first feature and the attention weight;

[0360] The eighth processing unit is used to obtain pause frame data in the speech frame data based on the energy difference between two adjacent speech frame data in the speech frame data.

[0361] Optionally, the eighth processing unit is specifically used for:

[0362] Obtain the frame energy of each voice frame in the voice frame data;

[0363] If the frame energy corresponding to the first voice frame data, the second voice frame data, and the third voice frame data is less than the preset energy value, the first voice frame data, the second voice frame data, and the third voice frame data are used as the pause frame data.

[0364] The first voice frame data, the second voice frame data, and the third voice frame data are three consecutive voice frame data in the voice frame data.

[0365] Optionally, the fourth processing module 404 includes:

[0366] The ninth processing unit is used to convert the speech frame data of the preset sliding time window into window text according to the prosody frame data and the speech frame data within the preset sliding time window.

[0367] The tenth processing unit is used to perform splicing and de-overlapping processing on the window text corresponding to the preset sliding time window to obtain the target text.

[0368] Optionally, the method further includes:

[0369] The first acquisition module is used to acquire the dynamic weight of the target text;

[0370] The fifth processing module is used to use the target text as the final text of the speech frame data when the dynamic weight is greater than the first dynamic weight.

[0371] The sixth processing module is used to perform semantic correction on the target text based on the acoustic features and the target text when the dynamic weight is less than the second dynamic weight, to obtain the corrected text, and to use the corrected text as the final text of the speech frame data.

[0372] The seventh processing module is used to perform semantic correction on the target text based on the acoustic features and the target text when the dynamic weight is less than or equal to the first dynamic weight and the dynamic weight is greater than or equal to the second dynamic weight, to obtain the corrected text, and to obtain the final text of the speech frame data based on the corrected text and the target text.

[0373] Wherein, the first dynamic weight is greater than the second dynamic weight.

[0374] It should be noted that the speech-to-text device provided in the embodiments of the present invention is a device capable of performing the above-described speech-to-text method. Therefore, all embodiments of the above-described speech-to-text method are applicable to this device and can achieve the same or similar technical effects.

[0375] like Figure 5 As shown, this embodiment of the invention also provides a client device, including: a processor 501; and a memory 503 connected to the processor 501 via a bus interface 502. The memory 503 is used to store programs and data used by the processor 501 when performing operations, and the processor 501 calls and executes the programs and data stored in the memory 503.

[0376] The transceiver 504 is connected to the bus interface 502 and is used to receive and send data under the control of the processor 501. Specifically, the processor 501 is used to read the program in the memory 503 and to execute the following processes:

[0377] The acoustic features of the speech frame data to be converted and the semantic context of the speech frame data are fused to obtain the fused features of different input channels;

[0378] The dimensionality reduction of the fusion features of the different input channels is performed to obtain the first feature;

[0379] Based on the attention weights of the input channel, the speech frame data, and the first feature, prosodic frame data in the speech frame data is obtained;

[0380] Based on the prosody frame data and the speech frame data, the speech frame data is converted into target text.

[0381] Optionally, the processor 501 is specifically used for:

[0382] Obtain the audio frame data to be converted;

[0383] The speech frame data is subjected to feature extraction processing to obtain the acoustic features of different input channels;

[0384] Predict the semantic context of the speech frame data;

[0385] The acoustic features of different input channels and the semantic context are fused to obtain the fused features of the response input channels.

[0386] Optionally, the processor 501 is specifically used for:

[0387] The speech to be converted is segmented into frames to obtain initial frame data;

[0388] The initial frame data is windowed to obtain the speech frame data.

[0389] Optionally, the processor 501 is specifically used for:

[0390] The speech frame data is converted into frequency domain features, wherein the frequency domain features include frequency domain features of different input channels;

[0391] By utilizing the channel self-attention mechanism, the frequency domain features of each input channel are processed separately to obtain the second feature of the corresponding input channel;

[0392] Using a temporal attention mechanism, a third feature of the corresponding input channel is obtained based on the frequency domain features of each input channel, the number of frames of the speech frame data, and the dimension of the input channel.

[0393] The second and third features are compressed and reduced in dimensionality to obtain the fourth feature;

[0394] The fourth feature is subjected to time-preservation processing to obtain the fifth feature;

[0395] The fifth feature is subjected to frequency domain feature extraction, and the fifth feature is subjected to time domain feature extraction to obtain the acoustic feature.

[0396] Optionally, the processor 501 is specifically used for:

[0397] Using a channel self-attention mechanism, the frequency domain features of each input channel are compressed according to the number of frames in the speech frame data to obtain the first target features;

[0398] The first feature of the corresponding input channel is obtained based on the attention weight of the input channel and the first target feature.

[0399] Optionally, the processor 501 is specifically used for:

[0400] Using a temporal attention mechanism, the second target feature of each input channel is obtained based on the frequency domain features of each input channel and the dimension of the input channel;

[0401] The second feature of the corresponding input channel is obtained based on the second target feature of each input channel and the number of frames of the speech frame data.

[0402] Optionally, the processor 501 is specifically used for:

[0403] The second feature and the third feature are fused based on the number of frames in the speech frame data, the number of input channels, and the number of frequency bands of the frequency domain feature to obtain the third target feature;

[0404] The third target feature is compressed and reduced in dimensionality according to a preset number of dimensions to obtain the third feature.

[0405] Optionally, the processor 501 is specifically used for:

[0406] Based on the fifth feature, construct the LFMMI objective function for maximum mutual information of the closed grid;

[0407] Based on the LFMMI objective function, frequency domain features are extracted from the fifth feature, and time domain features are extracted from the fifth feature to obtain the acoustic feature.

[0408] Optionally, the processor 501 is specifically used for:

[0409] The acoustic features are linearly projected onto the target dimension to obtain the fourth target feature, and the semantic context is linearly projected onto the target dimension to obtain the target context;

[0410] The fusion feature is obtained based on the fourth target feature and the target context.

[0411] Optionally, the processor 501 is specifically used for:

[0412] Obtain the attention weights of the input channels;

[0413] Based on the attention weights, obtain the fusion features corresponding to key input channels and the fusion features corresponding to non-key input channels in the input channels;

[0414] The fusion features corresponding to the non-critical input channels are rounded down to obtain the sixth feature;

[0415] Based on the attention score matrix, obtain the fusion feature corresponding to the key input channel and the first feature corresponding to the weak dependency interval in the sixth feature.

[0416] Optionally, the processor 501 is specifically used for:

[0417] Based on the normal distribution information of the attention weights, determine the key input channels and non-key input channels in the input channels;

[0418] Obtain the fusion features corresponding to the key input channels and the fusion features corresponding to the non-key input channels in the input channels.

[0419] Optionally, the processor 501 is specifically used for:

[0420] Based on the attention score matrix, obtain the inter-frame dependency score within the preset division interval;

[0421] Based on the inter-frame dependency score, obtain the fusion features corresponding to the key input channels and the weak dependency intervals in the sixth feature;

[0422] Obtain the fifth target feature corresponding to the weak dependency interval;

[0423] The fifth target feature is decentralized to obtain the sixth target feature;

[0424] Based on the sixth target feature, a covariance matrix is ​​constructed to obtain the seventh target feature;

[0425] The first feature is obtained based on the principal component matrix of the seventh target feature and the fifth target feature matrix.

[0426] Optionally, the prosodic frame data includes accent frame data and pause frame data;

[0427] The processor 501 is specifically used for:

[0428] Based on the first feature and the attention weight, the accented frame data in the speech frame data is obtained;

[0429] Based on the energy difference between two adjacent audio frames in the audio frame data, pause frame data in the audio frame data is obtained.

[0430] Optionally, the processor 501 is specifically used for:

[0431] Obtain the frame energy of each voice frame in the voice frame data;

[0432] If the frame energy corresponding to the first voice frame data, the second voice frame data, and the third voice frame data is less than the preset energy value, the first voice frame data, the second voice frame data, and the third voice frame data are used as the pause frame data.

[0433] The first voice frame data, the second voice frame data, and the third voice frame data are three consecutive voice frame data in the voice frame data.

[0434] Optionally, the processor 501 is specifically used for:

[0435] According to the preset sliding time window, the prosody frame data and the speech frame data within the preset sliding time window are converted into window text.

[0436] The target text is obtained by splicing and de-overlapping the text corresponding to the preset sliding time window.

[0437] Optionally, the processor 501 is further configured to:

[0438] Obtain the dynamic weight of the target text;

[0439] If the dynamic weight is greater than the first dynamic weight, the target text is taken as the final text of the speech frame data;

[0440] When the dynamic weight is less than the second dynamic weight, the target text is semantically corrected according to the acoustic features and the target text to obtain the corrected text, and the corrected text is used as the final text of the speech frame data.

[0441] When the dynamic weight is less than or equal to the first dynamic weight and the dynamic weight is greater than or equal to the second dynamic weight, the target text is semantically corrected according to the acoustic features and the target text to obtain the corrected text. The final text of the speech frame data is obtained according to the corrected text and the target text.

[0442] Wherein, the first dynamic weight is greater than the second dynamic weight.

[0443] Among them, Figure 5In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 501) and memory (memory 503). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides a user interface 505. A transceiver 504 may be multiple elements, including transmitters and receivers, providing units for communicating with various other devices over a transmission medium. Processor 501 is responsible for managing the bus architecture and general processing, and memory 503 may store data used by processor 501 during operation.

[0444] In addition, specific embodiments of the present invention also provide a readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, it implements the steps of the speech-to-text method as described above or the steps of the file receiving method as described above.

[0445] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0446] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.

[0447] The integrated unit implemented as a software functional unit described above can be stored in a computer-readable storage medium. This software functional unit, stored in a storage medium, includes several instructions that cause a computer device (which may be a personal computer, server, or network device, etc.) to execute partial steps of the resource selection method described in the various embodiments of the present invention, or to execute partial steps of the information transmission method described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0448] A specific embodiment of the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described functionality. Figure 1 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0449] The above describes the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also within the scope of protection of the present invention.

Claims

1. A method of converting speech to text, characterized by, The method comprises: fusing the acoustic features of the speech frame data to be converted and the semantic context of the speech frame data to obtain fused features of different input channels; dimension reduction processing is performed on the fused features of the different input channels to obtain first features; obtaining prosodic frame data in the speech frame data according to the attention weight of the input channel, the speech frame data and the first features; converting the speech frame data into target text according to the prosodic frame data and the speech frame data.

2. The method of claim 1, wherein, The method comprises: obtaining the speech frame data to be converted; performing feature extraction processing on the speech frame data to obtain acoustic features of different input channels; predicting the semantic context of the speech frame data; fusing the acoustic features of different input channels and the semantic context to obtain fused features of the response input channel.

3. The method of claim 2, wherein, The method comprises: performing frame processing on the speech to be converted to obtain initial frame data; performing windowing processing on the initial frame data to obtain the speech frame data.

4. The method of claim 2, wherein, The method comprises: converting the speech frame data into frequency domain features, wherein the frequency domain features include frequency domain features of different input channels; processing the frequency domain features of each input channel respectively by using a channel self-attention mechanism to obtain second features of the corresponding input channel; obtaining third features of the corresponding input channel according to the frequency domain features of each input channel, the number of frames of the speech frame data and the dimension of the input channel by using a timing attention mechanism; performing compression and dimension reduction processing on the second features and the third features to obtain fourth features; performing timing reservation processing on the fourth features to obtain fifth features; performing frequency domain feature extraction on the fifth features, and performing time domain feature extraction on the fifth features to obtain the acoustic features.

5. The method of claim 4, wherein, The method comprises: performing compression processing on the frequency domain features of each input channel respectively by using a channel self-attention mechanism according to the number of frames of the speech frame data to obtain first target features; obtaining the first features of the corresponding input channel according to the attention weight of the input channel and the first target features.

6. The method of claim 4, wherein, The method comprises: obtaining second target features of the corresponding input channel according to the frequency domain features of each input channel and the dimension of the input channel by using a timing attention mechanism; obtaining the second features of the corresponding input channel according to the second target features of each input channel and the number of frames of the speech frame data.

7. The method of claim 4, wherein, The compression dimension reduction processing on the second feature and the third feature is performed to obtain a fourth feature, including: According to the number of frames of the speech frame data, the number of input channels, and the number of frequency bands of the frequency domain feature, the second feature and the third feature are fused to obtain a third target feature; According to a preset dimension number, the third target feature is subjected to compression dimension reduction processing to obtain the third feature.

8. The method of claim 4, wherein, The frequency domain feature extraction on the fifth feature and the time domain feature extraction on the fifth feature are performed to obtain the acoustic feature, including: According to the fifth feature, a lattice maximum mutual information LFMMI target function is constructed; According to the LFMMI target function, the frequency domain feature extraction on the fifth feature and the time domain feature extraction on the fifth feature are performed to obtain the acoustic feature.

9. The method of claim 2, wherein, The acoustic feature of different input channels and the semantic context are fused to obtain a fusion feature of a response input channel, including: The acoustic feature is projected to a target dimension by linear projection to obtain a fourth target feature, and the semantic context is projected to a target dimension by linear projection to obtain a target context; According to the fourth target feature and the target context, the fusion feature is obtained.

10. The method of claim 1, wherein, The fusion feature of the different input channels is subjected to dimension reduction processing to obtain a first feature, including: An attention weight of an input channel is obtained; According to the attention weight, a fusion feature corresponding to a key input channel and a fusion feature corresponding to a non-key input channel in the input channel are obtained; The fusion feature corresponding to the non-key input channel is subjected to rounding processing to obtain a sixth feature; According to an attention score matrix, the fusion feature corresponding to the key input channel and the first feature corresponding to a weak dependence interval in the sixth feature are obtained.

11. The method of claim 10, wherein, According to the normal distribution information of the attention weight, the key input channel and the non-key input channel in the input channel are determined; The fusion feature corresponding to the key input channel and the fusion feature corresponding to the non-key input channel in the input channel are obtained. According to the attention score matrix, an inter-frame dependence score in a preset division interval is obtained; 12. The method of claim 10, wherein, According to the inter-frame dependence score, the fusion feature corresponding to the key input channel and the weak dependence interval in the sixth feature are obtained; The fifth target feature corresponding to the weak dependence interval is obtained; The fifth target feature is subjected to decentralization processing to obtain a sixth target feature; A covariance matrix is constructed according to the sixth target feature to obtain a seventh target feature; According to the principal component matrix of the seventh target feature and the fifth target feature matrix, the first feature is obtained. The prosody frame data includes accent frame data and pause frame data; ​ 13. The method of claim 1, wherein, ​ According to the attention weight of the input channel, the speech frame data and the first feature, the prosody frame data in the speech frame data is obtained, including: According to the first feature and the attention weight, the stress frame data in the speech frame data is obtained; According to the energy difference between the speech frame data of adjacent two frames in the speech frame data, the pause frame data in the speech frame data is obtained.

14. The method of claim 13, wherein, According to the energy difference between the speech frame data of adjacent two frames in the speech frame data, the pause frame data in the speech frame data is obtained, including: Obtaining the frame energy of each speech frame data in the speech frame data; In the case that the frame energy corresponding to the first speech frame data, the second speech frame data and the third speech frame data is less than the preset energy value, the first speech frame data, the second speech frame data and the third speech frame data are taken as the pause frame data; Wherein, the first speech frame data, the second speech frame data and the third speech frame data are three continuous speech frame data in the speech frame data.

15. The method of claim 1, wherein, According to the prosody frame data and the speech frame data, the speech frame data is converted into target text, including: According to the prosody frame data and the speech frame data in the preset sliding time window, the speech frame data in the preset sliding time window is converted into window text according to the preset sliding time window; The window text corresponding to the preset sliding time window is spliced and de-overlapped to obtain the target text.

16. The method of claim 1, wherein, The method further includes: Obtaining the dynamic weight of the target text; In the case that the dynamic weight is greater than the first dynamic weight, the target text is taken as the final text of the speech frame data; In the case that the dynamic weight is less than the second dynamic weight, the target text is semantically corrected according to the acoustic feature and the target text to obtain a corrected text, and the corrected text is taken as the final text of the speech frame data; In the case that the dynamic weight is less than or equal to the first dynamic weight, and the dynamic weight is greater than or equal to the second dynamic weight, the target text is semantically corrected according to the acoustic feature and the target text to obtain a corrected text, and the final text of the speech frame data is obtained according to the corrected text and the target text; Wherein, the first dynamic weight is greater than the second dynamic weight.

17. An apparatus for converting speech to text, the apparatus comprising: The device includes: The first processing module is configured to fuse the acoustic feature of the speech frame data to be converted and the semantic context of the speech frame data to obtain fusion features of different input channels; The second processing module is configured to reduce the dimension of the fusion features of different input channels to obtain first features; The third processing module is configured to obtain prosody frame data in the speech frame data according to the attention weight of the input channel, the speech frame data and the first feature; The fourth processing module is configured to convert the speech frame data into target text according to the prosody frame data and the speech frame data.

18. A device for converting speech to text, characterized by including: A processor, a memory, and a program stored on the memory and executable on the processor, the program, when executed by the processor, implements the method for converting speech into text according to any one of claims 1 to 16.

19. A readable storage medium, characterized by, The program is stored on the readable storage medium, and when executed by the processor, implements the method for converting speech into text according to any one of claims 1 to 16.

20. A computer program product, characterised in that, The program comprises computer instructions, and when executed by the processor, implements the method for converting speech into text according to any one of claims 1 to 16.

Images