Speech recognition feature extraction and reasoning method for artificial intelligence dialogue system

Abstract

The present invention discloses a speech recognition streaming feature extraction and batch reasoning method for a high-concurrency artificial intelligence real-time dialogue system; the method adopts a heterogeneous computing architecture in which a CPU and a GPU work together, performs operations such as preprocessing, feature extraction and normalization of speech data by the CPU, and then uses the GPU to perform batch feature reasoning; specifically, the method comprises: after segmenting and preprocessing the speech data on the CPU side, performing Mel filter group feature extraction and cepstral mean variance normalization to obtain standardized acoustic feature vectors; on the GPU side, extracting feature vectors in batches from a feature buffer, performing padding and alignment, and then sending them to a neural network model for parallel reasoning calculation; the method realizes streaming processing and batch reasoning of the speech recognition process, and effectively improves the concurrent processing capability and real-time performance of the system.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a speech recognition feature extraction and reasoning method for an artificial intelligence dialogue system. Background Art

[0002] With the development of deep learning technology, automatic speech recognition (ASR) technology has been widely used in artificial intelligence dialogue systems. Current speech recognition systems usually adopt the architecture of front-end feature extraction and back-end neural network inference, where feature extraction is mainly performed on the CPU, including framing speech signals and extracting acoustic features, while neural network inference is performed on the GPU to accelerate the processing process through parallel computing.

[0003] However, there are still some problems in the existing technology: first, the feature extraction and model inference processes are often executed serially, and the CPU and GPU resources cannot be fully utilized; second, when processing continuous speech streams, due to the lack of an effective batch processing mechanism, the parallel computing capabilities of the GPU cannot be fully utilized; third, the data preprocessing and normalization methods in the feature extraction process often rely on fixed parameter settings, which are difficult to adapt to the needs of different scenarios.

[0004] Therefore, there is an urgent need for a speech recognition method that can effectively coordinate CPU and GPU resources and support streaming processing. Summary of the invention

[0005] The present application provides a speech recognition feature extraction and reasoning method for an artificial intelligence dialogue system to improve the concurrent processing capability and real-time performance of speech recognition.

[0006] The present application provides a speech recognition streaming feature extraction and batch reasoning method for a high-concurrency artificial intelligence real-time dialogue system, including:

[0007] In the CPU execution environment, receiving user voice data, and segmenting and preprocessing the user voice data according to a preset time window or audio frame size;

[0008] Performing Mel filter bank feature extraction and cepstral mean variance normalization processing on the segmented and preprocessed data to obtain a standardized acoustic feature vector; storing the standardized acoustic feature vector in a feature buffer;

[0009] In a GPU execution environment, feature vectors are extracted from the feature buffer in batches, and feature vectors of different lengths are padded to meet uniform input dimension requirements;

[0010] The padded feature vector batches are sent to the neural network model for parallel inference calculation to obtain the speech recognition results.

[0011] The beneficial effects of the technical solution provided by this application include:

[0012] (1) The feature buffer is used to decouple CPU feature extraction and GPU model inference, so that feature extraction and inference calculations can be performed in parallel, significantly improving the overall processing efficiency of the system. (2) A mechanism for extracting feature vectors in batches is adopted, and feature vectors of different lengths are padded, which fully utilizes the parallel computing capabilities of the GPU and effectively improves the concurrent processing capabilities of the system. (3) The combination of Mel filter bank feature extraction and cepstral mean variance normalization is adopted to improve the stability and robustness of acoustic features, enabling the system to better adapt to different speakers and acoustic environments. (4) A streaming processing mechanism is adopted to segment speech data through preset windows, realizing real-time processing of continuous speech and meeting the low latency requirements of real-time dialogue systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a flowchart of a speech recognition feature extraction and reasoning method for an artificial intelligence dialogue system provided in the first embodiment of the present application. DETAILED DESCRIPTION

[0014] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0015] The first embodiment of the present application provides a method for extracting and reasoning speech recognition features for an artificial intelligence dialogue system. Figure 1 , which is a schematic diagram of the first embodiment of the present application. Figure 1 The first embodiment of the present application provides a method for speech recognition feature extraction and reasoning for an artificial intelligence dialogue system, which is described in detail.

[0016] Step S101: In a CPU execution environment, user voice data is received, and the user voice data is segmented and pre-processed according to a preset time window or audio frame size.

[0017] In this embodiment, the user's voice signal is obtained by an audio acquisition device, and the voice signal is transmitted to the CPU execution environment in the form of a digital audio stream. For the continuous audio stream input, the system uses a sliding window to perform segmentation processing. Among them, the length of the window and the overlapping length between adjacent windows can be flexibly configured according to the actual application scenario to strike a balance between real-time and accuracy. The setting of the overlapping part can effectively ensure the continuity between audio frames and avoid information loss.

[0018] For each audio frame, pre-emphasis processing is first performed to increase the high-frequency energy of the speech signal through a first-order high-pass filter. Pre-emphasis processing can compensate for the high-frequency attenuation of the speech signal during the utterance process, making the extracted features more accurate. Subsequently, the Hamming window function is applied to the pre-emphasized signal for windowing processing. The Hamming window can effectively reduce spectrum leakage and improve the accuracy of spectrum analysis.

[0019] In order to achieve efficient data management, the system uses a ring buffer to store and manage audio frames. When new audio data arrives, the buffer content is dynamically updated according to the first-in-first-out principle. The size of the ring buffer can be dynamically adjusted according to the system's memory capacity and real-time requirements to ensure that sufficient context information is stored while avoiding excessive system resources.

[0020] The preprocessing process is executed in the main thread of the CPU, and the continuity and real-time performance of data processing are ensured through efficient memory management and thread scheduling mechanisms. The delay of the entire preprocessing stage needs to be strictly controlled within a reasonable range to meet the requirements of real-time speech recognition. At the same time, the system will adaptively adjust the processing strategy according to the CPU load to achieve optimal resource utilization while ensuring processing quality.

[0021] Through the above processing, the original speech signal can be converted into a normalized audio frame sequence suitable for subsequent feature extraction. These pre-processed data lay the foundation for the next stage of feature extraction and model inference, which is an important link in achieving high-quality speech recognition.

[0022] Furthermore, the receiving of user voice data and segmenting and preprocessing the user voice data according to a preset time window or audio frame size includes:

[0023] The segment length is dynamically adjusted according to the real-time arrival speed of the speech segment, and the segment is immediately sent to the feature buffer after the segmentation is completed to achieve parallel feature extraction of the speech input that has not yet ended.

[0024] When receiving user voice data and segmenting and preprocessing it according to a preset time window or audio frame size, the input speed of the voice data can be measured and evaluated through a real-time monitoring module pre-set in the system, such as counting the number of collected audio samples per unit time, or judging the cumulative duration of a continuous voice signal when it is detected.

[0025] In order to balance real-time performance and accuracy in high-concurrency scenarios, it is necessary to dynamically adjust the length of voice segments according to the speech speed, network fluctuations, and hardware resource usage of different users, combined with the pre-set time window or audio frame size. Specifically, when the voice input stream arrives, the system will first segment the voice stream according to the initially set time window. If the accumulation of audio samples observed at a certain moment exceeds a certain threshold, the segmentation length of subsequent segments will be automatically shortened to process them in time and send the voice segments to the feature buffer; if it is detected that the voice input becomes sparse or the packet delay caused by the network condition is large, the segmentation length will be appropriately relaxed to reduce the overhead caused by excessive segmentation. Through this real-time monitoring and dynamic adjustment strategy, the system can perform fine-grained and flexible segmentation of continuous and unknown duration voice input, which not only ensures that the system can quickly obtain and process front-end voice segments in parallel, but also moderately relax the segmentation frequency when network and computing resources are tight, thereby maintaining smooth operation in high-concurrency application scenarios.

[0026] At the moment when the above dynamic segmentation is completed, the obtained speech fragment will be immediately sent to the feature buffer for subsequent feature extraction and processing. In this way, the Mel filter group feature extraction and other operations can be performed on the segmented fragments before the user's voice ends, avoiding the situation where feature extraction must be carried out only after the entire speech ends. Since the speech segmentation information of different users and in different states is centrally managed in the feature buffer, the system can use parallel or pipelined processing strategies to maximize the use of CPU and other resources to synchronously complete speech filtering, windowing, and cepstral mean variance normalization in the background.

[0027] In this way, even though the user continues to input voice, the system can continuously update the feature buffer, and segment new voice segments according to preset rules or current concurrency and send them to the feature buffer, truly realizing parallel feature extraction of voice input that has not yet ended. Since the timing of segmentation and feature extraction is dynamically evaluated and there is no fixed time rigid waiting cycle, it is possible to complete the processing of voice information in a timely manner while ensuring high concurrent throughput, bringing a unified structure and moderately normalized acoustic feature vector for subsequent GPU batch inference, thereby taking into account both response speed and recognition accuracy in large-scale real-time dialogue applications.

[0028] Furthermore, the receiving of user voice data and segmenting and preprocessing the user voice data according to a preset time window or audio frame size includes:

[0029] The segment length is dynamically adjusted according to the real-time arrival speed of the speech segment, and the segment is immediately sent to the feature buffer after the segmentation is completed to achieve parallel feature extraction of the speech input that has not yet ended.

[0030] The technical solution of receiving user voice data and segmenting and preprocessing it according to a preset time window or audio frame size is to solve the common real-time and continuity challenges in speech recognition systems, especially in streaming voice input scenarios in high-concurrency environments.

[0031] When user voice data enters the system through an audio acquisition device (such as a microphone), the system monitors the arrival speed of the voice signal in real time. Voice signals are usually transmitted in the form of a continuous data stream, and the system needs to dynamically manage it to ensure the efficiency and accuracy of subsequent feature extraction and reasoning. The preset time window or audio frame size is the initial parameter for processing, which is usually a fixed time length (such as 25 milliseconds) or a fixed frame size (such as 400 sampling points). However, due to the dynamic characteristics of voice signals (such as changes in user speech speed and the influence of background noise), fixed segmentation schemes may cause signal integrity loss or processing delays.

[0032] In order to solve this problem, this embodiment provides a method for dynamically adjusting the segment length. When receiving the voice data stream, the system analyzes the arrival speed of the current voice segment in real time. The arrival speed can be dynamically determined by detecting the interval time of the peak energy in the voice signal, the frequency of change of the zero-crossing rate, or other time domain and frequency domain features. Through these analyses, the system can determine the speed of the current speech. For example, when the user speaks faster, the system will automatically shorten the segment length to ensure that each segment can capture complete voice information without losing details; on the contrary, when the speech speed is slower, the system will appropriately extend the segment length, thereby reducing the number of segmentations and reducing the system's computational overhead.

[0033] After dynamic segmentation, each segmented speech data is immediately preprocessed and sent to the feature buffer. The preprocessing steps include but are not limited to removing silent segments, normalizing the audio signal amplitude, and applying anti-noise algorithms to eliminate background noise interference. After preprocessing, the segmented speech data is input into the feature buffer as a standardized signal. The feature buffer is a queue structure used to store speech segment features that have not yet been extracted, and is subsequently processed one by one according to the first-in-first-out principle.

[0034] By immediately sending the segmented data into the feature buffer, rather than waiting for the entire voice input to be completed before starting feature extraction, this embodiment realizes parallel feature extraction of voice input that has not yet ended. This parallel feature extraction mechanism can significantly improve the system processing efficiency, allowing the voice recognition system to respond to the user's voice input request in real time. At the same time, through the mechanism of dynamically adjusting the segment length and sending it into the buffer in real time, the system can adapt to the diversity and complexity of user voice input in high-concurrency scenarios, avoiding resource waste or delays caused by fixed segmentation strategies.

[0035] Furthermore, the receiving user voice data includes:

[0036] By calculating the signal-to-noise ratio and speech clarity index of the signal, the quality of each speech segment is evaluated in real time, and the feature extraction parameters are dynamically adjusted according to the evaluation results to enhance the feature extraction capability of low-quality speech, thereby improving the overall speech recognition accuracy.

[0037] In this embodiment, a technical solution of real-time signal quality evaluation and dynamic adjustment of feature extraction parameters is introduced in the process of receiving user voice data, aiming to solve the problem of reduced voice recognition accuracy due to low signal-to-noise ratio, fuzzy voice or environmental noise interference in voice input. The present invention dynamically evaluates the quality of each voice segment by calculating the signal-to-noise ratio and voice clarity index of the signal, and adjusts the parameter setting of feature extraction based on this, thereby enhancing the system's ability to process low-quality voice and improving the overall recognition effect.

[0038] After the speech signal reaches the system, the key feature information of the speech segment is first extracted through the signal analysis module. These feature information include the energy distribution, spectral characteristics and time domain properties of the signal, which can fully reflect the quality of the speech. The calculation of the signal-to-noise ratio is based on the power ratio between the signal and the background noise. By separating the speech and noise components, the strength of the speech signal relative to the noise in the current speech segment is calculated. For example, the energy within the main frequency range of the speech can be extracted using an adaptive filtering method or a frequency domain analysis method, and compared with the energy of the non-speech frequency band to obtain the signal-to-noise ratio value.

[0039] The speech clarity index evaluates the clarity of speech signals based on the distribution of speech spectrum, resonance peak characteristics, and time variation patterns. Specifically, the clarity of speech signals can be quantified by calculating the harmonic-to-noise ratio, zero-crossing rate, and other indicators of speech segments. For example, the harmonic-to-noise ratio is used to distinguish between the periodic part and the random noise part of speech, while the zero-crossing rate reflects the complexity of the frequency components in the speech signal. Combining these indicators, the system can comprehensively evaluate the quality of each speech segment.

[0040] After completing the quality assessment, the system dynamically adjusts the parameters of feature extraction based on the comprehensive results of the signal-to-noise ratio and clarity indicators. These parameters include but are not limited to the frequency band distribution, sampling frequency, frame length, and frame shift of the Mel filter group. For example, for speech segments with low signal-to-noise ratio, the number of filter groups can be appropriately increased or the frequency band division can be adjusted to better capture the effective components in the speech signal while suppressing the interference of background noise; for speech segments with low clarity, the frame length can be shortened or the frame shift step can be increased to improve the temporal resolution of the speech signal, thereby more accurately capturing the dynamic changes of the speech.

[0041] In addition, the system can also select different feature extraction algorithms based on the quality of different speech segments. For example, for high-quality segments, the standard Mel-spectrogram feature extraction method can be directly used; for low-quality segments, enhancement algorithms such as speech enhancement filtering or spectrum reconstruction technology can be introduced to improve the reliability of features. After feature extraction is completed, the system will store the processed feature vectors in the feature buffer for subsequent batch inference tasks.

[0042] By introducing a real-time evaluation mechanism for signal-to-noise ratio and speech clarity indicators, the present invention can not only dynamically adapt to speech inputs of different qualities, but also perform targeted optimization in the feature extraction stage, fundamentally improving the processing effect of low-quality speech. This method is particularly suitable for high-concurrency speech recognition tasks in complex speech environments, such as multi-user real-time conversations or speech recognition in public places with strong background noise.

[0043] Step S102: performing Mel filter bank feature extraction and cepstral mean variance normalization processing on the segmented and preprocessed data to obtain a standardized acoustic feature vector; and storing the standardized acoustic feature vector in a feature buffer.

[0044] After the audio data is preprocessed, feature extraction is performed on each preprocessed audio frame to obtain a more compact and information-rich acoustic feature representation. The feature extraction process is first processed using a Mel filter bank because the Mel frequency scale is closer to the auditory characteristics of the human ear and can better express the key features of the speech signal.

[0045] Specifically, the system first performs a fast Fourier transform on the preprocessed audio frame to obtain a frequency domain representation. The spectrum is then filtered through a Mel filter bank, which consists of a series of triangular filters uniformly distributed on the Mel frequency scale. The output of each filter represents the energy distribution characteristics within the frequency band. The number of filters can be configured according to actual application requirements to strike a balance between feature dimensionality and information retention.

[0046] After obtaining the energy features of the filter bank, the system further performs cepstral mean variance normalization. The purpose of this step is to eliminate the systematic differences in acoustic features caused by factors such as recording equipment and environmental noise. Normalization calculates the mean and variance of the features in the time dimension, and then standardizes the features to have distribution characteristics of zero mean and unit variance. This processing can significantly improve the robustness of acoustic features and enable the system to adapt to different speakers and acoustic environments.

[0047] After feature extraction and normalization, the system stores the obtained standardized acoustic feature vectors in the feature buffer. The feature buffer is managed using a queue structure to ensure that the feature vectors are organized in chronological order. When a new feature vector is generated, the system appends it to the end of the buffer and removes the earliest feature vector when necessary, thereby maintaining a sliding feature window. The size of the feature buffer can be adjusted dynamically to balance system resource usage and the integrity of the feature sequence.

[0048] The entire feature extraction and normalization process is executed in the CPU environment. Through optimized algorithm implementation and parallel processing strategy, the processing efficiency is ensured to meet the real-time requirements. At the same time, the system will perform necessary quality monitoring on the intermediate results of feature extraction in order to detect and handle abnormal situations in a timely manner. These processes provide high-quality input data for subsequent neural network model reasoning.

[0049] Furthermore, the feature buffer also includes a scheduling module for monitoring the number and length distribution of feature vectors in the buffer. When it is detected that the amount of feature vector data reaches a preset threshold, a batch inference task trigger signal is sent to the GPU, and the inference task load of the next batch of feature vectors is automatically increased when the GPU is idle, so as to improve the overall utilization of the GPU.

[0050] When a scheduling module capable of monitoring the number and length distribution of vectors is set in the feature buffer, it is possible to determine whether a batch inference task needs to be initiated by observing the total number of acoustic feature vectors in the buffer, the average length of each vector, and the current GPU load status in real time when the system is running.

[0051] In order to make full use of computing resources in high-concurrency scenarios, the scheduling module will first identify whether a sufficient number of feature vectors have been accumulated according to the strategy or threshold set by the system management end. For example, when it is observed that a certain scale of feature data generated by multiple real-time voice inputs exists in the buffer at the same time, it is determined that a critical value that can trigger batch reasoning has been reached or is close to it. At this time, the scheduling module will send a trigger signal for the batch reasoning task to the GPU, so that the GPU can start to perform parallel calculations on the collected batch of feature vectors in a timely manner, thereby concentrating the reasoning capabilities of the neural network model on unified batch processing. At the same time, if the scheduling module detects that the GPU is idle or nearly idle, or the load drops significantly after completing the reasoning of the previous batch of feature vectors, it will automatically schedule the next batch of feature vectors from the buffer to the GPU to maintain the efficient operation of the GPU and maximize its overall utilization.

[0052] Through this dynamic monitoring and triggering mechanism, the system can accurately capture at any time whether the speech feature vector data has accumulated enough to form batch processing conditions, so as to reasonably arrange the GPU's batch inference tasks. It will not trigger too early and cause the batch size to be too small to affect performance, nor will it trigger too late and cause excessive accumulation and extend the inference delay.

[0053] In implementation, the threshold parameters of the scheduling module can be flexibly configured so that the system can still appropriately exert the parallel computing potential of the GPU under different network environments, user scales or hardware resource conditions, and implement stable and orderly management of each batch of feature vectors in the buffer. Therefore, when the system faces high concurrent requests, the scheduling module can effectively avoid resource idling or overloading, allowing the speech recognition process provided in this embodiment to always maintain high throughput and real-time response performance at the GPU level, thereby providing users with fast and accurate recognition services in large-scale real-time conversation scenarios.

[0054] Furthermore, the cepstrum mean variance normalization process comprises the following steps:

[0055] A preliminary weighted energy calculation is performed on the acoustic feature vector of the input speech segment signal to eliminate the interference of background noise and the influence of dynamic signal changes. The weighted energy calculation is implemented by the following formula 1:

[0056] ;

[0057] in, is the weighted energy value of the current frame, indicating the The energy intensity of the speech signal on ;

[0058] It is Frame feature values ​​represent key characteristics of speech signals in the frequency domain or time domain, such as spectral energy or Mel-band energy value.

[0059] It is the total number of frames of feature vectors, which defines the context frame range involved in the current calculation.

[0060] is the weight coefficient, which determines the contribution of each frame to the current time frame and is calculated using the following formula 2:

[0061] ;

[0062] in, The temporal distance and the dynamic adjustment of the feature value are comprehensively considered to ensure that the contribution of each frame is not only determined by the temporal distance, but also related to the deviation of the frame features.

[0063] It is a parameter used to control weight attenuation. The recommended value is 5 to 10, which is selected according to the time resolution of the speech signal.

[0064] It is Frame feature value; It is the characteristic value of multiple frames of content in a specified time period The average value of It is the characteristic value of multiple frames of content in a specified time period The standard deviation of To prevent small positive numbers with zero denominator; To adjust the parameters;

[0065] According to the following formula 3, the energy value of each frame is calculated Rate of change , to evaluate the energy fluctuation of the current frame compared to the context frame:

[0066] ;

[0067] in, It is the average energy in the specified time period, obtained by calculating the energy of all frames; It is the energy standard deviation in the specified time period, which is used to measure the fluctuation range of energy. is a small positive number that prevents the denominator from being zero;

[0068] According to the following formula 4, calculate the weighting factor of the current frame :

[0069] ;

[0070] in, It is a parameter for adjusting the speed of change of the weighting factor, and the recommended value is 5 to 10; is the threshold of the rate of change, used to distinguish between drastically changing frames and stable frames of speech signals, and the recommended value is 0.5;

[0071] According to the following formula 5, the feature vector is smoothed and corrected: ;

[0072] in, Is the time frame The acoustic feature vector represents the short-time spectrum characteristics of the speech signal; is the smoothed corrected eigenvector; is the weighted mean of the context frame, calculated according to the following formula 6:

[0073] ;

[0074] in, It is Frame feature value; is the total number of frames of the feature vector; is the weight coefficient, calculated using Formula 2.

[0075] Step S103: in the GPU execution environment, extracting feature vectors from the feature buffer in batches, and padding feature vectors of different lengths to meet the uniform input dimension requirement.

[0076] The system implements a batch processing mechanism for feature vectors in the GPU execution environment. When reading data from the feature buffer, the system determines the number of feature vectors to extract each time based on a pre-set batch processing strategy. The batch size setting needs to take into account factors such as the GPU's computing power, memory capacity, and real-time requirements. Through a reasonable batch processing mechanism, the parallel computing characteristics of the GPU can be fully utilized to improve the overall processing efficiency of the system.

[0077] When extracting feature vectors, the system may encounter inconsistent lengths of feature vectors generated by different speech segments. In order for the neural network to process these feature vectors in batches, they need to be normalized. The system uses padding operations to make all feature vectors reach a unified input dimension. Specifically, for feature vectors that are not long enough, padding values ​​are added to the end of the sequence; for feature vectors that exceed the target length, truncation is required.

[0078] The padding operation requires special attention to the selection of padding values. The setting of padding values ​​will affect the performance of the neural network. Usually, a specific padding marker or zero value can be selected. At the same time, the system also needs to record the effective length information of each feature vector. This information will be used to generate a suitable attention mask in the subsequent model inference process to ensure that the padding part does not have a negative impact on the recognition result.

[0079] The batch processing and filling process of feature vectors are completed in GPU memory to reduce data transmission overhead. The system will dynamically adjust the batch size according to the usage of GPU video memory to ensure processing efficiency while avoiding video memory overflow. In addition, the system will also perform pipeline optimization on the batch processing of feature vectors so that feature extraction and model inference can be performed in parallel, further improving the system's throughput.

[0080] Furthermore, the feature buffer also includes a scheduling module for monitoring the number and length distribution of feature vectors in the buffer. When it is detected that the amount of feature vector data reaches a preset threshold, a batch inference task trigger signal is sent to the GPU, and the inference task load of the next batch of feature vectors is automatically increased when the GPU is idle, so as to improve the overall utilization of the GPU.

[0081] The scheduling module included in the feature buffer is a key component for optimizing system resource utilization, especially in high-concurrency AI real-time dialogue systems, aiming to maximize GPU computing efficiency and reduce task latency. The design of this module solves the problem of resource waste caused by uneven distribution of feature vector quantity and length in streaming speech recognition scenarios, while ensuring that the GPU can operate efficiently in a multi-task high-load environment.

[0082] In the implementation process, the feature buffer acts as a queue structure to store acoustic feature vectors after feature extraction in real time. These feature vectors come from the voice data stream received by the system, and their lengths may vary due to the dynamic adjustment of voice segments. In order to ensure the efficiency of batch inference tasks, the scheduling module needs to continuously monitor the number and length distribution of feature vectors in the buffer and perform dynamic scheduling according to preset rules.

[0083] The scheduling module counts the number of feature vectors in the buffer and immediately triggers the batch inference task when the number reaches the preset threshold. This threshold is set based on the GPU's computing power and the priority of task processing to ensure that each inference task can run with maximum parallelism. For example, if the input batch size corresponding to the GPU's maximum computing efficiency is 64, then when the number of feature vectors in the buffer reaches or approaches 64, the scheduling module will send a trigger signal to the GPU to combine these feature vectors as a batch and pass them to the GPU for inference.

[0084] In addition, the scheduling module will dynamically analyze the length distribution of feature vectors to ensure that the feature vectors in a batch are consistent in length. If some feature vectors are short, the scheduling module will fill them with uniform length according to the preset filling strategy to meet the dimensional requirements of GPU input. The filling data will use methods such as noise suppression or context mean interpolation to minimize interference with the inference results.

[0085] After the GPU completes the current batch of inference tasks, the scheduling module is also responsible for checking the GPU usage status in real time. When the GPU is idle, the scheduling module automatically increases the inference task load for the next batch of feature vectors. This mechanism avoids the waste of GPU idle time while leaving space in the task queue for high-priority requests. For example, when a high-priority request arrives, the scheduling module can ensure the real-time and flexibility of the system by adjusting the size of the next batch of tasks or rearranging the scheduling order.

[0086] Through this design, the scheduling module achieves the following goals: first, ensure that the GPU computing resources are fully utilized; second, optimize the efficiency of batch reasoning through dynamic filling and task scheduling; third, in a high-concurrency environment, ensure that low-priority tasks do not occupy too many resources to avoid affecting the system's responsiveness to critical tasks. This intelligent scheduling method based on the number and length distribution of feature vectors not only improves the overall throughput of the system, but also provides strong technical support for real-time processing of speech recognition tasks.

[0087] Furthermore, in the GPU execution environment, extracting feature vectors in batches from the feature buffer includes:

[0088] Based on the arrival time of feature vectors in the buffer, the criticality of feature vectors, and the current GPU load status, high-density data batches are prioritized and inference tasks of low-priority batches are delayed to optimize resource allocation and improve system throughput.

[0089] The step of extracting feature vectors from the feature buffer in batches in the GPU execution environment is designed to optimize resource allocation through an intelligent scheduling mechanism to ensure efficient inference computing in high-concurrency artificial intelligence real-time dialogue systems while maximizing system throughput and response speed.

[0090] During system operation, the feature buffer continuously receives feature vectors extracted from speech signals. These feature vectors are the result of speech segmentation and processing by the feature extraction module, and have different arrival times and criticalities. To achieve efficient inference task scheduling, the system first analyzes the data in the feature buffer, including the arrival time, importance evaluation, and current GPU load status of each feature vector. The arrival time is the timestamp information when the feature vector is stored in the buffer, which is used to determine the priority of the feature vector; the criticality evaluation is based on the specific attributes of the feature vector, such as signal energy or context importance, reflecting its impact on the speech recognition result; the GPU load status is monitored by the system in real time, including the number of tasks being executed, the remaining computing power, and the resource occupancy rate.

[0091] Based on these data, the system dynamically generates a scheduling plan for inference tasks. For feature vectors with higher criticality or earlier arrival time, the system prioritizes them to the current inference task and forms high-density data batches to be submitted to the GPU for processing. This priority strategy ensures that important tasks can be responded to quickly, especially in scenarios with high concurrency of multiple tasks, significantly improving the overall service quality of the system.

[0092] On the other hand, for feature vectors with lower criticality or later arrival time, the system can appropriately delay the scheduling of their inference tasks. The delay strategy dynamically adjusts the priority of tasks to make the GPU's computing resource allocation more balanced and avoid delays of high-priority tasks due to interference from low-priority tasks. In this process, the system will also comprehensively consider the current load status of the GPU and dynamically adjust the batch size and task scheduling frequency to further optimize computing efficiency.

[0093] When the GPU load is light, the system can actively increase the batch size of inference tasks or increase the scheduling frequency of low-priority tasks, so as to make full use of idle resources and avoid wasting GPU computing power. When the GPU load is heavy, the system will give priority to the execution of high-priority tasks, by reducing the batch size of low-priority tasks or extending their waiting time, to ensure that the entire system can still maintain stable operation under high-load conditions.

[0094] Through this dynamic scheduling mechanism based on arrival time, criticality, and GPU load status, the system can achieve intelligent batch extraction of feature vectors and efficient management of inference tasks. On the one hand, prioritizing high-density data batches significantly improves the real-time performance of speech recognition; on the other hand, delaying the inference of low-priority tasks effectively reduces resource conflicts and delay risks, and improves the overall throughput of the GPU.

[0095] Furthermore, extracting feature vectors from the feature buffer in batches comprises the following steps:

[0096] Monitors the number of feature vectors stored in the feature buffer And the length distribution of each eigenvector , in ;

[0097] According to the following formula 7, calculate the optimal number of feature vectors for the current batch :

[0098] ;

[0099] in, is the candidate batch size, which indicates the number of feature vectors in the batch that can be selected;

[0100] is the standard deviation of the feature vector lengths within the selected batch, calculated according to the following formula 8:

[0101] ;

[0102] in, is the average length of the feature vectors in the batch, obtained by calculating the length of all selected feature vectors;

[0103] Extracting the optimal batch size from the feature buffer , perform dynamic filling operation;

[0104] The padding operation is based on the length of the longest feature vector in the batch. Determine the position and number of fill values. The fill value is calculated using the following formula 9 to reduce the damage to the continuity of feature data caused by fill:

[0105] ;

[0106] in, Is the filling position The value used to insert data at the end of the feature vector.

[0107] This is the first eigenvector values;

[0108] After padding, the input tensor is formed into a uniform size , and send it into the neural network model for parallel reasoning calculation.

[0109] Step S104: sending the filled feature vector batches to the neural network model for parallel reasoning calculation to obtain speech recognition results.

[0110] After completing the batch processing and padding of feature vectors, the system passes the processed feature batches directly in the GPU memory to the pre-loaded neural network model. The model is a specially trained deep neural network used to convert acoustic feature sequences into corresponding text sequences. In the GPU environment, the system fully utilizes its parallel computing capabilities to process multiple feature vectors in a batch at the same time.

[0111] After receiving the input features, the neural network model first generates an attention mask based on the effective length information of each sequence to ensure that the padding part is correctly handled during the processing. Subsequently, the model performs forward propagation calculations according to the predefined network structure, including feature encoding, attention calculation, and decoding steps. During this process, the system maintains the decoding state and determines the optimal recognition result through cumulative probability distribution.

[0112] In order to improve the accuracy of recognition, the system uses a beam search strategy for decoding. During the decoding process, the system will maintain multiple candidate paths at the same time, and dynamically update and trim these paths based on the cumulative probability. By setting an appropriate beam width, the system can find a balance between decoding quality and computational overhead. Finally, the system selects the path with the highest probability from all candidate paths as the recognition result.

[0113] The entire inference process is completely executed on the GPU, avoiding unnecessary data transfer between the CPU and GPU. The system ensures efficient use of GPU resources through optimized memory management and computational scheduling. At the same time, due to the use of a batch processing mechanism, the recognition tasks of multiple audio clips can be executed in parallel, significantly improving the concurrent processing capabilities of the system. The final recognition result will be returned to the upper-level application to complete the entire speech recognition process.

[0114] Furthermore, after the GPU completes the inference calculation of the batch feature vectors, it includes the steps of caching and timing control of the inference results, so that when the user has not yet finished the voice input, some voice recognition results can be obtained and returned to the client in real time, thereby shortening the user's waiting time and ensuring the real-time performance of the system under high concurrent requests.

[0115] After the batch feature vectors are inferred and calculated by the GPU, the speech recognition results obtained by inference can be placed in a special cache area and managed with the timing control algorithm, so as to provide phased recognition information in real time when the user's voice input has not yet been completed.

[0116] Specifically, when the system receives the notification that GPU inference is complete, it will immediately write the generated recognition result into the cache, and then determine whether the result can be pushed locally to the client based on the pre-set time or frame length control mechanism. If the user continues to transmit voice data and has not yet marked the end of the voice, such as when the user does not have an obvious silent segment or a network disconnection signal, the system will first return the relatively complete segment recognition result that has completed the inference to the client. After the newly generated voice segment enters the feature buffer and completes the GPU inference, the recognition text can be appended or updated, so that the user can see the staged recognition progress at any time during the long voice conversation. At the same time, if it is detected that the user has stopped voice input, or the silent interval exceeds a certain length, all cached segment recognition results will be spliced ​​or merged to form the final complete voice recognition result and returned to the client to ensure that the user can get timely and continuously updated recognition feedback even in high concurrency scenarios.

[0117] Through this buffering and timing control mechanism, the system can balance the GPU inference speed and the user front-end experience in the background, effectively shortening the user's waiting time. Because in common real-time dialogue applications, if you need to wait for the entire voice to be fully input and all feature extraction and inference to provide results, it often leads to a long response delay. With cache and timing management, the system can provide real-time feedback when each stage of the inference task is completed, so that users can understand to what extent their voice input has been recognized by the system. This is crucial for user experience in high concurrency situations, and can avoid long blank waits for a large number of users during network congestion and resource competition.

[0118] During implementation, the cache size, time step of timing control or automatic detection rules can be flexibly set according to actual business needs, so as to ensure recognition accuracy while pushing intermediate recognition results to users appropriately, thereby making fuller use of GPU parallel resources and improving overall speech recognition efficiency and real-time performance.

[0119] Furthermore, after the GPU completes the inference calculation of the batch feature vectors, it includes the steps of caching and timing control of the inference results, so that when the user has not yet finished the voice input, some voice recognition results can be obtained and returned to the client in real time, thereby shortening the user's waiting time and ensuring the real-time performance of the system under high concurrent requests.

[0120] The GPU cache and timing control steps after completing the inference calculation of batch feature vectors are designed to achieve rapid response and efficient management of speech recognition results in a high-concurrency artificial intelligence real-time dialogue system. This step is designed to solve the problem of real-time feedback when the user's voice input is not yet completed, thereby significantly shortening the user's waiting time while maintaining the system's processing efficiency and stability in high-concurrency request scenarios.

[0121] After the GPU completes the inference calculation of a batch of feature vectors, the system will immediately cache the intermediate results of the inference into the specified storage area. The cached storage area can be high-performance memory or shared memory, and the specific choice depends on the system architecture and hardware conditions. The cached content includes the output results of the inference task, such as the predicted probability distribution of phonemes for each time frame or its corresponding partial speech recognition text. The cache module attaches timestamps and frame index information to each batch of inference results to ensure that the subsequent processing stage can correctly organize these intermediate results in time order and content.

[0122] At the same time, the system dynamically manages the cached inference results through the timing control module. This module monitors the status of the voice input and the completion progress of the inference task in real time to determine whether the user's voice input has ended. If the user's voice input is not yet completed, the timing control module will gradually generate part of the speech recognition text based on the cached intermediate results. This process is usually implemented through phrase-level speech splicing or context-based text generation methods to ensure that the generated partial text can naturally and coherently reflect the user's intentions.

[0123] After generating some speech recognition results, the system will return these results to the client in real time through the network interface. In order to ensure that the client can receive and correctly display or play these results in time, the timing control module needs to accurately manage the frequency and format of the returned data. For example, when the frequency of generating some results is too high, the timing control module will automatically merge or delay the return of some data to avoid the processing burden of the client; when the frequency is too low, the module will give priority to transmitting the latest results in the current cache to improve the smoothness of the user experience.

[0124] In addition, in order to adapt to the needs of high-concurrency scenarios, the timing control module also includes a priority management function. When the system receives voice input requests from multiple users at the same time, the module will dynamically adjust the cache and return strategy of the inference results according to the urgency of each request and the processing status of the task queue. For example, for high-priority requests, the module will reduce the cumulative time of cached data and give priority to returning partial results of these requests; for low-priority requests, the return of some results may be appropriately delayed to ensure the real-time performance of high-priority tasks.

[0125] Through such cache and timing control design, the system can provide real-time feedback of some speech recognition results before the user's speech input is completed, significantly shortening the user's waiting time. At the same time, this design effectively balances the processing efficiency and response speed in high-concurrency scenarios, ensuring that the system can run stably and provide fast and accurate services to each user.

[0126] Furthermore, the neural network model is based on the Conformer architecture, including an acoustic front-end feature enhancement module, a multi-head self-attention and convolution fusion encoding module, a dynamic phoneme prediction module, and a semantic enhancement decoding module based on a language model;

[0127] The input of the acoustic front-end feature enhancement module is the standardized acoustic feature vector, and the output is the enhanced acoustic feature vector; the acoustic front-end feature enhancement module performs multi-scale analysis on the input features through a convolutional neural network and an adaptive spectrum filter, extracts the local spectrum pattern of the speech signal, and enhances the expression of the high-frequency signal; the convolutional neural network part adopts hierarchical convolution to capture short-term and long-term patterns, and the adaptive spectrum filter dynamically adjusts the enhancement coefficient to match the energy distribution of the input feature;

[0128] The input of the multi-head self-attention and convolution fusion encoding module is the enhanced acoustic feature vector, and the output is the context encoding representation after the fusion of global and local features; the multi-head self-attention and convolution fusion encoding module is implemented by using the multi-head self-attention mechanism and convolution module of Conformer; the multi-head self-attention mechanism is used to model the global time dependency of the input features, and the input features are divided into multiple attention heads, each of which independently captures the global pattern in different time series; the one-dimensional depth-separable convolution module is used to capture the local context of the input features to enhance the expression ability of short-term information, wherein the multi-head self-attention mechanism and the convolution module are used in a serial manner;

[0129] The input of the dynamic phoneme prediction module is the context encoding representation, and the output is the probabilistic phoneme prediction distribution; the dynamic phoneme prediction module uses a prediction network based on a dynamic gating mechanism to dynamically model the phoneme distribution of the current time step in combination with the context encoding information;

[0130] The input of the language model-based semantic enhancement decoding module is the probabilistic phoneme prediction distribution and the pre-trained language model embedding representation, and the output is the speech recognition text result; the language model-based semantic enhancement decoding module adopts a hybrid language model decoding strategy to fuse the dynamic phoneme distribution with the embedding of the pre-trained language model; and at each time step, the weights of the phoneme distribution and the language model prediction are dynamically adjusted through the self-attention mechanism, so as to generate a text result with higher semantic consistency.

[0131] In this embodiment, the neural network model is based on the Conformer architecture, including an acoustic front-end feature enhancement module, a multi-head self-attention and convolution fusion encoding module, a dynamic phoneme prediction module, and a semantic enhancement decoding module based on a language model.

[0132] The input of the acoustic front-end feature enhancement module is the standardized acoustic feature vector, which is obtained by framing the speech signal, Fourier transform, Mel filter bank calculation, and cepstral mean variance normalization. The feature vector of each time frame includes multi-dimensional components, each dimension represents the energy or spectral characteristics of the speech signal in different frequency bands. This module performs multi-scale analysis on the input features through convolutional neural network (CNN) and adaptive spectral filter, focusing on extracting the local spectral pattern of the speech signal and enhancing the expression ability of high-frequency signals. The CNN part adopts a layer-by-layer hierarchical convolution structure, in which the first layer of convolution captures short-term patterns, and the subsequent convolution layers gradually expand the receptive field to capture long-term patterns. In addition, the adaptive spectral filter adjusts the enhancement coefficient in real time according to the energy distribution of the speech signal, for example, by comparing the energy changes in different frequency bands to dynamically amplify the high-frequency component, thereby generating an enhanced feature vector. The output enhanced feature vector not only retains the time-spectral characteristics of the original feature, but also further optimizes the signal-to-noise ratio, which is suitable for subsequent encoding and reasoning tasks.

[0133] The input of the multi-head self-attention and convolution fusion encoding module is the above-mentioned enhanced acoustic feature vector. The design of this module is based on the Conformer architecture, combining the multi-head self-attention mechanism and the one-dimensional deep separable convolution module to model the global temporal dependency and local contextual information of the input features respectively. In the self-attention mechanism, the input features are divided into several attention heads, and each attention head independently calculates the temporal dependency between features. Specifically, the correlation score between a specific time step and other time steps is determined by calculating the dot product of the three vectors of query, key, and value, and the feature values ​​are weighted and combined. This step can effectively capture the pattern and dependency of the speech signal over a long time span. The convolution module uses a one-dimensional deep separable convolution to extract the local contextual information of the input features, and enhances the ability to capture short-term changing features by reducing the number of parameters and computational complexity. The two submodules are combined in a serial manner, that is, global temporal dependency modeling is performed first, and then the local features are optimized through the convolution module, so as to output a contextual encoding representation that contains both global and local information.

[0134] The dynamic phoneme prediction module takes the context encoding representation as input and is mainly used to generate the phoneme prediction distribution of the current time step. This module adopts a dynamic gating mechanism and introduces a gating unit based on the context encoding to dynamically model the phoneme distribution of each time step by adjusting the response of the input features. The gating unit generates weights based on the input context information and performs a weighted combination of features, so that the model can adaptively adjust the output of the prediction distribution according to the feature changes at different time steps. In scenarios with fast speech rates or complex signals, this dynamic adjustment mechanism can significantly improve the accuracy and robustness of the prediction results. The output of the module is a probabilistic phoneme prediction distribution, in which each dimension corresponds to a possible phoneme category and its value represents the probability that the current time step prediction belongs to that category.

[0135] The semantically enhanced decoding module based on the language model is responsible for combining the phoneme prediction distribution with the semantic embedding of the pre-trained language model to generate the final speech recognition text result. The input includes dynamic phoneme distribution and language model embedding, and the output is the text generated step by time. The module adopts a hybrid language model decoding strategy, taking the phoneme distribution as the initial input of the decoder and correcting its output probability through the embedding information of the language model. At each time step, the decoder dynamically adjusts the weight distribution of phoneme prediction and language model embedding through the self-attention mechanism, for example, according to the context consistency of the current time step, the contribution of the two is calculated in real time. The text results generated in this way are more semantically reasonable and coherent, especially in complex multi-intention speech scenarios, which can significantly improve the decoding quality.

[0136] In summary, this embodiment realizes an efficient, accurate and highly concurrent speech recognition system through modular design, using acoustic feature enhancement, global and local feature modeling, dynamic phoneme prediction and semantically enhanced decoding.

[0137] Furthermore, the dynamic phoneme prediction module implements dynamic modeling of phoneme distribution through the following steps:

[0138] Extract the current time step from the context encoded representation Local context features and global context features ; Local context features Represents the short-term feature pattern near the current time frame, which can be obtained by intercepting a fixed-length sequence from the encoded representation through a sliding window and performing convolution calculation; global context feature The long-term feature pattern representing the entire speech segment is usually extracted from the global encoding through weighted pooling or attention mechanism. These two features reflect the different time scale information of the speech signal.

[0139] According to the following formula 10, the dynamic feature vector is calculated :

[0140] ;

[0141] in, Represents the activation function, usually Sigmoid function or Tanh function is selected to limit the output range;

[0142] is the fusion weight matrix of local features; is the fusion weight matrix of the global features; both are trainable parameters, which control the fusion ratio of local and global features respectively and are obtained through training. Represents element-wise product. and The initial value of can adopt the standard Gaussian distribution, and the activation function can choose Sigmoid, which can achieve stable feature weighting.

[0143] The attention score for each time step is calculated according to the following formula 11:

[0144] ;

[0145] in, Represents the time step and The attention weight between The length of the eigenvector is used to scale the inner product value to maintain numerical stability. The recommended value is usually 64 or 128. is the time step The query vector is calculated using the following formula 12:

[0146] ;

[0147] in, is a trainable weight matrix; is the time step The key vector is calculated using the following formula 13:

[0148] ;

[0149] in, is a trainable weight matrix; the weight matrix and The Xavier initialization method can be used to initialize to ensure the convergence of the training process.

[0150] According to the following formula 14, the phoneme distribution of the current time step is calculated :

[0151] ;

[0152] in, is the output weight matrix, which is used to map the attention-weighted dynamic feature vector to the phoneme category space. The Softmax function converts the output into a probability distribution, where each dimension corresponds to the probability value of a phoneme category. The matrix The dimension of depends on the number of phoneme categories. For example, if the target phoneme set contains 40 categories, then The dimension of is [d,40]. The initial value of the weight matrix can adopt a random normal distribution.

[0153] The reference implementation code of the neural network model is as follows:

[0154] import torch

[0155] import torch.nn as nn

[0156] import torch.nn.functional as F

[0157] # Acoustic front-end feature enhancement module

[0158] class AcousticFeatureEnhancement(nn.Module):

[0159] def __init__(self, input_dim, cnn_hidden_dim):

[0160] super(AcousticFeatureEnhancement, self).__init__()

[0161] # Convolutional layers are used to extract local and global features

[0162] self.conv1 = nn.Conv1d(input_dim, cnn_hidden_dim, kernel_size=3, padding=1)

[0163] self.conv2 = nn.Conv1d(cnn_hidden_dim, cnn_hidden_dim, kernel_size=5, padding=2)

[0164] # Adaptive spectral filter

[0165] self.adaptive_filter = nn.Parameter(torch.randn(cnn_hidden_dim))

[0166] # Activation function

[0167] self.activation = nn.ReLU()

[0168] def forward(self, x):

[0169] # The input shape is (batch_size, seq_len, input_dim), which needs to be converted to (batch_size, input_dim, seq_len)

[0170] x = x.permute(0, 2, 1)

[0171] # Layered convolutions capture short-term and long-term patterns

[0172] x = self.activation(self.conv1(x))

[0173] x = self.activation(self.conv2(x))

[0174] # Adaptive spectrum enhancement

[0175] x = x self.adaptive_filter.unsqueeze(0).unsqueeze(-1)

[0176] # Restore the dimension to (batch_size, seq_len, cnn_hidden_dim)

[0177] x = x.permute(0, 2, 1)

[0178] return x

[0179] # Multi-head self-attention and convolution fusion encoding module

[0180] class ConformerBlock(nn.Module):

[0181] def __init__(self, input_dim, num_heads, conv_hidden_dim):

[0182] super(ConformerBlock, self).__init__()

[0183] # Multi-head self-attention mechanism

[0184] self.attention = nn.MultiheadAttention(input_dim, num_heads,batch_first=True)

[0185] # Depthwise Separable Convolution

[0186] self.depthwise_conv = nn.Conv1d(input_dim, conv_hidden_dim,kernel_size=3, padding=1, groups=input_dim)

[0187] self.pointwise_conv = nn.Conv1d(conv_hidden_dim, input_dim,kernel_size=1)

[0188] # Activation function and normalization

[0189] self.activation = nn.ReLU()

[0190] self.norm = nn.LayerNorm(input_dim)

[0191] def forward(self, x):

[0192] # Multi-head self-attention mechanism models global dependencies

[0193] attn_output, _ = self.attention(x, x, x)

[0194] x = self.norm(x + attn_output)

[0195] # One-dimensional depth-wise separable convolution captures local context information

[0196] conv_input = x.permute(0, 2, 1) # Convert to (batch_size, input_dim, seq_len)

[0197] conv_output = self.pointwise_conv(self.activation(self.depthwise_conv(conv_input)))

[0198] conv_output = conv_output.permute(0, 2, 1) # Restore to (batch_size, seq_len, input_dim)

[0199] x = self.norm(x + conv_output)

[0200] return x

[0201] # Dynamic phoneme prediction module

[0202] class PhonemePrediction(nn.Module):

[0203] def __init__(self, input_dim, phoneme_dim):

[0204] super(PhonemePrediction, self).__init__()

[0205] self.W_c = nn.Linear(input_dim, input_dim)

[0206] self.W_g = nn.Linear(input_dim, input_dim)

[0207] self.W_q = nn.Linear(input_dim, input_dim)

[0208] self.W_k = nn.Linear(input_dim, input_dim)

[0209] self.W_o = nn.Linear(input_dim, phoneme_dim)

[0210] def forward(self, context_local, context_global):

[0211] # Calculate the dynamic feature vector F_dyn(t)

[0212] F_dyn_local = torch.sigmoid(self.W_c(context_local)) context_local

[0213] F_dyn_global = torch.sigmoid(self.W_g(context_global)) context_global

[0214] F_dyn = F_dyn_local + F_dyn_global

[0215] # Compute query and key vectors

[0216] Q = self.W_q(F_dyn)

[0217] K = self.W_k(F_dyn)

[0218] # Calculate attention weights

[0219] d = Q.size(-1) # feature vector dimension

[0220] A = torch.softmax(torch.matmul(Q, K.transpose(-1, -2)) / torch.sqrt(torch.tensor(d)), dim=-1)

[0221] # Calculate phoneme distribution

[0222] F_weighted = torch.matmul(A, F_dyn)

[0223] P_phoneme = F.softmax(self.W_o(F_weighted), dim=-1)

[0224] return P_phoneme

[0225] # Semantic enhancement decoding module based on language model

[0226] class SemanticDecoder(nn.Module):

[0227] def __init__(self, phoneme_dim, embedding_dim, vocab_size):

[0228] super(SemanticDecoder, self).__init__()

[0229] self.phoneme_to_embedding = nn.Linear(phoneme_dim, embedding_dim)

[0230] self.language_model = nn.GRU(embedding_dim, embedding_dim,batch_first=True)

[0231] self.fc = nn.Linear(embedding_dim, vocab_size)

[0232] def forward(self, phoneme_distribution, lm_embedding):

[0233] # Mapping phoneme distribution to embedding space

[0234] phoneme_embedding = self.phoneme_to_embedding(phoneme_distribution)

[0235] # Dynamic decoding combined with language model embedding

[0236] combined_embedding = phoneme_embedding + lm_embedding

[0237] lm_output, _ = self.language_model(combined_embedding)

[0238] logits = self.fc(lm_output)

[0239] return logits

[0240] # Overall model packaging

[0241] class ConformerSpeechRecognition(nn.Module):

[0242] def __init__(self, input_dim, cnn_hidden_dim, num_heads, conv_hidden_dim, phoneme_dim, embedding_dim, vocab_size):

[0243] super(ConformerSpeechRecognition, self).__init__()

[0244] self.acoustic_module = AcousticFeatureEnhancement(input_dim,cnn_hidden_dim)

[0245] self.conformer_block = ConformerBlock(cnn_hidden_dim, num_heads, conv_hidden_dim)

[0246] self.phoneme_prediction = PhonemePrediction(conv_hidden_dim,phoneme_dim)

[0247] self.semantic_decoder = SemanticDecoder(phoneme_dim,embedding_dim, vocab_size)

[0248] def forward(self, x, lm_embedding):

[0249] # Acoustic front-end enhancement

[0250] enhanced_features = self.acoustic_module(x)

[0251] # Multi-head self-attention and convolution fusion

[0252] encoded_features = self.conformer_block(enhanced_features)

[0253] # Dynamic phoneme prediction

[0254] phoneme_distribution = self.phoneme_prediction(encoded_features, encoded_features)

[0255] # Semantic decoding based on language model

[0256] output = self.semantic_decoder(phoneme_distribution, lm_embedding)

[0257] return output

[0258] To train the neural network model, we need to start with data preparation, including processing audio files and text labels. The audio data is converted into acoustic feature vectors through Mel spectrum, and the text labels are converted into the target format. The training set, validation set, and test set are used to train, validate, and evaluate the model performance, respectively.

[0259] When initializing the model, randomly set the weights and bias values, and use padding to align the length of the input audio data. The language model part can load a pre-trained model or fine-tune it according to the task to improve performance.

[0260] During the training process, the model receives feature input and generates output text through each module. Use a loss function (such as cross entropy or CTC) to measure the difference between the output and the true label. Select an optimizer (such as Adam) to adjust the model parameters and dynamically adjust the learning rate to accelerate convergence. To improve robustness, data augmentation or noise enhancement can be introduced.

[0261] After each round of training, evaluate the model on the validation set and adjust hyperparameters if necessary. When the validation performance is stable, evaluate the model on the test set to ensure good application performance.

[0262] A second embodiment of the present application provides an electronic device, the electronic device comprising:

[0263] processor;

[0264] The memory is used to store a program. When the program is read and executed by the processor, it executes a speech recognition feature extraction and reasoning method for an artificial intelligence dialogue system provided in the first embodiment of the present application.

[0265] The third embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, a speech recognition feature extraction and reasoning method for an artificial intelligence dialogue system provided in the first embodiment of the present application is executed.

[0266] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A speech recognition streaming feature extraction and batch inference method for a high-concurrency artificial intelligence real-time dialogue system, characterized in that: include: In the CPU execution environment, receiving user voice data, and segmenting and preprocessing the user voice data according to a preset time window or audio frame size; Performing Mel filter bank feature extraction and cepstral mean variance normalization processing on the segmented and preprocessed data to obtain a standardized acoustic feature vector; storing the standardized acoustic feature vector in a feature buffer; In a GPU execution environment, feature vectors are extracted from the feature buffer in batches, and feature vectors of different lengths are padded to meet uniform input dimension requirements; The filled feature vector batches are sent to the neural network model for parallel reasoning calculation to obtain the speech recognition results; The receiving of user voice data and segmenting and preprocessing the user voice data according to a preset time window or audio frame size includes: The segment length is dynamically adjusted according to the real-time arrival speed of the speech segment, and the segment is immediately sent to the feature buffer after the segmentation is completed to achieve parallel feature extraction of the speech input that has not yet ended.

2. The method for speech recognition streaming feature extraction and batch reasoning for a high-concurrency artificial intelligence real-time dialogue system according to claim 1 is characterized in that: The feature buffer also includes a scheduling module for monitoring the number and length distribution of feature vectors in the buffer. When it is detected that the amount of feature vector data reaches a preset threshold, a batch inference task trigger signal is sent to the GPU, and the inference task load of the next batch of feature vectors is automatically increased when the GPU is idle, so as to improve the overall utilization of the GPU.

3. The speech recognition streaming feature extraction and batch reasoning method for a high-concurrency artificial intelligence real-time dialogue system according to claim 1 is characterized in that: After the GPU completes the inference calculation of the batch feature vectors, it includes the steps of caching and timing control of the inference results, so that when the user has not yet finished the voice input, some voice recognition results can be obtained and returned to the client in real time, thereby shortening the user's waiting time and ensuring the real-time performance of the system under high concurrent requests.

4. The speech recognition streaming feature extraction and batch reasoning method for a high-concurrency artificial intelligence real-time dialogue system according to claim 1 is characterized in that: Extracting feature vectors in batches from the feature buffer in the GPU execution environment includes: Based on the arrival time of feature vectors in the buffer, the criticality of feature vectors, and the current GPU load status, high-density data batches are prioritized and inference tasks of low-priority batches are delayed to optimize resource allocation and improve system throughput.

5. The speech recognition streaming feature extraction and batch reasoning method for a high-concurrency artificial intelligence real-time dialogue system according to claim 1 is characterized in that: The receiving user voice data comprises: By calculating the signal-to-noise ratio and speech clarity index of the signal, the quality of each speech segment is evaluated in real time, and the feature extraction parameters are dynamically adjusted according to the evaluation results to enhance the feature extraction capability of low-quality speech, thereby improving the overall speech recognition accuracy.

6. The speech recognition streaming feature extraction and batch reasoning method for a high-concurrency artificial intelligence real-time dialogue system according to claim 1 is characterized in that: The neural network model is based on the Conformer architecture, including an acoustic front-end feature enhancement module, a multi-head self-attention and convolution fusion encoding module, a dynamic phoneme prediction module, and a semantic enhancement decoding module based on a language model; The input of the acoustic front-end feature enhancement module is the standardized acoustic feature vector, and the output is the enhanced acoustic feature vector; the acoustic front-end feature enhancement module performs multi-scale analysis on the input features through a convolutional neural network and an adaptive spectrum filter, extracts the local spectrum pattern of the speech signal, and enhances the expression of the high-frequency signal; the convolutional neural network part adopts hierarchical convolution to capture short-term and long-term patterns, and the adaptive spectrum filter dynamically adjusts the enhancement coefficient to match the energy distribution of the input feature; The input of the multi-head self-attention and convolution fusion encoding module is the enhanced acoustic feature vector, and the output is the context encoding representation after the fusion of global and local features; the multi-head self-attention and convolution fusion encoding module is implemented by using the multi-head self-attention mechanism and convolution module of Conformer; the multi-head self-attention mechanism is used to model the global time dependency of the input features, and the input features are divided into multiple attention heads, each of which independently captures the global pattern in different time series; a one-dimensional depth-separable convolution module is used to capture the local context of the input features to enhance the expression ability of short-term information, wherein the multi-head attention mechanism and the convolution module are used in a serial manner; The input of the dynamic phoneme prediction module is the context encoding representation, and the output is the probabilistic phoneme prediction distribution; the dynamic phoneme prediction module uses a prediction network based on a dynamic gating mechanism to dynamically model the phoneme distribution of the current time step in combination with the context encoding information; The input of the language model-based semantic enhancement decoding module is the probabilistic phoneme prediction distribution and the pre-trained language model embedding representation, and the output is the speech recognition text result; the language model-based semantic enhancement decoding module adopts a hybrid language model decoding strategy to fuse the dynamic phoneme distribution with the embedding of the pre-trained language model; and at each time step, the weights of the phoneme distribution and the language model prediction are dynamically adjusted through the self-attention mechanism, so as to generate a text result with higher semantic consistency.

7. The method for speech recognition streaming feature extraction and batch reasoning for a high-concurrency artificial intelligence real-time dialogue system according to claim 6 is characterized in that: The dynamic phoneme prediction module implements dynamic modeling of phoneme distribution through the following steps: Extract the current time step from the context encoded representation Local context features and global context features ; According to the following formula 10, the dynamic feature vector is calculated : ； in, represents the activation function; is the fusion weight matrix of local features; is the fusion weight matrix of global features; represents element-wise product; The attention score for each time step is calculated according to the following formula 11: ； in, Represents the time step and The attention weight between is the scaling factor of the feature vector dimension; is the time step The query vector is calculated using the following formula 12: ； in, is a trainable weight matrix; is the time step The key vector is calculated using the following formula 13: ； in, is a trainable weight matrix; According to the following formula 14, the phoneme distribution of the current time step is calculated : ； in, is the output weight matrix.

8. The speech recognition streaming feature extraction and batch reasoning method for a high-concurrency artificial intelligence real-time dialogue system according to claim 1 is characterized in that: The cepstrum mean variance normalization process comprises the following steps: A preliminary weighted energy calculation is performed on the acoustic feature vector of the input speech segment signal to eliminate the interference of background noise and the influence of dynamic signal changes. The weighted energy calculation is implemented by the following formula 1: ； in, is the weighted energy value of the current frame, indicating the The energy intensity of the speech signal on ; It is Frame feature value; is the total number of frames of the feature vector; is the weight coefficient, calculated using the following formula 2: ； in, Indicates Frame to Time No. The contribution of the frame; is a parameter used to control weight decay; It is Frame feature value; It is the characteristic value of multiple frames of content in a specified time period The average value of It is the characteristic value of multiple frames of content in a specified time period The standard deviation of To prevent small positive numbers with zero denominator; To adjust the parameters; According to the following formula 3, the energy value of each frame is calculated Rate of change , to evaluate the energy fluctuation of the current frame compared to the context frame: ； in, is the average energy in a specified time period; is the energy standard deviation of the specified time period; is a small positive number that prevents the denominator from being zero; According to the following formula 4, calculate the weighting factor of the current frame : ； in, is the weighting factor; is the threshold of the rate of change, used to distinguish between the drastically changing frames and stable frames of the speech signal; According to the following formula 5, the feature vector is smoothed and corrected: ； in, Is the time frame The acoustic feature vector represents the short-time spectrum characteristics of the speech signal; is the smoothed corrected eigenvector; is the weighted mean of the context frame, calculated according to the following formula 6: ； in, It is Frame feature value; is the total number of frames of the feature vector; is the weight coefficient, calculated using Formula 2.

9. The method for speech recognition streaming feature extraction and batch reasoning for a high-concurrency artificial intelligence real-time dialogue system according to claim 1, characterized in that: Extracting feature vectors in batches from the feature buffer comprises the following steps: Monitors the number of feature vectors stored in the feature buffer And the length distribution of each eigenvector , in ; According to the following formula 7, calculate the optimal number of feature vectors for the current batch : ； in, is the candidate batch size; is the standard deviation of the feature vector lengths within the selected batch, calculated according to the following formula 8: ； in, is the average length of the feature vectors in the batch; Extracting the optimal batch size from the feature buffer , perform dynamic filling operation; The padding operation is based on the length of the longest feature vector in the batch. Determine the position and number of fill values. The fill value is calculated using the following formula 9 to reduce the damage to the continuity of feature data caused by fill: ； in, Is the filling position The value of This is the first eigenvector values; After padding, the input tensor is formed into a uniform size , and send it into the neural network model for parallel reasoning calculation.

Images