Multilingual speech recognition method and system based on generative learning model

Through the parallel processing of generative learning model and three-dimensional convolution kernel, combined with cross-layer jump connection and dynamic programming algorithm, the problem of insufficient information integration in multilingual speech recognition technology is solved, and the recognition accuracy and generalization ability are significantly improved.

CN118841000BActive Publication Date: 2024-11-26SHENZHEN BEIBO INTELLIGENT TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411315924.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-20
Publication Date
2024-11-26
Estimated Expiration
2044-09-20

AI Technical Summary

Technical Problem

Current multilingual speech recognition technology has defects in recognition accuracy and cannot effectively integrate time, frequency and semantic information, resulting in the loss of important features of encoded vectors and affecting the recognition accuracy.

Method used

Using a multilingual speech recognition method based on a generative learning model, the input data is processed in parallel through a three-dimensional convolution kernel, combined with cross-layer jump connections and dynamic programming algorithms and attention mechanisms, comprehensively capture time, frequency and semantic information, and dynamically adjust and fusion Weight, realize efficient information transmission and decoding.

Benefits of technology

It improves the accuracy of multilingual speech recognition, overcomes the defects of inaccurate recognition in traditional methods, can better adapt to the characteristics and speech changes of different languages, and improves the generalization ability and recognition performance of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118841000B_ABST
    Figure CN118841000B_ABST
Patent Text Reader

Abstract

The present invention provides a multilingual speech recognition method and system based on a generative learning model, including: extracting features from multilingual speech data to obtain a high-dimensional feature vector; inputting into a target coding model for deep coding processing to obtain a coding vector; the target coding model is a pre-trained generative learning model; based on a dynamic programming algorithm and an attention mechanism, the coding vector is decoded to obtain a corresponding multilingual text sequence as a multilingual speech recognition result; wherein the generative learning model includes a plurality of neural network layers, and cross-layer jump connections are constructed between each neural network layer; each neural network layer includes a three-dimensional convolution kernel and a fusion module, and the three-dimensional convolution kernel is a parallel time convolution kernel, a frequency convolution kernel, and a semantic association convolution kernel. In the present invention, the multi-dimensional key information of time, frequency, and semantics is fully captured to overcome the defect of inaccurate recognition of current multilingual speech recognition technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of speech recognition, and in particular to a multilingual speech recognition method and system based on a generative learning model. Background Art

[0002] At present, common speech recognition technologies mainly rely on a combination of general acoustic models and language models. In terms of encoding models, traditional encoding methods usually use a single-structure neural network, such as a simple feedforward neural network or a basic recurrent neural network. When processing multilingual speech data, these network structures cannot effectively integrate information in multiple key dimensions such as time, frequency, and semantics, resulting in the encoded vectors losing important features, affecting subsequent recognition accuracy. For the decoding process, most existing methods rely on decoding algorithms with fixed rules and lack effective attention to speech context information and dynamic changes. For example, traditional decoding methods based on statistical models are not flexible enough when faced with complex grammatical structures and semantic diversity in multiple languages, and it is difficult to accurately convert encoding vectors into correct multilingual text sequences.

[0003] Therefore, current multilingual speech recognition technology has defects in recognition accuracy. Summary of the invention

[0004] The main purpose of the present invention is to provide a multilingual speech recognition method and system based on a generative learning model, aiming to overcome the defect of inaccurate recognition of current multilingual speech recognition technology.

[0005] To achieve the above object, the present invention provides a multilingual speech recognition method based on a generative learning model, comprising the following steps:

[0006] Extract features from the collected multilingual speech data to obtain a high-dimensional feature vector with speech information features;

[0007] Inputting the high-dimensional feature vector into a target coding model for deep coding processing to obtain a coding vector; the target coding model is a pre-trained generative learning model;

[0008] Based on a dynamic programming algorithm and an attention mechanism, the encoding vector is decoded to obtain a corresponding multilingual text sequence as a multilingual speech recognition result;

[0009] Among them, the generative learning model includes multiple neural network layers, and cross-layer jump connections are constructed between each neural network layer; each neural network layer includes a three-dimensional convolution kernel and a fusion module, and the three-dimensional convolution kernel is a parallel time convolution kernel, a frequency convolution kernel and a semantic association convolution kernel; in each convolution operation, the three convolution kernels process the input data simultaneously, and the feature information obtained by the three convolution kernels is dynamically fused through the fusion module; the fusion module automatically learns and adjusts the fusion weight according to the importance of the processing results of each convolution kernel.

[0010] Furthermore, the cross-layer skip connection is a selective information transmission channel established between adjacent odd-numbered neural network layers and even-numbered neural network layers;

[0011] After the input data of one neural network layer is processed, the processing result is passed directly to the next layer or passed to a further layer across the intermediate layer according to the preset evaluation index;

[0012] If the evaluation results show that skip transfer is more conducive to model convergence, the processing results of this layer are directly transferred to the target layer through cross-layer skip connections; during the transfer process, an adaptive information enhancement mechanism is used to adjust the transferred processing results according to the needs of the target layer, so that the processing results can be effectively transferred between different layers.

[0013] Furthermore, the feature extraction of the collected multilingual speech data to obtain a high-dimensional feature vector having speech information features includes:

[0014] The collected multilingual speech data is framed and divided into a plurality of speech segments, each segment containing a speech signal of a preset time length, to obtain a plurality of speech frames;

[0015] Perform windowing processing on each speech frame, and use the Hamming window function to smooth the edge of the speech frame to obtain a windowed speech frame;

[0016] Perform fast Fourier transform on the windowed speech frame to convert it from the time domain to the frequency domain to obtain a frequency domain signal;

[0017] The Mel-frequency cepstral coefficient features are extracted from the frequency domain signal, and the frequency domain signal is passed through a set of Mel filter banks. Logarithmic operations and discrete cosine transforms are performed on the filtered signal to obtain a high-dimensional feature vector with speech information characteristics.

[0018] Furthermore, the feature extraction of the collected multilingual speech data to obtain a high-dimensional feature vector having speech information features includes:

[0019] The collected multilingual speech data is simultaneously input into a mel spectrum analyzer and a linear prediction coding analyzer for processing to obtain corresponding mel spectrum feature data and linear prediction coding feature data;

[0020] Automatically calculating a weighting coefficient of a Mel spectrum feature according to the speech characteristics of the language and the position information of the speech segment in the entire speech data, and multiplying the Mel spectrum feature data by the weighting coefficient to obtain enhanced Mel spectrum feature data;

[0021] The linear prediction coding feature data is optimized by using the K-Means clustering algorithm to obtain the optimized linear prediction coding feature data;

[0022] The enhanced Mel spectrum feature data and the optimized linear prediction coding feature data are fused to obtain a high-dimensional feature vector with speech information characteristics.

[0023] Furthermore, the encoding vector is decoded based on the dynamic programming algorithm and the attention mechanism to obtain a corresponding multilingual text sequence, including:

[0024] Using a pre-trained semantic segmentation model, the coding vector is preliminarily divided into a number of sub-vector segments with latent semantic units according to the potential distribution of semantic information in the coding vector, and a set of preliminarily segmented sub-vector segments is obtained;

[0025] For each sub-vector segment, a search graph with it as a node is constructed. The optimal path from the starting sub-vector segment to each possible ending sub-vector segment is calculated in the search graph according to the dynamic programming algorithm. The path combination with the minimum cost is found through search to obtain a preliminary decoding path sequence.

[0026] The preliminary decoding path sequence is optimized based on the multi-head attention mechanism to obtain the optimized decoding path sequence;

[0027] Convert the optimized decoding path sequence into the corresponding multilingual text sequence.

[0028] Furthermore, before extracting features from the collected multilingual speech data to obtain a high-dimensional feature vector having speech information features, the process includes:

[0029] Obtain multilingual reference speech pre-stored in the speech recognition device and the current encoding model;

[0030] Get the latest encoding model from the server;

[0031] Based on the current coding model and the latest coding model, encoding processing is performed on the multilingual reference speech respectively, and corresponding coding evaluation effects are obtained;

[0032] The coding model with better coding evaluation effect is used as the target coding model.

[0033] Further, the obtaining of the latest encoding model in the server includes:

[0034] Integrate and encode the hardware performance parameters of the speech recognition device and the current model parameters of the current encoding model, generate a device status code with a unique identifier, and send it to the server;

[0035] The server parses the device status code and uses a pre-trained adaptive model matching network to analyze the parsed hardware performance parameters and current model parameters; the adaptive model matching network is built based on a deep neural network and is used to customize and generate an optimized generative learning model configuration file for the speech recognition device;

[0036] The server sends the generated configuration file to the speech recognition device, and the speech recognition device updates the current encoding model according to the information in the configuration file to obtain the latest encoding model.

[0037] The present invention also provides a multilingual speech recognition system based on a generative learning model, comprising:

[0038] An extraction unit, used to extract features from the collected multilingual speech data to obtain a high-dimensional feature vector having speech information features;

[0039] An encoding unit, used for inputting the high-dimensional feature vector into a target encoding model for deep encoding processing to obtain an encoding vector; the target encoding model is a pre-trained generative learning model;

[0040] A recognition unit, configured to decode the encoding vector based on a dynamic programming algorithm and an attention mechanism to obtain a corresponding multilingual text sequence as a multilingual speech recognition result;

[0041] Among them, the generative learning model includes multiple neural network layers, and cross-layer jump connections are constructed between each neural network layer; each neural network layer includes a three-dimensional convolution kernel and a fusion module, and the three-dimensional convolution kernel is a parallel time convolution kernel, a frequency convolution kernel and a semantic association convolution kernel; in each convolution operation, the three convolution kernels process the input data simultaneously, and the feature information obtained by the three convolution kernels is dynamically fused through the fusion module; the fusion module automatically learns and adjusts the fusion weight according to the importance of the processing results of each convolution kernel.

[0042] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0043] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0044] The multilingual speech recognition method and system based on the generative learning model provided by the present invention include: extracting features from the collected multilingual speech data to obtain a high-dimensional feature vector with speech information features; inputting the high-dimensional feature vector into the target coding model for deep coding processing to obtain a coding vector; the target coding model is a pre-trained generative learning model; decoding the coding vector based on a dynamic programming algorithm and an attention mechanism to obtain a corresponding multilingual text sequence as a multilingual speech recognition result; wherein the generative learning model includes multiple neural network layers, and cross-layer jump connections are constructed between each neural network layer; each neural network layer includes a three-dimensional convolution kernel and a fusion module, and the three-dimensional convolution kernel is a parallel time convolution kernel, a frequency convolution kernel, and a semantic association convolution kernel; in each convolution operation, the three convolution kernels process the input data at the same time, and the feature information obtained by the three convolution kernels is dynamically fused through the fusion module; the fusion module automatically learns and adjusts the fusion weight according to the importance of the processing results of each convolution kernel. In the present invention, the input data is processed in parallel by the three-dimensional convolution kernel to fully capture the multi-dimensional key information of time, frequency and semantics. The fusion module dynamically adjusts the fusion weights according to the importance, and the cross-layer jump connections between layers efficiently transmit information. The dynamic programming algorithm and attention mechanism are combined for decoding, and the context association is fully considered to accurately convert the encoding vector into a multilingual text sequence, which improves the accuracy of multilingual speech recognition and overcomes the inaccurate recognition defects of current multilingual speech recognition technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a multilingual speech recognition method based on a generative learning model in one embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of the steps of a multilingual speech recognition method based on a generative learning model in one embodiment of the present invention;

[0047] Figure 3 is a structural block diagram of a multilingual speech recognition system based on a generative learning model in one embodiment of the present invention;

[0048] Figure 4 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0049] The implementation, functional features and advantages of the present invention will be further described in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] Reference Figure 1 as well as Figure 2 In one embodiment of the present invention, a multilingual speech recognition method based on a generative learning model is provided, comprising the following steps:

[0052] Step S1, extracting features from the collected multilingual speech data to obtain a high-dimensional feature vector having speech information features;

[0053] Step S2, inputting the high-dimensional feature vector into a target coding model for deep coding processing to obtain a coding vector; the target coding model is a pre-trained generative learning model;

[0054] Step S3, decoding the encoding vector based on a dynamic programming algorithm and an attention mechanism to obtain a corresponding multilingual text sequence as a multilingual speech recognition result;

[0055] Among them, the generative learning model includes multiple neural network layers, and cross-layer jump connections are constructed between each neural network layer; each neural network layer includes a three-dimensional convolution kernel and a fusion module, and the three-dimensional convolution kernel is a parallel time convolution kernel, a frequency convolution kernel and a semantic association convolution kernel; in each convolution operation, the three convolution kernels process the input data simultaneously, and the feature information obtained by the three convolution kernels is dynamically fused through the fusion module; the fusion module automatically learns and adjusts the fusion weight according to the importance of the processing results of each convolution kernel.

[0056] In this embodiment, as described in step S1 above, voice data is collected from a real multilingual environment, and these data cover a variety of situations such as various accents, speaking speeds, and contexts of different languages. Based on signal processing and feature extraction technology, the collected multilingual voice data is processed. First, the voice data is preprocessed, including noise reduction, framing and other operations to remove background noise and divide the continuous voice signal into small segments that are easy to process. Then, a feature extraction algorithm is used to extract a high-dimensional feature vector with voice information characteristics. In one embodiment, not only the frequency characteristics of the voice can be paid attention to, but also the dynamic change characteristics and semantic association characteristics in the time dimension can be deeply explored. For example, by analyzing the time series changes of the voice signal, features that can reflect the temporal characteristics of the voice rhythm, prosody, etc. are extracted; at the same time, advanced semantic analysis technology is used to extract features related to the voice semantics, so that the feature vector can contain rich voice information.

[0057] As described in step S2 above, the extracted high-dimensional feature vector is input into the target encoding model in the pre-trained generative learning model for processing. The target encoding model has a unique structure and function. It contains multiple neural network layers, each of which is composed of a three-dimensional convolution kernel and a fusion module. The three-dimensional convolution kernel includes a time convolution kernel, a frequency convolution kernel, and a semantic association convolution kernel, and these three convolution kernels work in parallel. The time convolution kernel focuses on capturing the change pattern and dynamic characteristics of the speech signal in the time dimension, such as time series information such as speech connection and pause. The frequency convolution kernel is responsible for analyzing the frequency distribution and spectral characteristics of the speech signal, and extracting information related to the pronunciation characteristics and audio frequency characteristics of different languages. The semantic association convolution kernel interacts with the pre-built semantic knowledge base to mine the semantic connotation and association information in the speech data, and closely combines the speech features with the semantic information. The fusion module automatically learns and dynamically adjusts the fusion weights according to the importance of the processing results of the three convolution kernels. In each convolution operation, the fusion module adjusts the fusion ratio of the output features of the three convolution kernels in real time according to the characteristics of the current input data, the training status of the model, and the requirements of the task. For example, when processing a speech rich in semantic information, the fusion module will automatically increase the weight of the output features of the semantic association convolution kernel to highlight the role of semantic features in encoding; when processing speech with a faster rhythm, the weight of the output features of the temporal convolution kernel will be appropriately increased to better capture subtle changes in time.

[0058] As described in step S3 above, the encoding vector is decoded based on the dynamic programming algorithm and the attention mechanism, and the encoded vector is converted into a corresponding multilingual text sequence. The above dynamic programming algorithm is used to search for the optimal decoding path. In the decoding process, a search space is constructed based on the characteristics of the encoding vector and multilingual grammatical rules, language models and other information. In this search space, the dynamic programming algorithm calculates the cost of each possible decoding path and finds the path with the minimum cost as the optimal decoding result. The design of the cost function comprehensively considers various factors, such as the semantic coherence of the speech, the grammatical rationality, and the degree of matching with the language model. For example, when selecting a word as part of the decoding result, the algorithm considers whether the word is semantically coherent with the previously decoded part, whether it conforms to the grammatical rules of the current language, and the probability of occurrence in the statistical language model. The above attention mechanism plays a key role in the decoding process. It can dynamically focus on the key parts of the input data based on the current decoding state and the information in the encoding vector. For example, when decoding a sentence, the attention mechanism can automatically adjust the degree of attention to different positions in the encoding vector based on the part of the text that has been generated. When generating a new word, the attention mechanism focuses on the information in the encoding vector that is relevant to the word, thereby improving the accuracy and coherence of decoding.

[0059] In this embodiment, traditional speech recognition methods usually only focus on single-dimensional features of speech, such as frequency features or simple time features, while the three-dimensional convolution kernel in this solution can comprehensively analyze and process speech data from three key dimensions: time, frequency, and semantics. The parallel working mode of the time convolution kernel, the frequency convolution kernel, and the semantic association convolution kernel enables the model to deeply explore different aspects of speech data at the same time and fully capture the complex information in multilingual speech. This parallel processing method improves the processing efficiency and information extraction ability of the model, and can more comprehensively and accurately characterize the essential characteristics of speech compared to traditional serial processing or single-dimensional processing methods. Thereby better adapting to the characteristics of different languages ​​and improving the recognition accuracy of multilingual speech.

[0060] In traditional multi-feature fusion methods, fixed weight distribution or simple linear weighting is usually used, which cannot be dynamically adjusted according to the changes in input data and the requirements of tasks. The fusion module in this solution can perceive the characteristics of input speech data and the training status of the model in real time by using advanced machine learning algorithms, and automatically adjust the fusion ratio of the output features of the three convolution kernels. For example, when the model processes a speech with strong emotional color, the emotional semantic features obtained by the semantic association convolution kernel become more important, and the fusion module will automatically increase its weight so that the encoding result focuses more on the expression of emotional information; when processing some speech with stable rhythm and clear pronunciation, the output features of the time convolution kernel and the frequency convolution kernel are relatively more critical, and the fusion module will adjust the weight accordingly to highlight the role of time and frequency features in encoding. This adaptive weight adjustment mechanism enables the model to adapt to various types of multilingual speech data more flexibly, improving the generalization ability and recognition performance of the model.

[0061] Cross-layer skip connections are constructed between the neural network layers of the generative learning model, breaking the limitation of traditional neural networks that transmit information sequentially layer by layer. Cross-layer skip connections enable information to be directly transmitted between neural network layers at different levels, achieving efficient information circulation and fusion. This connection method helps the model better capture long-distance dependencies and global features in speech data, avoiding information loss and distortion during the layer-by-layer transmission process. For example, when processing a multilingual speech with complex semantic structure, the bottom neural network layer extracts the basic acoustic features of the speech, while the high-level neural network layer is responsible for processing semantic and grammatical information. Through cross-layer skip connections, the high-level semantic information can directly affect the feature extraction process of the bottom layer, so that the bottom-level convolution kernel can better extract features related to the current task according to the semantic guidance of the high-level; at the same time, the original features of the bottom layer can also be quickly transmitted to the high-level layer, providing a richer information basis for the semantic analysis of the high-level layer. This can greatly improve the model's ability to understand and recognize multilingual speech, especially when processing speech data with complex structure and semantics, which can significantly improve the performance of the model.

[0062] The above dynamic programming algorithm can comprehensively consider multiple factors such as the semantic coherence, grammatical rationality and matching degree of the language model of multilingual speech when searching for the decoding path, and find the optimal decoding result. Compared with the traditional decoding method based on rules or simple statistics, this decoding method based on global optimization can better handle the complex grammatical structure and semantic diversity in multilingual speech, and improve the accuracy and reliability of decoding. The introduction of the attention mechanism makes the decoding process more flexible and intelligent. It can dynamically focus on the key parts of the input data according to the current decoding state and the information in the encoding vector, thereby improving the efficiency and accuracy of decoding. For example, when decoding a multilingual sentence, the attention mechanism can automatically adjust the degree of attention to different positions in the encoding vector based on the knowledge of the generated partial text and language model, focusing on the information related to the current decoding task, and avoiding the processing and interference of irrelevant information. At the same time, the attention mechanism can also help the model better handle the problems of vocabulary translation and semantic alignment in multilingual, and improve the quality and effect of multilingual speech recognition.

[0063] In one embodiment, the cross-layer skip connection is a selective information transmission channel established between adjacent odd-numbered neural network layers and even-numbered neural network layers;

[0064] After the input data of one neural network layer is processed, the processing result is passed directly to the next layer or passed to a further layer across the intermediate layer according to the preset evaluation index;

[0065] If the evaluation results show that skip transfer is more conducive to model convergence, the processing results of this layer are directly transferred to the target layer through cross-layer skip connections; during the transfer process, an adaptive information enhancement mechanism is used to adjust the transferred processing results according to the needs of the target layer, so that the processing results can be effectively transferred between different layers.

[0066] In this embodiment, the cross-layer jump connection is precisely established between the adjacent odd-numbered neural network layers and the even-numbered neural network layers. In the neural network model of multilingual speech recognition, the odd-numbered layers and the even-numbered layers often have different feature processing characteristics and functional emphases. For example, the odd-numbered layers focus more on the preliminary time feature extraction and basic frequency analysis of the speech data, while the even-numbered layers play a key role in semantic association and high-level feature integration. By establishing cross-layer jump connections between such adjacent layers, direct interaction and information transmission of different types of feature processing results can be achieved, breaking the limitation of traditional neural networks that transmit information layer by layer in sequence. This connection mode enables information to flow quickly between network layers with different functional emphases, promoting the fusion and collaborative processing of multi-dimensional feature information such as time, frequency and semantics. For example, when processing a multilingual speech containing complex semantics and rapid speech rate changes, the time feature information extracted by the odd-numbered layers can be directly transmitted to the even-numbered layers responsible for semantic processing through cross-layer jump connections, helping the even-numbered layers to better understand the association between the changes in speech in the time dimension and semantics, thereby improving the model's comprehensive understanding of multilingual speech.

[0067] The above-mentioned cross-layer jump connection is not a simple direct connection, but a selective information transmission channel. This means that in the process of information transmission, intelligent decisions will be made on the information transmission path based on preset evaluation indicators. This selective design is one of the core innovations of this technical solution, which enables the model to dynamically select the optimal information transmission method based on the current input data characteristics and the training status of the model. The preset evaluation indicators are a set of preset algorithm rules that comprehensively consider multiple key factors, such as the convergence speed of the model, the importance and relevance of information, and the feature differences between different layers. For example, the evaluation indicators are quantitatively evaluated based on factors such as the amount of information of the processing results of the current neural network layer, the degree of match with the information required by the target layer, and the degree of impact on the overall performance of the model. In this way, the model can automatically determine whether to pass the processing results directly to the next layer for conventional layer-by-layer processing each time the information is transmitted, or to choose to pass it directly to the farther layer across the intermediate layer to achieve more efficient information utilization.

[0068] When the evaluation results show that skipping is more conducive to model convergence, the system will decisively pass the processing results of the layer directly to the target layer through cross-layer jump connections. Model convergence is one of the key indicators to measure the training effect and performance of the model. Fast and stable convergence can enable the model to reach the optimal state faster, improve training efficiency and recognition accuracy. For example, during the model training process, if it is found that the parameter update of some intermediate layers has no obvious effect on the overall performance of the model, or even leads to overfitting or performance degradation, and directly skipping these intermediate layers to pass information to a specific target layer can effectively improve the training effect and convergence speed of the model, then skipping will be chosen. This decision-making mechanism allows the model to automatically adjust the information transmission path when facing complex multilingual voice data, find the optimal model parameter configuration at the fastest speed, and achieve efficient speech recognition.

[0069] In the process of information transmission of cross-layer jump connections, the adaptive information enhancement mechanism is used to ensure that the transmitted processing results can better meet the needs of the target layer. Since different neural network layers have different functions and feature processing methods, the target layer has specific requirements for the format, scale and feature representation of the input information. Therefore, the directly transmitted processing results need to be properly adjusted and optimized to better integrate and work with the target layer. For example, the target layer requires the input information to have a specific scale or intensity, or to highlight certain key features. If the transmitted processing results do not meet these requirements, it will affect the processing effect of the target layer and the overall performance of the model. Therefore, the introduction of the information enhancement mechanism can effectively solve this problem and ensure the effective transmission and utilization of information between different layers.

[0070] In one embodiment, the feature extraction of the collected multilingual speech data to obtain a high-dimensional feature vector having speech information features includes:

[0071] The collected multilingual speech data is framed and divided into a plurality of speech segments, each segment containing a speech signal of a preset time length, to obtain a plurality of speech frames;

[0072] Perform windowing processing on each speech frame, and use the Hamming window function to smooth the edge of the speech frame to obtain a windowed speech frame;

[0073] Perform fast Fourier transform on the windowed speech frame to convert it from the time domain to the frequency domain to obtain a frequency domain signal;

[0074] The Mel-frequency cepstral coefficient features are extracted from the frequency domain signal, and the frequency domain signal is passed through a set of Mel filter banks. Logarithmic operations and discrete cosine transforms are performed on the filtered signal to obtain a high-dimensional feature vector with speech information characteristics.

[0075] In this embodiment, the collected multilingual speech data is a continuous signal stream, and it is very difficult to analyze and process it directly. Therefore, dividing it into multiple speech segments, that is, performing frame processing, is the first key operation of feature extraction. In this way, the continuous speech signal can be converted into a series of relatively independent small units, each unit containing a speech signal of a preset time length, which is convenient for subsequent detailed analysis. Different languages ​​have differences in pronunciation speed, rhythm and rhythm, etc., and considering the short-term stability assumption of speech signals, it is necessary to determine a suitable time length based on a large amount of multilingual speech data statistical analysis. For example, for languages ​​with a faster speaking speed, the frame length can be appropriately shortened to capture more subtle speech changes; for languages ​​with a slower speaking speed, a slightly longer frame length can be selected to ensure that sufficient speech information is contained.

[0076] The frame processing adopts a sliding window method to intercept speech segments of preset time length in sequence according to a certain step length on the time axis to obtain multiple speech frames. This method can ensure that there is a certain overlap between adjacent speech frames, thereby avoiding the loss of important speech information at the boundary. For example, when the pronunciation of a word spans two adjacent speech frames, due to the existence of overlapping parts, the complete pronunciation information of the word can be continued in the adjacent frames, which is conducive to the subsequent accurate extraction of the pronunciation features of the word. Through frame processing, the complex continuous speech signal is converted into a series of discrete speech frames, so that the subsequent processing algorithm can analyze each speech frame separately more efficiently, greatly reducing the computational complexity, and also laying the foundation for further extraction of various features.

[0077] The purpose of windowing each speech frame is to reduce spectrum leakage and improve the accuracy of frequency domain analysis. Among the many window functions, it is of great significance to choose the Hamming window function. The Hamming window function is a window function with smooth transition characteristics, which shows good performance in speech signal processing. When smoothing the edge of the speech frame, the Hamming window function can gradually reduce the signal amplitude at the edge, so that it can achieve a smooth transition with the adjacent frame or background noise. It can avoid spectrum leakage caused by sudden truncation of the signal when performing frequency domain analysis such as fast Fourier transform. Spectrum leakage will affect the accuracy of frequency domain features, and then affect the subsequent feature extraction and speech recognition effect. By using the Hamming window function, this effect can be effectively reduced and the quality of frequency domain signals can be improved.

[0078] The speech signal in the time domain represents the change of the signal over time, while in actual speech processing, the frequency domain features play a vital role in speech recognition and analysis. Different languages ​​have different feature distributions in the frequency domain. For example, the frequency components of vowels and consonants in different languages ​​are different, and changes in intonation are also reflected in the frequency domain. Therefore, converting speech frames from the time domain to the frequency domain is a key step in extracting representative features.

[0079] Fast Fourier Transform (FFT) is an efficient algorithm for converting time domain signals into frequency domain signals. Through FFT, the signal of the speech frame can be converted from the time dimension to the frequency dimension to obtain the frequency domain signal, so that the frequency composition and distribution of the speech signal can be analyzed more intuitively.

[0080] The implementation process of FFT is based on the mathematical principle of Fourier transform, which decomposes the signal of the speech frame into the superposition of sine and cosine waves of different frequencies. By calculating the amplitude and phase of these sine and cosine waves at different frequencies, the spectrum information of the frequency domain signal is obtained. The obtained frequency domain signal provides the basis for subsequent feature extraction. In the frequency domain, the frequency characteristics of the speech signal can be more easily analyzed, such as extracting frequency peaks, bandwidth and other information. These frequency domain features are of great value in distinguishing different speech pronunciations, recognizing the rhythm and intonation of the language, etc. At the same time, the high efficiency of FFT makes it possible to perform frequency domain conversion on a large number of speech frames in a short time, meeting the needs of real-time speech processing.

[0081] The above-mentioned Mel Frequency Cepstral Coefficient (MFCC) is a feature representation method applied to speech processing. When extracting MFCC features, the frequency domain signal is first passed through a set of Mel filter banks. The Mel filter bank is a set of bandpass filters designed based on the auditory characteristics of the human ear. The human ear's perception of sound is not linear in frequency, but follows the Mel scale. The Mel filter bank divides the frequency domain signal according to the Mel scale to simulate the difference in sensitivity of the human ear to sounds of different frequencies. This can better capture the speech features related to human auditory perception and improve the accuracy and robustness of speech recognition. The design of the Mel filter bank usually includes multiple filters, each of which covers a certain frequency range on the Mel frequency axis. The bandwidth and center frequency of the filter are determined according to the distribution of the Mel scale, so that the filter bank can perform targeted filtering processing on signals in different frequency regions.

[0082] The purpose of performing logarithmic operations on the filtered signal is to simulate the human ear's perception of sound intensity. The human ear's perception of sound intensity is approximately logarithmic within a certain range, so performing logarithmic operations on the filtered signal can make the features more consistent with the human ear's auditory perception model.

[0083] The discrete cosine transform (DCT) is then performed to convert the signal after filtering and logarithmic operation into the cepstrum domain. DCT can concentrate the energy of the signal on a few coefficients and has good decorrelation and energy compression characteristics. The MFCC feature coefficients obtained by DCT have low correlation and high information content, and can effectively represent the characteristics of the speech signal.

[0084] The final high-dimensional feature vector with speech information characteristics contains multiple MFCC coefficients, which reflect the characteristic information of speech signals in different frequency ranges and time. For example, the first few MFCC coefficients usually reflect the overall energy distribution and low-frequency characteristics of speech, while the subsequent coefficients can represent the detailed characteristics and high-frequency information of speech. These high-dimensional feature vectors can be used as input for subsequent speech recognition and analysis algorithms, providing a rich information foundation for accurately recognizing multilingual speech.

[0085] In one embodiment, the feature extraction of the collected multilingual speech data to obtain a high-dimensional feature vector having speech information features includes:

[0086] The collected multilingual speech data is simultaneously input into a mel spectrum analyzer and a linear prediction coding analyzer for processing to obtain corresponding mel spectrum feature data and linear prediction coding feature data;

[0087] Automatically calculating a weighting coefficient of a Mel spectrum feature according to the speech characteristics of the language and the position information of the speech segment in the entire speech data, and multiplying the Mel spectrum feature data by the weighting coefficient to obtain enhanced Mel spectrum feature data;

[0088] The linear prediction coding feature data is optimized by using the K-Means clustering algorithm to obtain the optimized linear prediction coding feature data;

[0089] The enhanced Mel spectrum feature data and the optimized linear prediction coding feature data are fused to obtain a high-dimensional feature vector with speech information characteristics.

[0090] In this embodiment, the collected multilingual speech data is simultaneously input into the Mel spectrum analyzer and the linear prediction coding analyzer for processing, which is an innovative method for multimodal feature extraction. The Mel spectrum analyzer is mainly used to extract the spectral features of the speech signal at the Mel frequency scale, while the linear prediction coding analyzer focuses on analyzing the time domain correlation and vocal tract model parameters of the speech signal. By using these two analyzers at the same time, the feature information of the speech data can be fully captured from different angles. Different languages ​​have their own unique speech characteristics, such as pronunciation, intonation, speaking speed, etc. Mel spectrum features are very effective in describing the frequency distribution and timbre characteristics of speech, while linear prediction coding features can reflect the generation mechanism and dynamic changes of speech. Simultaneous processing can obtain rich multidimensional features at an early stage, laying the foundation for subsequent accurate analysis and recognition.

[0091] Automatically calculating the weighting coefficient of the Mel-spectrogram feature based on the phonetic characteristics of the language and the position information of the speech segment in the entire speech data is a key step. The phonetic characteristics of a language include the pronunciation characteristics of vowels and consonants in different languages, the changing rules of tones, etc. For example, for some tonal languages, the change of tones is crucial to the expression of semantics. Therefore, when processing the speech data of these languages, the weighting coefficient can be adjusted according to the characteristics of the tones to highlight the Mel-spectrogram features related to the tones.

[0092] The position information of the speech segment in the entire speech data also has an important impact on the calculation of the weighting coefficient. For example, in a sentence, the beginning and end of the speech segment may have special contextual information, or in a continuous speech, some key speech segments contain important key words. Adjusting the weighting coefficient according to this position information can make the model pay more attention to the Mel spectrum features of these key parts, improving the expressiveness and discrimination of the features.

[0093] The Mel-spectrogram feature data is multiplied by the weighting coefficient to obtain enhanced Mel-spectrogram feature data. This method can dynamically adjust the strength and importance of the Mel-spectrogram feature according to different languages ​​and location information. By enhancing key features, the model can pay more attention to these representative information in subsequent processing and improve the recognition accuracy of speech in different languages. For example, when processing speech data with dialect accents, by enhancing the Mel-spectrogram features related to dialect characteristics, these special speech patterns can be better adapted and recognized.

[0094] Using K-Means clustering algorithm to optimize linear prediction coding feature data is a data processing method. K-Means clustering algorithm is an unsupervised learning algorithm that can divide linear prediction coding feature data into different clusters according to the similarity of data. In speech processing, linear prediction coding features reflect the time domain characteristics and vocal tract model parameters of speech signals. The clustering algorithm can cluster speech segments with similar features. The clustering process is based on the distance metric between features. The algorithm first randomly selects some initial cluster centers, and then assigns each linear prediction coding feature data point to the cluster with the nearest cluster center. Then, the center of each cluster is recalculated and the position of the cluster center is updated. Through multiple iterations, until the cluster center no longer changes significantly or the preset number of iterations is reached.

[0095] After being optimized by the K-Means clustering algorithm, the optimized linear prediction coding feature data is better organized and representative. Different clusters can correspond to different speech patterns or pronunciation types, such as different speakers, different speaking speeds, different emotional states, etc. In this way, in subsequent processing, the linear prediction coding features can be analyzed and utilized in a targeted manner according to the clustering results to improve the quality and availability of the features. The optimized features can better adapt to the changes and diversity of speech in a multilingual environment. The linear prediction coding features of different languages ​​have different distributions and characteristics. The clustering algorithm can classify similar features into one category, which is convenient for the model to better understand and process speech data in different languages.

[0096] The fusion of enhanced Mel spectrum feature data and optimized linear prediction coding feature data is to integrate two different types of features to obtain a high-dimensional feature vector with richer speech information features. The fusion method can be adopted in many ways, such as simple splicing fusion, neural network-based fusion, etc. In this technical solution, a suitable fusion method can be selected according to specific application requirements and data characteristics. For example, a fusion network can be used to take the enhanced Mel spectrum feature data and the optimized linear prediction coding feature data as input, and automatically learn the best fusion method and weight distribution between the two features through network learning and training. The fusion network can contain multiple hidden layers, and the two features are deeply fused through nonlinear transformation and weight adjustment to generate a high-dimensional feature vector with stronger expressiveness and discrimination. The obtained high-dimensional feature vector with speech information features contains rich information from both Mel spectrum and linear prediction coding, and has been enhanced and optimized. This high-dimensional feature vector can better describe the essential characteristics of multilingual speech data and provide more powerful support for subsequent tasks such as speech recognition, classification, and sentiment analysis.

[0097] In one embodiment, the decoding process of the encoding vector based on the dynamic programming algorithm and the attention mechanism to obtain the corresponding multilingual text sequence includes:

[0098] Using a pre-trained semantic segmentation model, the coding vector is preliminarily divided into a number of sub-vector segments with latent semantic units according to the potential distribution of semantic information in the coding vector, and a set of preliminarily segmented sub-vector segments is obtained;

[0099] For each sub-vector segment, a search graph with it as a node is constructed. The optimal path from the starting sub-vector segment to each possible ending sub-vector segment is calculated in the search graph according to the dynamic programming algorithm. The path combination with the minimum cost is found through search to obtain a preliminary decoding path sequence.

[0100] The preliminary decoding path sequence is optimized based on the multi-head attention mechanism to obtain the optimized decoding path sequence;

[0101] Convert the optimized decoding path sequence into the corresponding multilingual text sequence.

[0102] In this embodiment, the pre-trained semantic segmentation model is trained with a large amount of multilingual text and its corresponding encoding vector data, and can deeply understand and analyze the potential distribution of semantic information in the encoding vector. By learning from a large amount of data, the model has mastered the expression, structure and potential rules of semantics in different languages.

[0103] When faced with an input encoding vector, the model can preliminarily divide the encoding vector into several sub-vector segments with potential semantic units based on its inherent semantic structure and features. These sub-vector segments represent different semantic fragments or concepts in the encoding vector, providing a preliminary semantic framework for subsequent detailed decoding. For example, when processing a multilingual speech encoding vector containing multiple lexical and grammatical structures, the semantic segmentation model can divide it into sub-vector segments corresponding to different words or phrases based on the semantic associations and grammatical rules between the lexical elements.

[0104] The set of sub-vector segments obtained after the preliminary segmentation provides a more refined processing unit for the subsequent decoding step. Compared with directly decoding the entire encoding vector, dividing it into sub-vector segments can better adapt to the complex semantic structure and grammatical rules in multiple languages. Different languages ​​have different word orders, vocabulary combinations and semantic expressions. Through preliminary segmentation, each sub-vector segment can be processed separately according to the characteristics of each language, which improves the accuracy and adaptability of decoding. Preliminary segmentation also helps to reduce the search space and computational complexity in the decoding process. If the entire encoding vector is decoded directly, it is necessary to find the optimal decoding path in a very large search space, which will be very time-consuming and difficult. By dividing it into sub-vector segments, the complex decoding problem can be decomposed into multiple relatively small sub-problems, each sub-problem corresponding to the decoding of a sub-vector segment, thereby greatly improving the efficiency of decoding.

[0105] Constructing a search graph with each sub-vector segment as a node is a key step in the application of the dynamic programming algorithm. Each node in the search graph represents a sub-vector segment, and the connection between the nodes represents the conversion or combination relationship from one sub-vector segment to another. These connections can be determined based on factors such as the grammatical rules, semantic associations, and statistical information of the language. For example, when constructing a search graph for English, according to the grammatical structure and collocation rules of English, certain sub-vector segments are more likely to be adjacent to or combined with specific other sub-vector segments. By constructing such a search graph, the decoding problem can be transformed into a problem of finding the optimal path in the graph, thereby taking advantage of the dynamic programming algorithm for efficient solution.

[0106] The optimal path from the starting sub-vector segment to each possible ending sub-vector segment is calculated in the search graph according to the dynamic programming algorithm. The dynamic programming algorithm is a mathematical method for solving the optimization problem of multi-stage decision-making process. Its core idea is to improve the computational efficiency by decomposing complex problems into multiple overlapping sub-problems and saving the solutions of the sub-problems to avoid repeated calculations. In the decoding process, the dynamic programming algorithm starts from the starting sub-vector segment and gradually calculates the optimal path cost to each intermediate node and the ending sub-vector segment. The cost function of the path comprehensively considers multiple important factors, such as the semantic coherence between sub-vector segments, the conformity of language grammatical rules, and the degree of matching with the pre-built multilingual language model. Through continuous iteration and updating, the path combination with the minimum cost is found, that is, the preliminary decoding path sequence is obtained. The advantage of this method is that it can comprehensively consider various factors and is efficient and accurate in searching for the optimal path. Compared with blind search or simple greedy algorithms, the dynamic programming algorithm can ensure that the global optimal decoding path is found, especially when processing complex multilingual speech data, it can better adapt to the characteristics and grammatical structures of different languages ​​and improve the quality of decoding.

[0107] The multi-head attention mechanism described above is an advanced deep learning technique that plays a key role in optimizing the decoding path sequence. It consists of multiple independent attention heads, each of which can be regarded as an independent information processor that can focus on different aspects or features of the input data. When optimizing the preliminary decoding path sequence, the multi-head attention mechanism first takes the decoding path sequence and the encoding vector as input. Each attention head determines the degree of attention to different parts of the encoding vector at the current position by calculating its own attention weight distribution. These attention weights reflect the relevance and importance of each element in the encoding vector to the current decoding position. For example, one attention head focuses on the temporal information in the encoding vector, while another attention head focuses on the semantic information.

[0108] Then, according to the calculated attention weights, the information in the encoded vector is weighted and summed to obtain a re-adjusted and integrated feature representation. This feature representation contains the most relevant information for the current decoding position, providing more accurate and rich context information for the next step of decoding.

[0109] The process of optimizing the preliminary decoding path sequence based on the multi-head attention mechanism is an iterative process. At each decoding position, the multi-head attention mechanism dynamically adjusts the attention weight according to the current state and input information, and continuously optimizes the decoding path sequence. Through the collaborative work of multiple attention heads, the decoding path can be corrected and improved from different angles. The effect of the optimization is mainly reflected in the improvement of the accuracy and coherence of the decoding path sequence. The multi-head attention mechanism can better capture long-distance dependencies and contextual information, avoiding the local optimal solution and information loss problems that are prone to occur in traditional decoding methods. For example, when processing a multilingual sentence with a complex clause structure, the multi-head attention mechanism can accurately determine the position and meaning of each word based on the contextual information, thereby improving the accuracy of decoding. At the same time, by dynamically paying attention to and adjusting the information at different positions, the decoding path sequence is made more coherent and natural, in line with the expression habits of the language.

[0110] Finally, the optimized decoding path sequence is converted into the corresponding multilingual text sequence. This step mainly involves mapping each node or element in the decoding path sequence into actual language text. According to the pre-established mapping relationship between the encoding vector and the multilingual text, as well as the vocabulary and grammatical rules of the language, each sub-vector segment in the decoding path sequence is converted into a corresponding word, phrase or symbol. In the conversion process, many factors need to be considered, such as the encoding method of the language, the polysemy and ambiguity of the vocabulary, etc. For example, for some words with multiple meanings, their correct meaning needs to be determined based on contextual information and semantic analysis. At the same time, it is also necessary to make appropriate adjustments and combinations according to the grammatical rules of the language to ensure that the converted multilingual text sequence conforms to the grammatical norms and semantic logic. The final multilingual text sequence is the output result of the entire decoding process and has important application value. In the fields of multilingual speech recognition, translation, voice interaction, etc., accurate multilingual text sequences can realize information exchange and understanding between different languages. For example, in the intelligent voice assistant application, the user's multilingual voice instructions can be converted into understandable text sequences, so as to perform corresponding operations or provide accurate answers.

[0111] In one embodiment, before extracting features from the collected multilingual speech data to obtain a high-dimensional feature vector having speech information features, the process includes:

[0112] Obtain multilingual reference speech pre-stored in the speech recognition device and the current encoding model;

[0113] Get the latest encoding model from the server;

[0114] Based on the current coding model and the latest coding model, encoding processing is performed on the multilingual reference speech respectively, and corresponding coding evaluation effects are obtained;

[0115] The coding model with better coding evaluation effect is used as the target coding model.

[0116] In this embodiment, the multilingual reference speech pre-stored in the speech recognition device is the key reference data for evaluating the performance of the coding model. The above-mentioned reference speech usually covers various typical pronunciations, accents, speech speeds and contexts of multiple languages. It represents the common speech patterns and features of different languages, and provides a basic comparison standard for subsequent coding model evaluation. For example, it can include daily conversation fragments, news broadcast speech, literary works recitation, etc. in different languages, covering a wide range of language expressions and content types. By pre-storing these reference speech, it can be used for evaluation and comparison of coding models at any time at different times and scenes, without the need to re-acquire the original data every time, thereby improving efficiency.

[0117] On the one hand, the current encoding model in the speech recognition device is obtained. This is the encoding model that the device is already using or has loaded, which is a previously trained or updated version. Understanding the structure, parameters, and performance characteristics of the current encoding model is crucial to evaluating its differences and advantages and disadvantages with the latest encoding model.

[0118] On the other hand, the latest encoding model is obtained from the server. The server is usually a central node that centrally stores and manages various data and models, and can continuously receive new model updates, improvements and optimizations. By connecting to the server, the speech recognition device can obtain the latest encoding model in a timely manner to maintain its advancement and adaptability in multilingual speech recognition capabilities.

[0119] In this embodiment, the current coding model and the latest coding model are used to encode the multilingual reference speech. The coding process is to convert and compress the features of the reference speech according to the structure and algorithm of the model to obtain the corresponding coding representation. Different coding models have different structures and parameters, so different results will be produced when encoding the same reference speech. For example, some coding models focus more on capturing the temporal features of speech, while other models focus more on the encoding of semantic information. By encoding the reference speech, it can be converted from the original speech signal to the coding vector or feature representation inside the model for subsequent performance evaluation.

[0120] In order to determine which encoding model is better, the encoding results need to be evaluated. The evaluation indicators usually include multiple aspects, such as encoding accuracy, adaptability to different languages, degree of retention of speech features, encoding efficiency, etc. Accuracy can be measured by comparing the encoded results with the real annotations or known semantic information of the reference speech. If the encoding results can accurately reflect the content and features of the reference speech, then the model has high accuracy. Adaptability to different languages ​​can be tested on reference speech in multiple languages ​​to observe whether the model is stable and effective in different language environments. The degree of retention of speech features can be measured by analyzing whether the encoded features can fully retain the key information in the original speech, such as intonation, speech speed, emotion, etc. The efficiency of encoding can be evaluated by calculating the time and computing resources required for the encoding process. For example, some quantitative indicators such as accuracy, recall, F1 value and other statistical indicators can be used to measure the accuracy of encoding; cross-language test sets can be used to evaluate the adaptability of the model to different languages; and the degree of feature retention can be evaluated by comparing the similarity of speech features before and after encoding.

[0121] According to the coding evaluation effect, a better coding model is selected as the target coding model. When comparing the coding evaluation effects, the importance and weight of each indicator need to be considered comprehensively. For example, if in an application scenario with high real-time requirements, the coding efficiency will be given a higher weight; while in a speech recognition system with extremely high accuracy requirements, the coding accuracy is the primary consideration. By comprehensively evaluating and comparing various indicators, it can be determined which coding model has better overall performance. The selected target coding model will be used for feature extraction and subsequent processing of the collected multilingual speech data. A target coding model with excellent performance can better capture the key features in the speech data and improve the quality and accuracy of feature extraction.

[0122] In one embodiment, obtaining the latest encoding model in the server includes:

[0123] Integrate and encode the hardware performance parameters of the speech recognition device and the current model parameters of the current encoding model, generate a device status code with a unique identifier, and send it to the server;

[0124] The server parses the device status code and uses a pre-trained adaptive model matching network to analyze the parsed hardware performance parameters and current model parameters; the adaptive model matching network is built based on a deep neural network and is used to customize and generate an optimized generative learning model configuration file for the speech recognition device;

[0125] The server sends the generated configuration file to the speech recognition device, and the speech recognition device updates the current encoding model according to the information in the configuration file to obtain the latest encoding model.

[0126] In this embodiment, the hardware performance parameters of the speech recognition device include processor performance, memory capacity, storage speed, etc. These parameters directly affect the operating efficiency and performance of the model on the device. The current model parameters include information such as the structure, weights, and hyperparameters of the current encoding model, which reflects the current state and characteristics of the model. Through integrated coding, these key information are merged into a device status code with a unique identifier. The encoding method can use an advanced hash algorithm or a custom encoding rule to ensure that the generated device status code is unique and stable. For example, the hardware performance parameters and model parameters can be combined in a certain format and order, and then a unique value is calculated by a hash function as the device status code. Such a device status code not only contains the hardware characteristics of the device, but also reflects the current state of the model, providing comprehensive information for the server to accurately understand the device status.

[0127] The purpose of sending the generated device status code to the server is to enable the server to provide the most suitable latest encoding model for the device according to its specific situation. During the sending process, encryption technology can also be used to encrypt the device status code to protect the security and privacy of device information. For example, the communication channel is encrypted using the SSL / TLS encryption protocol to prevent the device status code from being stolen or tampered with during transmission.

[0128] After receiving the device status code, the server uses the pre-trained adaptive model matching network to parse and analyze it. The adaptive model matching network is built based on a deep neural network and has powerful data analysis and processing capabilities. The network contains multiple hidden layers, each of which is composed of a large number of neurons. Through complex connections and weight adjustments, it can learn the complex relationship between device hardware performance parameters and model parameters. The network is trained using a large amount of device data and model data, enabling it to customize and generate the most optimized generative learning model configuration file for speech recognition devices based on different device states and requirements. For example, the neurons in the hidden layer can learn the optimal parameter configuration of the model under different hardware performance, as well as the impact of different model parameter combinations on performance. By continuously adjusting weights and optimizing network structures, the adaptive model matching network can accurately predict the most suitable model configuration for the device based on the information in the device status code. After parsing the device status code, the hardware performance parameters and current model parameters are extracted. These parameters are then input into the adaptive model matching network for analysis. The network performs comprehensive processing and calculations on these parameters based on its learned knowledge and experience. During the analysis process, the network will consider multiple factors, such as the limitation of hardware performance on the model operation speed, the impact of different model parameters on recognition accuracy, etc. Based on these factors, the network generates an optimized generative learning model configuration file. The configuration file contains key information such as model structure parameter adjustment suggestions for the device, the initial value of the optimized weights of each layer of the neural network, and the dynamic adjustment strategy of cross-layer jump connections. For example, if the device's processor performance is low, the network will recommend reducing the complexity of the model, reducing the number of layers of the neural network or reducing the number of neurons; if the current model parameters show poor performance in the recognition of certain languages, the network will adjust the weights of the corresponding languages ​​or optimize the relevant convolution kernel parameters. In this way, the generated configuration file can adapt to the hardware conditions of the device and the status of the current model to the greatest extent, providing the best model update solution for the device.

[0129] After receiving the configuration file sent by the server, the speech recognition device will first parse the configuration file and extract key information, such as model structure parameter adjustment suggestions, weight initial values, etc. Then, the current encoding model will be adjusted and optimized accordingly based on this information. For example, if the configuration file recommends adjusting the weight of a certain layer of neural network, the device will reinitialize and optimize the weight of the layer according to the provided weight initial value; if the configuration file recommends adding or reducing certain neural network layers, the device will modify the model structure accordingly. During the update process, the device can also use some optimization algorithms, such as gradient descent method, Adam optimizer, etc., to quickly and accurately adjust the parameters of the model. Through continuous iteration and optimization, the model gradually converges to a more optimal state to obtain the latest encoding model.

[0130] In one embodiment, the hardware performance parameters of the speech recognition device and the current model parameters of the current encoding model are integrated and encoded to generate a device status code with a unique identifier, including:

[0131] Integrate the hardware performance parameters of the speech recognition device and the current model parameters of the current encoding model into a preset template, and encode them to obtain an encoded string;

[0132] Obtain the number of types of hardware performance parameters as the first number; obtain the number of types of current model parameters as the second number;

[0133] Arrange the characters in the encoded string in order and assign serial numbers to each character in order to generate a character sequence;

[0134] Constructing a two-dimensional coordinate system, using the value corresponding to the first quantity as the horizontal coordinate of the coordinate point, using the value corresponding to the second quantity as the vertical coordinate of the coordinate point, and generating a straight line based on the coordinate point and the coordinate origin;

[0135] Obtain the first number and the last number in the character sequence, and substitute them into the straight line as horizontal coordinate values ​​for calculation, thereby obtaining a first vertical coordinate value and a second vertical coordinate value;

[0136] Obtaining a first target character corresponding to a first target sequence number and a second target character corresponding to a second target sequence number in the character sequence; wherein the first target sequence number is a first ordinate value, and the second target sequence number is a second ordinate value; when the first ordinate value or the second ordinate value is greater than the value of the character sequence, the sequence number is continuously allocated from the head of the character sequence;

[0137] The first digit, the last digit, the first target character, and the second target character are combined into the device status code.

[0138] In this embodiment, the preset template is a pre-designed structural framework for accommodating parameters. Its function is to ensure that these parameters can be integrated according to certain rules and orders for subsequent processing and encoding. Integrating hardware performance parameters and current model parameters into the preset template can achieve the standardization and unification of parameters. For example, hardware performance parameters may include the model, frequency, memory capacity, etc. of the processor, and current model parameters may include the number of layers, number of nodes, weights, etc. of the neural network. Through the preset template, these different types of parameters can be arranged and combined in a certain format to provide an orderly data basis for subsequent encoding. Coding can adopt various algorithms and techniques, such as hash algorithms, encoding mapping tables, etc. Through encoding, complex parameter information can be converted into a concise string form, which is convenient for subsequent processing and transmission. The encoding process needs to ensure uniqueness and stability, that is, for the same parameter combination, the encoding result should always be consistent, and different parameter combinations should generate different encoding strings.

[0139] Obtaining the number of types of hardware performance parameters as the first number and the number of types of current model parameters as the second number are of great significance for constructing a unique device status code. The statistical method can be implemented by classifying and counting parameters. For example, for hardware performance parameters, classification and counting can be performed according to different hardware component types; for current model parameters, counting can be performed according to different structural elements or parameter types of the model. By accurately counting these quantities, key basic information can be provided for subsequent coordinate construction and data processing.

[0140] By constructing a two-dimensional coordinate system, taking the value corresponding to the first quantity as the horizontal coordinate of the coordinate point, and taking the value corresponding to the second quantity as the vertical coordinate of the coordinate point, a unique coordinate point can be determined on the two-dimensional plane. A straight line is generated based on the coordinate point and the coordinate origin. This straight line plays a key connecting role in subsequent processing. The equation of the straight line can be determined according to the two-point formula or other mathematical methods. It connects the coordinate point with the coordinate origin to form a mathematical object with a specific slope and position relationship. The parameters and characteristics of the straight line will be interrelated with the subsequent character sequence processing to jointly construct the device status code.

[0141] Arrange the characters in the encoded string in order and assign serial numbers to each character in order to generate a character sequence. This process converts the encoded string from a simple string to a sequence structure with order and numbering. The generation of the character sequence can be achieved by traversing and numbering the encoded string. For example, assign a continuous serial number to each character in order from left to right, starting from 1 and increasing. In this way, each character has a unique serial number corresponding to it, which facilitates the subsequent extraction and processing of data according to the serial number.

[0142] Get the first and last digits in the character sequence, and substitute them into the line as the horizontal coordinate values ​​for calculation, and get the first and second vertical coordinate values. Substitute the numbers in the character sequence into the equation of the line, and obtain the corresponding vertical coordinate values ​​through mathematical calculations based on parameters such as the slope and intercept of the line. The vertical coordinate values ​​will serve as the key basis for subsequent extraction of target characters. They establish a mathematical connection between the character sequence and the two-dimensional coordinate system, making the generation process of the device status code more logical and unique.

[0143] The first target sequence number and the second target sequence number are determined in the character sequence according to the calculated first ordinate value and the second ordinate value. If the ordinate value is greater than the length of the character sequence, the sequence number is continuously assigned from the head of the character sequence to ensure that the target character can be correctly located. The corresponding first target character and the second target character are extracted according to the target sequence number. These target characters are characters with specific positions and meanings selected from the character sequence, and they will form the device status code together with other data. The extraction process of the target character needs to ensure accuracy and uniqueness, that is, the corresponding character can be accurately found according to the determined sequence number, and each sequence number corresponds to only one character.

[0144] The first digit, the last digit, the first target character, and the second target character are combined into a device status code. This combination process integrates the various data elements obtained previously to form a device status code with a unique identifier. The significance of the device status code is that it can comprehensively reflect the hardware performance parameters of the speech recognition device, the current model parameters of the current encoding model, and the associated information between them. Through this status code, the server can quickly and accurately understand the status and characteristics of the device, providing a key basis for subsequent model matching and updating. At the same time, the unique generation method of the device status code ensures its uniqueness and non-replicability, and improves the security and reliability of communication between the device and the server.

[0145] Reference Figure 3 In another embodiment of the present invention, a multilingual speech recognition system based on a generative learning model is provided, comprising:

[0146] An extraction unit, used to extract features from the collected multilingual speech data to obtain a high-dimensional feature vector having speech information features;

[0147] An encoding unit, used for inputting the high-dimensional feature vector into a target encoding model for deep encoding processing to obtain an encoding vector; the target encoding model is a pre-trained generative learning model;

[0148] A recognition unit, configured to decode the encoding vector based on a dynamic programming algorithm and an attention mechanism to obtain a corresponding multilingual text sequence as a multilingual speech recognition result;

[0149] Among them, the generative learning model includes multiple neural network layers, and cross-layer jump connections are constructed between each neural network layer; each neural network layer includes a three-dimensional convolution kernel and a fusion module, and the three-dimensional convolution kernel is a parallel time convolution kernel, a frequency convolution kernel and a semantic association convolution kernel; in each convolution operation, the three convolution kernels process the input data simultaneously, and the feature information obtained by the three convolution kernels is dynamically fused through the fusion module; the fusion module automatically learns and adjusts the fusion weight according to the importance of the processing results of each convolution kernel.

[0150] In this embodiment, for the specific implementation of each unit in the above system embodiment, please refer to the description in the above method embodiment, which will not be repeated here.

[0151] Reference Figure 4 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0152] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0153] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0154] In summary, the multilingual speech recognition method and system based on the generative learning model provided by the present invention in the embodiment of the present invention include: extracting features from the collected multilingual speech data to obtain a high-dimensional feature vector with speech information features; inputting the high-dimensional feature vector into the target coding model for deep coding processing to obtain a coding vector; the target coding model is a pre-trained generative learning model; based on a dynamic programming algorithm and an attention mechanism, the coding vector is decoded to obtain a corresponding multilingual text sequence as a multilingual speech recognition result; wherein the generative learning model includes multiple neural network layers, and cross-layer jump connections are constructed between each neural network layer; each neural network layer includes a three-dimensional convolution kernel and a fusion module, and the three-dimensional convolution kernel is a parallel time convolution kernel, a frequency convolution kernel, and a semantic association convolution kernel; in each convolution operation, the three convolution kernels simultaneously process the input data, and dynamically fuse the feature information obtained by the three convolution kernels through the fusion module; the fusion module automatically learns and adjusts the fusion weight according to the importance of the processing results of each convolution kernel. In the present invention, input data is processed in parallel by three-dimensional convolution kernels to fully capture multi-dimensional key information of time, frequency and semantics. The fusion module dynamically adjusts the fusion weight according to the importance, and the cross-layer jump connection between layers efficiently transmits information. The dynamic programming algorithm and the attention mechanism are combined for decoding, and the context association is fully considered, so as to accurately convert the encoding vector into a multilingual text sequence, improve the accuracy of multilingual speech recognition, and overcome the defect of inaccurate recognition of current multilingual speech recognition technology.

[0155] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0156] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0157] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multilingual speech recognition method based on a generative learning model, characterized in that: The following steps are involved: Extract features from the collected multilingual speech data to obtain a high-dimensional feature vector with speech information features; Inputting the high-dimensional feature vector into a target coding model for deep coding processing to obtain a coding vector; the target coding model is a pre-trained generative learning model; Based on a dynamic programming algorithm and an attention mechanism, the encoding vector is decoded to obtain a corresponding multilingual text sequence as a multilingual speech recognition result; The generative learning model includes multiple neural network layers, and cross-layer jump connections are constructed between each neural network layer; each neural network layer includes a three-dimensional convolution kernel and a fusion module, and the three-dimensional convolution kernel is a parallel time convolution kernel, a frequency convolution kernel, and a semantic association convolution kernel; in each convolution operation, the three convolution kernels process the input data at the same time, and the feature information obtained by the three convolution kernels is dynamically fused through the fusion module; the fusion module automatically learns and adjusts the fusion weight according to the importance of the processing results of each convolution kernel; The cross-layer jump connection is a selective information transmission channel established between adjacent odd-numbered neural network layers and even-numbered neural network layers; After the input data of one neural network layer is processed, the processing result is passed directly to the next layer or passed to a further layer across the intermediate layer according to the preset evaluation index; If the evaluation results show that skipping is more conducive to model convergence, the processing results of the layer are directly transferred to the target layer through the cross-layer skip connection. During the transfer process, an adaptive information enhancement mechanism is used to adjust the transferred processing results according to the needs of the target layer, so that the processing results can be effectively transferred between different layers. The decoding process of the encoding vector based on the dynamic programming algorithm and the attention mechanism to obtain the corresponding multilingual text sequence includes: Using a pre-trained semantic segmentation model, the coding vector is preliminarily divided into a number of sub-vector segments with latent semantic units according to the potential distribution of semantic information in the coding vector, and a set of preliminarily segmented sub-vector segments is obtained; For each sub-vector segment, a search graph with it as a node is constructed. The optimal path from the starting sub-vector segment to each possible ending sub-vector segment is calculated in the search graph according to the dynamic programming algorithm. The path combination with the minimum cost is found through search to obtain a preliminary decoding path sequence. The preliminary decoding path sequence is optimized based on the multi-head attention mechanism to obtain the optimized decoding path sequence; Convert the optimized decoding path sequence into the corresponding multilingual text sequence.

2. The multilingual speech recognition method based on a generative learning model according to claim 1, characterized in that: The feature extraction of the collected multilingual speech data to obtain a high-dimensional feature vector having speech information features includes: The collected multilingual speech data is framed and divided into a plurality of speech segments, each segment containing a speech signal of a preset time length, to obtain a plurality of speech frames; Perform windowing processing on each speech frame, and use the Hamming window function to smooth the edge of the speech frame to obtain a windowed speech frame; Perform fast Fourier transform on the windowed speech frame to convert it from the time domain to the frequency domain to obtain a frequency domain signal; The Mel-frequency cepstral coefficient features are extracted from the frequency domain signal, and the frequency domain signal is passed through a set of Mel filter banks. Logarithmic operations and discrete cosine transforms are performed on the filtered signal to obtain a high-dimensional feature vector with speech information characteristics.

3. The multilingual speech recognition method based on a generative learning model according to claim 1, characterized in that: The feature extraction of the collected multilingual speech data to obtain a high-dimensional feature vector having speech information features includes: The collected multilingual speech data is simultaneously input into a mel spectrum analyzer and a linear prediction coding analyzer for processing to obtain corresponding mel spectrum feature data and linear prediction coding feature data; Automatically calculating a weighting coefficient of a Mel spectrum feature according to the speech characteristics of the language and the position information of the speech segment in the entire speech data, and multiplying the Mel spectrum feature data by the weighting coefficient to obtain enhanced Mel spectrum feature data; The linear prediction coding feature data is optimized by using the K-Means clustering algorithm to obtain the optimized linear prediction coding feature data; The enhanced Mel spectrum feature data and the optimized linear prediction coding feature data are fused to obtain a high-dimensional feature vector with speech information characteristics.

4. The multilingual speech recognition method based on a generative learning model according to claim 1, characterized in that: Before extracting features from the collected multilingual speech data to obtain a high-dimensional feature vector having speech information features, the method includes: Obtain multilingual reference speech pre-stored in the speech recognition device and the current encoding model; Get the latest encoding model from the server; Based on the current coding model and the latest coding model, encoding processing is performed on the multilingual reference speech respectively, and corresponding coding evaluation effects are obtained; The coding model with better coding evaluation effect is used as the target coding model.

5. The multilingual speech recognition method based on a generative learning model according to claim 4, characterized in that: The obtaining of the latest encoding model in the server includes: Integrate and encode the hardware performance parameters of the speech recognition device and the current model parameters of the current encoding model, generate a device status code with a unique identifier, and send it to the server; The server parses the device status code and uses a pre-trained adaptive model matching network to analyze the parsed hardware performance parameters and current model parameters; the adaptive model matching network is built based on a deep neural network and is used to customize and generate an optimized generative learning model configuration file for the speech recognition device; The server sends the generated configuration file to the speech recognition device, and the speech recognition device updates the current encoding model according to the information in the configuration file to obtain the latest encoding model.

6. A multilingual speech recognition system based on a generative learning model, characterized in that: include: An extraction unit, used to extract features from the collected multilingual speech data to obtain a high-dimensional feature vector having speech information features; An encoding unit, used for inputting the high-dimensional feature vector into a target encoding model for deep encoding processing to obtain an encoding vector; the target encoding model is a pre-trained generative learning model; A recognition unit, configured to decode the encoding vector based on a dynamic programming algorithm and an attention mechanism to obtain a corresponding multilingual text sequence as a multilingual speech recognition result; The generative learning model includes multiple neural network layers, and cross-layer jump connections are constructed between each neural network layer; each neural network layer includes a three-dimensional convolution kernel and a fusion module, and the three-dimensional convolution kernel is a parallel time convolution kernel, a frequency convolution kernel, and a semantic association convolution kernel; in each convolution operation, the three convolution kernels process the input data at the same time, and the feature information obtained by the three convolution kernels is dynamically fused through the fusion module; the fusion module automatically learns and adjusts the fusion weight according to the importance of the processing results of each convolution kernel; The cross-layer jump connection is a selective information transmission channel established between adjacent odd-numbered neural network layers and even-numbered neural network layers; After the input data of one neural network layer is processed, the processing result is passed directly to the next layer or passed to a further layer across the intermediate layer according to the preset evaluation index; If the evaluation results show that skipping is more conducive to model convergence, the processing results of the layer are directly transferred to the target layer through the cross-layer skip connection. During the transfer process, an adaptive information enhancement mechanism is used to adjust the transferred processing results according to the needs of the target layer, so that the processing results can be effectively transferred between different layers. The decoding process of the encoding vector based on the dynamic programming algorithm and the attention mechanism to obtain the corresponding multilingual text sequence includes: Using a pre-trained semantic segmentation model, the coding vector is preliminarily divided into a number of sub-vector segments with latent semantic units according to the potential distribution of semantic information in the coding vector, and a set of preliminarily segmented sub-vector segments is obtained; For each sub-vector segment, a search graph with it as a node is constructed. The optimal path from the starting sub-vector segment to each possible ending sub-vector segment is calculated in the search graph according to the dynamic programming algorithm. The path combination with the minimum cost is found through search to obtain a preliminary decoding path sequence. The preliminary decoding path sequence is optimized based on the multi-head attention mechanism to obtain the optimized decoding path sequence; Convert the optimized decoding path sequence into the corresponding multilingual text sequence.

7. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.