Voice-to-text conversion system and method based on personalized lexicon

Abstract

The invention discloses a voice-to-text conversion system and method based on a personalized lexicon. The method comprises the following steps: collecting personal data and voice data of a current user; performing feature extraction on the voice data by using a multi-band linear predictive coding method to obtain a feature vector group; inputting the feature vector group into a voice-to-character large model to obtain a first text; screening from a professional database according to the personal data to obtain a first word library; obtaining a shared lexicon of the related user according to the personal data, and obtaining a second lexicon; combining the first lexicon with the second lexicon to obtain a personalized lexicon of the current user; after the first text is automatically modified according to the personalized word bank, manual modification is carried out, and manual modification vocabularies and a final text are obtained; and updating the personalized lexicon of the current user according to the manually modified vocabularies. The invention relates to the technical field of voice combing, and solves the technical problems of low voice feature recognition accuracy and insufficient voice-to-character conversion accuracy in an existing system.

Description

Technical Field

[0001] The present invention belongs to the field of speech processing and relates to deep learning technology, and specifically to a speech-to-text system and method based on a personalized vocabulary. Background Art

[0002] Speech-to-text systems are widely used in modern life, greatly improving the convenience and efficiency of information exchange and processing, and playing a key role in many scenarios. For example, in daily office work, it can help people quickly record the content of meetings, so that participants do not need to manually record every key point, so that they can focus more on meeting discussions and improve meeting efficiency; in the learning process, students can use the speech-to-text system to convert the teacher's lecture voice into text notes for subsequent review and organization; for people with disabilities, speech-to-text systems are indispensable communication tools that can help them better understand other people's voice information and achieve barrier-free communication; in the era of mobile Internet, speech-to-text functions are also widely used in voice assistants and voice input of smartphones, allowing users to easily generate and process text information when their hands are busy (such as driving, doing housework, etc.).

[0003] However, in terms of accuracy, existing speech-to-text systems often fail to fully and precisely extract speech features, which leads to errors in distinguishing speech elements with similar pronunciations. In addition, although the vocabulary in the speech-to-text system is highly versatile, it is insufficient in the recognition of professional terms and regional vocabulary, which affects the accuracy of recognition. In terms of efficiency, due to the lack of optimization mechanisms, existing systems often consume unnecessary computing resources when processing speech feature extraction, and are inefficient in the vocabulary matching process, which seriously affects the overall processing speed, especially when faced with large-scale vocabulary. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a speech-to-text system and method based on a personalized vocabulary, which is used to solve the technical problems of low accuracy in speech feature recognition and insufficient precision in speech-to-text in the existing speech-to-text system.

[0005] To achieve the above object, the first aspect of the present invention provides a speech-to-text system based on a personalized vocabulary, comprising:

[0006] Data collection module: used to collect the current user's personal data and obtain the current user's voice data; personal data includes professional information and geographic location;

[0007] Feature extraction module: used to extract features from the current user's voice data using a multi-band linear predictive coding method to obtain a feature vector group;

[0008] Speech recognition module: used to input the feature vector group into the speech-to-text model to obtain the first text; wherein the speech-to-text model is constructed based on a deep learning algorithm;

[0009] The word library construction module is used to filter from a professional database according to the personal data of the current user to obtain a first word library, obtain a shared word library of related users according to the personal data of the current user, and obtain a second word library; and merge the first word library with the second word library to obtain a personalized word library for the current user;

[0010] Text modification module: used to automatically modify the first text according to the personalized vocabulary to obtain the second text, and manually modify the second text to obtain the manually modified vocabulary and the final text; and update the personalized vocabulary of the current user according to the manually modified vocabulary.

[0011] Based on the above technical modules, the speech-to-text function based on personalized vocabulary is realized, which improves the accuracy of speech-to-text and the degree of meeting the personalized needs of users. First, the data acquisition module is responsible for collecting the user's voice data and personal data. Then, the feature extraction module uses a multi-band linear predictive coding method to perform feature processing on the voice data to capture the acoustic features in the voice, thereby improving the accuracy of speech-to-text. Subsequently, the speech recognition module generates a preliminary text. On this basis, the vocabulary construction module creates a personalized vocabulary that matches the user's professional field and geographical environment based on personal data. Then, the text modification module automatically and manually modifies the preliminary text to obtain more accurate speech recognition results. At the same time, based on the manually modified vocabulary, the system will update the personalized vocabulary of the current user, realize dynamic optimization of the vocabulary, and further improve the accuracy of the system's speech-to-text conversion for the current user.

[0012] Furthermore, the multi-band linear predictive coding method extracts features from the voice data of the current user, including:

[0013] A1, using a first-order digital filter to perform pre-emphasis processing on the voice data of the current user to obtain a pre-emphasized voice signal sequence s(n); wherein n represents a time index;

[0014] A2, using wavelet basis functions to perform L-layer wavelet decomposition on s(n) to obtain low-frequency coefficients dL(n) and high-frequency coefficients dl(n); where l represents the number of decomposition layers of wavelet decomposition, and l = 1, 2, ..., L;

[0015] A3, using a multi-band linear prediction coding method to extract features of the low-frequency coefficient dL(n), and obtain low-frequency feature vectors Vlow in a number of time windows;

[0016] A4, extracting features from the high-frequency coefficients dl(n) to obtain high-frequency feature vectors Vhigh within a number of time windows;

[0017] A5, concatenate the low-frequency feature vector and the high-frequency feature vector in the i-th time window to obtain a number of fusion features Vi = [Vlow(i), Vhigh(i)]; where i = 1, 2, ..., N, and N represents the number of time windows;

[0018] A6, arranging and integrating the plurality of fused features in chronological order to obtain a feature vector group V; wherein the feature vector group V = [V1, V2, ..., VN].

[0019] The multi-band linear prediction coding method can extract the low-frequency and high-frequency features of speech signals at the same time. The low-frequency coefficients and high-frequency coefficients are obtained by wavelet basis function decomposition, and then feature extraction is performed on them to obtain low-frequency feature vectors and high-frequency feature vectors. Among them, the low-frequency part often contains information such as the fundamental frequency and formant of the speech, while the high-frequency part contains detailed information such as the fricative sound of the speech. This multi-dimensional feature extraction helps to more comprehensively characterize the speech signal.

[0020] In addition, the low-frequency feature vector and the high-frequency feature vector are concatenated to obtain the fused feature, which completely retains the information of the speech signal in different frequency ranges and time windows, providing a rich and complete speech feature representation for the subsequent speech recognition module, which is conducive to improving the accuracy of speech recognition.

[0021] Furthermore, the method of extracting features of the low-frequency coefficient dL(n) by using a multi-band linear prediction coding method includes:

[0022] A31, divide dL(n) into windows according to preset time lengths to obtain a number of time windows;

[0023] A32, for each time window, using discrete Fourier transform to convert the low-frequency coefficients in the time window from the time domain to the frequency domain, to obtain a frequency domain signal X(i,k); wherein i represents the time window index and k represents the frequency index in the frequency domain;

[0024] A33, divide the frequency domain signal X(i,k) into M sub-bands, calculate the center frequency fcm of each sub-band, and assign the components of the corresponding frequency range in X(i,k) to the corresponding sub-band according to fcm, to obtain M sub-band signals;

[0025] A34, using a linear prediction coding method to extract the features of the M sub-band signals respectively, and obtain the feature vector elements of the M sub-bands;

[0026] A35, combining the feature vector elements of the M sub-bands to obtain a low-frequency feature vector Vlow(i) of a single time window;

[0027] A36, repeat A32-A35 until a number of time windows are traversed to obtain low-frequency feature vectors Vlow in a number of time windows.

[0028] The frequency domain signal is divided into M subbands, and signal processing and feature extraction are performed for each subband, so that the model can focus on the characteristics of the speech signal in different frequency ranges, rather than processing the entire frequency domain as a whole. The feature vector elements of each subband are combined into a low-frequency feature vector of a single time window, which further refines the feature representation and helps to more accurately describe the spectral structure of the speech signal.

[0029] Furthermore, the M subbands are adjusted by using an adaptive characteristic parameter optimization method, including:

[0030] A33-1, according to the formula E(n) = ∑|s(n+m)|^2 and the formula Z(n) = (1 / 2T) × ∑|sgn(s(n+m))-sgn(s(n+m-1))|, respectively calculate the short-time energy and zero-crossing rate of the pre-emphasized speech signal to obtain the short-time energy E(n) and the zero-crossing rate Z(n); wherein m represents the index of the sampling point in the short-time window, the summation range is m = 0, 1, ..., (T-1), T represents the short-time window length, and sgn represents the step function;

[0031] A33-2, according to the formula ER = ∑ (E (n) - E (n - 1)) ^ 2 / ∑ (E (n)) ^ 2 and the formula ZCR = ∑ (Z (n) - Z (n - 1)) ^ 2 / ∑ (Z (n)) ^ 2 respectively calculate the energy change rate and the zero-crossing change rate, and obtain the energy change rate ER and the zero-crossing change rate ZCR; wherein the summation range is the start time point to the end time point of the speech signal;

[0032] A33-3, calculate the complexity index CI of speech according to the formula CI = αER + (1-α) ZCR; wherein α represents the weight coefficient;

[0033] A33-4, determine whether CI is greater than a preset threshold; if yes, then M=M+ΔM; if no, then M=M-ΔM; wherein ΔM represents the change in the number of subbands.

[0034] By calculating the short-time energy and zero-crossing rate of the pre-emphasized speech signal, the energy change rate and zero-crossing change rate are obtained, and finally the complexity index of the speech is calculated, and the number of subbands M is adjusted according to the comparison between the complexity index and the preset threshold. When the speech signal is more complex, the number of subbands will be automatically increased, which helps to capture the detailed features of the speech more finely; when the speech signal is relatively simple, the number of subbands will be automatically reduced to improve the calculation efficiency and effectively avoid overfitting. This adaptive mechanism enables the feature extraction process to dynamically adjust the parameters according to the complexity of the speech signal, optimizing the performance and efficiency of speech processing.

[0035] Furthermore, the feature extraction of the high frequency coefficient dl(n) includes:

[0036] A41, divide dl(n) into windows according to preset time lengths to obtain a plurality of time windows;

[0037] A42, for each time window, the energy of the high frequency coefficient is calculated according to the formula El(i)=∑|dl(n)|^2 to obtain the energy feature El(i); wherein the summation range of the energy feature is a number of sampling points in the i-th time window;

[0038] A43, perform Fourier transform on the high frequency coefficient dl(n) in each time window to obtain the spectrum feature Xl(i,k);

[0039] A44, combining the energy feature El(i) and the spectrum feature Xl(i,k) to obtain a high-frequency feature vector Vhigh(i) of a single time window;

[0040] A45, repeat A42-A43 until a number of time windows are traversed to obtain high-frequency feature vectors Vhigh in the number of time windows.

[0041] Furthermore, the speech-to-text model is constructed based on a deep learning algorithm, including:

[0042] B1, collect a number of voice data, and obtain corresponding text labels according to the number of voice data to obtain a number of label data;

[0043] B2, preprocessing a number of speech data, and extracting features from the preprocessed speech data using a multi-band linear predictive coding method to obtain a number of feature data;

[0044] B3, build a speech recognition model based on deep learning, input a number of label data and a number of feature data into the speech recognition model for training, verification and testing, and obtain the speech recognition model with the highest test accuracy;

[0045] B4, taking the speech recognition model with the highest test accuracy as the final output, and obtaining a large speech-to-text model; wherein the input of the large speech-to-text model is the feature vector group of the speech data, and the output is the text data.

[0046] Furthermore, obtaining a shared vocabulary of related users according to the personal data of the current user includes:

[0047] C1, based on the geographic location in the personal data, the spatial distance between the current user and other users is calculated using the Euclidean distance formula, and users whose spatial distance is less than the preset distance threshold are marked as associated users;

[0048] C2, based on the professional information in the personal data, uses the cosine similarity calculation formula to calculate the professional similarity between the current user and the associated users, and marks the associated users whose professional similarity is greater than the preset similarity threshold as related users;

[0049] C3, counting the usage frequencies of words in the shared vocabulary of related users, screening and combining words whose usage frequencies exceed a preset frequency threshold, and obtaining a second vocabulary.

[0050] Use geographic location information to identify related users and meet the user's speech-to-text needs in the local context. At the same time, by analyzing professional information, related users are obtained from related users, and the shared vocabulary of related users is obtained to build a second vocabulary. Therefore, the shared vocabulary of related users increases the vocabulary of personalized vocabulary and ensures that the vocabulary can cover a wider range of expressions. At the same time, the introduction of shared vocabulary can also maintain the professionalism and timeliness of the vocabulary, reflecting the latest popular vocabulary and idioms in the professional field.

[0051] Furthermore, the automatically modifying the first text according to the personalized word library includes:

[0052] D1, performing word segmentation on the first text to obtain a number of word segments;

[0053] D2, grouping a number of word segmentations and a number of words in the personalized word library according to the length of the words to obtain a number of groups;

[0054] D3, for each group, the similarity between the segmentation word and the vocabulary is calculated using the cosine similarity formula, and the vocabulary with a similarity greater than the preset similarity value is marked as a matching vocabulary;

[0055] D4, using matching words to replace corresponding word segments in the first text to obtain an automatically modified second text.

[0056] Grouping by word length avoids calculating the similarity between each word segment and all words in the personalized word library one by one. Usually, words with similar lengths are more likely to have higher similarity. In this way, after grouping, only similarity calculation needs to be performed within the group, which greatly narrows the calculation scope and reduces unnecessary calculations, thereby improving the efficiency of word matching.

[0057] A second aspect of the present invention provides a speech-to-text method based on a personalized word library, comprising:

[0058] S1, collect personal data of the current user and obtain voice data of the current user; wherein the personal data includes professional information and geographic location;

[0059] S2, extracting features from the current user's voice data using a multi-band linear predictive coding method to obtain a feature vector group;

[0060] S3, inputting the feature vector group into a speech-to-text model to obtain a first text; wherein the speech-to-text model is constructed based on a deep learning algorithm;

[0061] S4, filtering from a professional database according to the personal data of the current user to obtain a first vocabulary;

[0062] S5, obtaining a shared vocabulary of related users according to the personal data of the current user, and obtaining a second vocabulary;

[0063] S6, merging the first word library with the second word library to obtain a personalized word library for the current user;

[0064] S7, automatically modifying the first text according to the personalized vocabulary to obtain a second text, and manually modifying the second text to obtain manually modified vocabulary and a final text;

[0065] S8, updating the personalized vocabulary of the current user according to the manually modified vocabulary.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] The multi-band linear predictive coding method is used for feature extraction, which can simultaneously extract the low-frequency and high-frequency features of the speech signal. The multi-dimensional feature extraction can more comprehensively characterize the speech signal. The frequency domain signal is divided into multiple sub-bands, and signal processing and feature extraction are performed on each sub-band, which refines the feature representation and more accurately describes the spectral structure of the speech signal. The extracted features can better reflect the essential characteristics of the speech, which is conducive to improving the accuracy of speech recognition.

[0068] At the same time, in the multi-band linear prediction coding method, a mechanism for adaptively adjusting the number of sub-bands is adopted to avoid using a feature extraction method with a fixed complexity for all speech situations. While ensuring the extraction of effective features, it is flexibly adjusted according to the complexity of the speech, reducing unnecessary calculations in simple speech situations, and improving the operating efficiency of the feature extraction link and the speech-to-text system.

[0069] The word library construction module builds a personalized word library based on the user's personal data, which includes a first word library selected from a professional database and a second word library composed of related users and shared words of related users, covering the latest popular words, idioms and regional characteristics in the professional field, so that the word library fits the user's personalized needs. In the text modification after speech-to-text conversion, the text can be corrected more accurately based on the personalized word library, thereby improving the overall accuracy of speech-to-text conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0071] Figure 1 A flowchart of a speech-to-text system based on a personalized vocabulary provided by the present invention;

[0072] Figure 2 A schematic diagram of a framework of a speech-to-text system based on a personalized vocabulary provided by the present invention;

[0073] Figure 3 The present invention provides a schematic diagram of the operation of a feature extraction module in a speech-to-text system based on a personalized vocabulary. DETAILED DESCRIPTION

[0074] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] See also Figure 1-Figure 3 The first aspect of the present invention provides a speech-to-text system based on a personalized vocabulary, including:

[0076] Data collection module: used to collect the current user's personal data and obtain the current user's voice data; personal data includes professional information and geographic location;

[0077] Feature extraction module: used to extract features from the current user's voice data using a multi-band linear predictive coding method to obtain a feature vector group;

[0078] Speech recognition module: used to input the feature vector group into the speech-to-text model to obtain the first text; wherein the speech-to-text model is constructed based on a deep learning algorithm;

[0079] The word library construction module is used to filter from a professional database according to the personal data of the current user to obtain a first word library, obtain a shared word library of related users according to the personal data of the current user, and obtain a second word library; and merge the first word library with the second word library to obtain a personalized word library for the current user;

[0080] Text modification module: used to automatically modify the first text according to the personalized vocabulary to obtain the second text, and manually modify the second text to obtain the manually modified vocabulary and the final text; and update the personalized vocabulary of the current user according to the manually modified vocabulary.

[0081] It should be noted that, in this system, the data acquisition module, feature extraction module, speech recognition module, vocabulary building module and text modification module are communicatively connected, and the vocabulary building module is communicatively connected to the data acquisition module and the text modification module respectively.

[0082] In this embodiment, in order to build a highly accurate and personalized vocabulary that meets the specific needs of users, the data collection module will not only collect the user's voice data, but also obtain the user's geographical location and professional information, so as to mine vocabulary resources closely related to the user from multiple dimensions;

[0083] Geographic location information can be used to explore the common vocabulary characteristics of user groups in the same area, such as dialect vocabulary specific to certain regions, local industry terms or local popular slang, etc. These words can enable personalized vocabulary to better adapt to the speech-to-text needs in the regional cultural context; while professional information focuses on the professional vocabulary system of the user's professional field. Through precise positioning of the professional field, professional vocabulary such as professional disease names and treatment terms in the medical field, or programming language keywords and specific algorithm names in the computer field are screened out, thereby enhancing the vocabulary richness and accuracy of the personalized vocabulary in the professional context. Therefore, the accuracy and adaptability of the system of the present invention enable it to better serve user groups in different regions and with different professional backgrounds, and provide users with a better quality, more efficient and more in line with their own expression habits speech-to-text service experience.

[0084] In the feature extraction module, the multi-band linear predictive coding method is used to extract features from the collected speech data to obtain a feature vector group, which specifically includes:

[0085] A1, pre-emphasis processing:

[0086] First, the collected voice data of the current user is pre-emphasized using a first-order digital filter to obtain a pre-emphasized voice signal sequence s(n); the purpose is to enhance the high-frequency part of the voice signal so that the characteristics of the high-frequency part of the voice signal can be more prominent in the subsequent processing process, which is convenient for analyzing and extracting effective information; where n represents a time index, for example, s(1), s(2), etc. represent the signal values ​​corresponding to different moments respectively;

[0087] Next, the pre-emphasized speech signal sequence s(n) is subjected to L-layer wavelet decomposition using wavelet basis functions. Wavelet decomposition can divide the speech signal according to different frequency scales, and decompose it into low-frequency coefficients dL(n) and high-frequency coefficients dl(n), where l represents the number of decomposition layers of the wavelet decomposition, and l = 1, 2, ..., L. Different levels of decomposition correspond to information in different frequency ranges. The low-frequency coefficients mainly contain the characteristics of the relatively low-frequency part of the speech signal, such as the fundamental frequency, resonance peak and other basic characteristics of the speech. The high-frequency coefficients cover the detailed characteristics of the high-frequency part such as fricative sounds. Such decomposition prepares for the subsequent extraction of the characteristics of different frequency bands.

[0088] A3, low frequency coefficient feature extraction:

[0089] A31, window division:

[0090] For the low-frequency coefficient dL(n), first divide it into windows according to the preset time length, and divide the entire low-frequency coefficient sequence into several time windows. For example, assuming the preset time length is 20 milliseconds, then divide the low-frequency coefficient into a window unit of 20 milliseconds to obtain multiple time windows, which is convenient for analyzing the characteristic changes of the low-frequency coefficient within the local time range;

[0091] A32, time domain to frequency domain conversion:

[0092] For each divided time window, the low-frequency coefficients in the time window are converted from the time domain to the frequency domain using discrete Fourier transform (DFT) to obtain the frequency domain signal X(i,k); through this conversion, the signal characteristics originally reflected in the time domain can be displayed in the frequency domain, and the distribution of low-frequency coefficients at different frequencies can be seen more clearly; where i represents the time window index, which is used to distinguish different time windows, and k represents the frequency index in the frequency domain, which is used to mark the signal values ​​corresponding to different frequency points in the frequency domain;

[0093] A33, sub-band division and adjustment:

[0094] Subband division: divide the frequency domain signal X(i,k) into M subbands. The specific operation is to first calculate the center frequency fcm of each subband, and then classify the components of the corresponding frequency range in X(i,k) into the corresponding subband according to fcm, so as to obtain M subband signals; each subband corresponds to a specific frequency interval, and the speech signal has different frequency characteristics in different subbands;

[0095] Here, since speech signals of different complexity have different information richness in different frequency ranges, in order to more accurately capture the key features of each frequency range in the speech signal and make the feature extraction process adapt to the actual complexity of the speech signal, in this system, the adaptive feature parameter optimization method is used to automatically adjust the number of subbands M:

[0096] First, according to the formula E(n)=∑|s(n+m)|^2 and the formula Z(n)=(1 / 2T)×∑|sgn(s(n+m))-sgn(s(n+m-1))|, the short-time energy E(n) and the zero-crossing rate Z(n) of the pre-emphasized speech signal are calculated respectively; wherein m represents the index of the sampling point in the short-time window, the summation range is m=0,1,...,(T-1), T represents the short-time window length, and sgn represents the step function;

[0097] Short-time energy reflects the energy of a speech signal in a short period of time, while the zero-crossing rate reflects the number of times a signal crosses the zero value per unit time. These two indicators can describe the characteristics of speech signals from different angles.

[0098] Next, the energy change rate ER and the zero-crossing change rate ZCR are calculated according to the formula ER = ∑(E(n)-E(n-1))^2 / ∑(E(n))^2 and the formula ZCR = ∑(Z(n)-Z(n-1))^2 / ∑(Z(n))^2, respectively, and the summation range is from the start time point to the end time point of the speech signal; by analyzing the changes of these indicators at adjacent moments, the dynamic characteristics of the speech signal are grasped;

[0099] Then, the complexity index CI of speech is calculated according to the formula CI = αER + (1-α) ZCR, where α represents the weight coefficient, which is determined through practical experience;

[0100] Finally, it is determined whether CI is greater than the preset threshold; if yes, then M = M + ΔM, that is, the number of subbands is increased, which means that when the speech signal is more complex, the features are extracted more finely by increasing the subband division; if no, then M = M - ΔM, the number of subbands is reduced, and when the speech is relatively simple, excessive division is avoided, the calculation efficiency is improved while ensuring the effectiveness of feature extraction; where ΔM represents the change value of the number of subbands, which can be set to 1 or 2 according to practical experience, and the preset threshold is obtained through practical experience, usually 0.3-0.5, and the specific value can be determined according to the complexity distribution of speech samples in actual application scenarios and the comprehensive trade-off between feature extraction effect and calculation efficiency. For example, if most of the speech samples are relatively simple, the threshold can be appropriately lowered to 0.3; if the proportion of complex speech samples is relatively large, the threshold can be appropriately increased to close to 0.5;

[0101] A34, sub-band feature extraction:

[0102] The linear prediction coding method is used to extract the features of the M sub-band signals respectively, and obtain the feature vector elements of the M sub-bands; the linear prediction coding method is based on the linear prediction model of the speech signal, and predicts the current signal value by analyzing the past signal samples, and then extracts the relevant parameters that can reflect the signal characteristics as the feature vector elements;

[0103] A35, combined to obtain low-frequency eigenvectors:

[0104] Combine the feature vector elements of the M sub-bands to obtain the low-frequency feature vector Vlow(i) of a single time window; this vector integrates the feature information of different sub-bands in the time window and is used to describe the speech features of the low-frequency part within the local time range;

[0105] A36, traverse all time windows:

[0106] Finally, repeat steps A32-A35 until all time windows divided according to the preset time length are traversed, so that the low-frequency feature vectors Vlow in the time windows are obtained, covering the characteristics of the entire speech signal in different time windows in the low-frequency band;

[0107] A4, high frequency coefficient feature extraction:

[0108] A41, window division:

[0109] Similarly, the high frequency coefficient dL(n) is divided into windows according to a preset time length to obtain a number of time windows;

[0110] A42, calculate energy characteristics:

[0111] For each time window, the energy of the high-frequency coefficient is calculated according to the formula El(i) = ∑|dl(n)|^2 to obtain the energy feature El(i), and the summation range of the energy feature is a number of sampling points in the i-th time window, reflecting the energy strength of the high-frequency coefficient in this time window;

[0112] A43, Fourier transform to obtain the spectrum characteristics:

[0113] Perform Fourier transform on the high-frequency coefficients dL(n) in each time window to obtain the spectrum features Xl(i,k); through the spectrum features, we can observe the distribution characteristics of the high-frequency coefficients in the frequency domain and further explore the detailed information of the high-frequency part of the speech signal;

[0114] A44: Combine to get high frequency feature vector**

[0115] Combine the energy feature El(i) and the spectrum feature Xl(i,k) to obtain the high-frequency feature vector Vhigh(i) of a single time window;

[0116] A45, traverse all time windows:

[0117] Repeat steps A42-A43 until all the divided time windows are traversed, thereby obtaining high-frequency feature vectors Vhigh in the time windows, and completely obtaining feature information of the speech signal in different time windows in the high-frequency band;

[0118] A5, construct fusion features:

[0119] The low-frequency feature vector and the high-frequency feature vector in the i-th time window are concatenated, and the features of the low-frequency and high-frequency parts in the same time window are integrated together to obtain a fusion feature vector Vi = [Vlow(i), Vhigh(i)] that can more comprehensively reflect the speech features of the time window; wherein i = 1, 2, ..., N, and N represents the number of time windows;

[0120] A6, permutation and integration to obtain the eigenvector group:

[0121] Finally, several fusion features are arranged and integrated in chronological order to obtain a feature vector group V = [V1, V2, ..., VN];

[0122] The feature vector group obtained based on the multi-band linear predictive coding method completely covers the comprehensive feature information of the entire speech signal in different time windows, providing effective feature input for subsequent speech recognition and other processing, facilitating further analysis of speech content and conversion into corresponding text information.

[0123] The speech signal itself has complex time-varying and frequency-varying characteristics, and traditional feature extraction methods may not be able to fully capture these characteristics. The multi-band linear predictive coding method can more accurately extract the key features of the speech signal through the above-mentioned multi-dimensional, time-domain-frequency domain combination, sub-band division and other sophisticated operations. It is crucial for distinguishing different speech content and speaker characteristics, and is the key to improving speech recognition accuracy.

[0124] The speech recognition module inputs the feature vector group output by the feature extraction module into the constructed speech-to-text model. The model will calculate and process the input feature vector group layer by layer according to its pre-trained network structure and parameters, and finally output the corresponding text content to obtain the first text;

[0125] Among them, the speech-to-text model is built based on a deep learning algorithm, has strong pattern recognition capabilities, and can learn the mapping relationship between speech features and text. Its construction process is as follows:

[0126] First, collect a large amount of voice data from multiple channels. This voice data should cover different accents, speaking speeds, speaking styles, and voice content in various practical application scenarios. Build a rich and diverse voice data set to ensure that the model can learn the voice features in various situations.

[0127] Next, for each piece of collected speech data, it is necessary to obtain the corresponding accurate text content manually or through other reliable transcription methods as text labels, and obtain a number of label data for subsequent model training supervision;

[0128] Preprocess the collected speech data to make its format and features more suitable for subsequent feature extraction and model training. Common preprocessing operations include unified sampling rate (such as adjusting the sampling rate of all speech data to 16kHz), audio format conversion (such as conversion to PCM format) and frame processing (segmenting speech signals into short frames of fixed length, such as 25 milliseconds per frame). The purpose is to convert speech data into a regular, easy-to-process form to ensure data consistency.

[0129] Then, a multi-band linear predictive coding method is used to extract features from the pre-processed speech data to obtain a number of feature data;

[0130] Then build and train a speech recognition model based on deep learning:

[0131] There are many deep learning architectures that can be used to process sequence data. The following is an example of an embodiment of an optional speech recognition model architecture:

[0132] First, use the CNN layer to extract local pattern information in speech features; for example, set up several two-dimensional convolutional layers, perform convolution operations on the speech feature map, and automatically capture some local spectral features in the speech;

[0133] Then add the LSTM layer to process the timing information of the speech data; LSTM has a gating mechanism that can well capture the long-term dependencies in the speech sequence, such as the phoneme order of speech, the grammatical and semantic relationships between words, etc. You can set up multiple LSTM layers, each LSTM layer is configured with a certain number of hidden units, and the input feature sequence is gradually processed through the loop structure, the hidden state is continuously updated, and the timing information is transmitted and integrated;

[0134] After the LSTM layer, several fully connected layers are connected to further integrate and transform feature information. The neurons in the fully connected layer are connected to all neurons in the previous layer. For example, two fully connected layers are set, the first fully connected layer has 512 neurons, and the second fully connected layer has 256 neurons. In each layer, activation functions such as ReLU are used to introduce nonlinear factors to enhance the model's expressiveness, so that the model can learn more complex speech-text mapping relationships. Finally, the output is converted into the probability distribution of each word in the vocabulary through the Softmax function;

[0135] Then, the prepared label data and feature data are input into the constructed speech recognition model for training;

[0136] During the training process, the cross entropy loss function can be used to measure the difference between the probability distribution of the model output and the true label; at the same time, a suitable optimization algorithm (such as the Adam optimization algorithm) is selected to update the weight parameters of the model according to the gradient calculated by the loss function, so that the model is continuously optimized and converges in the direction of minimizing the loss; during the training process, a part of the data is also divided as a validation set. After a certain number of training rounds (for example, every 10 training rounds), the validation set is used to evaluate the performance of the model, such as calculating the accuracy and loss value on the validation set. If it is found that the performance on the validation set is no longer improving or begins to decline, the early stopping strategy is used to stop the training to prevent the model from overfitting; finally, after completing the training on the entire training data set, an independent test set is used to perform a final performance test on the model to obtain the speech recognition model with the highest test accuracy;

[0137] Finally, the speech recognition model with the highest test accuracy obtained after the above training, verification and testing process is used as the final output and determined as the speech-to-text model. The input of this model is the feature vector group of the speech data obtained by the feature extraction module, and the output is the corresponding text data, which is used to actually perform the speech-to-text task in the speech recognition module and convert the input speech features into corresponding text content.

[0138] The first text obtained by using the speech-to-text model may not be completely accurate due to differences in professional fields or regions, or the timeliness of the collected voice data. Therefore, in this system, a personalized vocabulary that can include user professional vocabulary and shared vocabulary of related users is constructed through the vocabulary building module to improve the accuracy of speech-to-text.

[0139] Specifically, first, the vocabulary building module obtains the professional information in the current user's personal data, filters it from the professional database, and obtains the first vocabulary that matches the user's profession;

[0140] Then, the shared vocabulary of related users is obtained based on the geographic location in the current user's personal data:

[0141] C1, extracts geographic location information from the user's personal data, then uses the Euclidean distance formula to calculate the spatial distance between the current user and other users, and marks users whose spatial distance is less than a preset distance threshold as associated users;

[0142] C2, then uses the cosine similarity calculation formula to measure the professional similarity between the current user and these related users based on the professional information of the related users; then marks the related users whose professional similarity is greater than the preset similarity threshold as related users. For example, the preset similarity threshold is set to 0.85. If the calculated professional similarity between a related user and the current user reaches 0.9, then the related user is marked as a related user;

[0143] C3, counting the usage frequencies of words in the personalized word library shared by relevant users, screening out words whose usage frequencies exceed a preset frequency threshold, and combining these screened words to obtain a second word library;

[0144] Then, the first word library and the second word library are merged, all the words in the two word libraries are integrated together, and after removing duplicate words, a personalized word library for the current user is finally formed. This personalized word library contains both professional words directly related to the user's profession and some high-frequency words commonly used by other users in the same region, the same profession, or similar professions, which can better meet the user's personalized needs in the process of voice-to-text conversion.

[0145] In the text modification module, the first text is automatically corrected and modified according to the personalized vocabulary to replace the misrecognized words:

[0146] D1, using an existing natural language processing word segmentation tool to perform a word segmentation operation on the first text output by the speech recognition module to obtain a plurality of word segments;

[0147] D2, grouping the several segmented words obtained after the word segmentation processing and the several words in the personalized word library according to the word length;

[0148] Specifically, different length ranges are set to divide the groups, for example, the segmentation words and vocabulary with a length of 2-4 characters are grouped together, and the words with a length of 4-6 characters are grouped together. By using this grouping method, the calculation scope can be narrowed when similarity is calculated later, avoiding the comparison of all words in the personalized vocabulary one by one, thus improving processing efficiency;

[0149] D3, for each group, use the cosine similarity formula to calculate the similarity between the word segmentation and the vocabulary. That is, for each word segmentation and the personalized vocabulary vocabulary in the same group, convert them into word vectors, and then use the cosine similarity calculation formula to calculate the similarity between the two;

[0150] When the calculated similarity value is greater than the preset similarity value, the words in the personalized word library are marked as matching words for this word segmentation and collected into a temporary candidate set;

[0151] D4, using the marked matching words, according to their corresponding positions in the text, to replace the original participles in the first text, thereby completing the automatic modification of the corresponding part of the first text and obtaining the automatically modified second text.

[0152] In addition, after obtaining the second text, the text modification module will continue to manually check and modify the second text, and then record the words that need to be modified found in the manual modification process to obtain the manually modified words;

[0153] Finally, the personalized vocabulary of the current user is updated based on the manually modified vocabulary, and the personalized vocabulary is continuously improved to better adapt to subsequent speech-to-text and text modification tasks, thereby improving the accuracy and applicability of the entire system.

[0154] A second embodiment of the present invention provides a speech-to-text method based on a personalized vocabulary, comprising:

[0155] S1, collect personal data of the current user and obtain voice data of the current user; wherein the personal data includes professional information and geographic location;

[0156] S2, extracting features from the current user's voice data using a multi-band linear predictive coding method to obtain a feature vector group;

[0157] S3, inputting the feature vector group into a speech-to-text model to obtain a first text; wherein the speech-to-text model is constructed based on a deep learning algorithm;

[0158] S4, filtering from a professional database according to the personal data of the current user to obtain a first vocabulary;

[0159] S5, obtaining a shared vocabulary of related users based on the personal data of the current user, and obtaining a second vocabulary

[0160] S6, merging the first word library with the second word library to obtain a personalized word library for the current user;

[0161] S7, automatically modifying the first text according to the personalized vocabulary to obtain a second text, and manually modifying the second text to obtain manually modified vocabulary and a final text;

[0162] S8, updating the personalized vocabulary of the current user according to the manually modified vocabulary.

[0163] Part of the data in the above formula is calculated by removing the dimension and taking its numerical value. The formula is a formula closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0164] Working principle of the present invention:

[0165] First, the data collection module is responsible for collecting users’ voice data and personal data;

[0166] Next, the feature extraction module uses a multi-band linear predictive coding method to pre-emphasize the speech data, and then uses wavelet decomposition to obtain high and low frequency coefficients. It then extracts high and low frequency feature vectors respectively and concatenates and fuses them to obtain a feature vector group to capture the acoustic features in the speech and improve the accuracy of speech-to-text conversion.

[0167] Subsequently, the feature vector group is input into a speech-to-text model trained based on deep learning, and the first text is output;

[0168] At the same time, the word library construction module selects the first word library from the professional database, determines the relevant users based on the geographical location and professional similarity, and counts the high-frequency words in their shared word library to obtain the second word library, and then merges the first word library with the second word library to obtain a personalized word library;

[0169] Finally, the text modification module automatically and manually modifies the first text to obtain more accurate speech recognition results, and automatically updates the current user's personalized vocabulary based on the manually modified vocabulary, realizes dynamic optimization of the vocabulary, and further improves the accuracy of the system's speech-to-text conversion for the current user.

[0170] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A speech-to-text system based on a personalized vocabulary, characterized in that: Data collection module: used to collect the current user's personal data and obtain the current user's voice data; personal data includes professional information and geographic location; Feature extraction module: used to extract features from the current user's voice data using a multi-band linear predictive coding method to obtain a feature vector group; Speech recognition module: used to input the feature vector group into the speech-to-text model to obtain the first text; wherein the speech-to-text model is constructed based on a deep learning algorithm; The word library construction module is used to filter from a professional database according to the personal data of the current user to obtain a first word library, obtain a shared word library of related users according to the personal data of the current user, and obtain a second word library; and merge the first word library with the second word library to obtain a personalized word library for the current user; Text modification module: used to automatically modify the first text according to the personalized vocabulary to obtain the second text, and manually modify the second text to obtain the manually modified vocabulary and the final text; and update the personalized vocabulary of the current user according to the manually modified vocabulary.

2. A speech-to-text system based on a personalized vocabulary according to claim 1, characterized in that: The multi-band linear predictive coding method extracts features from the voice data of the current user, including: A1, using a first-order digital filter to perform pre-emphasis processing on the voice data of the current user to obtain a pre-emphasized voice signal sequence s(n); wherein n represents a time index; A2, using wavelet basis functions to perform L-layer wavelet decomposition on s(n) to obtain low-frequency coefficients dL(n) and high-frequency coefficients dl(n); where l represents the number of decomposition layers of wavelet decomposition, and l = 1, 2, ..., L; A3, using a multi-band linear prediction coding method to extract features of the low-frequency coefficient dL(n), and obtain low-frequency feature vectors Vlow in a number of time windows; A4, extracting features from the high-frequency coefficients dl(n) to obtain high-frequency feature vectors Vhigh within a number of time windows; A5, concatenate the low-frequency feature vector and the high-frequency feature vector in the i-th time window to obtain a number of fusion features Vi = [Vlow(i), Vhigh(i)]; where i = 1, 2, ..., N, and N represents the number of time windows; A6, arranging and integrating the plurality of fused features in chronological order to obtain a feature vector group V; wherein the feature vector group V = [V1, V2, ..., VN].

3. A speech-to-text system based on a personalized vocabulary according to claim 2, characterized in that: The method of extracting features of the low-frequency coefficient dL(n) by using a multi-band linear prediction coding method comprises: A31, divide dL(n) into windows according to preset time lengths to obtain a number of time windows; A32, for each time window, using discrete Fourier transform to convert the low-frequency coefficients in the time window from the time domain to the frequency domain, to obtain a frequency domain signal X(i,k); wherein i represents the time window index and k represents the frequency index in the frequency domain; A33, divide the frequency domain signal X(i,k) into M sub-bands, calculate the center frequency fcm of each sub-band, and assign the components of the corresponding frequency range in X(i,k) to the corresponding sub-band according to fcm, to obtain M sub-band signals; A34, using a linear prediction coding method to extract the features of the M sub-band signals respectively, and obtain the feature vector elements of the M sub-bands; A35, combining the feature vector elements of the M sub-bands to obtain a low-frequency feature vector Vlow(i) of a single time window; A36, repeat A32-A35 until a number of time windows are traversed to obtain low-frequency feature vectors Vlow in a number of time windows.

4. A speech-to-text system based on a personalized vocabulary according to claim 3, characterized in that: The M subbands are adjusted by using an adaptive characteristic parameter optimization method, including: A33-1, according to the formula E(n) = ∑|s(n+m)|^2 and the formula Z(n) = (1 / 2T) × ∑|sgn(s(n+m))-sgn(s(n+m-1))|, respectively calculate the short-time energy and zero-crossing rate of the pre-emphasized speech signal to obtain the short-time energy E(n) and the zero-crossing rate Z(n); wherein m represents the index of the sampling point in the short-time window, the summation range is m = 0, 1, ..., (T-1), T represents the short-time window length, and sgn represents the step function; A33-2, according to the formula ER = ∑ (E (n) - E (n - 1)) ^ 2 / ∑ (E (n)) ^ 2 and the formula ZCR = ∑ (Z (n) - Z (n - 1)) ^ 2 / ∑ (Z (n)) ^ 2 respectively calculate the energy change rate and the zero-crossing change rate, and obtain the energy change rate ER and the zero-crossing change rate ZCR; wherein the summation range is the start time point to the end time point of the speech signal; A33-3, calculate the complexity index CI of speech according to the formula CI = αER + (1-α) ZCR; wherein α represents the weight coefficient; A33-4, determine whether CI is greater than a preset threshold; if yes, then M=M+ΔM; if no, then M=M-ΔM; wherein ΔM represents the change in the number of subbands.

5. A speech-to-text system based on a personalized vocabulary according to claim 2, characterized in that: The feature extraction of the high frequency coefficient dl(n) comprises: A41, divide dl(n) into windows according to preset time lengths to obtain a plurality of time windows; A42, for each time window, the energy of the high frequency coefficient is calculated according to the formula El(i)=∑|dl(n)|^2 to obtain the energy feature El(i); wherein the summation range of the energy feature is a number of sampling points in the i-th time window; A43, perform Fourier transform on the high frequency coefficient dl(n) in each time window to obtain the spectrum feature Xl(i,k); A44, combining the energy feature El(i) and the spectrum feature Xl(i,k) to obtain a high-frequency feature vector Vhigh(i) of a single time window; A45, repeat A42-A43 until a number of time windows are traversed to obtain high-frequency feature vectors Vhigh in the number of time windows.

6. A speech-to-text system based on a personalized vocabulary according to claim 1 or 2, characterized in that: The speech-to-text model is built based on a deep learning algorithm and includes: B1, collect a number of voice data, and obtain corresponding text labels according to the number of voice data to obtain a number of label data; B2, preprocessing a number of speech data, and extracting features from the preprocessed speech data using a multi-band linear predictive coding method to obtain a number of feature data; B3, build a speech recognition model based on deep learning, input a number of label data and a number of feature data into the speech recognition model for training, verification and testing, and obtain the speech recognition model with the highest test accuracy; B4, taking the speech recognition model with the highest test accuracy as the final output, and obtaining a large speech-to-text model; wherein the input of the large speech-to-text model is the feature vector group of the speech data, and the output is the text data.

7. A speech-to-text system based on a personalized vocabulary according to claim 1, characterized in that: The step of obtaining a shared vocabulary of related users according to the personal data of the current user includes: C1, based on the geographic location in the personal data, the spatial distance between the current user and other users is calculated using the Euclidean distance formula, and users whose spatial distance is less than the preset distance threshold are marked as associated users; C2, based on the professional information in the personal data, uses the cosine similarity calculation formula to calculate the professional similarity between the current user and the associated users, and marks the associated users whose professional similarity is greater than the preset similarity threshold as related users; C3, counting the usage frequencies of words in the shared vocabulary of related users, screening and combining words whose usage frequencies exceed a preset frequency threshold, and obtaining a second vocabulary.

8. The speech-to-text system based on a personalized vocabulary according to claim 1, characterized in that: The automatically modifying the first text according to the personalized word library includes: D1, performing word segmentation on the first text to obtain a number of word segments; D2, grouping a number of word segmentations and a number of words in the personalized word library according to the length of the words to obtain a number of groups; D3, for each group, the similarity between the segmentation word and the vocabulary is calculated using the cosine similarity formula, and the vocabulary with a similarity greater than the preset similarity value is marked as a matching vocabulary; D4, using matching words to replace corresponding word segments in the first text to obtain an automatically modified second text.

9. A speech-to-text method based on a personalized word library, applied to a speech-to-text system based on a personalized word library as claimed in any one of claims 1 to 8, characterized in that: include S1, collect personal data of the current user and obtain voice data of the current user; wherein the personal data includes professional information and geographic location; S2, extracting features from the current user's voice data using a multi-band linear predictive coding method to obtain a feature vector group; S3, inputting the feature vector group into a speech-to-text model to obtain a first text; wherein the speech-to-text model is constructed based on a deep learning algorithm; S4, filtering from a professional database according to the personal data of the current user to obtain a first vocabulary; S5, obtaining a shared vocabulary of related users according to the personal data of the current user, and obtaining a second vocabulary; S6, merging the first word library with the second word library to obtain a personalized word library for the current user; S7, automatically modifying the first text according to the personalized vocabulary to obtain a second text, and manually modifying the second text to obtain manually modified vocabulary and a final text; S8, updating the personalized vocabulary of the current user according to the manually modified vocabulary.

Images