Continuous Learning Methods and Products for Large-Scale Speech Models Based on Intrinsic Information Dimension

Abstract

This application proposes a continuous learning method and product for a large speech model based on intrinsic information dimensions, relating to the field of speech recognition technology. First, the intrinsic information dimension of each speech data point in K speech datasets is determined. Then, based on the total number of speech data points and the intrinsic information dimension of the k-th speech dataset, the intrinsic information dimension of the k-th speech dataset is determined. Based on the intrinsic information dimensions of each of the K speech datasets, the number of model parameters for the training model is determined. The training model with the specified number of parameters is then continuously learned to obtain a large speech model for speech recognition. Finally, the speech to be recognized is input into the large speech model to obtain the recognition result. The large speech model obtained through continuous learning using the above method can significantly improve the ability to recognize speech generated in new dataset scenarios, while reducing the forgetting of speech type detection capabilities in previous dataset scenarios.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a continuous learning method and product for large speech models based on intrinsic information dimensions. Background Technology

[0002] In recent years, deep learning has driven the rapid development of speech conversion and synthesis technology, and the generated speech has become close to the level of a real person. It is widely used in human-computer interaction, smart home and other fields, but it also brings the risk of abuse, prompting attention to the generated speech detection technology.

[0003] Current deep learning-based fake speech detection models demonstrate excellent performance on known datasets, typically achieving high accuracy. However, these models show significant limitations in generalization when faced with speech generated by new or unknown algorithms, resulting in a marked drop in accuracy. This is particularly evident in cross-dataset applications; a detection model trained on the source dataset often experiences a substantial performance decrease when directly applied to the target dataset for speech detection. Summary of the Invention

[0004] This application provides a continuous learning method and product for large speech models based on intrinsic information dimensions, in order to solve the problem of decreased speech detection accuracy in cross-dataset scenarios of existing speech identification methods.

[0005] In a first aspect, this application proposes a continuous learning method for large speech models based on intrinsic information dimensions, the method comprising:

[0006] Determine the intrinsic information dimension of each speech data in the K speech datasets used for continuous learning. The intrinsic information dimension of each speech data is the minimum dimension required to describe the features of that speech data. Each dimension in the minimum dimension contains a subset of the features of that speech data, and the features contained in any two dimensions in the minimum dimension are unrelated. K is an integer greater than or equal to 2.

[0007] The intrinsic information dimension of the k-th speech dataset is determined based on the total number of speech data contained in the k-th speech dataset and the intrinsic information dimension of each speech data in the k-th speech dataset; k is an integer between 1 and K.

[0008] Based on the inherent information dimensions of each of the K speech datasets, determine the number of model parameters for the training model to be continuously learned from the K speech datasets.

[0009] Using the K speech datasets, the training model with the specified number of model parameters is continuously learned to obtain a large speech model for speech recognition.

[0010] The speech to be identified is input into the speech model to obtain the identification result of the speech to be identified. The identification result indicates whether the speech to be identified is real speech or synthetic speech.

[0011] Optionally, the intrinsic information dimension of each voice data point in the k-th voice dataset is determined according to the following steps:

[0012] For the i-th voice data in the k-th voice dataset, extract N voice data from the k-th voice dataset, where the N voice data include the i-th voice data and N is an integer greater than or equal to 3;

[0013] From the N voice data, determine the h voice data that are most similar to the i-th voice data in terms of features, where h is an integer greater than or equal to 2 and less than N;

[0014] Based on the features of the i-th speech data and the distance between them and the features of the corresponding h speech data, the intrinsic information dimension of the i-th speech data is determined.

[0015] Based on the intrinsic information dimension of each of the N voice data sets, the average intrinsic information dimension of the N voice data sets is determined, and used as the intrinsic information dimension of each voice data set in the k-th voice dataset.

[0016] Optionally, the method further includes:

[0017] The speech content, timbre, and generation algorithm of different speech data in the k-th speech dataset are compared to determine the information overlap of the k-th speech dataset.

[0018] Obtain the information overlap threshold of the user terminal configuration running the model to be trained with the specified number of model parameters;

[0019] If the information overlap of the k-th speech dataset is less than the information overlap threshold, the intrinsic information dimension of the k-th speech dataset remains unchanged.

[0020] If the information overlap of the k-th speech dataset is not less than the information overlap threshold, the difference between the intrinsic information dimension and the information overlap of the k-th speech dataset is taken as the new intrinsic information dimension of the k-th speech dataset.

[0021] Optionally, based on the inherent information dimensions of each of the K speech datasets, the number of model parameters for the training model to be continuously learned from the K speech datasets is determined, including:

[0022] With the goal of ensuring that the model to be trained does not forget the knowledge learned from the previously learned speech datasets during the continuous learning process, the lower limit of the number of model parameters is determined to be the maximum value among the intrinsic information dimensions of the K speech datasets;

[0023] The number of model parameters of the model to be trained is determined based on the lower limit of the number of model parameters.

[0024] Optionally, based on the inherent information dimensions of each of the K speech datasets, the number of model parameters for the training model to be continuously learned from the K speech datasets is determined, including:

[0025] With the goal of avoiding overfitting during the continuous learning process of the model to be trained, the upper limit of the number of model parameters is determined to be the sum of the intrinsic information dimensions of K speech datasets;

[0026] The number of model parameters of the model to be trained is determined based on the upper and lower limits of the number of model parameters.

[0027] Optionally, the number of model parameters of the model to be trained is determined based on the upper and lower limits of the number of model parameters, including:

[0028] Iterate through each candidate model parameter quantity between the upper and lower limits of the stated model parameter quantity;

[0029] Using the K speech datasets, the candidate training models with the number of candidate model parameters are continuously learned;

[0030] Determine the overfitting and underfitting information of the candidate model to be trained during the continuous learning process;

[0031] Obtain the overfitting and underfitting target conditions configured on the user terminal running the model to be trained with the specified number of model parameters;

[0032] The number of candidate model parameters that satisfy both the overfitting and underfitting target conditions is determined as the number of model parameters of the model to be trained.

[0033] Optionally, the method further includes:

[0034] If there are no candidate model parameters that satisfy both the overfitting and underfitting objective conditions, the value of h is continuously adjusted until a candidate model parameter that satisfies both the overfitting and underfitting objective conditions is determined.

[0035] In a second aspect, this application proposes a speech large model continuous learning device based on intrinsic information dimension, the device comprising:

[0036] The first determining module is used to determine the intrinsic information dimension of each speech data in the K speech datasets used for continuous learning. The intrinsic information dimension of each speech data is the minimum dimension required to describe the features of the speech data. Each dimension in the minimum dimension contains a portion of the features of the speech data, and the features contained in any two dimensions in the minimum dimension are unrelated. K is an integer greater than or equal to 2.

[0037] The second determining module is used to determine the intrinsic information dimension of the kth speech dataset based on the total number of speech data contained in the kth speech dataset and the intrinsic information dimension of each speech data in the kth speech dataset; k is an integer between 1 and K.

[0038] The third determining module is used to determine the number of model parameters of the model to be trained for continuous learning on the K speech datasets based on the inherent information dimensions of each of the K speech datasets.

[0039] The continuous learning module is used to continuously learn the training model with the number of model parameters using the K speech datasets to obtain a large speech model for speech recognition.

[0040] The identification module is used to input the speech to be identified into the speech model and obtain the identification result of the speech to be identified. The identification result indicates whether the speech to be identified is real speech or synthetic speech.

[0041] In a third aspect of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the speech large model continuous learning method based on the intrinsic information dimension as described in any of the first aspects above.

[0042] In a fourth aspect of this application, a computer-readable storage medium is provided, on which a computer program / instruction is stored, which, when executed by a processor, implements the speech large model continuous learning method based on the intrinsic information dimension as described in any of the first aspects above.

[0043] This application includes the following advantages: This application proposes a continuous learning method and product for a large speech model based on intrinsic information dimensions. First, the intrinsic information dimension of each speech data in the K speech datasets used for continuous learning is determined. Then, based on the total number of speech data in the k-th speech dataset and the intrinsic information dimension of each speech data in the k-th speech dataset, the intrinsic information dimension of the k-th speech dataset is determined. Further, based on the intrinsic information dimensions of each of the K speech datasets, the number of model parameters for the training model to be continuously learned from the K speech datasets is determined. Then, the training model with the specified number of model parameters is continuously learned using the K speech datasets to obtain a large speech model for speech recognition. Finally, the speech to be recognized is input into the large speech model to obtain the recognition result of the speech to be recognized. The recognition result indicates whether the speech to be recognized is real speech or synthetic speech.

[0044] This method determines the number of model parameters for continuous learning on multiple different datasets based on the inherent information dimensions of the learning speech data. Based on the number of model parameters, the model to be trained is continuously learned. The resulting large speech model can greatly improve the detection accuracy of generated speech in new dataset scenarios, while reducing the forgetting of the ability to detect speech types in previous dataset scenarios. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the steps of a continuous learning method for a large speech model based on the intrinsic information dimension proposed in this application.

[0047] Figure 2 This is a schematic diagram of a process for continuously learning a source speech dataset and a target speech dataset using a model to be trained, provided in an embodiment of this application.

[0048] Figure 3 This is a flowchart illustrating a method for estimating the intrinsic information dimension provided in an embodiment of this application;

[0049] Figure 4 This is a schematic diagram illustrating the process of determining model parameter quantities provided in an embodiment of this application;

[0050] Figure 5This is a schematic diagram of the functional modules of a speech large model continuous learning device based on the intrinsic information dimension provided in an embodiment of this application;

[0051] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] Currently, fake speech detection achieves high accuracy on most speech datasets. However, in real-world scenarios or across different speech datasets—that is, when training a model on the source speech dataset to detect the speech type in the target dataset—accuracy drops significantly. Fine-tuning the model parameters directly using data from the target dataset causes the model to "forget" the knowledge learned on the source speech dataset, resulting in a substantial decrease in the model's recognition accuracy on the source speech dataset. Existing continuous learning methods for generated speech identification mostly involve complex gradient operations from the model's perspective, making them difficult to apply effectively and widely in real-world scenarios.

[0054] Current research results indicate that fine-tuning the model on the target speech dataset can significantly improve the model's recognition accuracy on the target speech dataset, but it can also significantly reduce the model's recognition accuracy on the source speech dataset.

[0055] Speech recognition based on continuous learning has two learning objectives: to achieve good recognition accuracy on the target speech dataset, while minimizing the decrease in recognition accuracy on the source speech dataset. Based on this, this application aims to provide a continuous learning method that considers the intrinsic information dimensions of the speech dataset to be trained. First, the intrinsic information dimensions of different speech datasets are calculated. Then, the number of model parameters is determined based on these intrinsic information dimensions. Finally, a model with the determined number of model parameters is used to continuously learn from different speech datasets, thereby improving the overall detection performance of the model on various speech datasets.

[0056] In a first aspect, embodiments of this application propose a continuous learning method for large speech models based on intrinsic information dimensions, see [reference]. Figure 1 , Figure 1 This is a flowchart illustrating the steps of a continuous learning method for a large speech model based on intrinsic information dimension proposed in this application. The method includes the following steps:

[0057] Step 101: Determine the intrinsic information dimension of each speech data in the K speech datasets used for continuous learning.

[0058] In this embodiment, continuous learning refers to enabling the model to be trained to continuously learn new tasks or data in a dynamic environment, while avoiding forgetting old knowledge.

[0059] In this embodiment, in order to enable the model to be trained to learn continuously in multiple speech datasets, the number of speech datasets K used is an integer greater than or equal to 2.

[0060] It should be noted that all speech datasets used for training in this application contain both real speech data and generated speech data. These speech datasets can be actually acquired speech data or speech data from multiple datasets.

[0061] The continuous learning method proposed in this application determines the model parameters of the model to be trained based on the inherent information dimensions of the speech data in different speech datasets, and then uses the model to be trained containing the determined model parameters to continuously learn multiple different speech datasets.

[0062] In this context, the intrinsic information dimension of each speech data is the minimum dimension required to describe the features of that speech data. Each dimension in the minimum dimension contains a subset of the features of that speech data, and the features contained in any two dimensions in the minimum dimension are unrelated.

[0063] Intrinsic information dimension refers to the number of independent information dimensions in speech data or feature space. It represents the minimum dimension required to describe the speech data in that space. In other words, intrinsic information dimension is the amount of completely independent information in the data. Each intrinsic information dimension represents an independent feature that has no direct correlation or redundant information with other dimensions.

[0064] This means that each intrinsic information dimension contains a portion of the data, and this information is not "blurred" or "repetitive" due to changes in other dimensions. Therefore, the intrinsic information dimension is the minimum number of dimensions required to describe the data, and each intrinsic information dimension carries independent feature information.

[0065] Step 102: Determine the intrinsic information dimension of the k-th speech dataset based on the total number of speech data contained in the k-th speech dataset and the intrinsic information dimension of each speech data in the k-th speech dataset.

[0066] After obtaining the intrinsic information dimension of each speech data point in the speech dataset, for the k-th speech dataset out of the K speech datasets, where k is an integer between 1 and K, the intrinsic information dimension of the k-th speech dataset can be determined based on the total number of speech data points contained in the k-th speech dataset and the intrinsic information dimension of each speech data point in the k-th speech dataset.

[0067] Step 103: Determine the number of model parameters for the training model to be continuously learned from the K speech datasets based on the inherent information dimensions of each of the K speech datasets.

[0068] Based on the method for determining the intrinsic information dimension of the k-th speech dataset, the intrinsic information dimension of each of the k speech datasets can be obtained.

[0069] Then, based on the inherent information dimensions of each of the K speech datasets, the number of model parameters for the training model to be continuously learned from the K speech datasets is determined.

[0070] Introducing more model parameters helps the model being trained to learn from speech data in previously learned speech datasets, reducing the forgetting of speech data in new speech datasets.

[0071] The model obtained after continuous learning of the above-mentioned training model is the final model deployed to the user terminal for speech recognition.

[0072] Step 104: Using the K speech datasets, continuously learn the training model with the specified number of model parameters to obtain a large speech model for speech recognition.

[0073] The calculated model parameters are deployed into the above-mentioned training model, and K speech datasets are used to continuously learn the training model with the above-mentioned model parameters deployed, so as to obtain a large speech model for speech recognition.

[0074] The intrinsic information dimensionality of multiple speech datasets is estimated, and the number of model parameters for continuous learning across multiple datasets is determined. Furthermore, parameter constraints are introduced into the model to be trained (i.e., continuous learning is performed using a model to be trained with the determined number of model parameters), which reduces the accuracy drop of the model to be trained on previously learned speech datasets and improves the accuracy of the model to be trained on new speech datasets.

[0075] Step 105: Input the speech to be identified into the speech model to obtain the identification result of the speech to be identified. The identification result indicates whether the speech to be identified is real speech or synthetic speech.

[0076] By continuously learning the training model on actual acquired or cross-dataset speech data, the resulting large speech model can greatly improve the ability to identify speech generated in new scenarios, while reducing the forgetting of the ability to detect speech types in previous scenarios.

[0077] Based on this, by inputting the speech to be identified into the large speech model, the identification result of the identified speech can be obtained. This identification result can accurately characterize whether the speech to be identified is real speech or synthetic speech.

[0078] For example, when K is 2, the first speech dataset is the source speech dataset, and the second speech dataset is the target speech dataset. The process of continuously learning the source and target speech datasets using the trainable model is as follows: Figure 2 As shown ( Figure 2 This is a flowchart illustrating a process proposed in this application of continuously learning a source speech dataset and a target speech dataset using a model to be trained. First, the intrinsic information dimensions of the source speech dataset and the target speech dataset are calculated according to steps 101 and 102. Then, the number of model parameters of the model to be trained is determined based on the intrinsic information dimensions of the source speech dataset and the target speech dataset. Furthermore, the model to be trained, which contains the number of model parameters of the model to be trained, is continuously learned based on the source speech dataset, and the model to be trained, which contains the number of model parameters of the model to be trained, is continuously learned based on the target speech dataset, to obtain a large speech model for identifying newly synthesized speech.

[0079] This application proposes a continuous learning method for a large speech model based on intrinsic information dimensions. First, the intrinsic information dimension of each speech data point in the K speech datasets used for continuous learning is determined. Then, based on the total number of speech data points in the k-th speech dataset and the intrinsic information dimension of each speech data point in the k-th speech dataset, the intrinsic information dimension of the k-th speech dataset is determined. Further, based on the intrinsic information dimensions of each of the K speech datasets, the number of model parameters for the training model to be continuously trained on the K speech datasets is determined. Then, the training model with the specified number of model parameters is continuously trained using the K speech datasets to obtain a large speech model for speech recognition. Finally, the speech to be recognized is input into the large speech model to obtain the recognition result of the speech to be recognized. The recognition result indicates whether the speech to be recognized is real speech or synthetic speech.

[0080] This method designs a number of model parameters for continuous learning on multiple different datasets based on the inherent information dimensions of the learning speech data. The model to be trained is continuously learned based on the number of model parameters, and the resulting large speech model can greatly improve the ability to identify speech generated in new scenarios, while reducing the forgetting of the ability to detect speech types in previous scenarios.

[0081] Based on the above embodiments, in an optional implementation, see [reference needed]. Figure 3 , Figure 3 This is a flowchart illustrating an intrinsic information dimension estimation method proposed in this application. First, the data volume of the speech dataset to be trained is estimated. Then, the audio features of each speech data are extracted, and the intrinsic information dimension is estimated using the manifold learning method on these features. In this application, maximum likelihood estimation is used to estimate the intrinsic information dimension of the speech dataset.

[0082] Specifically, for the k-th speech dataset, the inherent information dimension of each speech data point in the k-th speech dataset is determined according to the following steps:

[0083] Step 1: For the i-th voice data in the k-th voice dataset, extract N voice data from the k-th voice dataset.

[0084] In this embodiment, for the i-th voice data in the k-th voice dataset, N voice data are extracted from the k-th voice dataset for internal information dimension calculation.

[0085] Among them, the N voice data extracted from the k-th voice dataset include the i-th voice data, where N is an integer greater than or equal to 3.

[0086] For example, if for the first voice data in the k-th voice dataset, the N voice data extracted from it are data that are close to the i-th voice data, but these N extracted voice data still need to include the i-th voice data.

[0087] Step 2: From the N voice data, determine the h voice data that are most similar to the i-th voice data in terms of features.

[0088] Based on the features of the extracted N speech data, the similarity between the features of these N speech data and the features of the i-th speech data is sorted from low to high. The h speech data with the highest similarity to the features of the i-th speech data are determined, where h is a hyperparameter and its value is an integer greater than or equal to 2 and less than N. For example, the value of h can be 10, 20, 50, etc.

[0089] Among them, the features of speech data can be time-domain features (such as short-time energy), frequency-domain features (such as MFCCs, Mel-Frequency Cepstral Coefficients), high-frequency semantic features (such as x-vector, speaker embedding), dynamic features (such as fundamental frequency (F0) trajectory), and nonlinear features (such as t-SNE two-dimensional embedding).

[0090] The degree of proximity can be calculated using the distance between features, such as Euclidean distance, discrete distance, etc. This implementation does not limit this, but only uses the actual calculation.

[0091] Step 3: Determine the intrinsic information dimension of the i-th speech data based on the features of the i-th speech data and the distance between the features of the corresponding h speech data.

[0092] Based on the h speech data points that are selected as having features closest to the i-th speech data point, the distances between the features of the i-th speech data point and the features of the h speech data points are calculated, thus determining the intrinsic information dimension of the i-th speech data point.

[0093] Step 4: Based on the intrinsic information dimension of each of the N voice data sets, determine the average intrinsic information dimension of the N voice data sets, and use it as the intrinsic information dimension of each voice data set in the k-th voice dataset.

[0094] Calculate the intrinsic information dimension of each of the N speech data based on the intrinsic information dimension of the i-th speech data, and then determine the average intrinsic information dimension of the N speech data, which is used as the intrinsic information dimension of each speech data in the k-th speech dataset.

[0095] It should be noted that, for each speech data in the k-th speech dataset, when determining the h speech data with the highest similarity of features among each speech data, if the N speech data extracted from the k-th speech dataset are different, then the h speech data with the highest similarity of features among the corresponding speech data will change accordingly.

[0096] For example, if N is 5 and h is 3, for the first speech data in the first speech dataset, extract 5 speech data (let's say the 1st, 2nd, 3rd, 4th, and 5th) from the first speech dataset. These 5 extracted speech data contain the first speech data in the first speech dataset. Extract the audio features of these 5 speech data to calculate the intrinsic information dimension of the first speech data.

[0097] Then, from the five speech data, the three speech data that are most similar to the features of the first speech data are identified, let's say the second, third and fourth speech data.

[0098] Then, based on the features of the first speech data and the distances between them and the features of the corresponding three speech data, the intrinsic information dimension of the first speech data is determined. .in This represents the distance between the features of the first speech data and the features of the second speech data. This represents the distance between the features of the second speech data and the features of the third speech data. This represents the distance between the features of the first speech data and the features of the fourth speech data. < < .

[0099] Then, for the second speech data in the first speech dataset, extract five speech data points from the first speech dataset (let's say the 2nd, 3rd, 4th, 5th, and 6th). Assume the speech data points closest to the 2nd speech data point are the 3rd, 4th, and 5th. Following the process of calculating the intrinsic information dimension of the 1st speech data point, determine the intrinsic information dimension of the 2nd speech data point as follows. Here, 'r' represents the distance between features of different speech data, and its meaning is the same as described above. < < .

[0100] For the third speech data point in the first speech dataset, extract five more speech data points from the first speech dataset (let's say the 3rd, 4th, 5th, 6th, and 7th). Assume the speech data points closest to the 3rd speech data point are the 4th, 5th, and 6th. Following the process of calculating the intrinsic information dimension of the 1st speech data point, determine the intrinsic information dimension of the 3rd speech data point as follows. Here, 'r' represents the distance between features of different speech data, and its meaning is the same as described above. < < .

[0101] For the fourth speech data point in the first speech dataset, extract five more speech data points from the first speech dataset (let's say the fourth, fifth, sixth, seventh, and eighth). Assume the speech data points closest to the fourth speech data point are the fifth, sixth, and seventh. Following the process of calculating the intrinsic information dimension of the first speech data point, determine the intrinsic information dimension of the fourth speech data point as follows: Here, 'r' represents the distance between features of different speech data, and its meaning is the same as described above. < < .

[0102] For the fifth speech data point in the first speech dataset, extract five more speech data points from the first speech dataset (let's say the 5th, 6th, 7th, 8th, and 9th). Assume the speech data points closest to the 5th speech data point are the 6th, 7th, and 8th. Following the process of calculating the intrinsic information dimension of the first speech data point, determine the intrinsic information dimension of the 4th speech data point as follows. Here, 'r' represents the distance between features of different speech data, and its meaning is the same as described above. < < .

[0103] Finally, based on the intrinsic information dimension of each of the five speech data points, the average intrinsic information dimension of the five speech data points was determined to be: This is used as the intrinsic information dimension of each voice data in the first voice dataset.

[0104] Based on the above embodiments, in an optional implementation, a continuous learning method for a large speech model based on intrinsic information dimension is also proposed. This method considers that the data in the speech dataset may be duplicated. For example, the first speech data is "Tomorrow I'm going to Shanghai," and the second speech data is "Tomorrow I'm going to Beijing." These two speech data are identical except for the location. The intrinsic information dimension of each speech data is the minimum dimension required to describe the features of that speech data. Each dimension in the minimum dimension contains a subset of the features of that speech data, and the features contained in any two dimensions in the minimum dimension are unrelated. Therefore, if there are many duplicate speech data, the degree of duplication between the speech data needs to be considered when calculating the intrinsic information dimension. The method specifically includes the following steps:

[0105] Step 106: Compare the speech content, timbre, and generation algorithm of different speech data in the k-th speech dataset to determine the information overlap of the k-th speech dataset.

[0106] For the k-th speech dataset, the speech content and / or timbre and / or generation algorithm of different speech data in the k-th speech dataset can be compared to determine the information overlap of the k-th speech dataset.

[0107] For example, if the speech content of the speech data has a high degree of repetition, there is no need to further compare the timbre or generation algorithm; however, if the speech content of the speech data has a low degree of repetition, the timbre of the speech data should be further compared, and if the timbre of the speech data has a low degree of repetition, the generation algorithm of the speech data should be further compared.

[0108] The information overlap of speech data can be calculated using the similarity between different speech data segments. For example, to calculate the overlap based on speech content, a specific method is to calculate cosine similarity. The calculation process includes: first, extracting the speech content of two speech data segments, and then calculating the cosine similarity of the speech content.

[0109] Step 107: Obtain the information overlap threshold of the user terminal configuration running the model to be trained with the specified model parameter amount.

[0110] After determining the information overlap of the k-th speech dataset, the information overlap is compared with the information overlap threshold configured for the user terminal that finally runs the training model with the specified number of model parameters.

[0111] Based on this, the information overlap threshold of the user terminal configuration for running the model to be trained with the above model parameter amounts is first obtained.

[0112] The user terminal here can be a computer, server, embedded device, smartphone, or tablet. Based on the different hardware and software configurations of the user terminal, different information overlap thresholds are determined, with values ​​ranging from [0, 1].

[0113] Step 108: If the information overlap of the kth speech dataset is less than the information overlap threshold, keep the intrinsic information dimension of the kth speech dataset unchanged.

[0114] In this embodiment, if the information overlap of the k-th speech dataset is less than the information overlap threshold, the intrinsic information dimension of the k-th speech dataset remains unchanged. This indicates that the information overlap of the k-th speech dataset does not affect the calculation of the intrinsic information dimension of the speech dataset.

[0115] Step 109: If the information overlap of the kth speech dataset is not less than the information overlap threshold, the difference between the intrinsic information dimension and the information overlap of the kth speech dataset is taken as the new intrinsic information dimension of the kth speech dataset.

[0116] In this embodiment, if the information overlap of the k speech datasets is not less than the information overlap threshold, the difference between the intrinsic information dimension and the information overlap of the kth speech dataset is taken as the new intrinsic information dimension of the kth speech dataset.

[0117] That is, the new intrinsic information dimension of the kth speech dataset = the intrinsic information dimension of the kth speech dataset - information overlap.

[0118] Based on the above embodiments, in an optional implementation, a continuous learning method for a large speech model based on the intrinsic information dimension is proposed. In this method, step 103 further includes the following steps:

[0119] Step 1031: With the goal of ensuring that the model to be trained does not forget the knowledge learned from the previously learned speech datasets during the continuous learning process, the lower limit of the number of model parameters is determined to be the maximum value among the intrinsic information dimensions of the K speech datasets.

[0120] To reduce the forgetting of knowledge from previously learned speech datasets by the model during continuous learning, the number of speech datasets to be trained is estimated, and then the number of model parameters required for training on the speech datasets is given based on the inherent information dimensions of the speech datasets to be trained.

[0121] For the k-th speech dataset, the number of model parameters The calculation formula is: ,in, This represents the information overlap of the speech data in the k-th speech dataset. , This represents the similarity between the speech content, timbre, or generation algorithm of speech data i and speech data j in the k-th speech dataset. This represents the intrinsic information dimension of the k-th speech dataset. This represents the total number of speech data in the k-th speech dataset.

[0122] Specifically, with the goal of ensuring that the model to be trained does not forget the knowledge learned from the previously learned speech datasets during the continuous learning process, the lower limit of the number of model parameters is determined to be the maximum value among the intrinsic information dimensions of the K speech datasets.

[0123] Since the content, timbre, and generation algorithm of each training speech data are mostly independent of each other during training, the information overlap in actual generated speech is small, generally less than the information overlap threshold, and can be ignored.

[0124] Therefore, for each speech dataset, the number of parameters needed to capture the complexity is required to ensure that the model to be trained can learn all the information on the most complex speech dataset. Specifically, this can be expressed as: for the k-th speech dataset, the number of model parameters... The lower limit is: .

[0125] Based on this, the lower bound for the number of model parameters for K speech datasets can be: .

[0126] Step 1032: Determine the number of model parameters of the model to be trained based on the lower limit of the number of model parameters.

[0127] Based on the lower limit of the number of model parameters mentioned above, the number of model parameters for the model to be trained can be determined. If the number of model parameters is less than the lower limit, the model may not have sufficient capacity to represent the complex structure of the most informative dataset, leading to underfitting.

[0128] Based on the above embodiments, in an optional implementation, a continuous learning method for a large speech model based on intrinsic information dimension is proposed. In this method, step 103 further includes:

[0129] Step 1033: With the goal of avoiding overfitting during the continuous learning process of the model to be trained, the upper limit of the number of model parameters is determined to be the sum of the intrinsic information dimensions of the K speech datasets.

[0130] When learning from multiple speech datasets in succession, the number of model parameters needs to be allocated to each speech dataset. The upper limit of the allocated number of model parameters is used to ensure that the capacity of the model to be trained is properly distributed across all training datasets and is sufficient to extract all information from each speech dataset.

[0131] Based on this, in order to avoid overfitting during the continuous learning process of the model to be trained, the upper limit of the number of model parameters is determined to be the sum of the intrinsic information dimensions of K speech datasets.

[0132] Specifically, this can be expressed as follows: For the k-th speech dataset, the upper limit of the number of model parameters is: .

[0133] Based on this, the upper limit of the number of model parameters for K speech datasets can be: .

[0134] If the number of parameters of the model to be trained exceeds the above limit, the model to be trained may overfit to the last learned speech dataset, resulting in a decrease in the performance of the model to be trained on other speech datasets, that is, forgetting the information of the speech datasets that have been learned.

[0135] Step 1034: Determine the number of model parameters of the model to be trained based on the upper and lower limits of the number of model parameters.

[0136] Finally, based on the upper and lower limits of the model parameter count, the number of model parameters to be trained is determined.

[0137] The formula for limiting the number of model parameters in the model to be trained can be expressed as: .

[0138] The number of parameters of the model to be trained can be determined from the range of these upper and lower limits.

[0139] Based on the above embodiments, in an optional implementation, see [reference needed]. Figure 4 , Figure 4 This is a schematic diagram illustrating a process for determining the number of model parameters proposed in an embodiment of this application. Using the aforementioned formula for limiting the number of model parameters, the range of model parameters for the model to be trained can be determined, and a suitable value between the upper and lower limits of the parameters can be selected as the number of model parameters. Then, continuous learning is performed on multiple speech datasets based on the model to be trained containing this number of model parameters. Specifically, determining the number of model parameters for the model to be trained based on the upper and lower limits of the model parameters includes the following steps:

[0140] Step 5: Iterate through each candidate model parameter quantity between the upper and lower limits of the model parameter quantity.

[0141] Based on the constraint formula for the number of model parameters, multiple candidate model parameter quantities can be determined. By iterating through each candidate model parameter quantity using the model to be trained, multiple candidate models to be trained can be obtained.

[0142] Step 6: Using the K speech datasets, continuously learn the candidate training models with the number of candidate model parameters.

[0143] Then, using K speech datasets, continuous learning is performed on candidate trainable models with candidate model parameter counts to determine whether the candidate trainable models with candidate model parameter counts can be applied to the target device.

[0144] Step 7: Determine the overfitting and underfitting information of the candidate model to be trained during the continuous learning process.

[0145] To confirm whether candidate trainable models, including their parameter sets, can be applied to the target device, overfitting and underfitting information for each candidate model during continuous learning is determined. Overfitting information refers to information about a model that performs well on previously learned speech datasets but performs poorly on new, unlearned speech datasets. Underfitting information refers to information about a candidate trainable model that performs poorly on previously learned speech datasets during continuous learning.

[0146] Step 8: Obtain the overfitting and underfitting target conditions configured on the user terminal running the model to be trained with the specified number of model parameters.

[0147] The overfitting and underfitting information is compared with the overfitting and underfitting target conditions configured on the target terminal to determine whether the candidate model to be trained with the above-mentioned number of candidate model parameters can be applied to the user terminal.

[0148] Step 9: The number of candidate model parameters that satisfy both the overfitting and underfitting target conditions is determined as the number of model parameters of the model to be trained.

[0149] Only when both overfitting and underfitting information meet the overfitting and underfitting conditions configured by the user terminal will the candidate model parameter count of the candidate model to be trained be determined as the final model parameter count of the model to be trained.

[0150] Specifically, the number of candidate model parameters that satisfy both the overfitting and underfitting target conditions is determined as the number of model parameters to be trained.

[0151] Based on the above embodiments, in an optional implementation, this application also proposes a continuous learning method for a large speech model based on the intrinsic information dimension. The method further includes: continuously adjusting the value of h until a candidate model parameter quantity that satisfies both the overfitting and underfitting objective conditions is determined when there are no candidate model parameters that satisfy both the overfitting and underfitting objective conditions.

[0152] In a second aspect, embodiments of this application propose a continuous learning device for a large speech model based on intrinsic information dimensions, see [reference]. Figure 5 , Figure 5 This is a functional module diagram of a speech large model continuous learning device based on intrinsic information dimension proposed in an embodiment of this application. The device includes:

[0153] The first determining module 501 is used to determine the intrinsic information dimension of each speech data in the K speech datasets used for continuous learning. The intrinsic information dimension of each speech data is the minimum dimension required to describe the features of the speech data. Each dimension in the minimum dimension contains a portion of the features of the speech data, and the features contained in any two dimensions in the minimum dimension are unrelated. K is an integer greater than or equal to 2.

[0154] The second determining module 502 is used to determine the intrinsic information dimension of the kth speech dataset based on the total number of speech data contained in the kth speech dataset and the intrinsic information dimension of each speech data in the kth speech dataset; k is an integer between 1 and K.

[0155] The third determining module 503 is used to determine the number of model parameters of the model to be trained for continuous learning of the K speech datasets based on the inherent information dimensions of each of the K speech datasets.

[0156] The continuous learning module 504 is used to continuously learn the training model with the number of model parameters using the K speech datasets to obtain a large speech model for speech recognition.

[0157] The identification module 505 is used to input the speech to be identified into the speech model and obtain the identification result of the speech to be identified. The identification result indicates whether the speech to be identified is real speech or synthetic speech.

[0158] The second determining module further includes a determining submodule, which is used to determine the intrinsic information dimension of each voice data in the k-th voice dataset. The determining submodule also includes:

[0159] The speech data extraction unit is used to extract N speech data from the k-th speech dataset for the i-th speech data in the k-th speech dataset, wherein the N speech data includes the i-th speech data and N is an integer greater than or equal to 3;

[0160] The proximity determination unit is used to determine, from the N speech data, the h speech data that are most closely related to the i-th speech data in terms of features, where h is an integer greater than or equal to 2 and less than N;

[0161] The first intrinsic information dimension calculation unit is used to determine the intrinsic information dimension of the i-th speech data based on the feature of the i-th speech data and the distance between the feature of the corresponding h speech data.

[0162] The second intrinsic information dimension calculation unit is used to determine the average intrinsic information dimension of the N voice data based on the intrinsic information dimension of each voice data in the N voice data, and use it as the intrinsic information dimension of each voice data in the k-th voice dataset.

[0163] The device further includes:

[0164] The comparison module is used to compare the speech content, timbre, and generation algorithm of different speech data in the kth speech dataset to determine the information overlap of the kth speech dataset.

[0165] The information overlap threshold acquisition module is used to acquire the information overlap threshold of the user terminal configuration running the model to be trained with the number of model parameters.

[0166] The intrinsic information dimension determination module is used to keep the intrinsic information dimension of the kth speech dataset unchanged when the information overlap of the kth speech dataset is less than the information overlap threshold.

[0167] The intrinsic information dimension update module is used to take the difference between the intrinsic information dimension and the information overlap of the kth speech dataset as the new intrinsic information dimension of the kth speech dataset, provided that the information overlap of the kth speech dataset is not less than the information overlap threshold.

[0168] The third determining module includes:

[0169] The model parameter quantity lower limit determination submodule is used to determine the lower limit of the model parameter quantity as the maximum value among the intrinsic information dimensions of K speech datasets, with the goal of ensuring that the model to be trained does not forget the knowledge learned from the previously learned speech datasets during continuous learning.

[0170] The first model parameter determination submodule is used to determine the number of model parameters of the model to be trained based on the lower limit of the number of model parameters.

[0171] The third determining module further includes:

[0172] The model parameter upper limit determination submodule is used to determine the upper limit of the model parameter quantity as the sum of the intrinsic information dimensions of K speech datasets, with the goal of avoiding overfitting during the continuous learning process of the model to be trained.

[0173] The second model parameter determination submodule is used to determine the number of model parameters of the model to be trained based on the upper and lower limits of the number of model parameters.

[0174] The second model parameter determination submodule includes:

[0175] A traversal unit is used to traverse each candidate model parameter quantity between the upper and lower limits of the model parameter quantity;

[0176] A continuous learning unit is used to continuously learn a candidate model to be trained with the number of candidate model parameters using the K speech datasets.

[0177] The fitting information determination unit is used to determine the overfitting and underfitting information of the candidate model to be trained during the continuous learning process.

[0178] The target condition acquisition unit is used to acquire the overfitting target conditions and underfitting target conditions configured on the user terminal running the model to be trained with the number of model parameters.

[0179] The model parameter determination unit is used to determine the number of candidate model parameters that satisfy both the overfitting and underfitting target conditions as the number of model parameters of the model to be trained.

[0180] The device further includes:

[0181] The adjustment module is used to continuously adjust the value of h until a number of candidate model parameters that satisfy both the overfitting and underfitting target conditions are determined when there are no candidate model parameters that satisfy both the overfitting and underfitting target conditions.

[0182] Based on the same concept, this application discloses an electronic device in a third aspect. Figure 6 A schematic diagram of an electronic device disclosed in an embodiment of this application is shown, such as... Figure 6 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory of the electronic device is not less than 12G, and the main frequency of the processor is not less than 2.4GHz. The memory 110 and the processor 120 are connected by a bus communication. The memory 110 stores a computer program, which can run on the processor 120 to implement a speech large model continuous learning method based on intrinsic information dimension disclosed in the embodiments of this application.

[0183] Based on the same concept, this application discloses a computer-readable storage medium storing a computer program / instruction thereon in a fourth aspect. When the computer program / instruction is executed by a processor, it implements a speech large model continuous learning method based on intrinsic information dimension disclosed in this application.

[0184] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0185] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0186] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0188] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0189] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0190] The above provides a detailed description of the continuous learning method and product for a large speech model based on the intrinsic information dimension provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A continuous learning method for large speech models based on intrinsic information dimension, characterized in that, The method includes: Determine the intrinsic information dimension of each speech data in the K speech datasets used for continuous learning. The intrinsic information dimension of each speech data is the minimum dimension required to describe the features of that speech data. Each dimension in the minimum dimension contains a subset of the features of that speech data, and the features contained in any two dimensions in the minimum dimension are unrelated. K is an integer greater than or equal to 2. The intrinsic information dimension of the k-th speech dataset is determined based on the total number of speech data contained in the k-th speech dataset and the intrinsic information dimension of each speech data in the k-th speech dataset; k is an integer between 1 and K. Based on the inherent information dimensions of each of the K speech datasets, determine the number of model parameters for the training model to be continuously learned from the K speech datasets. Using the K speech datasets, the training model with the specified number of model parameters is continuously learned to obtain a large speech model for speech recognition. The speech to be identified is input into the speech model to obtain the identification result of the speech to be identified. The identification result indicates whether the speech to be identified is real speech or synthetic speech.

2. The continuous learning method for large speech models based on intrinsic information dimension according to claim 1, characterized in that, The intrinsic information dimension of each voice data point in the k-th voice dataset is determined according to the following steps: For the i-th voice data in the k-th voice dataset, extract N voice data from the k-th voice dataset, where the N voice data include the i-th voice data and N is an integer greater than or equal to 3; From the N voice data, determine the h voice data that are most similar to the i-th voice data in terms of features, where h is an integer greater than or equal to 2 and less than N; Based on the features of the i-th speech data and the distance between them and the features of the corresponding h speech data, the intrinsic information dimension of the i-th speech data is determined. Based on the intrinsic information dimension of each of the N voice data sets, the average intrinsic information dimension of the N voice data sets is determined, and used as the intrinsic information dimension of each voice data set in the k-th voice dataset.

3. The continuous learning method for a large speech model based on intrinsic information dimension according to claim 2, characterized in that, The method further includes: The speech content, timbre, and generation algorithm of different speech data in the k-th speech dataset are compared to determine the information overlap of the k-th speech dataset. Obtain the information overlap threshold of the user terminal configuration running the model to be trained with the specified number of model parameters; If the information overlap of the k-th speech dataset is less than the information overlap threshold, the intrinsic information dimension of the k-th speech dataset remains unchanged. If the information overlap of the k-th speech dataset is not less than the information overlap threshold, the difference between the intrinsic information dimension and the information overlap of the k-th speech dataset is taken as the new intrinsic information dimension of the k-th speech dataset.

4. The continuous learning method for large speech models based on intrinsic information dimension according to claim 2, characterized in that, Based on the inherent information dimensions of each of the K speech datasets, the number of model parameters for the training model to be continuously learned from the K speech datasets is determined, including: With the goal of ensuring that the model to be trained does not forget the knowledge learned from the previously learned speech datasets during the continuous learning process, the lower limit of the number of model parameters is determined to be the maximum value among the intrinsic information dimensions of the K speech datasets; The number of model parameters of the model to be trained is determined based on the lower limit of the number of model parameters.

5. The continuous learning method for a large speech model based on intrinsic information dimension according to claim 4, characterized in that, Based on the inherent information dimensions of each of the K speech datasets, the number of model parameters for the training model to be continuously learned from the K speech datasets is determined, including: With the goal of avoiding overfitting during the continuous learning process of the model to be trained, the upper limit of the number of model parameters is determined to be the sum of the intrinsic information dimensions of K speech datasets; The number of model parameters of the model to be trained is determined based on the upper and lower limits of the number of model parameters.

6. The continuous learning method for a large speech model based on intrinsic information dimension according to claim 5, characterized in that, The number of model parameters of the model to be trained is determined based on the upper and lower limits of the number of model parameters, including: Iterate through each candidate model parameter quantity between the upper and lower limits of the stated model parameter quantity; Using the K speech datasets, the candidate training models with the number of candidate model parameters are continuously learned; Determine the overfitting and underfitting information of the candidate model to be trained during the continuous learning process; Obtain the overfitting and underfitting target conditions configured on the user terminal running the model to be trained with the specified number of model parameters; The number of candidate model parameters that satisfy both the overfitting and underfitting target conditions is determined as the number of model parameters of the model to be trained.

7. The continuous learning method for a large speech model based on intrinsic information dimension according to claim 6, characterized in that, The method further includes: If there are no candidate model parameters that satisfy both the overfitting and underfitting objective conditions, the value of h is continuously adjusted until a candidate model parameter that satisfies both the overfitting and underfitting objective conditions is determined.

8. A continuous learning device for a large speech model based on intrinsic information dimension, characterized in that, The device includes: The first determining module is used to determine the intrinsic information dimension of each speech data in the K speech datasets used for continuous learning. The intrinsic information dimension of each speech data is the minimum dimension required to describe the features of the speech data. Each dimension in the minimum dimension contains a portion of the features of the speech data, and the features contained in any two dimensions in the minimum dimension are unrelated. K is an integer greater than or equal to 2. The second determining module is used to determine the intrinsic information dimension of the kth speech dataset based on the total number of speech data contained in the kth speech dataset and the intrinsic information dimension of each speech data in the kth speech dataset; k is an integer between 1 and K. The third determining module is used to determine the number of model parameters of the model to be trained for continuous learning on the K speech datasets based on the inherent information dimensions of each of the K speech datasets. The continuous learning module is used to continuously learn the training model with the number of model parameters using the K speech datasets to obtain a large speech model for speech recognition. The identification module is used to input the speech to be identified into the speech model and obtain the identification result of the speech to be identified. The identification result indicates whether the speech to be identified is real speech or synthetic speech.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the continuous learning method for a large speech model based on the intrinsic information dimension as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program / instruction, which, when executed by a processor, implements the speech large model continuous learning method based on the intrinsic information dimension as described in any one of claims 1-7.

Images