Tone library expansion method and device, electronic equipment and storage medium
By selecting sample voiceprint features in the tone library for mixing and similarity calculation, the tone library is expanded, and the problem of limited number of copyrighted tones of the tone library is solved, achieving the multi-character dubbing needs and cost reduction effects.
Patent Information
- Application Number
- CN202411264939.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, the number of copyrighted tones in the tone library is limited, which cannot meet the dubbing needs of film and television dramas with more characters. Due to the high cost of sound sampling and the objection of some sound source providers, it is difficult to add the tone library through personal sound sampling. copyright tone.
By selecting at least two sample voiceprint features from the voiceprint features of the voiceprint library for mixing, the target voiceprint features are generated, and the preset test audio is transformed based on the feature to calculate the voiceprint of the audio to be detected and the sample audio to be detected. Similarity. If the similarity is less than the threshold, the target voiceprint feature is added to the tone library to expand the number of tones in the tone library.
By mixing existing tones, increasing the number and types of tones in the tone library, meeting the dubbing needs of multiple characters in the film and television dramas, reducing the cost and complexity of adding new tones.
Smart Images

Figure CN119028357A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of audio processing, and in particular to a method, device, electronic device and storage medium for expanding a timbre library. Background Art
[0002] In the dubbing of film and television dramas, in order to improve the recognition between different characters, different characters often need to be dubbed by different dubbing artists. With the rise of short videos, the scope of dubbing applications has become more extensive. Many people are reluctant to use their own voices to dub videos, so VC (Voice Conversion) technology is used to convert the timbre in the dubbing audio into the timbre already in the timbre library.
[0003] In the prior art, the conversion of timbre using VC technology requires reliance on a timbre library, and one copyrighted timbre in the timbre library can only generate one timbre of dubbing. Limited by the cost of sound sampling and the opposition of some sound source providers to AI (Artificial Intelligence) products, it is difficult to increase the copyrighted timbre in the timbre library by sampling personal voices. The number of timbre that can be obtained in the timbre library is limited, and it cannot meet the dubbing needs of TV dramas with more characters. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for expanding the capacity of a timbre library, so as to increase the number of timbres in the timbre library. The specific technical solution is as follows:
[0005] In a first aspect, a method for expanding a timbre library is provided, comprising:
[0006] Select at least two sample voiceprint features from each voiceprint feature in the timbre library;
[0007] Mixing the sample voiceprint features to obtain a target voiceprint feature;
[0008] Performing timbre transformation on a preset test audio based on the target voiceprint feature to obtain an audio to be tested;
[0009] Acquire sample audio under each voiceprint feature in the timbre library, wherein the sample audio is obtained by performing timbre transformation on the preset test audio;
[0010] Calculating the voiceprint similarity between the audio to be detected and each of the sample audios respectively;
[0011] If each of the similarities is less than a preset similarity threshold, the target voiceprint feature is added to the timbre library.
[0012] Optionally, mixing the sample voiceprint features to obtain a target voiceprint feature includes:
[0013] Get the preset weight coefficient;
[0014] The current weight coefficient is used to weight each of the sample voiceprint features to obtain the target voiceprint feature.
[0015] Optionally, after respectively calculating the voiceprint similarities between the audio to be detected and each of the sample audios, the method further includes:
[0016] If there is a first similarity among the similarities that is greater than a preset similarity threshold, the weight coefficient is updated, and the process returns to the step of weighting the sample voiceprint features using the current weight coefficient to obtain a target voiceprint feature.
[0017] Optionally, updating the weight coefficient includes:
[0018] If the first voiceprint feature corresponding to the first similarity is a part of the sample voiceprint features, the weight coefficient of the first voiceprint feature is reduced.
[0019] Optionally, the method further comprises:
[0020] If the first voiceprint feature corresponding to the first similarity includes each of the sample voiceprint features, then reselect at least two sample voiceprint features from each of the voiceprint features in the timbre library, and return to the execution step: mix each of the sample voiceprint features to obtain a target voiceprint feature, wherein the first voiceprint feature is a voiceprint feature used when converting a preset test audio timbre into a first sample audio, and the first sample audio is a sample audio whose similarity to the audio to be detected is the first similarity.
[0021] Optionally, the method further comprises:
[0022] If the first voiceprint feature corresponding to the first similarity includes each of the sample voiceprint features, then the voiceprint feature corresponding to the minimum similarity is selected from the voiceprint features in the timbre library and added to the sample voiceprint feature, and the execution step is returned to: mixing each of the sample voiceprint features to obtain the target voiceprint feature.
[0023] Optionally, respectively calculating the voiceprint similarity between the audio to be detected and each of the sample audios includes:
[0024] Extracting the voiceprint of the audio to be detected and each of the sample audios to obtain a plurality of voiceprint data;
[0025] The cosine similarities between the voiceprint data of the audio to be detected and the voiceprint data of each of the sample audios are calculated respectively to obtain multiple similarities.
[0026] In a second aspect, a device for expanding a timbre library is provided, the device comprising:
[0027] Voiceprint feature selection module, sound conversion module, timbre confusion module, voiceprint comparison module:
[0028] The voiceprint feature selection module is used to select at least two sample voiceprint features from each voiceprint feature in the timbre library;
[0029] The timbre confusion module is used to mix the sample voiceprint features to obtain the target voiceprint features;
[0030] The sound conversion module is used to perform timbre conversion on the preset test audio based on the target voiceprint feature to obtain the audio to be detected; obtain the sample audio under each voiceprint feature in the timbre library, wherein the sample audio is obtained after the timbre conversion of the preset test audio;
[0031] The voiceprint comparison module is used to respectively calculate the voiceprint similarity between the audio to be detected and each of the sample audios; if each of the similarities is less than a preset similarity threshold, the target voiceprint feature is added to the timbre library.
[0032] Optionally, the timbre confusion module is specifically used to: obtain a preset weight coefficient; and weight each of the sample voiceprint features using the current weight coefficient to obtain a target voiceprint feature;
[0033] Optionally, the device further includes: a weight updating module, configured to update the weight coefficient if there is a first similarity among the similarities that is greater than a preset similarity threshold, and return to the execution step of weighting each of the sample voiceprint features using the current weight coefficient to obtain a target voiceprint feature.
[0034] Optionally, the weight updating module is specifically used for:
[0035] If the first voiceprint feature corresponding to the first similarity is a part of the sample voiceprint feature, the weight coefficient of the first voiceprint feature is reduced, wherein the first voiceprint feature is the voiceprint feature used when converting the preset test audio timbre into the first sample audio, and the first sample audio is the sample audio whose similarity with the audio to be detected is the first similarity.
[0036] Optionally, the device further includes: a sample updating module, configured to:
[0037] If the first voiceprint feature corresponding to the first similarity includes each of the sample voiceprint features, then reselect at least two sample voiceprint features from each of the voiceprint features in the timbre library, and return to the execution step: mix each of the sample voiceprint features to obtain a target voiceprint feature.
[0038] The sample update module is also used for:
[0039] If the first voiceprint feature corresponding to the first similarity includes each of the sample voiceprint features, then the voiceprint feature corresponding to the minimum similarity is selected from the voiceprint features in the timbre library and added to the sample voiceprint feature, and the execution step is returned to: mixing each of the sample voiceprint features to obtain the target voiceprint feature.
[0040] Optionally, the voiceprint comparison module is specifically used to:
[0041] Extracting the voiceprint of the audio to be detected and each of the sample audios to obtain a plurality of voiceprint data;
[0042] The cosine similarities between the voiceprint data of the audio to be detected and the voiceprint data of each of the sample audios are calculated respectively to obtain multiple similarities.
[0043] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0044] Memory, used to store computer programs;
[0045] The processor is used to implement any method step described in the first aspect when executing a program stored in the memory.
[0046] In another aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned methods for expanding the timbre library is implemented.
[0047] In another aspect of the implementation of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned methods for expanding the timbre library.
[0048] The method for expanding the timbre library provided in the embodiment of the present application selects at least two sample voiceprint features from each voiceprint feature in the timbre library, mixes each of the sample voiceprint features to obtain a target voiceprint feature, and performs timbre transformation on a preset test audio based on the target voiceprint feature to obtain the audio to be detected; obtains sample audio under each voiceprint feature in the timbre library, wherein the sample audio is obtained after timbre transformation of the preset test audio; respectively calculates the voiceprint similarity between the audio to be detected and each of the sample audios; if each of the similarities is less than a preset similarity threshold, the target voiceprint feature is added to the timbre library. By mixing the existing timbres in the timbre library, the number of timbres in the timbre library can be increased, and by calculating the similarity, it can be effectively ensured that each timbre in the timbre library is different, which can meet the dubbing needs of film and television dramas with a large number of characters. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.
[0050] Figure 1 This is a first flow chart of the method for expanding the timbre library in the embodiment of the present application;
[0051] Figure 2 This is a flow chart of a mixing method in a method for expanding a timbre library in an embodiment of the present application;
[0052] Figure 3 This is a second flow chart of the method for expanding the timbre library in the embodiment of the present application;
[0053] Figure 4 This is a third flow chart of the method for expanding the timbre library in the embodiment of the present application;
[0054] Figure 5 This is a fourth flow chart of the method for expanding the timbre library in the embodiment of the present application;
[0055] Figure 6 This is a fifth flow chart of the method for expanding the timbre library in the embodiment of the present application;
[0056] Figure 7 This is a flow chart of a voiceprint similarity comparison method in a timbre library expansion method in an embodiment of the present application;
[0057] Figure 8 This is a schematic diagram of a device for expanding the timbre library in an embodiment of the present application;
[0058] Fig. 9 Schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION
[0059] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0060] In order to protect the copyright of sound, the timbres in the timbre library need to be genuine licensed timbres. However, due to the cost of sound sampling and the opposition of some sound source providers to AI (Artificial Intelligence) products, it is difficult to add copyrighted timbres to the timbre library by sampling personal voices. The number of timbres that can be obtained in the timbre library is limited, which cannot meet the dubbing needs of TV dramas with more characters. In order to solve this problem, the embodiment of the present application provides a method for expanding the timbre library, such as Figure 1 As shown, including:
[0061] S101, selecting at least two sample voiceprint features from each voiceprint feature in the timbre library;
[0062] S102, mixing the sample voiceprint features to obtain a target voiceprint feature;
[0063] S103, performing timbre transformation on a preset test audio based on the target voiceprint feature to obtain an audio to be tested;
[0064] S104, obtaining a sample audio under each voiceprint feature in the timbre library, wherein the sample audio is obtained by performing timbre transformation on the preset test audio;
[0065] S105, respectively calculating the voiceprint similarity between the audio to be detected and each of the sample audios;
[0066] S106: If each of the similarities is less than a preset similarity threshold, the target voiceprint feature is added to the timbre library.
[0067] By mixing the existing timbres in the timbre library, the number of timbres in the timbre library can be increased. By calculating the similarity, it can be effectively ensured that the timbres in the timbre library are different, which can meet the dubbing needs of film and television dramas with more characters.
[0068] This application mainly solves the problem of the need to expand the timbre library. On the basis of a certain number of existing timbres, the timbre confusion technology is used to mix the existing timbres and then generate new timbres. The similarity between the confused timbre and the existing timbre is calculated using the voiceprint similarity, and the timbre with the least similarity is selected and added to the timbre library. In this way, the number and type of timbres in the timbre library are supplemented to meet the demand for the number of timbres for dubbing. The following is a detailed explanation:
[0069] Figure 1 A method for expanding a timbre library provided in an embodiment of the present application includes:
[0070] S101, selecting at least two sample voiceprint features from each voiceprint feature in the timbre library.
[0071] In the embodiment of the present application, the voiceprint feature may be speaker embedding, which is a matrix model feature that can be used for VC (Voice Conversion) model training. One timbre corresponds to one voiceprint feature, that is, each timbre corresponds to one feature vector / matrix. During each expansion of the timbre library, the voiceprint feature selected from the timbre library for mixing is called a sample voiceprint feature. The number of sample voiceprint features selected each time can be set according to actual conditions, for example, it can be 2, 3, 4 or more.
[0072] S102, mixing the voiceprint features of the samples to obtain the target voiceprint features.
[0073] In one example, mixing can be performed by weighted averaging, in which case the target voiceprint feature is the voiceprint feature obtained by weighted mixing of the sample voiceprint features. In other possible implementations, other mixing techniques besides weighted averaging can also be used, all of which are within the scope of protection of this application.
[0074] S103, performing timbre transformation on a preset test audio based on the target voiceprint feature to obtain an audio to be tested.
[0075] In S103, VC technology is used for timbre conversion. Prepare multiple preset tests for timbre conversion, use VC technology to extract audio information such as audio text content, emotion, and rhythmic features from the preset test audio, and then combine the extracted audio information with the target voiceprint features to obtain the audio to be tested. Audio information such as text content, emotion, and rhythmic features can be extracted using general speech emotion representation models such as Whisper's encoder model, emotin2vec model, and vqwav2vec model, which are not limited in the embodiments of this application.
[0076] S104, obtaining sample audio under each voiceprint feature in the timbre library, wherein the sample audio is obtained by performing timbre transformation on a preset test audio.
[0077] In S104, the sample audio is the audio obtained by converting the timbre of the preset test audio into a timbre specified in the timbre library through VC technology. The sample audio can be pre-generated and stored audio, or it can be obtained by transforming the timbre of the preset test audio using the voiceprint features in the timbre library in real time. This is not specifically limited in the present application.
[0078] S105, respectively calculating the voiceprint similarity between the audio to be detected and each sample audio.
[0079] The cosine similarity of the audio to be detected and each sample audio can be compared in sections, and the average of the obtained cosine similarities can be calculated to obtain the voiceprint similarity in S105.
[0080] S106: If each similarity is less than a preset similarity threshold, the target voiceprint feature is added to the timbre library.
[0081] If all of the similarities are less than the preset similarity threshold, that is, the voiceprint similarities between the target voiceprint feature and the voiceprint feature corresponding to each timbre in the timbre library are less than the preset similarity threshold, it can be considered that the timbre corresponding to the target voiceprint feature is different from any timbre in the timbre library, and the target voiceprint feature is added to the timbre library as the voiceprint feature of the new timbre.
[0082] In an embodiment of the present application, a new timbre is generated by selecting existing timbres in the timbre library and mixing and superimposing them, the generated new timbre is compared with the existing timbre in the sound library, and the new timbre with a similarity lower than a similarity threshold is selected and added to the timbre library, thereby expanding the available timbre in the timbre library.
[0083] The method for expanding the timbre library of the embodiment of the present application is simpler than the traditional method of adding new timbres through vocal sampling. The generation of new timbres no longer depends on specific people, and the requirements for the implementers of the method are lower, while reducing the cost of adding new timbres.
[0084] In a possible implementation, S102, each sample voiceprint feature is mixed to obtain a target voiceprint feature, such as Figure 2 As shown, including:
[0085] S1021, obtaining a preset weight coefficient;
[0086] S1022: Use the current weight coefficient to weight each sample voiceprint feature to obtain a target voiceprint feature.
[0087] The weight coefficient is a manually set weight ratio, which determines the proportion of the sample voiceprint feature in the target voiceprint feature. The higher the weight coefficient, the greater the proportion of the corresponding sample voiceprint feature in the target voiceprint feature. The initial value of the weight coefficient can be customized according to the actual situation. For example, the initial value of the weight coefficient of each sample voiceprint feature can be the same, or it can be an arithmetic progression or geometric progression. The sum of the weight coefficients of each sample voiceprint feature is always equal to 1.
[0088] The number of sample voiceprint features selected each time can be set according to actual conditions. In one embodiment of the present application, taking two sample voiceprint features as an example, each sample voiceprint feature corresponds to a weight coefficient, and the weight coefficient of the first sample voiceprint feature E1 is preset as the first weight coefficient W1, and the weight coefficient of the second sample voiceprint feature E2 is preset as the second weight coefficient W2. The target voiceprint feature is: W1*E1+W2*E2.
[0089] In another embodiment of the present application, taking three sample voiceprint features as an example, each sample voiceprint feature corresponds to a weight coefficient, the weight coefficient of the first sample voiceprint feature E1 is preset as the first weight coefficient W1, the weight coefficient of the second sample voiceprint feature E2 is preset as the second weight coefficient W2, the weight coefficient of the third sample voiceprint feature E3 is preset as the third weight coefficient W3, and the target voiceprint feature is: W1*E1+W2*E2+W3*E3.
[0090] The weight coefficient is preset manually, which increases the freedom of adjusting the weight coefficient, makes the mixing operation easier, and is conducive to timely adjustment of the target voiceprint characteristics as needed.
[0091] In a possible implementation, in S105, after respectively calculating the voiceprint similarity between the audio to be detected and each sample audio, as shown in FIG. Figure 3 As shown, the method also includes:
[0092] S1071, if there is a first similarity among the similarities that is greater than the preset similarity threshold, then update the weight coefficient and return to the execution step: S1022, use the current weight coefficient to weight the voiceprint features of each sample to obtain the target voiceprint feature.
[0093] The preset similarity threshold can be customized according to actual conditions. The voiceprint similarity greater than the preset similarity threshold is called the first similarity. The existence of the first similarity means that there is a voiceprint feature in the timbre library that is highly similar to the target voiceprint feature. At this time, the timbre corresponding to the target voiceprint feature is not considered to be a new timbre, and the target voiceprint feature needs to be adjusted.
[0094] By adjusting the weight coefficient, the target voiceprint features can be adjusted more freely and accurately. The operation is relatively simple and increases the probability of successful mixing.
[0095] In a possible implementation, in S1071, the weight coefficient is updated, such as Figure 4 As shown, including:
[0096] If the first voiceprint feature corresponding to the first similarity is a part of the sample voiceprint features, the weight coefficient of the first voiceprint feature is reduced.
[0097] The first voiceprint feature is the voiceprint feature used when the preset test audio timbre is transformed into the first sample audio, and the first sample audio is the sample audio whose similarity to the audio to be detected is the first similarity. For example, if sample audio A is the audio obtained by performing timbre transformation on the preset test audio using voiceprint feature 1, and the similarity between sample audio A and the audio to be detected is greater than the preset similarity threshold (i.e., the first similarity), then voiceprint feature 1 is considered to be the first voiceprint feature.
[0098] When the first voiceprint feature is a part of the sample voiceprint feature used to synthesize the target voiceprint feature, it means that the target voiceprint feature is greatly affected by the first voiceprint feature and their audios are relatively similar, so the weight of the first voiceprint feature needs to be reduced.
[0099] In one embodiment of the present application, taking two sample voiceprint features as an example, if the first sample voiceprint feature E1 is the first voiceprint feature, that is, the similarity between the target voiceprint feature and the first sample voiceprint feature E1 is greater than the preset similarity threshold, at this time, the first weight coefficient W1 is reduced, that is, the proportion of the first sample voiceprint feature E1 in the target voiceprint feature is reduced; if the second sample voiceprint feature E2 is the first voiceprint feature, that is, the similarity between the target voiceprint feature and the second sample voiceprint feature E2 is greater than the preset similarity threshold, at this time, the second weight coefficient W2 is reduced, that is, the proportion of the second sample voiceprint feature E2 in the target voiceprint feature is reduced.
[0100] In another embodiment of the present application, taking three sample voiceprint features as an example, if the first sample voiceprint feature E1 is the first voiceprint feature, that is, the similarity between the target voiceprint feature and the first sample voiceprint feature E1 is greater than the preset similarity threshold, at this time, the first weight coefficient W1 is reduced, that is, the proportion of the first sample voiceprint feature E1 in the target voiceprint feature is reduced; if the second sample voiceprint feature E2 is the first voiceprint feature, that is, the similarity between the target voiceprint feature and the second sample voiceprint feature E2 is greater than the preset similarity threshold, at this time, the second weight coefficient W2 is reduced, that is, the proportion of the second sample voiceprint feature E2 in the target voiceprint feature is reduced; if the third sample voiceprint feature E3 is the first voiceprint feature, that is, the similarity between the target voiceprint feature and the third sample voiceprint feature E3 is greater than the preset similarity threshold, at this time, the third weight coefficient W3 is reduced, that is, the proportion of the third sample voiceprint feature E3 in the target voiceprint feature is reduced; if the first sample voiceprint feature E1 and the second sample voiceprint feature E2 are the first voiceprint feature, that is, the target voiceprint feature and the first sample voiceprint feature E1 and the third sample voiceprint feature E3 are greater than the preset similarity threshold, at this time, the third weight coefficient W3 is reduced, that is, the proportion of the third sample voiceprint feature E3 in the target voiceprint feature is reduced. If the similarity between the two sample voiceprint features E2 is greater than the preset similarity threshold, at this time, the first weight coefficient W1 and the second weight coefficient W2 are reduced, that is, the proportion of the first sample voiceprint feature E1 and the second sample voiceprint feature E2 in the target voiceprint feature is reduced; if the first sample voiceprint feature E1 and the third sample voiceprint feature E3 are the first voiceprint features, that is, the similarity between the target voiceprint features and the first sample voiceprint features E1 and the third sample voiceprint features E3 is greater than the preset similarity threshold, at this time, the first weight coefficient W1 and the third weight coefficient W3 are reduced, that is, the proportion of the first sample voiceprint features E1 and the third sample voiceprint features E3 in the target voiceprint features is reduced; if the second sample voiceprint features E2 and the third sample voiceprint features E3 are the first voiceprint features, that is, the similarity between the target voiceprint features and the second sample voiceprint features E2 and the third sample voiceprint features E3 is greater than the preset similarity threshold, at this time, the second weight coefficient W2 and the third weight coefficient W3 are reduced, that is, the proportion of the second sample voiceprint features E2 and the third sample voiceprint features E3 in the target voiceprint features is reduced.
[0101] By reducing the weight coefficient of the first voiceprint feature in the sample feature, the target voiceprint feature can be adjusted more freely, the operation is relatively simple, and the probability of successful mixing is increased.
[0102] In one possible implementation, Figure 5 As shown, the method also includes:
[0103] S1072, if the first voiceprint feature corresponding to the first similarity includes each sample voiceprint feature, reselect at least two sample voiceprint features from each voiceprint feature in the timbre library, and return to execute step: S102, mix each sample voiceprint feature to obtain a target voiceprint feature.
[0104] The first voiceprint feature includes each sample voiceprint feature, indicating that the sample voiceprint features used to synthesize the target voiceprint feature are relatively similar to the target voiceprint feature. If only the weight is changed, the similarity between the newly synthesized target voiceprint feature and each sample voiceprint feature will still be high, so the sample voiceprint feature can be reselected.
[0105] The number of the re-selected sample voiceprint features may be the same as or different from that of the sample voiceprint features selected in the previous mixing. For example, if two sample voiceprint features were selected in the previous mixing, the number of the re-selected sample voiceprint features may be 2, 3, 4 or more.
[0106] This method eliminates timbre combinations that are difficult to mix into new timbres, improves the distinction between new timbres and existing timbres in the timbre library, and makes it less likely to confuse different characters when used as dubbing timbres for film and television works.
[0107] In another possible implementation, Figure 6 As shown, the method also includes:
[0108] S1073, if the first voiceprint feature corresponding to the first similarity includes all sample voiceprint features, then select the voiceprint feature corresponding to the minimum similarity from among the voiceprint features in the timbre library and add it to the sample voiceprint feature, and return to execute step: S102, mix all sample voiceprint features to obtain the target voiceprint feature.
[0109] The first voiceprint feature includes each sample voiceprint feature, indicating that the sample voiceprint features used to synthesize the target voiceprint feature are relatively similar to the target voiceprint feature. If only the weight is changed, the similarity between the newly synthesized target voiceprint feature and each sample voiceprint feature will still be high. Therefore, the voiceprint feature corresponding to the minimum similarity can be selected to resynthesize the target voiceprint feature. The voiceprint feature corresponding to the minimum similarity is the sample voiceprint feature used when the preset test audio timbre is transformed into the second sample audio. The second sample audio is the sample audio with the minimum similarity to the audio to be detected.
[0110] In one embodiment of the present application, taking two sample voiceprint features as an example, if the first sample voiceprint feature E1 and the second sample voiceprint feature E2 are both first voiceprint features, the voiceprint feature Es with the smallest similarity to the target voiceprint feature in the timbre library is added as a new sample voiceprint feature to the sample voiceprint feature, E3 corresponds to the weight Ws, and the target voiceprint feature changes from W1*E1+W2*E2 to W1*E1+W2*E2+Ws*Es.
[0111] In another embodiment of the present application, taking three sample voiceprint features as an example, if the first sample voiceprint feature E1, the second sample voiceprint feature E2, and the third sample voiceprint feature E3 are all first voiceprint features, then the voiceprint feature Es with the smallest similarity to the target voiceprint feature in the timbre library is added as a new sample voiceprint feature to the sample voiceprint feature, Es corresponds to the weight Ws, and the target voiceprint feature changes from W1*E1+W2*E2+W3*E3 to W1*E1+W2*E2+W3*E3+Ws*Es.
[0112] Through this method, the existing timbre with the smallest similarity is used as a new sample voiceprint feature and mixed with the target voiceprint feature, thereby increasing the probability of successful secondary mixing.
[0113] In a possible implementation, S105, the voiceprint similarity between the audio to be detected and each sample audio is calculated respectively, such as Figure 7 As shown, including:
[0114] S1051, extracting the voiceprint of the audio to be detected and each sample audio to obtain a plurality of voiceprint data;
[0115] S1052, respectively calculating the cosine similarity between the voiceprint data of the audio to be detected and the voiceprint data of each sample audio to obtain multiple similarities.
[0116] In one embodiment of the present application, the voiceprint data of the audio to be detected is divided into the first audio voiceprint data D1 to be detected, the second audio voiceprint data D2 to be detected, and the third audio voiceprint data D3 to be detected, and the voiceprint data of each sample audio is divided into the first sample audio voiceprint data Y1, the second sample audio voiceprint data Y2, and the third sample audio voiceprint data Y3, wherein the first audio voiceprint data D1 to be detected has the same length as the first sample audio voiceprint data Y1, the second audio voiceprint data D2 to be detected has the same length as the second sample audio voiceprint data Y2, and the third audio voiceprint data D3 to be detected has the same length as the third sample audio voiceprint data Y3. The cosine similarities of D1 and Y1, D2 and Y2, and D3 and Y3 are compared respectively to obtain the first cosine similarity S1, the second cosine similarity S2, and the third cosine similarity S3, and the average of S1, S2, and S3 is obtained to obtain the cosine similarity described in S1052.
[0117] The present application embodiment provides a device for expanding the capacity of a timbre library, such as Figure 8 As shown, the device comprises:
[0118] Voiceprint feature selection module 201, sound conversion module 203, timbre confusion module 202, voiceprint comparison module 204:
[0119] The voiceprint feature selection module 201 is used to select at least two sample voiceprint features from each voiceprint feature in the timbre library;
[0120] The timbre confusion module 202 is used to mix the sample voiceprint features to obtain the target voiceprint features;
[0121] The sound conversion module 203 is used to perform timbre conversion on the preset test audio based on the target voiceprint feature to obtain the audio to be detected; obtain the sample audio under each voiceprint feature in the timbre library, wherein the sample audio is obtained after the timbre conversion of the preset test audio;
[0122] The voiceprint comparison module 204 is used to respectively calculate the voiceprint similarity between the audio to be detected and each of the sample audios; if each of the similarities is less than a preset similarity threshold, the target voiceprint feature is added to the timbre library. When performing timbre conversion, the sound conversion module 203 extracts the audio information of the preset test audio, combines the extracted audio information with the target voiceprint feature, and obtains the audio to be detected. The audio information includes text content, emotion, and rhythmic features. The audio information such as text content, emotion, and rhythmic features can be extracted using general speech emotion representation models such as Whisper's encoder model, emotin2vec model, and vqwav2vec model, which is not limited in the embodiments of the present application.
[0123] When executing S101-S103, the voiceprint feature selection module 201 is used to select at least two sample voiceprint features from each voiceprint feature in the timbre library and provide them to the timbre confusion module 202; the sound conversion module 203 is used to convert the voiceprint features of the preset test audio into target voiceprint features and combine the target voiceprint features with the audio information to generate the audio to be detected.
[0124] When generating the sample audio in S104, the voiceprint feature selection module 201 is used to select a sample voiceprint feature from the timbre library and provide it to the sound conversion module 203; the sound conversion module 203 is used to convert the voiceprint feature of the preset test audio into a sample voiceprint feature and combine the sample voiceprint feature with the audio information to generate the sample audio. The timbre confusion module 202 is specifically used to:
[0125] Obtain a preset weight coefficient; and use the current weight coefficient to weight each of the sample voiceprint features to obtain a target voiceprint feature.
[0126] In one embodiment of the present application, taking two sample voiceprint features as an example, each sample voiceprint feature corresponds to a weight coefficient, the weight coefficient of the first sample voiceprint feature E1 is preset as the first weight coefficient W1, the weight coefficient of the second sample voiceprint feature E2 is preset as the second weight coefficient W2, and the target voiceprint feature is: W1*E1+W2*E2.
[0127] In another embodiment of the present application, taking three sample voiceprint features as an example, each sample voiceprint feature corresponds to a weight coefficient, the weight coefficient of the first sample voiceprint feature E1 is preset as the first weight coefficient W1, the weight coefficient of the second sample voiceprint feature E2 is preset as the second weight coefficient W2, the weight coefficient of the third sample voiceprint feature E3 is preset as the third weight coefficient W3, and the target voiceprint feature is: W1*E1+W2*E2+W3*E3.
[0128] Optionally, the device further comprises: a weight updating module for updating the weight coefficient if there is a first similarity greater than a preset similarity threshold among the similarities, and returning to execute step S1022: weighting each sample voiceprint feature using the current weight coefficient to obtain a target voiceprint feature.
[0129] Optionally, the weight updating module is specifically used for:
[0130] If the first voiceprint feature corresponding to the first similarity is a part of the sample voiceprint feature, the weight coefficient of the first voiceprint feature is reduced, wherein the first voiceprint feature is the voiceprint feature used when converting the preset test audio timbre into the first sample audio, and the first sample audio is the sample audio whose similarity with the audio to be detected is the first similarity.
[0131] Optionally, the device further comprises: a sample updating module.
[0132] If the first voiceprint feature corresponding to the first similarity includes the sample voiceprint features, at least two sample voiceprint features are reselected from the voiceprint features in the timbre library, and the process returns to step S102: mixing the sample voiceprint features to obtain a target voiceprint feature.
[0133] The sample update module is also used for:
[0134] If the first voiceprint feature corresponding to the first similarity includes the sample voiceprint features, then the voiceprint feature corresponding to the minimum similarity is selected from the voiceprint features in the timbre library and added to the sample voiceprint feature, and the process returns to step S102: mixing the sample voiceprint features to obtain the target voiceprint feature.
[0135] Optionally, the voiceprint comparison module is specifically used to:
[0136] Extracting the voiceprint of the audio to be detected and each of the sample audios to obtain a plurality of voiceprint data;
[0137] The cosine similarities between the voiceprint data of the audio to be detected and the voiceprint data of each of the sample audios are calculated respectively to obtain multiple similarities.
[0138] The timbre library expansion device of the embodiment of the present application can be applied to all the timbre library expansion methods described above, making the timbre library expansion method simple to implement and easy to operate.
[0139] The present application also provides an electronic device, such as Fig. 9 As shown, it includes a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304;
[0140] Memory 303, used for storing computer programs;
[0141] The processor 301 is used to implement any method step described in the first aspect when executing a program stored in the memory.
[0142] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0143] The communication interface is used for communication between the above terminal and other devices.
[0144] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0145] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0146] In another aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned methods for expanding the timbre library is implemented.
[0147] In another aspect of the implementation of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned methods for expanding the timbre library.
[0148] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.
[0149] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0150] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0151] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. A method for expanding a timbre library, characterized in that: The method comprises: Select at least two sample voiceprint features from each voiceprint feature in the timbre library; Mixing the sample voiceprint features to obtain a target voiceprint feature; Performing timbre transformation on a preset test audio based on the target voiceprint feature to obtain an audio to be tested; Acquire sample audio under each voiceprint feature in the timbre library, wherein the sample audio is obtained by performing timbre transformation on the preset test audio; Calculating the voiceprint similarity between the audio to be detected and each of the sample audios respectively; If each of the similarities is less than a preset similarity threshold, the target voiceprint feature is added to the timbre library.
2. The method according to claim 1, characterized in that The mixing of the sample voiceprint features to obtain the target voiceprint features includes: Get the preset weight coefficient; The current weight coefficient is used to weight each of the sample voiceprint features to obtain the target voiceprint feature.
3. The method according to claim 2, characterized in that After respectively calculating the voiceprint similarities between the audio to be detected and each of the sample audios, the method further includes: If there is a first similarity among the similarities that is greater than a preset similarity threshold, the weight coefficient is updated, and the process returns to the step of weighting the sample voiceprint features using the current weight coefficient to obtain a target voiceprint feature.
4. The method according to claim 3, characterized in that: The updating of the weight coefficient comprises: If the first voiceprint feature corresponding to the first similarity is a part of the sample voiceprint feature, the weight coefficient of the first voiceprint feature is reduced, wherein the first voiceprint feature is the voiceprint feature used when converting the preset test audio timbre into the first sample audio, and the first sample audio is the sample audio whose similarity with the audio to be detected is the first similarity.
5. The method according to claim 4, characterized in that The method further comprises: If the first voiceprint feature corresponding to the first similarity includes each of the sample voiceprint features, then reselect at least two sample voiceprint features from each of the voiceprint features in the timbre library, and return to the execution step: mix each of the sample voiceprint features to obtain a target voiceprint feature.
6. The method according to claim 4, characterized in that The method further comprises: If the first voiceprint feature corresponding to the first similarity includes each of the sample voiceprint features, then the voiceprint feature corresponding to the minimum similarity is selected from the voiceprint features in the timbre library and added to the sample voiceprint feature, and the execution step is returned to: mixing each of the sample voiceprint features to obtain the target voiceprint feature.
7. The method according to claim 1, characterized in that The respectively calculating the voiceprint similarity between the audio to be detected and each of the sample audios includes: Extracting the voiceprint of the audio to be detected and each of the sample audios to obtain a plurality of voiceprint data; The cosine similarities between the voiceprint data of the audio to be detected and the voiceprint data of each of the sample audios are calculated respectively to obtain multiple similarities.
8. A device for expanding the capacity of a timbre library, characterized in that: The device comprises: Voiceprint feature selection module, sound conversion module, timbre confusion module, voiceprint comparison module: The voiceprint feature selection module is used to select at least two sample voiceprint features from each voiceprint feature in the timbre library; The timbre confusion module is used to mix the sample voiceprint features to obtain the target voiceprint features; The sound conversion module is used to perform timbre conversion on the preset test audio based on the target voiceprint feature to obtain the audio to be detected; obtain the sample audio under each voiceprint feature in the timbre library, wherein the sample audio is obtained after the timbre conversion of the preset test audio; The voiceprint comparison module is used to respectively calculate the voiceprint similarity between the audio to be detected and each of the sample audios; if each of the similarities is less than a preset similarity threshold, the target voiceprint feature is added to the timbre library.
9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.
