Voice augmentation method and related method, apparatus, device, and storage medium

Abstract

The application discloses a voice augmentation method and related method, device, equipment and storage medium, wherein the voice augmentation method comprises: obtaining a to-be-augmented voice and obtaining a to-be-synthesized text; performing feature extraction based on the to-be-augmented voice to obtain a voiceprint feature of a speaker to which the to-be-augmented voice belongs and an environmental noise feature of a scene to which the to-be-augmented voice belongs; selecting, based on a first distribution of each speaker to which the to-be-augmented voice belongs, a voiceprint feature of at least one speaker as a target voiceprint feature, and selecting, based on a second distribution of each scene to which the to-be-augmented voice belongs, an environmental noise feature of at least one scene as a target environmental noise feature; and obtaining a synthesized voice synthesized by any target voiceprint feature, any target environmental noise feature and the to-be-synthesized text as an augmented voice. The above scheme can realize voice augmentation to improve the diversity of voice data.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech augmentation method and related methods, devices, equipment, and storage media. Background Art

[0002] With the rapid development of electronic technologies such as microcomputer chips, the popularity of smart devices has increased significantly. Voice, as a key component, is particularly important in these devices. For example, voice wake-up and voice recognition are among the services available.

[0003] Thanks to artificial intelligence (AI), voice services such as voice wake-up and speech recognition using deep learning models are gaining popularity. However, their accuracy relies on speech diversity. In real-world scenarios, the accuracy of voice services is also limited by the amount of speech that can be collected. For example, voice wake-up services are prone to problems such as difficulty waking up devices in noisy environments and being mistakenly woken by others. Therefore, how to implement voice augmentation to increase the diversity of voice data has become a pressing issue. Summary of the Invention

[0004] The main technical problem solved by this application is to provide a voice augmentation method and related methods, devices, equipment and storage media, which can realize voice augmentation to improve the diversity of voice data.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a speech augmentation method, including: obtaining a speech to be augmented and obtaining a text to be synthesized; performing feature extraction based on the speech to be augmented to obtain the voiceprint features of the speaker to which the speech to be augmented belongs and the environmental noise features of the scene to which the speech to be augmented belongs; based on a first distribution of the speakers to which each speech to be augmented belongs, selecting the voiceprint features of at least one speaker as the target voiceprint features, and based on a second distribution of the scenes to which each speech to be augmented belongs, selecting the environmental noise features of at least one scene as the target environmental noise features; and obtaining a speech synthesized by any target voiceprint feature, any target environmental noise feature and the text to be synthesized as the augmented speech.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a voice wake-up method, including: obtaining the collected voice of the terminal device; determining whether to wake up the terminal device based on the recognition result of the collected voice by the voiceprint recognition model; wherein, the voiceprint recognition model is trained based on a sample voice set, and the sample voice set is obtained based on the voice augmentation method in the above first aspect.

[0007] In order to solve the above technical problems, the third aspect of the present application provides a speech augmentation device, including: an acquisition module, an extraction module, a selection module and a synthesis module, the acquisition module is used to obtain the speech to be augmented and the text to be synthesized; the extraction module is used to perform feature extraction based on the speech to be augmented to obtain the voiceprint features of the speaker to which the speech to be augmented belongs and the environmental noise features of the scene to which the speech to be augmented belongs; the selection module is used to select the voiceprint features of at least one speaker as the target voiceprint features based on the first distribution of the speakers to which the speech to be augmented belongs, and select the environmental noise features of at least one scene as the target environmental noise features based on the second distribution of the scenes to which the speech to be augmented belongs; the synthesis module is used to obtain speech synthesized by any target voiceprint feature, any target environmental noise feature and the text to be synthesized as the augmented speech.

[0008] In order to solve the above technical problems, the fourth aspect of the present application provides a voice wake-up device, including: an acquisition module and a determination module, the acquisition module is used to acquire the collected voice of the terminal device; the determination module is used to determine whether to wake up the terminal device based on the recognition result of the collected voice by the voiceprint recognition model; wherein, the voiceprint recognition model is trained based on a sample voice set, and the sample voice set is obtained based on the voice augmentation device in the above third aspect.

[0009] In order to solve the above technical problems, the fifth aspect of the present application provides an electronic device, which includes at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor is used to execute the program instructions to implement the voice augmentation method in the above-mentioned first aspect, or the voice wake-up method in the second aspect.

[0010] In order to solve the above technical problems, the sixth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the voice augmentation method of the first aspect or the voice wake-up method in the second aspect.

[0011] The above scheme obtains a speech to be augmented and a text to be synthesized, performs feature extraction based on the speech to be augmented, obtains the voiceprint features of the speaker to which the speech to be augmented belongs and the environmental noise features of the scene to which the speech to be augmented belongs, then selects at least one speaker's voiceprint feature as the target voiceprint feature based on a first distribution of the speakers to which each speech to be augmented belongs, selects at least one scene's environmental noise feature as the target environmental noise feature based on a second distribution of the scene to which each speech to be augmented belongs, and obtains a speech synthesized from any target voiceprint feature, any target environmental noise feature, and the text to be synthesized as the augmented speech. This allows speech data augmentation to be achieved by selecting relevant voiceprint features and environmental noise features for speech synthesis with the text to be synthesized. Furthermore, because feature selection is performed from both the speaker and scene dimensions based on their respective distributions for subsequent speech synthesis during the speech augmentation process, the diversity of the augmented speech can be increased by combining different speakers and different scenes. Therefore, speech augmentation can be achieved to increase speech data diversity. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a flow chart of an embodiment of the speech augmentation method of the present application;

[0013] Figure 2 1 is a schematic diagram of a framework of an embodiment of a feature extraction model;

[0014] Figure 3 1 is a schematic diagram of a framework of an embodiment of a speech synthesis model;

[0015] Figure 4 This is a process diagram of an embodiment of the speech augmentation method of the present application;

[0016] Figure 5 This is a flow chart of an embodiment of the voice wake-up method of the present application;

[0017] Figure 6 This is a schematic diagram of the framework of an embodiment of the speech amplification device of the present application;

[0018] Figure 7 This is a schematic diagram of the framework of an embodiment of the voice wake-up device of the present application;

[0019] Figure 8 This is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0020] Figure 9 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0021] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0022] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0023] The terms "system" and "network" are often used interchangeably in this document. The term "and / or" is simply a description of an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the fragment " / " generally indicates that the related objects are in an "or" relationship. Furthermore, "multiple" in this document refers to two or more than two.

[0024] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the speech augmentation method of the present application. Specifically, it may include the following steps:

[0025] Step S11: Acquire the speech to be augmented and the text to be synthesized.

[0026] In one implementation scenario, the voice to be augmented can be collected based on the voice service to be applied for voice augmentation. For example, in a voice wake-up service, the voice to be augmented can be collected in the application location of the voice wake-up service, such as when the user wakes up the device in a shopping mall, station, home, office, etc., to obtain the voice to be augmented; or, in a voice recognition service, the voice to be augmented can be collected in the application location of the voice recognition service, such as when the user is in a conference room, exhibition hall, etc., to obtain the voice to be augmented. It should be noted that the above examples are only a few possible examples of the voice to be augmented, and the acquisition method of the augmented voice is not limited here. In addition, when the voice to be augmented is applied to other voice services, the above collection method can also be used by analogy to collect the voice to be augmented, and examples will not be given one by one here. In addition, when actually collecting voice data, the specific purpose of the voice collection can be clearly stated, and voice collection can be carried out only after obtaining authorization from the person being collected.

[0027] In one implementation scenario, the text to be synthesized can be set according to the voice service to which voice augmentation is to be applied. For example, in the voice wake-up service, the text to be synthesized can at least include the wake-up words used by the voice wake-up service, such as the text to be synthesized can include but is not limited to: "XX classmate", "Xiao X Xiao X", "hey XX", "XX classmate, please help me XXXXX", "Xiao X Xiao X, please check XXXXX", "hey XX, may I ask XXXXXX", etc.; or, in the voice recognition service, the text to be synthesized can at least include relevant words and sentences in the application field of the voice recognition service, such as in the application field of daily life, the text to be synthesized can include but is not limited to: "How is the weather today", "Is there rain in the forecast for tomorrow", etc. It should be noted that the above examples are only a few possible examples of the text to be synthesized, and the specific content of the text to be synthesized is not limited here. In addition, when voice augmentation is applied to other voice services, the text to be synthesized can also be set with reference to the above situation, and examples will not be given one by one here.

[0028] Step S12: performing feature extraction based on the speech to be augmented to obtain the voiceprint features of the speaker to which the speech to be augmented belongs and the environmental noise features of the scene to which the speech to be augmented belongs.

[0029] It should be noted that the voiceprint features of the speech to be augmented may at least include voiceprint-related feature information such as timbre, undulation, and stress of the speaker to whom the speech to be augmented belongs. For example, different speakers may have different voiceprint features, such as fast or slow vocalization habits, and different amounts of force. The environmental noise features of the speech to be augmented may at least include feature information related to background noise in the scene to which the speech to be augmented belongs, which is unrelated to the presence or absence of the speaker. For example, the environmental noise features of different scenes vary. For example, in generally quiet places such as libraries, the environmental noise features generally include characteristic information of environmental noise such as white noise, while in places such as factories (especially heavy industrial plants), the environmental noise features generally also include characteristic information of other noises such as machinery operation.

[0030] In one implementation scenario, voiceprint feature extraction and ambient noise feature extraction can be obtained by processing the augmented speech using different technical approaches. For example, a voiceprint extraction network for extracting voiceprint features and an ambient noise extraction network for extracting ambient noise features can be pre-trained. This allows the voiceprint feature extraction network to extract features from the augmented speech, obtaining the voiceprint features of the speaker to which the augmented speech belongs, while the ambient noise extraction network can be used to extract features from the augmented speech, obtaining the ambient noise features of the scene to which the augmented speech belongs. It should be noted that the voiceprint feature extraction network may include, but is not limited to, a convolutional neural network, and the network structure of the voiceprint feature extraction network is not limited herein. Similarly, the ambient noise extraction network may include, but is not limited to, a convolutional neural network, and the network structure of the ambient noise extraction network is not limited herein. As one possible example, the voiceprint extraction network and the ambient noise extraction network can be trained before using them for feature extraction, respectively. For example, when training a voiceprint extraction network, sample speech can be obtained in advance, and the sample speech can be annotated with the sample speaker to which the sample speech belongs (e.g., Zhang San, Li Si, Wang Wu, etc.). The voiceprint extraction network can then be used to extract features from the sample speech to obtain sample voiceprint features. Based on the sample voiceprint features, predictions can be made to obtain the predicted speaker to which the sample speech belongs. The network parameters of the voiceprint extraction network can then be adjusted based on the differences between the sample speaker and the predicted speaker. Similarly, when training an environmental noise extraction network, sample speech can be obtained in advance, and the sample speech can be annotated with the sample scene to which the sample speech belongs (e.g., shopping mall, office, home, etc.). The environmental noise extraction network can then be used to extract features from the sample speech to obtain sample environmental noise features. Based on the sample environmental noise features, predictions can be made to obtain the predicted scene to which the sample speech belongs. The network parameters of the environmental noise extraction network can then be adjusted based on the differences between the sample scene and the predicted scene.

[0031] In another implementation scenario, distinct from the aforementioned embodiment, as another possible implementation, voiceprint feature extraction and ambient noise feature extraction can also share at least partially the same technical approach to process the augmented speech. For example, speech features can be extracted based on the acoustic features of the augmented speech. Based on this, voiceprint features can be extracted based on the speech features, and ambient noise features can be extracted based on the speech features. It should be noted that acoustic features may include, but are not limited to, Fbank features, MFCC features, etc., and the specific types of acoustic features are not limited here. Furthermore, the acoustic features of the augmented speech can specifically include sub-acoustic features of each speech frame in the augmented speech. The speech features further extracted based on these sub-acoustic features may not only contain deep feature information for each speech frame, but also include correlated feature information between speech frames. This approach, which extracts speech features based on the acoustic features of the augmented speech, then extracts voiceprint features based on the speech features, and finally extracts ambient noise features based on the speech features, allows speech features to be shared between the voiceprint and ambient noise extraction processes, thereby improving feature extraction efficiency.

[0032] In another implementation scenario, as previously mentioned, voiceprint feature extraction and ambient noise feature extraction can also share at least some of the same technical routes for processing the augmented speech. For example, unlike the aforementioned implementation, voiceprint features and ambient noise features are extracted from the augmented speech using a feature extraction model trained on sample speech, and the sample speech is annotated with the sample speaker and sample scene to which the sample speech belongs. Therefore, the ability to simultaneously extract voiceprint features and ambient noise features using a unified model helps improve feature extraction efficiency.

[0033] In a specific implementation scenario, sample speech can be obtained by on-site collection in different scenarios. Alternatively, to reduce speech collection costs and improve speech collection efficiency, the noise-free speech of the sample speaker and the ambient noise of the sample scene can be first obtained. Then, based on the ambient noise of at least one sample scene, linear augmentation can be performed on the noise-free speech of any sample speaker to obtain the sample augmented speech. For example, the noise-free speech of sample speaker A can be linearly augmented using the ambient noise of a shopping mall to obtain the sample augmented speech of sample speaker A in the shopping mall scene. Based on this, sample speech for model training can be obtained based on at least the sample augmented speech. For example, each sample augmented speech can be directly used as the sample speech for model training, or each sample augmented speech and the noise-free speech of the sample speaker can be used as the sample speech for model training. It should be noted that the specific process of linear augmentation can be referred to the technical details of linear augmentation and will not be described in detail here. Of course, in actual applications, other augmentation methods can be used to augment the noise-free speech of the sample speaker using the ambient noise of the sample scene, and these examples will not be given one by one here. The above method obtains the noise-free speech of the sample speaker and the ambient noise of the sample scene in advance, and uses the ambient noise to linearly augment the noise-free speech to obtain the sample augmented speech, so as to obtain the sample speech for model training at least based on the sample augmented speech, which can reduce the model training cost as much as possible.

[0034] In a specific implementation scenario, please refer to Figure 2 , Figure 2 Schematic diagram of the framework of an embodiment of a feature extraction model. Figure 2 As shown, the feature extraction model may include a backbone network, a voiceprint extraction network, and a scene extraction network. The backbone network is used to extract speech features from the input speech, and the voiceprint extraction network and the scene extraction network share speech features. The voiceprint extraction network is used to extract voiceprint features, and the scene extraction network is used to extract ambient noise features. For example, the backbone network may include, but is not limited to, a convolutional neural network (e.g., ResNet), the voiceprint extraction network may include, but is not limited to, pooling layers (e.g., average pooling), fully connected layers, etc., and the scene extraction network may include, but is not limited to, pooling layers (e.g., average pooling), fully connected layers, etc. The specific structures of the backbone network, voiceprint extraction network, and scene extraction network are not limited here. In this approach, the voiceprint extraction network and the scene extraction network share the backbone network, which can improve feature extraction efficiency.

[0035] In a specific implementation scenario, to improve the feature extraction accuracy of the feature extraction model, the feature extraction model can be pre-trained. For example, feature extraction can be performed based on sample speech to obtain sample voiceprint features of the sample speaker to which the sample speech belongs and sample environmental noise features of the sample speaker to which the sample speech belongs (the specific feature extraction process can be found in the aforementioned description and will not be repeated here). Prediction is then performed based on the sample voiceprint features to obtain the predicted speaker to which the sample speech belongs, and prediction is also performed based on the sample voiceprint features to obtain the predicted scene to which the sample speech belongs. Based on the differences between the predicted speaker and the sample speaker, and between the predicted scene and the sample scene, the network parameters of the feature extraction model can be adjusted. For example, the difference between the predicted speaker and the sample speaker, as well as the difference between the predicted scene and the sample scene, can be measured based on a loss function such as AAM (Additive Angular Margin). For the specific calculation process, please refer to the technical details of loss functions such as AAM and will not be repeated here. The above method performs feature extraction based on the sample speech to obtain sample noise features and sample environmental noise features. The predicted speaker to which the sample speech belongs and the predicted scene to which the sample environmental noise features belong are then predicted based on the sample noise features. The network parameters of the feature extraction model are adjusted based on the difference between the prediction and the annotation. This helps force the feature extraction model to extract as much identifiable feature information of different speakers as possible during training, that is, feature information related to voiceprints, as voiceprint features, and to extract as much identifiable feature information of different scenes as possible, that is, feature information related to background noise, as environmental noise features.

[0036] Step S13: Based on the first distribution of the speakers to which the respective augmented speech belongs, select the voiceprint feature of at least one speaker as the target voiceprint feature, and based on the second distribution of the scenes to which the respective augmented speech belongs, select the environmental noise feature of at least one scene as the target environmental noise feature.

[0037] Specifically, the first distribution may include the statistical number of to-be-augmented speech sounds belonging to different speakers. For example, the first distribution may include: M for Speaker A (i.e., there are M to-be-augmented speech sounds belonging to Speaker A), N for Speaker B (i.e., there are N to-be-augmented speech sounds belonging to Speaker B), etc. Similarly, the second distribution may include the statistical number of to-be-augmented speech sounds belonging to different scenes. For example, the second distribution may include: P for Scene C (i.e., there are P to-be-augmented speech sounds belonging to Scene C), Q for Scene D (i.e., there are Q to-be-augmented speech sounds belonging to Scene D), etc. Of course, the above examples are merely possible scenarios of the first and second distributions, and the specific scenarios of the first and second distributions are not limited herein.

[0038] In one implementation scenario, as a possible example, after obtaining the first distribution and the second distribution, based on the purpose of sample balance, the first distribution can be used to determine speakers with relatively low distribution, and the voiceprint features of these speakers can be selected as target voiceprint features. The second distribution can also be used to determine scenes with relatively low distribution, and the environmental noise features of these scenes can be selected as target environmental noise features. For example, if the first distribution determines that speaker A has relatively low distribution, the voiceprint features of speaker A can be selected as the target voiceprint features. If speaker A has a total of K speech items to be augmented, the voiceprint features extracted from these K speech items to be augmented can be selected as the target voiceprint features. Similarly, if the second distribution determines that scene D has relatively low distribution, the environmental noise features of scene D can be selected as the target environmental noise features. If scene D has a total of L speech items to be augmented, the environmental noise features extracted from these L speech items to be augmented can be selected as the target environmental noise features. Of course, the above examples are only possible examples of selecting target voiceprint features and target environmental noise features, and are not limited to selecting other voiceprint features as target voiceprint features and selecting other environmental noise features as target environmental noise features. No further examples will be given here.

[0039] In another implementation scenario, as another possible example, after obtaining the first distribution and the second distribution, for the purpose of sample customization, the first distribution can be used to determine speakers that are relatively more relevant to the application requirements, and the voiceprint features of these speakers can be selected as target voiceprint features. The second distribution can also be used to determine scenes that are relatively more relevant to the application requirements, and the environmental noise features of these scenes can be selected as target environmental noise features. For example, if the first distribution determines that speaker B is relatively more relevant to the application requirements, the voiceprint features of speaker B can be selected as the target voiceprint features. For example, if speaker B has a total of I speech items to be augmented, the voiceprint features extracted from these I speech items to be augmented can be selected as the target voiceprint features. Alternatively, if the second distribution determines that scene C is relatively more relevant to the application requirements, the environmental noise features of scene C can be selected as the target environmental noise features. For example, if scene C has a total of J speech items to be augmented, the environmental noise features extracted from these J speech items to be augmented can be selected as the target environmental noise features. Of course, the above examples are only possible examples of selecting target voiceprint features and target environmental noise features, and are not limited to selecting other voiceprint features as target voiceprint features and selecting other environmental noise features as target environmental noise features. No further examples will be given here.

[0040] It should be noted that the above examples are only two possible examples of selecting voiceprint features according to the first distribution and selecting environmental noise features according to the second distribution. They do not limit the possible situations of selecting features according to distribution based on other principles, and no examples will be given one by one here.

[0041] Step S14: obtaining a speech synthesized by any target voiceprint feature, any target environmental noise feature and the text to be synthesized as augmented speech.

[0042] In one implementation scenario, a speech synthesis model can be pre-trained, based on which any target voiceprint feature, any target environmental noise feature, and the text to be synthesized can be input into the speech synthesis model to obtain the output speech of the speech synthesis model as augmented speech. It should be noted that the augmented speech has both the pronunciation characteristics of the speaker to whom the target voiceprint feature belongs and the environmental noise characteristics of the scene to which the target environmental noise feature belongs, that is, the augmented speech "represents" the speaker to whom the target voiceprint feature belongs speaking the content of the text to be synthesized in the scene to which the target environmental noise feature belongs. For example, the voiceprint feature of speaker A can be selected as the target voiceprint feature, and the environmental noise feature of scene C can be selected as the target environmental noise feature. Then, both can be input into the speech synthesis model together with the text to be synthesized, so as to obtain the synthesized speech that represents the content of the text to be synthesized spoken by speaker A in scene C, that is, as the augmented speech. As a possible example, in order to train a speech synthesis model, sample speech can be collected in advance, and based on the aforementioned method or other possible methods, the sample voiceprint features and sample environmental noise features of the sample speech can be extracted, so that the sample voiceprint features, sample environmental noise features and sample text corresponding to the sample speech (which represents the content of the sample speech) can be input into the speech synthesis model to obtain synthesized speech. Based on this, the network parameters of the speech synthesis model can be adjusted based on the difference between the sample speech and the synthesized speech, and the speech synthesis model can be forced to learn to synthesize speech through voiceprint features, environmental noise features and related text during the training process, which helps to improve the quality of speech synthesis.

[0043] In another implementation scenario, different from the aforementioned implementation, it is also possible to obtain a target scene sequence predicted by the target environmental noise feature and obtain a speech synthesis identifier. It should be noted that the target scene sequence includes the probability values ​​of the target environmental noise feature being predicted to belong to various scenes, and the speech synthesis identifier is used to indicate the relative importance of the target scene sequence and the text to be synthesized in the speech synthesis process. On this basis, the text to be synthesized, the target scene sequence and the speech synthesis identifier can be combined to obtain a text sequence, and based on the target voiceprint feature, the target environmental noise feature and the speech synthesis identifier, a feature sequence can be combined to obtain a feature sequence, so that the speech to be augmented from which the target voiceprint feature and the target environmental noise feature are derived can be used as the reference speech, and speech synthesis can be performed based on the text sequence and the feature sequence to obtain the synthesized speech as the augmented speech. The above method can improve the speech synthesis quality by combining the text sequence and the feature sequence and setting the speech synthesis identifier in the two sequences.

[0044] In a specific implementation scenario, the specific process of predicting the target scene sequence from the target environmental noise characteristics can be referred to the relevant description of obtaining the predicted scene to which the sample speech belongs based on the environmental noise characteristics prediction, which will not be repeated here. For the convenience of description, the target scene sequence can be represented as ,in, represents the probability value of the target environmental noise feature being predicted to belong to the i-th scene. It should be noted that, as a possible example, the value range of any probability value is between 0 and 1.

[0045] In a specific implementation scenario, for ease of description, the speech synthesis identifier can be recorded as Flag As a possible example, in the speech synthesis identifier Flag When set to 1, it indicates that the target scene sequence is less important than the text to be synthesized in the speech synthesis process. Flag When set to 0, it can indicate that the target scene sequence is more important than the text to be synthesized in the speech synthesis process. Of course, the above example is only a speech synthesis identifier. Flag A possible example of , which does not exclude speech synthesis identifiers Flag There are other possible configurations, which will not be listed here one by one.

[0046] In a specific implementation scenario, for the convenience of description, the text to be synthesized is recorded as For example, represents the i-th character in the text to be synthesized. In this case, the text sequence can be represented as . Similarly, Taking the concatenation feature representing the target voiceprint feature and the target environment noise feature as an example, the feature sequence can be expressed as ,in, Indicates the target voiceprint feature, Indicates the target environment noise characteristics.

[0047] In a specific implementation scenario, please refer to Figure 3 , Figure 3 Schematic diagram of the framework of an embodiment of a speech synthesis model. Figure 3As shown, CosyVoice can be used as the initialization model for the speech synthesis model. In this case, the speech synthesis model may include a text encoder (i.e., a text encoder) for encoding text sequences to obtain encoded representations of each component in the text sequence. The speech synthesis model also includes a speech tokenizer (i.e., a speech tokenizer) for obtaining speech tokens for the speech to be augmented. The speech synthesis model may also include a text-to-token language model (i.e., a text-to-token language model) and a token-to-speech stream matching network. Here, S, E, and T represent the start, end, and transition markers of a sequence, respectively. The dotted lines in CosyVoice represent autoregressive decoding during the inference phase. Furthermore, the speech tokenizer may specifically include a first encoder and a vector quantizer. It should be noted that the module indicated by the dotted line in the upper left corner is only used during the training phase to ensure that the speech tokens (i.e., speech tokens) output by the speech tokenizer are more accurate, sufficient to restore the input speech to the speech tokenizer. Of course, the above description only represents the technical principles of CosyVoice related to speech synthesis in this case. For more detailed technical details, please refer to CosyVoice and will not be elaborated here.

[0048] In an implementation scenario, in actual application, the aforementioned feature extraction model and speech synthesis model can also be used in combination. For example, the voiceprint features and environmental noise features can be extracted by the feature extraction model to obtain the augmented speech, and the augmented speech can be synthesized by the speech synthesis model. Figure 4 , Figure 4 This is a process diagram of an embodiment of the speech augmentation method of the present application. Figure 4 As shown, each speech to be augmented is subjected to a feature extraction model to extract voiceprint features and environmental noise features, which can then be used to select target voiceprint features and target environmental noise features. Based on this, at least the target voiceprint features, target environmental noise features, and the text to be synthesized can be input into a speech synthesis model to generate synthesized speech as the augmented speech. Furthermore, the feature extraction model and speech synthesis model are trained on sample speech, and then trained on the sample speech. The sample speech can be annotated with sample text, which represents the speech content of the sample speech.

[0049] In a specific implementation scenario, after the feature extraction model is trained, the feature extraction model can be used to assist the training process of the speech synthesis model. For example, the sample speech can be feature extracted based on the feature extraction model to obtain the sample voiceprint features of the sample speaker to which the sample speech belongs and the sample environmental noise features of the sample speaker to which the sample speech belongs. Then, speech synthesis is performed based on the sample voiceprint features, the sample environmental noise features and the sample text to obtain the sample synthesized speech. For details, please refer to the above-mentioned relevant descriptions and will not be repeated here. On this basis, the network parameters of the speech synthesis model can be adjusted based on the difference between the sample speech and the sample synthesized speech. For example, the frame-level difference between the sample speech and the sample synthesized speech can be measured, and the network parameters of the speech synthesis model can be adjusted accordingly to force the speech synthesis model to learn to synthesize speech as accurately as possible based on the voiceprint features, environmental noise features and the text to be synthesized.

[0050] In a specific implementation scenario, after the speech synthesis model is trained based on the sample speech, in order to further improve the speech synthesis quality, the sample synthesized speech synthesized by the speech synthesis model based on the sample voiceprint features of the sample speaker to which the sample speech belongs, the sample environmental noise features of the sample scene to which the sample speech belongs, and the sample text can be obtained as a new sample speech, and then the new sample speech is subjected to feature extraction based on the feature extraction model to at least obtain the sample voiceprint features of the new sample speech, so that a prediction can be made based on the sample voiceprint features of the new sample speech to obtain the predicted speaker to which the new sample speech belongs, and then based on the difference between the predicted speaker to which the new sample speech belongs and the sample speaker to which the new sample speech belongs, it can be determined whether to re-execute the steps of training the feature extraction model and the speech synthesis model respectively. For example, when the predicted speaker to which the new sample speech belongs is consistent with the sample speaker to which the sample speech from which the new sample speech is derived belongs, it can be determined that the steps of separately training the feature extraction model and the speech synthesis model are no longer required to be re-executed; conversely, when the predicted speaker to which the new sample speech belongs is inconsistent with the sample speaker to which the new sample speech is derived from, it can be determined that the steps of separately training the feature extraction model and the speech synthesis model are re-executed to further improve the feature extraction quality and the speech synthesis quality through model training. For ease of description, taking the example of the sample speaker to which the sample speech belongs being "Speaker A" and the sample scene to which the sample speech belongs being "Scene D", the feature extraction model can be used to extract features of the sample speech to obtain sample voiceprint features and sample environmental noise features, so that speech synthesis can be performed based on the sample voiceprint features, the sample environmental noise features, and the sample text (which may or may not be consistent with the sample text corresponding to the sample speech) to obtain sample synthesized speech as a new sample speech. The feature extraction model is then used to extract features of the new sample speech to at least obtain sample voiceprint features of the new sample speech, and prediction is performed based on the sample voiceprint features of the new sample speech to obtain the predicted speaker to which the new sample speech belongs. If the model performance of the feature extraction model and the speech synthesis model meet the requirements, then theoretically the predicted speaker to which the new sample speech belongs should be "Speaker A". Conversely, if the model performance of either the feature extraction model or the speech synthesis model does not meet the requirements, then theoretically the predicted speaker to which the new sample speech belongs may not be "Speaker A".Therefore, if the predicted speaker of the new sample speech is "Speaker A," it can be determined that the model performance of the feature extraction model and the speech synthesis model meets the requirements, and it can be determined that the steps of training the feature extraction model and the speech synthesis model separately and sequentially are no longer required. However, if the predicted speaker of the new sample speech is not "Speaker A," it can be determined that the model performance of either the feature extraction model or the speech synthesis model does not meet the requirements, and it can be determined that the steps of training the feature extraction model and the speech synthesis model separately and sequentially are required to be re-performed. The above method uses the sample synthesized speech of the speech synthesis model as the new sample speech, extracts the sample voiceprint features of the new sample speech using the feature extraction model, and predicts the predicted speaker of the new sample speech based on the features. Based on the differences between the predicted speaker of the new sample speech and the sample speaker of the sample speech from which the new sample speech originated, it is determined whether to re-perform the steps of training the feature extraction model and the speech synthesis model separately and sequentially. This allows the feature extraction model and the speech synthesis model to be iteratively trained until the performance requirements are met, thereby maximizing the model performance of the feature extraction model and the speech synthesis model, and thus improving the quality of speech augmentation.

[0051] The above scheme obtains a speech to be augmented and a text to be synthesized, performs feature extraction based on the speech to be augmented, obtains the voiceprint features of the speaker to which the speech to be augmented belongs and the environmental noise features of the scene to which the speech to be augmented belongs, then selects at least one speaker's voiceprint feature as the target voiceprint feature based on a first distribution of the speakers to which each speech to be augmented belongs, selects at least one scene's environmental noise feature as the target environmental noise feature based on a second distribution of the scene to which each speech to be augmented belongs, and obtains a speech synthesized from any target voiceprint feature, any target environmental noise feature, and the text to be synthesized as the augmented speech. This allows speech data augmentation to be achieved by selecting relevant voiceprint features and environmental noise features for speech synthesis with the text to be synthesized. Furthermore, because feature selection is performed from both the speaker and scene dimensions based on their respective distributions for subsequent speech synthesis during the speech augmentation process, the diversity of the augmented speech can be increased by combining different speakers and different scenes. Therefore, speech augmentation can be achieved to increase speech data diversity.

[0052] See also Figure 5 , Figure 5 This is a flow chart of an embodiment of the voice wake-up method of the present application. Specifically, it may include the following steps:

[0053] Step S51: Acquire the collected voice of the terminal device.

[0054] In one implementation scenario, the terminal device may include but is not limited to: a smart phone, a tablet computer, a learning machine, an office notebook, a smart speaker, etc., and there is no limitation on the type of the terminal device.

[0055] In one implementation scenario, the terminal device may keep the sound pickup device turned on in a sleep state (or in any other state, such as any other state) to obtain collected voice through the sound pickup device.

[0056] It should be noted that in actual application, the specific purpose of voice collection can be clearly stated in advance, and voice collection can be carried out after obtaining authorization from the person being collected.

[0057] Step S52: Determine whether to wake up the terminal device based on the recognition result of the collected voice by the voiceprint recognition model.

[0058] In the embodiment of the present disclosure, the voiceprint recognition model is trained based on a sample speech set. The sample speech set can be obtained based on the process steps in the above-mentioned speech augmentation method embodiment. For details, please refer to the above-mentioned speech augmentation method embodiment, which will not be repeated here. For example, the sample speech set can include the augmented speech obtained by the above-mentioned speech augmentation method embodiment, or the sample speech set can include the speech to be augmented and the augmented speech obtained by the above-mentioned speech augmentation method embodiment. As a possible example, in the voice wake-up service, speech augmentation can be performed based on the purpose of sample balance to provide more realistic and diverse data for voiceprint recognition model training, and to provide an effective solution for the voiceprint recognition model to optimize the threshold drift problem caused by the mismatch between registration and test scenarios. For details, please refer to the relevant description in the above-mentioned speech augmentation embodiment, which will not be repeated here. Of course, it can also be added in the registration stage to generate augmented data that fits the test scenario for the registered speech, further reducing the threshold difference caused by the scenario.

[0059] In one implementation scenario, the voiceprint recognition model can be based on, but not limited to, d-vector, x-vector, ResNet, ECAPA-TDNN, etc., and the model structure of the voiceprint recognition model is not limited here.

[0060] In one implementation scenario, the voiceprint recognition model can recognize the collected voice to obtain a recognition result, and the recognition result may include whether the collected voice and the registered voice belong to the same person. If so, it can be determined to wake up the terminal device, otherwise the terminal device may not be woken up.

[0061] The above scheme obtains the collected voice of the terminal device, and then determines whether to wake up the terminal device based on the recognition result of the collected voice by the voiceprint recognition model. The voiceprint recognition model is trained based on the sample voice set. The sample voice set can be obtained based on the process steps in the above-mentioned voice augmentation method embodiment, so it can improve the diversity of the sample voice set, thereby improving the voiceprint recognition accuracy of the voiceprint recognition model trained based on it, and then reducing the omission rate of voice wake-up and improving the accuracy of voice wake-up.

[0062] See also Figure 6 , Figure 6 The figure is a schematic diagram of the framework of an embodiment of the speech augmentation device of the present application. The speech augmentation device 60 includes: an acquisition module 61, an extraction module 62, a selection module 63, and a synthesis module 64. The acquisition module 61 is used to acquire the speech to be augmented and the text to be synthesized; the extraction module 62 is used to perform feature extraction based on the speech to be augmented to obtain the voiceprint features of the speaker to whom the speech to be augmented belongs and the environmental noise features of the scene to which the speech to be augmented belongs; the selection module 63 is used to select the voiceprint features of at least one speaker as the target voiceprint features based on a first distribution of the speakers to whom the speech to be augmented belongs, and to select the environmental noise features of at least one scene as the target environmental noise features based on a second distribution of the scenes to which the speech to be augmented belongs; and the synthesis module 64 is used to obtain a speech synthesized from any target voiceprint features, any target environmental noise features, and the text to be synthesized as the augmented speech.

[0063] In the above scheme, the speech augmentation device 60 obtains the speech to be augmented and the text to be synthesized, performs feature extraction based on the speech to be augmented, obtains the voiceprint features of the speaker to which the speech to be augmented belongs and the environmental noise features of the scene to which the speech to be augmented belongs, then selects at least one speaker's voiceprint feature as the target voiceprint feature based on the first distribution of the speakers to which the speech to be augmented belongs, selects at least one scene's environmental noise feature as the target environmental noise feature based on the second distribution of the scene to which the speech to be augmented belongs, and obtains a speech synthesized from any target voiceprint feature, any target environmental noise feature, and the text to be synthesized as the augmented speech. Therefore, speech data augmentation can be achieved by selecting relevant voiceprint features and environmental noise features for speech synthesis with the text to be synthesized. Furthermore, because feature selection is performed from both the speaker and scene dimensions based on their respective distributions for subsequent speech synthesis during the speech augmentation process, the diversity of the augmented speech can be increased by combining different speakers and different scenes. Therefore, speech augmentation can be achieved to increase speech data diversity.

[0064] In some disclosed embodiments, the extraction module 62 includes a first extraction submodule for extracting speech features based on the acoustic features of the speech to be augmented; the extraction module 62 includes a second extraction submodule for extracting voiceprint features based on the speech features, and extracting ambient noise features based on the speech features.

[0065] In some disclosed embodiments, voiceprint features and environmental noise features are obtained by extracting features from the augmented speech by a feature extraction model. The feature extraction model is trained based on sample speech, and the sample speech is annotated with a sample speaker and a sample scene to which the sample speech belongs.

[0066] In some disclosed embodiments, the speech augmentation device 60 includes a sample feature extraction module for performing feature extraction based on the sample speech to obtain sample voiceprint features of the sample speaker to which the sample speech belongs and sample environmental noise features of the sample speaker to which the sample speech belongs; the speech augmentation device 60 includes a sample feature prediction module for performing prediction based on the sample voiceprint features to obtain a predicted speaker to which the sample speech belongs, and performing prediction based on the sample voiceprint features to obtain a predicted scene to which the sample speech belongs; the speech augmentation device 60 includes an extraction network adjustment module for adjusting the network parameters of the feature extraction model based on the differences between the predicted speaker and the sample speaker, and the differences between the predicted scene and the sample scene.

[0067] In some disclosed embodiments, the speech augmentation device 60 includes a reference audio acquisition module for acquiring the noise-free speech of a sample speaker and the ambient noise of a sample scene; the speech augmentation device 60 includes a speech linear augmentation module for linearly augmenting the noise-free speech of any sample speaker based on the ambient noise of at least one sample scene to obtain sample augmented speech; the speech augmentation device 60 includes a sample speech acquisition module for obtaining sample speech for model training based at least on the sample augmented speech.

[0068] In some disclosed embodiments, the feature extraction model includes a backbone network, a voiceprint extraction network, and a scene extraction network. The backbone network is used to extract speech features of the input speech, and the voiceprint extraction network and the scene extraction network share speech features. The voiceprint extraction network is used to extract voiceprint features, and the scene extraction network is used to extract environmental noise features.

[0069] In some disclosed embodiments, the synthesis module 64 includes an input content acquisition submodule for acquiring a target scene sequence predicted by the target environmental noise feature and acquiring a speech synthesis identifier; wherein the target scene sequence includes the probability values ​​of the target environmental noise feature predicted to belong to various scenes, and the speech synthesis identifier is used to indicate the relative importance of the target scene sequence and the text to be synthesized in the speech synthesis process; the synthesis module 64 includes an input content combination submodule for combining a text sequence based on the text to be synthesized, the target scene sequence and the speech synthesis identifier, and combining a feature sequence based on the target voiceprint feature, the target environmental noise feature and the speech synthesis identifier; the synthesis module 64 includes a speech synthesis submodule for using the speech to be augmented from which the target voiceprint feature and the target environmental noise feature are derived as a reference speech, performing speech synthesis based on the text sequence and the feature sequence, and obtaining a synthesized speech as the augmented speech.

[0070] In some disclosed embodiments, voiceprint features and environmental noise features are obtained by extracting features from the augmented speech by a feature extraction model, and the augmented speech is synthesized by a speech synthesis model. The feature extraction model and the speech synthesis model are respectively trained based on sample speech and then trained based on the sample speech. The sample speech is annotated with sample text, and the sample text represents the speech content of the sample speech.

[0071] In some disclosed embodiments, the speech augmentation device 60 includes a model feature extraction module for performing feature extraction on the sample speech based on a feature extraction model to obtain sample voiceprint features of the sample speaker to which the sample speech belongs and sample environmental noise features of the sample speaker to which the sample speech belongs; the speech augmentation device 60 includes a sample speech synthesis module for performing speech synthesis based on the sample voiceprint features, the sample environmental noise features and the sample text to obtain sample synthesized speech; the speech augmentation device 60 includes a feature extraction network adjustment module for adjusting the network parameters of the speech synthesis model based on the difference between the sample speech and the sample synthesized speech.

[0072] In some disclosed embodiments, the speech augmentation device 60 includes a new sample acquisition module, which is used to obtain, after the speech synthesis model is trained based on the sample speech, a sample synthesized speech synthesized by the speech synthesis model based on the sample voiceprint features of the sample speaker to which the sample speech belongs, the sample environmental noise features of the sample scene to which the sample speech belongs, and the sample text, as a new sample speech; the speech augmentation device 60 includes a new voiceprint feature acquisition module, which is used to extract features of the new sample speech based on the feature extraction model, and at least obtain the sample voiceprint features of the new sample speech; the speech augmentation device 60 includes a predicted speaker acquisition module, which is used to make a prediction based on the sample voiceprint features of the new sample speech, and obtain the predicted speaker to which the new sample speech belongs; the speech augmentation device 60 includes a cyclic iterative training module, which is used to determine whether to re-execute the steps of training the feature extraction model and the speech synthesis model respectively based on the difference between the predicted speaker to which the new sample speech belongs and the sample speaker to which the sample speech from which the new sample speech originates.

[0073] See also Figure 7 , Figure 7 This is a schematic diagram of the framework of an embodiment of the voice wake-up device of the present application. The voice wake-up device 70 includes an acquisition module 71 and a determination module 72. Acquisition module 71 is used to acquire voice collected by a terminal device; determination module 72 is used to determine whether to wake up the terminal device based on the recognition result of the collected voice collected by a voiceprint recognition model. The voiceprint recognition model is trained based on a sample voice set, and the sample voice set is obtained based on the voice augmentation device described above.

[0074] The above scheme obtains the collected voice of the terminal device, and then determines whether to wake up the terminal device based on the recognition result of the collected voice by the voiceprint recognition model. The voiceprint recognition model is trained based on the sample voice set. The sample voice set can be obtained based on the above-mentioned voice augmentation device embodiment, so it can improve the diversity of the sample voice set, thereby improving the voiceprint recognition accuracy of the voiceprint recognition model trained based on it, and then reducing the omission rate of voice wake-up and improving the accuracy of voice wake-up.

[0075] See also Figure 8 , Figure 8 It is a schematic diagram of the framework of an embodiment of an electronic device of the present application. The electronic device 80 includes a memory 81 and a processor 82 coupled to each other, the memory 81 at least storing program instructions, and the processor 82 is used to execute the program instructions to implement the steps in any of the above-mentioned voice augmentation method embodiments, or the steps in any of the above-mentioned voice wake-up method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 80 may include but is not limited to a smartphone, a tablet computer, an office notebook, a learning machine, a server, etc., and the specific type of the electronic device 80 is not limited here.

[0076] Specifically, the processor 82 is used to control itself and the memory 81 to implement the steps of any of the aforementioned voice augmentation method embodiments or any of the aforementioned voice wake-up method embodiments. The processor 82 may also be referred to as a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip with signal processing capabilities. The processor 82 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor. Furthermore, the processor 82 may be implemented by an integrated circuit chip.

[0077] In the above scheme, electronic device 80 obtains the speech to be augmented and the text to be synthesized, performs feature extraction based on the speech to be augmented, obtains the voiceprint features of the speaker to which the speech to be augmented belongs and the environmental noise features of the scene to which the speech to be augmented belongs, then selects at least one speaker's voiceprint feature as the target voiceprint feature based on the first distribution of the speakers to which the speech to be augmented belongs, selects at least one scene's environmental noise feature as the target environmental noise feature based on the second distribution of the scene to which the speech to be augmented belongs, and obtains speech synthesized from any target voiceprint feature, any target environmental noise feature, and the text to be synthesized as the augmented speech. This allows speech data augmentation to be achieved by selecting relevant voiceprint features and environmental noise features for speech synthesis with the text to be synthesized. Furthermore, because feature selection is performed from both the speaker and scene dimensions based on their respective distributions for subsequent speech synthesis during the speech augmentation process, the diversity of the augmented speech can be increased by combining different speakers and different scenes. Therefore, speech augmentation can be achieved to increase speech data diversity. In addition, the collected voice of the terminal device is obtained, and then based on the recognition result of the collected voice by the voiceprint recognition model, it is determined whether to wake up the terminal device, and the voiceprint recognition model is trained based on the sample voice set. The sample voice set can be obtained based on the above-mentioned voice augmentation method embodiment, so the diversity of the sample voice set can be improved, thereby improving the voiceprint recognition accuracy of the voiceprint recognition model trained based on it, thereby reducing the omission rate of voice wake-up and improving the accuracy of voice wake-up.

[0078] See also Figure 9 , Figure 91 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 90 of the present application. The computer-readable storage medium 90 stores program instructions 91 that can be executed by a processor, and the program instructions 91 are used to implement the steps of any of the above-mentioned speech augmentation method embodiments.

[0079] In the above scheme, the computer-readable storage medium 90 obtains the speech to be augmented and the text to be synthesized, performs feature extraction based on the speech to be augmented, obtains the voiceprint features of the speaker to which the speech to be augmented belongs and the environmental noise features of the scene to which the speech to be augmented belongs, then selects at least one speaker's voiceprint feature as the target voiceprint feature based on a first distribution of the speakers to which each speech to be augmented belongs, selects at least one scene's environmental noise feature as the target environmental noise feature based on a second distribution of the scene to which each speech to be augmented belongs, and obtains a speech synthesized from any target voiceprint feature, any target environmental noise feature, and the text to be synthesized as the augmented speech. This allows speech data augmentation to be achieved by selecting relevant voiceprint features and environmental noise features for speech synthesis with the text to be synthesized. Furthermore, because feature selection is performed from both the speaker and scene dimensions based on their respective distributions for subsequent speech synthesis during the speech augmentation process, the diversity of the augmented speech can be increased by combining different speakers and different scenes. Therefore, speech augmentation can be achieved to increase speech data diversity. In addition, the collected voice of the terminal device is obtained, and then based on the recognition result of the collected voice by the voiceprint recognition model, it is determined whether to wake up the terminal device, and the voiceprint recognition model is trained based on the sample voice set. The sample voice set can be obtained based on the above-mentioned voice augmentation method embodiment, so the diversity of the sample voice set can be improved, thereby improving the voiceprint recognition accuracy of the voiceprint recognition model trained based on it, thereby reducing the omission rate of voice wake-up and improving the accuracy of voice wake-up.

[0080] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0081] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0083] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0084] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0085] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the various implementation methods of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0086] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A speech augmentation method, characterized in that: include: Obtain the speech to be augmented and the text to be synthesized; Performing feature extraction based on the speech to be augmented to obtain voiceprint features of a speaker to which the speech to be augmented belongs and environmental noise features of a scene to which the speech to be augmented belongs; Based on a first distribution of the speakers to which the to-be-augmented speech sounds belong, selecting at least one voiceprint feature of the speaker as a target voiceprint feature, and based on a second distribution of the scenes to which the to-be-augmented speech sounds belong, selecting at least one environmental noise feature of the scene as a target environmental noise feature; Obtaining speech synthesized by any of the target voiceprint features, any of the target environmental noise features and the text to be synthesized as augmented speech, specifically including: obtaining a target scene sequence predicted by the target environmental noise features, and obtaining a speech synthesis identifier; wherein the target scene sequence includes probability values ​​of the target environmental noise features predicted to belong to various scenes, and the speech synthesis identifier is used to indicate the relative importance of the target scene sequence and the text to be synthesized in the speech synthesis process; based on the text to be synthesized, the target scene sequence and the speech synthesis identifier, a text sequence is obtained by combination, and based on the target voiceprint features, the target environmental noise features and the speech synthesis identifier, a feature sequence is obtained by combination; using the speech to be augmented from which the target voiceprint features and the target environmental noise features are derived as the reference speech, performing speech synthesis based on the text sequence and the feature sequence, and obtaining synthesized speech as the augmented speech.

2. The method according to claim 1, characterized in that The extracting features based on the speech to be augmented to obtain the voiceprint features of the speaker to which the speech to be augmented belongs and the environmental noise features of the scene to which the speech to be augmented belongs includes: Extracting speech features based on the acoustic features of the speech to be augmented; The voiceprint feature is extracted based on the speech feature, and the environmental noise feature is extracted based on the speech feature.

3. The method according to claim 1 or 2, characterized in that The voiceprint features and the environmental noise features are obtained by extracting features from the speech to be augmented by a feature extraction model, wherein the feature extraction model is trained based on sample speech, and the sample speech is annotated with a sample speaker to which the sample speech belongs and a sample scene to which the sample speech belongs.

4. The method according to claim 3, characterized in that The training steps of the feature extraction model include: Performing feature extraction based on the sample speech to obtain a sample voiceprint feature of the sample speaker to which the sample speech belongs and a sample environmental noise feature of the sample speaker to which the sample speech belongs; Predicting based on the sample voiceprint features to obtain a predicted speaker to which the sample speech belongs, and predicting based on the sample voiceprint features to obtain a predicted scene to which the sample speech belongs; Based on the difference between the predicted speaker and the sample speaker, and the difference between the predicted scene and the sample scene, the network parameters of the feature extraction model are adjusted.

5. The method according to claim 3, characterized in that The step of obtaining the sample speech includes: Acquire the noise-free speech of the sample speaker and the ambient noise of the sample scene; performing linear augmentation on the noise-free speech of any of the sample speakers based on the environmental noise of at least one of the sample scenes to obtain a sample augmented speech; Based at least on the sample augmented speech, a sample speech for model training is obtained.

6. The method according to claim 3, characterized in that The feature extraction model includes a backbone network, a voiceprint extraction network and a scene extraction network. The backbone network is used to extract speech features of the input speech, and the voiceprint extraction network and the scene extraction network share the speech features. The voiceprint extraction network is used to extract voiceprint features, and the scene extraction network is used to extract environmental noise features.

7. The method according to claim 1, characterized in that The voiceprint features and the environmental noise features are obtained by extracting features from the speech to be augmented by a feature extraction model, and the augmented speech is synthesized by a speech synthesis model. The feature extraction model and the speech synthesis model are respectively trained based on sample speech and then trained based on the sample speech. The sample speech is annotated with sample text, and the sample text represents the speech content of the sample speech.

8. The method according to claim 7, characterized in that The training steps of the speech synthesis model include: Performing feature extraction on the sample speech based on the feature extraction model to obtain a sample voiceprint feature of a sample speaker to which the sample speech belongs and a sample environmental noise feature of the sample speaker to which the sample speech belongs; Performing speech synthesis based on the sample voiceprint feature, the sample environmental noise feature, and the sample text to obtain a sample synthesized speech; Based on the difference between the sample speech and the sample synthesized speech, the network parameters of the speech synthesis model are adjusted.

9. The method according to claim 7, characterized in that After the speech synthesis model is trained based on the sample speech, the method further includes: Obtaining a sample synthesized speech synthesized by the speech synthesis model based on a sample voiceprint feature of a sample speaker to which the sample speech belongs, a sample environmental noise feature of a sample scene to which the sample speech belongs, and the sample text as a new sample speech; Performing feature extraction on the new sample speech based on the feature extraction model to obtain at least a sample voiceprint feature of the new sample speech; Performing a prediction based on the sample voiceprint features of the new sample speech to obtain a predicted speaker to which the new sample speech belongs; Based on the difference between the predicted speaker to which the new sample speech belongs and the sample speaker from which the new sample speech originates, it is determined whether to re-execute the steps of training the feature extraction model and the speech synthesis model respectively.

10. A voice wake-up method, characterized in that: include: Get the collected voice of the terminal device; Determine whether to wake up the terminal device based on the recognition result of the collected voice by a voiceprint recognition model; wherein the voiceprint recognition model is trained based on a sample voice set, and the sample voice set is obtained based on the voice augmentation method according to any one of claims 1 to 9.

11. A speech amplification device, characterized in that: include: An acquisition module is used to acquire the speech to be augmented and the text to be synthesized; An extraction module, configured to perform feature extraction based on the speech to be augmented, to obtain a voiceprint feature of a speaker to which the speech to be augmented belongs and an environmental noise feature of a scene to which the speech to be augmented belongs; a selection module configured to select, based on a first distribution of the speakers to which the respective to-be-augmented speech sounds belong, at least one voiceprint feature of the speaker as a target voiceprint feature, and, based on a second distribution of the scenes to which the respective to-be-augmented speech sounds belong, at least one environmental noise feature of the scene as a target environmental noise feature; A synthesis module, for obtaining speech synthesized by any of the target voiceprint features, any of the target environmental noise features and the text to be synthesized as augmented speech, specifically comprising: obtaining a target scene sequence predicted by the target environmental noise features, and obtaining a speech synthesis identifier; wherein the target scene sequence includes probability values ​​of the target environmental noise features predicted to belong to various scenes, and the speech synthesis identifier is used to indicate the relative importance of the target scene sequence and the text to be synthesized in the speech synthesis process; based on the text to be synthesized, the target scene sequence and the speech synthesis identifier, a text sequence is obtained by combination, and based on the target voiceprint features, the target environmental noise features and the speech synthesis identifier, a feature sequence is obtained by combination; using the speech to be augmented from which the target voiceprint features and the target environmental noise features are derived as a reference speech, speech synthesis is performed based on the text sequence and the feature sequence to obtain synthesized speech as the augmented speech.

12. A voice wake-up device, characterized in that: include: An acquisition module is used to acquire the collected voice of the terminal device; A determination module is used to determine whether to wake up the terminal device based on the recognition result of the collected voice by a voiceprint recognition model; wherein, the voiceprint recognition model is trained based on a sample voice set, and the sample voice set is obtained based on the voice augmentation device according to claim 11.

13. An electronic device, characterized in that: The invention comprises at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor is used to execute the program instructions to implement the speech augmentation method according to any one of claims 1 to 9, or the speech wake-up method according to claim 10.

14. A computer-readable storage medium, characterized in that Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the speech augmentation method according to any one of claims 1 to 9, or the speech augmentation method according to claim 10.

Images