Scene acoustic style extractor training method, device, electronic device and computer-readable storage medium

Abstract

The embodiments of the present invention provide a scene acoustic style extractor training method, device, electronic device and computer-readable storage medium, which relate to the field of data processing technology. The method comprises: obtaining acoustic features of a home scene in a noisy acoustic environment, inputting the acoustic features of the home scene in the noisy acoustic environment into a scene acoustic style extractor, and obtaining training style embedding information corresponding to the home scene. Thus, reliable acquisition of the scene acoustic style corresponding to the home scene is achieved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a scene acoustic style extractor training method, device, electronic device and computer-readable storage medium. Background Art

[0002] In daily life, various noises are everywhere in the home, and people often need to communicate in noisy environments. In 1909, French otolaryngologist Etienne Lombard found through research that when communicating in a noisy environment, the speaker has to actively change the way of speaking to improve the effect of the sound in the hope that the other party can hear clearly. Studies have found that even if the same person speaks the same voice, the voice characteristics in different environments are different. The changed characteristics include improving the pitch, tone, loudness and resonance peak characteristics of the voice. This phenomenon is called the Lombard effect. With the rise of smart homes, how to improve the smoothness of human-computer interaction and achieve the purpose of better serving mankind has become a concern in this field.

[0003] At present, electronic devices in home scenes, such as smart devices, can sense changes in the state of the acoustic environment and give reasonable answers to users' questions. For example, when the smart device detects that the user has issued an instruction message "turn on the microwave oven", it controls the microwave oven to turn on and voice broadcasts "The microwave oven is on" to complete the response to the instruction message. However, research has found that the content of the voice broadcast of smart devices is often not accurately received by users. For example, in a noisy environment, the voice broadcast is drowned out by the ambient sound, making it impossible for users to clearly receive the broadcast voice, which affects the smoothness of human-computer interaction in smart home scenes and cannot meet actual application needs. Summary of the invention

[0004] Based on the above problems, the inventor analyzed and concluded that the premise for users to accurately receive reasonable content such as the above-mentioned voice broadcast of "the microwave oven is on" is that the smart device can change the way of speaking in different acoustic environments and actively improve the clarity and naturalness of the synthesized speech. However, existing research has not fully considered how smart devices can actively change the way of speaking like humans under the Lombard effect when synthesizing speech, thereby improving speech clarity and allowing users to receive accurate information, resulting in the voice broadcast feedback from smart devices may not be received by users. In view of this, the inventor proposed that one of the important factors for "imitating humans" in noisy home scenes, actively changing the way of speaking under the Lombard effect, and improving the clarity and naturalness of voice broadcasts (that is, emitting Lombard speech) is how to obtain the acoustic style corresponding to the home scene.

[0005] One of the purposes of the present invention includes, for example, providing a scene acoustic style extractor training method, device, electronic device and computer-readable storage medium to achieve reliable acquisition of the acoustic style corresponding to the home scene.

[0006] The embodiments of the present invention can be implemented as follows:

[0007] In a first aspect, an embodiment of the present invention provides a scene acoustic style extractor training method, comprising:

[0008] Obtain the acoustic characteristics of home scenes in noisy acoustic environments;

[0009] The acoustic features of the home scene in the noisy acoustic environment are input into a scene acoustic style extractor to obtain training style embedding information corresponding to the home scene; wherein the training style embedding information represents the scene acoustic style corresponding to the home scene.

[0010] Based on the acoustic feature training of home scenes in noisy acoustic environments, a scene acoustic style extractor is obtained, which can reliably obtain the scene style embedding information corresponding to each home scene. The smart device can then adaptively change the voice mode according to the scene style embedding information, and speak in accordance with the scene acoustic style corresponding to the home scene, ensuring that the voice broadcast in the corresponding voice mode can be accurately received by the user.

[0011] In a second aspect, an embodiment of the present invention provides a scene acoustic style extractor training device, comprising:

[0012] A noisy environment acoustic feature acquisition module is used to obtain the acoustic features of home scenes in a noisy acoustic environment;

[0013] A scene acoustic style extractor training module is used to input the acoustic features of the home scene in the noisy acoustic environment into the scene acoustic style extractor to obtain training style embedding information corresponding to the home scene; wherein the training style embedding information represents the scene acoustic style corresponding to the home scene.

[0014] Based on the acoustic feature training of home scenes in noisy acoustic environments, a scene acoustic style extractor is obtained, which can reliably obtain the scene style embedding information corresponding to each home scene. The smart device can then adaptively change the voice mode according to the scene style embedding information, and speak in accordance with the scene acoustic style corresponding to the home scene, ensuring that the voice broadcast in the corresponding voice mode can be accurately received by the user.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the scene acoustic style extractor training method described in any one of the aforementioned implementations is implemented. Accordingly, the electronic device includes the beneficial effects of the scene acoustic style extractor training method.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a computer program, and when the computer program is executed, the electronic device where the computer-readable storage medium is located executes the scene acoustic style extractor training method described in any one of the aforementioned implementations. Accordingly, the computer-readable storage medium includes the beneficial effects of the scene acoustic style extractor training method.

[0017] In order to make the above-mentioned purposes, features and advantages of the embodiments of the present invention more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present invention is shown.

[0020] Figure 2 A schematic flow chart of a scene acoustic style extractor training method provided by an embodiment of the present invention is shown.

[0021] Figure 3 A schematic diagram of a training architecture of a scene acoustic style extractor provided by an embodiment of the present invention is shown.

[0022] Figure 4 Another schematic diagram of a flow chart of a scene acoustic style extractor training method provided by an embodiment of the present invention is shown.

[0023] Figure 5 A schematic diagram of another training architecture of a scene acoustic style extractor provided by an embodiment of the present invention is shown.

[0024] Figure 6 Another schematic diagram of a flow chart of a scene acoustic style extractor training method provided by an embodiment of the present invention is shown.

[0025] Figure 7 A schematic diagram of another training architecture of a scene acoustic style extractor provided by an embodiment of the present invention is shown.

[0026] Figure 8 Another schematic flow chart of a scene acoustic style extractor training method provided by an embodiment of the present invention is shown.

[0027] Fig. 9 One of the schematic diagrams of a reference encoder provided by an embodiment of the present invention is shown.

[0028] Fig.10 A second schematic diagram of a reference encoder provided by an embodiment of the present invention is shown.

[0029] Fig.11 A third schematic diagram of a reference encoder provided by an embodiment of the present invention is shown.

[0030] Fig.12 A schematic diagram of another training architecture of a scene acoustic style extractor provided by an embodiment of the present invention is shown.

[0031] Fig.13 A schematic flow chart of a first scene classification model training method provided in an embodiment of the present invention is shown.

[0032] Fig.14 Another flowchart of a first scene classification model training method provided by an embodiment of the present invention is shown.

[0033] Fig.15 A schematic flow chart of an acoustic feature prediction model training method provided in an embodiment of the present invention is shown.

[0034] Fig.16 A schematic diagram of a training architecture of an acoustic feature prediction model provided by an embodiment of the present invention is shown.

[0035] Fig.17 A flow chart of a speech synthesis method provided by an embodiment of the present invention is shown.

[0036] Fig.18 A schematic diagram showing the implementation principle of a speech synthesis method provided by an embodiment of the present invention is shown.

[0037] Fig.19 Another schematic diagram of the implementation principle of a speech synthesis method provided by an embodiment of the present invention is shown.

[0038] Fig. 20 A schematic diagram of the implementation principle of a scene acoustic style extractor provided by an embodiment of the present invention is shown.

[0039] Fig.21Another schematic diagram of the implementation principle of a scene acoustic style extractor provided by an embodiment of the present invention is shown.

[0040] Fig. 22 A schematic diagram of the implementation principle of an acoustic feature prediction model provided by an embodiment of the present invention is shown.

[0041] Fig.23 A schematic diagram showing the implementation principle of a speech synthesis method provided by an embodiment of the present invention is shown.

[0042] Fig.24 A schematic diagram showing another implementation principle of synthesizing and outputting response voice data provided by an embodiment of the present invention is shown.

[0043] Fig.25 A flow chart of a speech synthesis model training method provided by an embodiment of the present invention is shown.

[0044] Fig.26 A schematic diagram of a training architecture of a speech synthesis model provided by an embodiment of the present invention is shown.

[0045] Fig. 27 An exemplary structural block diagram of a first speech synthesis device provided by an embodiment of the present invention is shown.

[0046] Fig.28 An exemplary structural block diagram of a second speech synthesis device provided by an embodiment of the present invention is shown.

[0047] Fig.29 An exemplary structural block diagram of a scene classification model training device provided by an embodiment of the present invention is shown.

[0048] Fig.30 An exemplary structural block diagram of a scene acoustic style extractor training device provided by an embodiment of the present invention is shown.

[0049] Fig.31 An exemplary structural block diagram of an acoustic feature prediction model training device provided by an embodiment of the present invention is shown.

[0050] Fig.32 An exemplary structural block diagram of a speech synthesis model training device provided by an embodiment of the present invention is shown.

[0051] Icons: 100-electronic device; 110-memory; 120-processor; 130-communication module; 140-first speech synthesis device; 141-information determination module; 142-response speech data synthesis module; 150-second speech synthesis device; 151-prediction acoustic feature acquisition module; 152-information synthesis module; 160-scene classification model training device; 161-environment acoustic feature acquisition module; 162-scene classification network training module; 170-scene acoustic style extractor training device; 171-noisy environment acoustic feature acquisition module; 172-scene acoustic style extractor training module; 180-acoustic feature prediction model training device; 181-data acquisition module; 182-acoustic feature prediction model training module; 190-speech synthesis model training device; 191-information acquisition module; 192-speech data synthesis module to be output. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0053] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] It should be noted that the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0055] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it will not be repeatedly defined and explained in subsequent drawings unless otherwise specified.

[0056] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0057] Please refer to Figure 1, is a block diagram of an electronic device 100 provided in this embodiment. The electronic device 100 in this embodiment may be a variety of processing devices capable of information interaction and processing. For example, the electronic device 100 may be an intelligent device capable of voice interaction with a user. For another example, the electronic device may be a server, processing platform, etc. capable of model training and speech synthesis.

[0058] The electronic device 100 may include a memory 110, a processor 120, and a communication module 130. The memory 110, the processor 120, and the communication module 130 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines.

[0059] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0060] The processor 120 is used to read / write data or programs stored in the memory 110 and execute corresponding functions.

[0061] The communication module 130 is used to establish a communication connection between the electronic device 100 and other communication terminals through the network, and to send and receive data through the network.

[0062] It should be understood that Figure 1 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 The components shown in can be implemented by hardware, software or a combination thereof. For example, when the electronic device 100 is a smart device capable of voice interaction with a user, the electronic device 100 can also include a voice module, and the electronic device 100 can collect and output voice through the voice module.

[0063] See also Figure 2, which is a flow chart of a scene acoustic style extractor training method provided by an embodiment of the present invention, can be Figure 1 The electronic device 100 shown executes, for example, the processor 120 in the electronic device 100. The scene acoustic style extractor training method includes S110 and S120.

[0064] S110, obtaining acoustic characteristics of a home scene in a noisy acoustic environment.

[0065] S120, input the acoustic features in the noisy acoustic environment into a scene acoustic style extractor to obtain training style embedding information corresponding to the home scene. The training style embedding information represents the scene acoustic style corresponding to the home scene.

[0066] The scene acoustic style in this embodiment may include one or more attributes, for example, any one or a combination of any two or more of the attributes such as pitch, sound intensity, speech speed, emotion, etc.

[0067] Among them, the home scenes can be flexibly divided, and the acoustic features of the home scenes in a noisy acoustic environment can be collected. For example, the kitchen, living room, and bathroom where noisy sounds often occur can be selected as home scenes and the acoustic features in a noisy acoustic environment can be collected. For another example, the home exercise gym, multimedia audio-visual room, study room, etc. can be selected as home scenes and the acoustic features in a noisy acoustic environment can be collected.

[0068] It is understandable that the finer the division of home scenes, the more speech can be prepared according to each subdivided home scene, supporting the selection of more acoustic features for classification, and using the acoustic features in a noisy acoustic environment to train the scene acoustic style extractor. The finer the training style embedding information obtained, the more delicate the scene acoustic style corresponding to each home scene represented. Correspondingly, the finer the scene style embedding information corresponding to the home scene determined by the trained scene acoustic style extractor, the clearer and more natural the response speech data synthesized by the electronic device using the scene style embedding information.

[0069] The acoustic features of the home scene in a noisy acoustic environment are obtained based on the Lombard speech in the noisy home scene. For example, the Lombard speech in the noisy home scene can be collected and acoustic features can be extracted, such as using an acoustic feature extraction module to extract acoustic features to obtain the acoustic features of the home scene in the noisy acoustic environment.

[0070] Lombard speech in noisy home scenes can be obtained in a variety of ways. For example, the user's speech can be recorded under the interference of environmental sounds in various home scenes as training data for the scene acoustic style extractor. For example, volunteers can be asked to wear removable headphones, and the headphones play the environmental sounds of different home scenes. The environmental noise induces the volunteers to change their voices, so that Lombard speech in different home scenes can be captured as training data for the scene acoustic style extractor.

[0071] In order to improve the training effect, the acoustic features obtained in noisy home scenes can be as rich as possible. For example, the number of volunteers and the amount of speech of each volunteer in each noisy home scene can be increased, and the Lombard speech captured in each noisy home scene can be increased, thereby improving the richness of the acoustic features obtained in the noisy home scene.

[0072] By obtaining the Lombard speech corresponding to the home scene as richly and comprehensively as possible, performing acoustic feature extraction, and obtaining the acoustic features of the home scene in a noisy acoustic environment, and inputting the acoustic features into the scene acoustic style extractor for training, the accuracy of the training style embedding information extracted by the scene acoustic style extractor can be effectively improved. Accordingly, the scene style embedding information corresponding to the home scene determined by the trained scene acoustic style extractor is more accurate and reliable.

[0073] In this embodiment, the acoustic features in a noisy acoustic environment may be multiple. For example, the acoustic features may be spectral envelope, fundamental frequency (F0) of sound, acoustic energy, spectrogram such as linear spectrogram, Mel Frequency Cepstrum Coefficient (MFCC), Mel spectrogram, etc.

[0074] In one implementation, the scene acoustic style extractor can be trained based only on Lombard speech. For example, for each home scene, Lombard speech in a noisy home scene can be collected and acoustic features can be extracted to obtain the acoustic features of the home scene in a noisy acoustic environment. The acoustic features in the noisy acoustic environment are input into the scene acoustic style extractor to extract the embedding representation of the corresponding scene acoustic style features (training style embedding information).

[0075] The scene acoustic style extractor can be implemented in many ways. For example, see Figure 3 and Figure 4 The scene acoustic style extractor may include: a reference encoder, a first attention module, and a fully connected layer. Among them, S120, the step of inputting the acoustic features in the noisy acoustic environment into the scene acoustic style extractor, and obtaining the training style embedding information corresponding to the home scene can be implemented by S121, S122, and S123.

[0076] S121, input the acoustic features of the home scene in a noisy acoustic environment into a reference encoder to obtain reference embedding information.

[0077] S122, input the reference embedding information into the first attention module to obtain the attention weight.

[0078] S123, input the attention weight into the fully connected layer to obtain the training style embedding information.

[0079] Based on this training method, the richest and most comprehensive acoustic features of the home scene in a noisy acoustic environment are input into the scene acoustic style extractor to obtain the training style embedding information, thereby realizing the training of the scene acoustic style extractor.

[0080] In another implementation, the training of the scene acoustic style extractor can be performed based on Lombard speech and scene type information output by the first scene classification model.

[0081] Exemplarily, when the acoustic features of a home scene in a noisy acoustic environment are input into a scene acoustic style extractor, the ambient acoustic features of the home scene can also be obtained, and the ambient acoustic features of the home scene are input into a first scene classification model to obtain scene type information corresponding to the home scene, and the scene type information is input into the scene acoustic style extractor for fusion, that is, the scene type information is introduced to learn the scene acoustic style extractor.

[0082] In order to improve the convenience and efficiency of fusion, the scene type information can be input into the scene acoustic style extractor in the form of weights for fusion.

[0083] For example, when the first scene classification model is a VGG16 network, the step of inputting the ambient acoustic features of the home scene into the first scene classification model to obtain scene type information corresponding to the home scene may include: inputting the ambient acoustic features of the home scene into the VGG16 network to obtain a Softmax probability value; determining a first scene type weight corresponding to the Softmax probability value; and using the first scene type weight as the scene type information.

[0084] Among them, the correspondence between the Softmax probability value and the first scene type weight can be set in advance. Accordingly, the step of determining the first scene type weight corresponding to the Softmax probability value may include: determining the first scene type weight corresponding to the Softmax probability value according to the correspondence between the Softmax probability value and the first scene type weight.

[0085] Of course, when the scene classification model is a VGG16 network, the label value can also be obtained.

[0086] See also Figure 5 and Figure 6 , where the scene acoustic style extractor includes a reference encoder, a first attention module, and a fully connected layer, Figure 2 Step S120 shown in the figure, that is, inputting the acoustic features in the noisy acoustic environment into the scene acoustic style extractor, and obtaining the training style embedding information corresponding to the home scene can be implemented by S124, S125 and S126.

[0087] S124, input the acoustic features of the home scene in a noisy acoustic environment into a reference encoder to obtain reference embedding information.

[0088] S125, input the reference embedding information into the first attention module to obtain the attention weight.

[0089] In the case where the first scene type weight is used as the scene type information, correspondingly, the step of inputting the scene type information into the scene acoustic style extractor for fusion is implemented through S226.

[0090] S126, input the attention weight and the first scene type weight into the fully connected layer for weighting to obtain training style embedding information.

[0091] For another example, when the first scene classification model is a ResNet network, the step of inputting the ambient acoustic features of the home scene into the first scene classification model to obtain scene type information corresponding to the home scene may include: inputting the ambient acoustic features of the home scene into the ResNet network to obtain a label value of the home scene; determining a second scene type weight corresponding to the label value; and using the second scene type weight as the scene type information.

[0092] Among them, the correspondence between the label value and the second scene type weight can be set in advance. Accordingly, the step of determining the second scene type weight corresponding to the label value may include: determining the second scene type weight corresponding to the label value according to the correspondence between the label value and the second scene type weight.

[0093] See also Figure 7 and Figure 8, in the case where the scene acoustic style extractor includes a reference encoder, a first attention module and a fully connected layer, S120, the step of inputting the acoustic features in the noisy acoustic environment into the scene acoustic style extractor, and obtaining the training style embedding information corresponding to the home scene can be implemented through S127, S128 and S129.

[0094] S127, input the acoustic features of the home scene in a noisy acoustic environment into a reference encoder to obtain reference embedding information.

[0095] S128, input the reference embedding information into the first attention module to obtain the attention weight.

[0096] In the case where the second scene type weight is used as the scene type information, correspondingly, the step of inputting the scene type information into the scene acoustic style extractor for fusion is implemented through S229.

[0097] S129, input the attention weight and the second scene type weight into the fully connected layer for weighting to obtain the training style embedding information.

[0098] The above training process of the scene acoustic style extractor and the implementation structure of the scene acoustic style extractor are merely examples. The training process and implementation structure of the scene acoustic style extractor may also be other, which are not described one by one in this embodiment.

[0099] In the above example of the scene acoustic style extractor training process, the acoustic features of the home scene in a noisy acoustic environment are input into the reference encoder, and the step of obtaining reference embedding information may include: inputting the acoustic features of variable-length Lombard speech (acoustic features of the home scene in a noisy acoustic environment) into the reference encoder, and the reference encoder compresses the acoustic features of the variable-length Lombard speech into reference embedding information of fixed length, such as a fixed-size vector. The reference embedding information is used to encode the overall acoustic style of a piece of audio.

[0100] The reference encoder may have a variety of implementation structures, which are described in the following examples in this embodiment.

[0101] For example, Fig. 9 As shown, the reference encoder may include a convolutional neural network (CNN), a bidirectional long short-term memory recurrent neural network (including: a backward recurrent neural network (backward LSTM) and a forward recurrent neural network (forward LSTM)) and a mapping layer.

[0102] The acoustic features of the home scene in a noisy acoustic environment are input into CNN to extract further acoustic features, and the further acoustic features are input into the bidirectional long short-term memory recurrent neural network to obtain the acoustic features related to the context, and the acoustic features related to the context are input into the mapping layer to output the reference embedding information of a fixed length to encode the overall acoustic style of a piece of audio, such as the Lombard speech acoustic style of the user in each home scene.

[0103] It can be understood that if different home scenes with similar environmental acoustic characteristics are distinguished, and accordingly, the reference encoder processes the acoustic characteristics of different home scenes with similar environmental acoustic characteristics in a noisy acoustic environment, it is possible to achieve refined processing of the acoustic styles corresponding to different home scenes with similar environmental acoustic characteristics.

[0104] For example, Fig.10 As shown, the reference encoder may include a first pre-trained model output layer, a CNN, and a mapping layer. The first pre-trained model may refer to the implementation of Audio word2vec, which is mainly implemented by a sequence-to-sequence autoencoder. Audio word2vec mainly has a recurrent neural network (RNN) structure. The first pre-trained model output layer outputs a pre-trained vector. The purpose of using Audio word2vec is to obtain a better expression of acoustic features. Further acoustic features are extracted through two to three convolutional neural networks, and then mapped to predefined dimensions through two to three fully connected networks as mapping layers.

[0105] For example, Fig.11 As shown, the reference encoder may include a second pre-trained model output layer, a bidirectional long short-term memory recurrent neural network, and a mapping layer. The second pre-trained model may refer to the implementation of an unsupervised pre-trained model (wav2vec), which is mainly implemented by a CNN-based encoder. The second pre-trained model output layer outputs a pre-trained vector. The purpose of using Wave2Vec is to obtain a better expression of acoustic features. Further context-related acoustic features are extracted through a bidirectional long short-term memory recurrent neural network, and then mapped to a predefined dimension through two to three fully connected networks as mapping layers.

[0106] Using the first pre-trained model and the second pre-trained model can obtain more expressions about sound features.

[0107] The first attention module can obtain attention weights in a variety of ways. Exemplarily, an acoustic style marker set can be formed by permutation and combination based on attributes such as pitch, sound intensity, speech rate, emotion, etc. In the process of obtaining the attention weights in the first attention module, a group of acoustic style markers are randomly called from the acoustic style marker set for embedding. Each acoustic style marker can include one or more of the attributes such as pitch, sound intensity, speech rate, emotion, etc. The number of embedded acoustic style markers can be flexibly set, for example, it can be set to one, three, five, ten, etc., to represent a small number of different acoustic dimensions in the training data of the scene acoustic style extractor, such as one or more of the attributes such as pitch, sound intensity, speech rate, emotion, etc. The reference embedding information is used as the query information of the first attention module, and the first attention module learns the similarity measure between the reference embedding information and each acoustic style marker in the embedded set of acoustic style markers. The first attention module then outputs a set of combined weights (attention weights), which represent the contribution of each acoustic style marker to the reference embedding information. Accordingly, the training style embedding information output by the scene acoustic style extractor may be information related to any one or more of the attributes such as pitch, sound intensity, speech rate, emotion, etc. For example, when a home scene is a living room, the training style embedding information corresponding to the living room may include increasing pitch, increasing sound intensity, and increasing speech rate.

[0108] In this embodiment, the training style embedding information can be presented in a variety of ways. For example, the training style embedding information can be a specific assignment of attributes such as pitch, sound intensity, speech speed, emotion, etc. For another example, the training style embedding information can be an adjustment value of attributes such as pitch, sound intensity, speech speed, emotion, etc. based on a normal voice. Among them, the normal voice can be a voice uttered in a quiet scene.

[0109] Please refer to Fig.12 When scene type information such as Softmax probability value or label value is introduced to learn the scene acoustic style extractor, the Softmax probability value and label value can be processed into information that can be recognized and used by the fully connected layer through the style embedding regulator. For example, the first scene type weight corresponding to the Softmax probability value is determined by the style embedding regulator, and the second scene type weight corresponding to the label value is determined. In this case, the home scene to which the acoustic features in the noisy acoustic environment belong is known.

[0110] When the scene type information is a label value, the fully connected layer can use the second scene type weight corresponding to the label value to multiply the attention weight, and only retain one acoustic style marker as the training style embedding information corresponding to the corresponding home scene, which can determine which attribute a certain acoustic style marker has a greater influence on in the home scene. During training, multiple sets of weight parameters can be manually defined and directly called during inference, thereby improving training efficiency.

[0111] When the scene type information is a Softmax probability value, the fully connected layer can use the first scene type weight corresponding to the softmax probability value to multiply the attention weight to balance the contribution of each embedded acoustic style marker to the reference embedding information, so that the scene acoustic style extractor can not only learn the acoustic style of Lombardspeech in the corresponding home scene, but also learn the impact of different home scenes on the acoustic style.

[0112] In order to improve the training efficiency of the scene acoustic style extractor and accelerate convergence, in another implementation method, a second scene classification model can also be called, and the second scene classification model can indicate the type of home scene to which the acoustic features in a noisy acoustic environment belong. By inputting the acoustic features of the home scene in a noisy acoustic environment into the second scene classification model, the scene type indication information corresponding to the home scene is obtained, so that the fully connected layer can be trained and fed back according to the scene type indication information. For example, when the acoustic features in a noisy acoustic environment are input into the scene acoustic style extractor and the scene acoustic style extractor is trained, the acoustic features in the noisy acoustic environment are input into the second scene classification model, and the second scene classification model outputs the scene type indication information, which is passed to the fully connected layer of the scene acoustic style extractor, and training feedback is provided to the fully connected layer, thereby assisting convergence and improving training efficiency. In this embodiment, when the second scene classification model is called to perform training feedback on the scene acoustic style extractor, the acoustic features input to the second scene classification model in a noisy acoustic environment and the acoustic features input to the scene acoustic style extractor in a noisy acoustic environment may be the same or different, as long as the acoustic features of the home scene input to the scene acoustic style extractor in a noisy acoustic environment and the acoustic features of the home scene input to the second scene classification model in a noisy acoustic environment correspond to the same home scene.

[0113] Please refer to Fig.13 , which is a flow chart of a first scene classification model training method provided by an embodiment of the present invention, can be Figure 1 The electronic device 100 is shown to execute, for example, it can be executed by the processor 120 in the electronic device 100. The first scene classification model training method includes S210 and S220.

[0114] S210, obtains environmental acoustic characteristics of various home scenes.

[0115] S220, input the environmental acoustic features of multiple home scenes into the scene classification network for training to obtain scene type information corresponding to each home scene.

[0116] Among them, the home scenes can be flexibly divided and the environmental acoustic features of the home scenes can be collected. For example, the kitchen, living room, and bathroom where noisy sounds often occur can be selected as home scenes and the environmental acoustic features of the home scenes can be collected. For another example, the home exercise gym, multimedia audio-visual room, study room, etc. can be selected as home scenes and the environmental acoustic features of the home scenes can be collected. Through the fine division of home scenes, the scene classification network is trained using the environmental acoustic features of different home scenes to obtain the scene type information corresponding to each home scene, and the type of home scene can be judged based on the scene type information.

[0117] In order to improve the training effect, the environmental acoustic features of each home scene can be as rich as possible. For example, the environmental acoustic features of each home scene can include the individual environmental acoustic features of each home appliance located in the home scene, and the environmental acoustic features of at least two combinations of each home appliance located in the home scene. For example, when the home scene is a kitchen, the environmental acoustic features of the kitchen can include the individual environmental acoustic features of each home appliance in the kitchen, such as a range hood, a dishwasher, a gas stove, pots and pans, etc., and the environmental acoustic features of two, three, four, etc. combinations of the range hood, dishwasher, gas stove, pots and pans, etc. in the kitchen.

[0118] By obtaining the environmental acoustic features of the home scene as richly and comprehensively as possible, and inputting them into the scene classification network for training, the sensitivity and reliability of the first scene classification model obtained through training can be effectively improved. For example, by inputting the environmental acoustic features of individual home appliances into the scene classification network for training, the type of home scene can be directly determined based on the environmental acoustic features of certain specific home appliances. For example, based on the acoustic features of the sound of a toilet flushing, the home scene can be directly determined to be a bathroom. For another example, by inputting the environmental acoustic features of multiple permutations and combinations of each home appliance in each home scene into the scene classification network for training, it is possible to reliably distinguish different home scenes with similar environmental acoustic features. For example, both the kitchen and the bathroom may be mixed with the acoustic features of water flow, however, the home scene can be determined to be a kitchen by combining the acoustic features of the sound of a wok cooking.

[0119] Please refer to Fig.14 In one implementation, S210, obtaining environmental acoustic characteristics of various home scenes can be achieved through S211 and S212.

[0120] S211, obtain environmental acoustic data of various home scenes.

[0121] S212, extracting acoustic features from the ambient acoustic data to obtain ambient acoustic features of various home scenes.

[0122] The acoustic feature extraction can be performed in a variety of ways, for example, the acoustic feature extraction module can be used to extract the acoustic features of the ambient acoustic data. The ambient acoustic data in this embodiment is the acoustic data generated by the use of various household appliances in the home scene, and does not include human voice.

[0123] The scene classification network can be flexibly selected, for example, it can be a set deep learning network. Exemplarily, the scene classification network can be a deep convolutional neural network, such as a VGG16 network, a ResNet network, etc. Depending on the scene classification network, different scene type information can be obtained. Exemplarily, in the case where the scene classification network is a VGG16 network, in S220, the environmental acoustic features are input into the VGG16 network, and the scene type information obtained is a Softmax probability value (of course, a label value such as one-hot encoding can also be obtained). In the case where the scene classification network is a ResNet network, in S220, the environmental acoustic features are input into the ResNet network, and the scene type information obtained is a label value such as one-hot encoding (of course, the scene type information obtained can also be a softmax probability value).

[0124] Based on the above design, according to the label value representing the home scene type, the home scene type to which the ambient acoustic characteristics of the home scene specifically belong can be determined, or according to the softmax probability value representing the home scene belonging to each type, different weight combinations of home scene styles can be determined, thereby achieving the refinement of the home scene recognition granularity, improving the accuracy of home scene recognition, and better "guiding" the extraction of scene style embedding.

[0125] Based on the detailed classification of the home scene by the first scene classification model, the scene acoustic style in the home scene can be divided more finely. And it can also better "guide" the extraction of scene style embedding. For example, when the home scene includes three types of kitchen, living room and bathroom, the scene acoustic style extractor can extract the scene acoustic style for the kitchen, living room and bathroom respectively in combination with the first scene classification model.

[0126] In order to more clearly explain the first scene classification model training method in the embodiment of the present invention, the following scene is taken as an example for illustration.

[0127] In the case where the home scene includes three types of rooms: kitchen, living room, and bathroom, the environmental acoustic data generated by each household appliance in the kitchen individually, and the environmental acoustic data generated by two permutations, three permutations, four permutations, etc. are collected, and all the environmental acoustic data collected in the kitchen are used as the first acoustic data set. The environmental acoustic data generated by each household appliance in the living room individually, and the environmental acoustic data generated by two permutations, three permutations, four permutations, etc. are collected, and all the environmental acoustic data collected in the living room are used as the second acoustic data set. The environmental acoustic data generated by each household appliance in the bathroom individually, and the environmental acoustic data generated by two permutations, three permutations, four permutations, etc. are collected, and all the environmental acoustic data collected in the bathroom are used as the third acoustic data set.

[0128] The acoustic features of the ambient acoustic data in the first acoustic data set are extracted to obtain a first acoustic feature set. The acoustic features of the ambient acoustic data in the second acoustic data set are extracted to obtain a second acoustic feature set. The acoustic features of the ambient acoustic data in the third acoustic data set are extracted to obtain a third acoustic feature set.

[0129] The scene type information corresponding to the kitchen is set to the first type, the scene type information corresponding to the living room is set to the second type, and the scene type information corresponding to the bathroom is set to the third type.

[0130] When the acoustic features in the first acoustic feature set are input into the scene classification network, the first type is used as the target output of the scene classification network. When the acoustic features in the second acoustic feature set are input into the scene classification network, the second type is used as the target output of the scene classification network. When the acoustic features in the third acoustic feature set are input into the scene classification network, the third type is used as the target output of the scene classification network. Based on this, the scene classification network is trained until the convergence condition is reached, and the desired first scene classification model is obtained.

[0131] The convergence condition can be set flexibly. For example, the accuracy of obtaining scene type information corresponding to each home scene can reach a preset value. For example, the acoustic features in the first acoustic feature set, the second acoustic feature set, and the third acoustic feature set can be divided into training data and test data, and the scene classification network is trained based on the training data until the accuracy of the target output obtained after the test data is input reaches a preset value, and it is determined that the convergence condition is met and the required first scene classification model is obtained. For another example, the number of training times can reach a set amount. This embodiment does not limit this.

[0132] After the first scene classification model is obtained by the scene classification model training method, the home scene or weight combination of home scene styles to which the environmental acoustic feature belongs can be identified based on the first scene classification model. For example, if the environmental acoustic feature of a home scene is input into the scene classification model, the scene classification model outputs the first type, and according to the first type, it can be determined that the environmental acoustic feature belongs to the kitchen.

[0133] After inputting the environmental acoustic features of various home scenes into the trained first scene classification model, the first scene classification model outputs scene type related information, such as label values ​​or softmax probability values. The scene type information output by the first scene classification model can be used as input to the scene acoustic style extractor, and combined with the training and use of the scene acoustic style extractor.

[0134] An embodiment of the present invention also provides a second scene classification model training method, including: obtaining acoustic features of multiple home scenes in a noisy acoustic environment, inputting the acoustic features of the multiple home scenes in the noisy acoustic environment into a scene classification network for training, and obtaining scene type indication information corresponding to each home scene.

[0135] The second scene classification model uses the acoustic features of various home scenes in a noisy acoustic environment as input. Accordingly, the second scene classification model can indicate the type of home scene to which the acoustic features in the noisy acoustic environment belong.

[0136] Since the difference between the training process of the second scene classification model and the first scene classification model is only that the input data of the two are different, due to the different input data, there are differences in the output data. The scene type indication information output by the second scene classification model is used as training feedback for the scene acoustic style extractor, such as performing training feedback on the fully connected layer of the scene acoustic style extractor according to the scene type indication information to accelerate convergence. The scene type information output by the first scene classification model is used for weighting, such as inputting the scene type information and the attention weight into the fully connected layer of the scene acoustic style extractor for weighting. The optional structure and training principle of the second scene classification model can refer to the relevant description of the first scene classification model mentioned above, so it will not be repeated here.

[0137] During the training and application process of the scene acoustic style extractor, you can choose to call the first scene classification model, you can choose to call the second scene classification model, you can choose to call the first scene classification model and the second scene classification model, or you can choose not to call either the first scene classification model or the second scene classification model.

[0138] See also Fig.15 , which is a flow chart of a method for training an acoustic feature prediction model provided by an embodiment of the present invention, can be Figure 1The electronic device 100 shown is executed, for example, by the processor 120 in the electronic device 100. The acoustic feature prediction model training method includes S310 and S320.

[0139] S310, obtaining training style embedding information corresponding to the text information to be output and the home scene.

[0140] S320, input the text information to be output and the training style embedding information into the acoustic feature prediction network to obtain the predicted features of the sound to be synthesized corresponding to the home scene.

[0141] The training style embedding information corresponding to the home scene can be obtained through the aforementioned scene acoustic style extractor, which will not be elaborated here.

[0142] In this embodiment, the training style embedding information corresponding to the home scene is cleverly introduced as a new dimension for speech synthesis consideration, and the text information to be output and the training style embedding information are both input into the acoustic feature prediction network to obtain the predicted features of the sound to be synthesized corresponding to the home scene, and the predicted features of the sound to be synthesized have the acoustic style of Lombard speech. Based on the two dimensions of the text information to be output and the training style embedding information, reliable prediction of the acoustic features corresponding to the home scene can be achieved, and the predicted acoustic features with the acoustic style of Lombard speech can be output.

[0143] In one implementation, the speech of the user in a quiet scene can be collected as normal speech, and the normal speech and the training style embedding information are used to jointly train the acoustic feature prediction network to obtain the prediction features of the sound to be synthesized corresponding to the home scene. For example, when the training style embedding information is to increase the pitch, increase the sound intensity, and increase the speech speed, the normal speech and the training style embedding information are used to jointly train the acoustic feature prediction network, and the obtained prediction features of the sound to be synthesized may include increasing the volume, sound intensity, and speech speed by set values ​​on the basis of the normal speech.

[0144] There are many ways to implement the acoustic feature prediction network. In order to improve the naturalness of the predicted features of the sound to be synthesized, Tacotron can be selected as the acoustic feature prediction network.

[0145] Please refer to Fig.16In one implementation, the acoustic feature prediction network may include: an encoder, a second attention module, and a decoder. Accordingly, S320, the step of inputting the text information to be output and the training style embedding information into the acoustic feature prediction network to obtain the prediction features of the sound to be synthesized corresponding to the home scene may be implemented in the following manner: inputting the text information to be output into the encoder to obtain the character embedding information of a fixed length; inputting the character embedding information of a fixed length and the training style embedding information into the second attention module to obtain the aligned acoustic features and character information; inputting the aligned acoustic features and character information into the decoder to obtain the prediction features of the sound to be synthesized.

[0146] When Tacotron is selected as the acoustic feature prediction network, the aligned acoustic features and character information are input into the decoder, and the predicted features of the sound to be synthesized output by the decoder can be a linear spectrum or a Mel spectrum or other acoustic features suitable for the vocoder.

[0147] The above lists the training process of the scene acoustic style extractor, the acoustic feature prediction model, the first scene classification model and the second scene classification model. Based on the above training process, the trained scene acoustic style extractor, the acoustic feature prediction model, the first scene classification model and the second scene classification model can be obtained. The above-mentioned first scene classification model training method, the second scene classification model training method, the scene acoustic style extractor training method and the acoustic feature prediction model training method can be executed in the same electronic device or in different electronic devices. The first scene classification model training method, the second scene classification model training method, the scene acoustic style extractor training method and the acoustic feature prediction model training method can be run separately or in combination using different matching methods as required. The trained scene acoustic style extractor, acoustic feature prediction model, the first scene classification model and the second scene classification model can be run separately or in combination using different matching methods as required.

[0148] See also Fig.17 , which is a flow chart of a speech synthesis method provided by an embodiment of the present invention, can be Figure 1 The electronic device 100 shown is executed, for example, may be executed by the processor 120 in the electronic device 100. The speech synthesis method includes S410, S420 and S430.

[0149] S410, determining scene style embedding information corresponding to the home scene; wherein the scene style embedding information represents the scene acoustic style corresponding to the home scene.

[0150] S420, determining the predicted acoustic features corresponding to the home scene according to the response text content information and the scene style embedding information.

[0151] S430, synthesizing the predicted acoustic features to obtain output response voice data.

[0152] In one implementation, the electronic device may be a smart device in a smart home scenario. The output response voice data synthesized by the smart device through S410 to S430 is Lombard speech with Lombard effect. The smart device outputs the output response voice data in a noisy home scenario so that it can be clearly transmitted to the user, ensuring smooth interaction. In another implementation, the electronic device may be a server, which can be connected to a smart device in a smart home scenario. The output response voice data synthesized by the server through S410 to S430 is Lombard speech with Lombard effect. The server sends the synthesized output response voice data to the smart device. The smart device plays the output response voice data in a noisy home scenario so that it can be clearly transmitted to the user, ensuring smooth interaction.

[0153] S410 to S430 may be implemented in a variety of ways, for example, the scene style embedding information corresponding to the home scene may be determined by a trained scene acoustic style extractor. For another example, the predicted acoustic features corresponding to the home scene may be determined by a trained acoustic feature prediction model.

[0154] Please refer to Fig.18 In one implementation, the electronic device can call a trained scene acoustic style extractor to obtain acoustic features of a home scene in a noisy acoustic environment, input the acoustic features of the home scene in a noisy acoustic environment into the scene acoustic style extractor, and obtain scene style embedding information corresponding to the home scene. The electronic device can call a trained acoustic feature prediction model, use the response text content information and the scene style embedding information as inputs to the acoustic feature prediction model, obtain predicted acoustic features corresponding to the home scene, and then synthesize and output response voice data.

[0155] Please refer to Fig.19In another implementation, the electronic device can call the trained scene acoustic style extractor and the first scene classification model, input the environmental acoustic features of the home scene into the first scene classification model, obtain the scene type information corresponding to the home scene, and input the scene type information into the scene acoustic style extractor for fusion, such as using the scene type information and the acoustic features of the home scene in a noisy acoustic environment as inputs to the scene acoustic style extractor, and the scene acoustic style extractor outputs the scene style embedding information corresponding to the home scene. The electronic device can call the trained acoustic feature prediction model, use the response text content information and the scene style embedding information as inputs to the acoustic feature prediction model, obtain the predicted acoustic features corresponding to the home scene, and then synthesize and output the response voice data.

[0156] For example, in the speech synthesis process, you can call the trained scene acoustic style extractor and acoustic feature prediction model. Fig. 20 , the scene acoustic style extractor may include: a reference encoder, a first attention module and a fully connected layer. When only the acoustic features of the home scene in a noisy acoustic environment are used as the input of the scene acoustic style extractor, the scene style embedding information corresponding to the home scene can be obtained in the following manner: input the acoustic features of the home scene in a noisy acoustic environment into the reference encoder to obtain reference embedding information; input the reference embedding information into the first attention module to obtain the attention weight; input the attention weight into the fully connected layer to obtain the scene style embedding information. The response text content information and the scene style embedding information are used as inputs to the acoustic feature prediction model to obtain the predicted acoustic features corresponding to the home scene, and the predicted acoustic features are synthesized to obtain the output response speech data.

[0157] For another example, in the speech synthesis process, the trained scene acoustic style extractor, the first scene classification model, and the acoustic feature prediction model can be called. Fig.21 , the scene acoustic style extractor may include: a reference encoder, a first attention module and a fully connected layer. When the scene type information and the acoustic features of the home scene in a noisy acoustic environment are used as the input of the scene acoustic style extractor, the scene style embedding information corresponding to the home scene can be obtained in the following manner: the acoustic features of the home scene in a noisy acoustic environment are input into the reference encoder to obtain reference embedding information; the reference embedding information is input into the first attention module to obtain the attention weight; the attention weight and the scene type information are input into the fully connected layer for weighting to obtain the scene style embedding information. The response text content information and the scene style embedding information are used as the input of the acoustic feature prediction model to obtain the predicted acoustic features corresponding to the home scene, the predicted acoustic features are synthesized, and the output response speech data is obtained.

[0158] When the first scene classification model is a VGG16 network, the environmental acoustic features of the home scene can be input into the VGG16 network to obtain a Softmax probability value; determine the first scene type weight corresponding to the Softmax probability value; and use the first scene type weight as scene type information. According to the corresponding relationship between the Softmax probability value and the first scene type weight, the first scene type weight corresponding to the Softmax probability value can be determined.

[0159] Accordingly, the scene style embedding information is obtained by inputting the attention weight and the first scene type weight into the fully connected layer for weighting.

[0160] When the first scene classification model is a ResNet network, the environmental acoustic features of the home scene can be input into the ResNet network to obtain a label value of the home scene; determine the second scene type weight corresponding to the label value; and use the second scene type weight as the scene type information. According to the corresponding relationship between the label value and the second scene type weight, the second scene type weight corresponding to the label value can be determined.

[0161] Accordingly, the scene style embedding information is obtained by inputting the attention weight and the second scene type weight into the fully connected layer for weighting.

[0162] In this embodiment, the acoustic features of the home scene in a noisy acoustic environment are used as input to the scene acoustic style extractor, or the acoustic features of the home scene in a noisy acoustic environment and the scene type information output by the first scene classification model are used as input to the scene acoustic style extractor, and the scene acoustic style extractor outputs scene style embedding information. The scene style embedding information output by the scene acoustic style extractor can be used as input to the acoustic feature prediction model described below.

[0163] Please refer to Fig. 22 , the acoustic feature prediction model may include an encoder, a second attention module and a decoder. Accordingly, taking the response text content information and the scene style embedding information as the input of the acoustic feature prediction model, obtaining the predicted acoustic features corresponding to the home scene can be achieved in the following manner: inputting the response text content information into the encoder to obtain fixed-length character embedding information; inputting the fixed-length character embedding information and the scene style embedding information into the second attention module to obtain aligned acoustic features and character information; inputting the aligned acoustic features and character information into the decoder to obtain the predicted acoustic features corresponding to the home scene.

[0164] In this embodiment, the response text content information and the scene style embedding information are used as the input of the acoustic feature prediction model, and the acoustic feature prediction model outputs predicted acoustic features. Based on the predicted acoustic features, output response voice data with Lombard effect can be synthesized, so that the output response voice data played by the electronic device in a noisy home scene can be reliably received by the user, thereby ensuring the smoothness of human-computer interaction in the smart home scene and better meeting the actual application needs.

[0165] In one implementation, a speech synthesis device may be used to synthesize the output response speech data. Fig.23 As shown, a vocoder can be used as a speech synthesis device to synthesize the predicted acoustic features output by the acoustic feature prediction model, and the predicted acoustic features are input into the vocoder to obtain output response speech data containing response text content information. In this embodiment, the vocoder can be flexibly selected. For example, when the predicted acoustic feature is Linear Spectrum, Griffin-Lim can be selected as a spectrum-to-waveform vocoder, or WaveNet can be selected as a vocoder. For another example, when the predicted acoustic feature is Mel Spectrum, WaveNet can be selected as a vocoder.

[0166] The above describes the optional embodiments of the speech synthesis method in the embodiment of the present invention. In other implementations, based on the same design concept, by "imitating" the communication mode of humans actively changing the way of speaking under the Lombard effect, for different home scenes, synthesizing speech data with corresponding scene acoustic characteristics for broadcasting, and synthesizing Lombard speech with good naturalness and clarity to ensure the smoothness of voice interaction with users in a noisy home environment. The speech synthesis method in the embodiment of the present invention can also have other implementations.

[0167] In another implementation, see Fig.24 , the Lombard speech can be input into the Lombard speech generation model, and the Lombard speech can be learned based on the Lombard speech generation model, so that the Lombard speech generation model can directly obtain output response speech data according to the response text content information. Based on this Lombard speech generation model, the smart device does not need to train and call the first scene classification model, the second scene classification model, the scene acoustic style extractor and the acoustic feature prediction model, and directly inputs the response text content information into the Lombard speech generation model to obtain the output response speech data corresponding to the home scene.

[0168] In this embodiment, a speech synthesis device may be used to synthesize the predicted acoustic features to obtain output response speech data. In one implementation, the speech synthesis device may be a vocoder, and the predicted acoustic features are input into the vocoder to obtain output response speech data.

[0169] In another implementation, the speech synthesis model may be directly trained, and then the output response speech data may be obtained based on the trained speech synthesis model.

[0170] See also Fig.25 , which is a flow chart of a speech synthesis model training method provided by an exemplary embodiment of the present invention, can be Figure 1 The electronic device 100 shown is executed, for example, can be executed by the processor 120 in the electronic device 100. The speech synthesis model training method includes S510, S520 and S530.

[0171] S510, obtaining training style embedding information corresponding to the text information to be output and the home scene.

[0172] S520, obtaining predicted features of the sound to be synthesized corresponding to the home scene according to the text information to be output and the training style embedding information;

[0173] S530, synthesizing the predicted features of the sound to be synthesized to obtain speech data to be output.

[0174] Among them, the training style embedding information corresponding to the home scene can be obtained through the scene acoustic style extractor. For example, the acoustic features in the noisy acoustic environment corresponding to the home scene can be input into the scene acoustic style extractor to obtain the training style embedding information corresponding to the home scene. The predicted features of the sound to be synthesized can be obtained through the acoustic feature prediction network. For example, the training style embedding information and the text information to be output can be used as the input of the acoustic feature prediction network to obtain the predicted features of the sound to be synthesized corresponding to the home scene.

[0175] Accordingly, see Fig.26 , the acoustic features of the home scene in a noisy acoustic environment can be input into the scene acoustic style extractor to obtain the training style embedding information corresponding to the home scene. The training style embedding information represents the scene acoustic style corresponding to the home scene. The training style embedding information and the text information to be output are used as inputs of the acoustic feature prediction network to obtain the predicted features of the sound to be synthesized corresponding to the home scene. The predicted features of the sound to be synthesized are synthesized to obtain the speech data to be output.

[0176] In this embodiment, the scene acoustic style extractor may include: a reference encoder, a first attention module, and a fully connected layer. The training style embedding information may be obtained by the following steps: inputting the acoustic features of the home scene in a noisy acoustic environment into the reference encoder to obtain reference embedding information; inputting the reference embedding information into the first attention module to obtain attention weights; inputting the attention weights into the fully connected layer to obtain training style embedding information. Please return to Figure 3 and Figure 4 One of the training processes of the scene acoustic style extractor can refer to the relevant descriptions of S110 and S121 to S123 above, which will not be repeated here.

[0177] The acoustic feature prediction network may include an encoder, a second attention module, and a decoder. Accordingly, the prediction features of the sound to be synthesized can be obtained by the following steps: input the text information to be output into the encoder to obtain fixed-length character embedding information; input the fixed-length character embedding information and the training style embedding information into the second attention module to obtain aligned acoustic features and character information; input the aligned acoustic features and character information into the decoder to obtain the prediction features of the sound to be synthesized. Please return to refer to Fig.15 and Fig.16 The training process of the acoustic feature prediction model can refer to the relevant description of S310 and S320 above, which will not be repeated here.

[0178] On the basis of combining the scene acoustic style extractor training method and the acoustic feature prediction model training method for speech synthesis model training, the above-mentioned first scene classification model training method can also be introduced to train the speech synthesis model. The training process of the first scene classification model can refer to the relevant descriptions of S210 and S220 above, which will not be repeated here. Accordingly, the speech synthesis model training method can also include: inputting the environmental acoustic features of the home scene into the first scene classification model, obtaining the scene type information corresponding to the home scene, and inputting the scene type information into the scene acoustic style extractor for fusion.

[0179] In one implementation, when the first scene classification model is a VGG16 network, the step of inputting the environmental acoustic features of the home scene into the first scene classification model, and obtaining the scene type information corresponding to the home scene may include: inputting the environmental acoustic features of the home scene into the VGG16 network to obtain a Softmax probability value; determining a first scene type weight corresponding to the Softmax probability value; and using the first scene type weight as the scene type information. Exemplarily, the first scene type weight corresponding to the Softmax probability value may be determined based on the corresponding relationship between the Softmax probability value and the first scene type weight.

[0180] The scene acoustic style extractor may include: a reference encoder, a first attention module, and a fully connected layer. The training style embedding information may be obtained by: inputting the acoustic features of the home scene in a noisy acoustic environment into the reference encoder to obtain reference embedding information; inputting the reference embedding information into the first attention module to obtain attention weights. Accordingly, the step of inputting the scene type information into the scene acoustic style extractor for fusion includes: inputting the attention weight and the first scene type weight into the fully connected layer for weighting to obtain the training style embedding information. Please refer to Figure 5 and Figure 6 One of the training processes of the scene acoustic style extractor can refer to the relevant descriptions of S110 and S124 to S126 above, which will not be repeated here.

[0181] In another implementation, when the scene classification model is a ResNet network, the step of inputting the environmental acoustic features of the home scene into the first scene classification model, and obtaining the scene type information corresponding to the home scene may include: inputting the environmental acoustic features of the home scene into the ResNet network to obtain a label value of the home scene; determining a second scene type weight corresponding to the label value; and using the second scene type weight as the scene type information. Exemplarily, the second scene type weight corresponding to the label value may be determined according to the corresponding relationship between the label value and the second scene type weight.

[0182] The scene acoustic style extractor may include: a reference encoder, a first attention module, and a fully connected layer. The training style embedding information may be obtained by: inputting the acoustic features of the home scene in a noisy acoustic environment into the reference encoder to obtain reference embedding information; inputting the reference embedding information into the first attention module to obtain attention weights. Accordingly, the step of inputting the scene type information into the scene acoustic style extractor for fusion may include: inputting the attention weight and the second scene type weight into the fully connected layer for weighting to obtain the training style embedding information. Please return to refer to Figure 7 and Figure 8 One of the training processes of the scene acoustic style extractor can refer to the relevant descriptions of S110 and S127 to S129 above, which will not be repeated here.

[0183] In this embodiment, a second scene classification model training method can also be introduced. The acoustic features of the home scene in a noisy acoustic environment are input into the above-mentioned second scene classification model to obtain scene type indication information corresponding to the home scene. The fully connected layer is trained and fed back according to the scene type indication information to accelerate convergence.

[0184] A speech synthesis device can be used to synthesize the predicted features of the sound to be synthesized to obtain the speech data to be output. The speech synthesis device can be a vocoder. Accordingly, a vocoder can be used as a speech synthesis device to synthesize the predicted features of the sound to be synthesized output by the acoustic feature prediction network, and the predicted features of the sound to be synthesized are input into the vocoder to obtain the speech data to be output.

[0185] Based on the above, it can be known that the speech synthesis model in this embodiment can be obtained by training based on the aforementioned scene acoustic style extractor and acoustic feature prediction network, and the speech synthesis model can also be obtained by training based on the aforementioned first scene classification model, second scene classification model, scene acoustic style extractor and acoustic feature prediction network. The training process and implementation principle of the first scene classification model, second scene classification model, scene acoustic style extractor and acoustic feature prediction model used in the speech synthesis model training method can directly refer to the corresponding descriptions in the above-mentioned scene classification model training method, scene acoustic style extractor training method and acoustic feature prediction model training method, and similar content will not be repeated here.

[0186] By training the speech synthesis model, after the acoustic characteristics of the home scene in a noisy acoustic environment and the response text content information are input into the trained speech synthesis model, the speech synthesis model can output output response speech data containing the response text content information. The output response speech data is Lombardspeech with good naturalness and clarity.

[0187] In order to more clearly explain the speech synthesis method in the embodiment of the present invention, the following scenario is taken as an example for illustration.

[0188] The electronic device is a smart device, which is located in a home scene and can interact with the user by voice, such as playing and outputting response voice data. The smart device is loaded with a first scene classification model, a second scene classification model, a scene acoustic style extractor, and an acoustic feature prediction model.

[0189] The smart device determines whether the user has performed pre-configuration. If it is determined that the user has performed pre-configuration, the smart device generates output response voice data according to the pre-configuration; if it is determined that the user has not performed pre-configuration, the smart device receives the surrounding sounds and determines whether the received sounds are only the ambient sounds of the home scene or the user's voice in a noisy acoustic environment.

[0190] If it is determined that the received sound is only the ambient sound of the home scene, the first scene classification model is called, and the ambient sound of the home scene is input into the first scene classification model to obtain the scene type information corresponding to the home scene, such as the label value or the softmax probability value, and classify the home scene.

[0191] If it is determined that the received sound is the user's voice in a noisy acoustic environment, the smart device calls the scene acoustic style extractor and the second scene classification model. If the first scene classification model has not obtained the scene type information corresponding to the home scene, the smart device inputs the user's voice in the noisy acoustic environment into the scene acoustic style extractor and the second scene classification model to generate corresponding scene style embedding information. If the first scene classification model has obtained the scene type information corresponding to the home scene, the smart device inputs the user's voice in the noisy acoustic environment and the scene type information corresponding to the home scene into the scene acoustic style extractor to generate corresponding scene style embedding information.

[0192] Before inputting the user voice in a noisy acoustic environment into the scene acoustic style extractor, denoising processing may be performed, and the denoised sound may be input into the scene acoustic style extractor to further improve the accuracy of scene style embedding information extraction.

[0193] When the intelligent device detects that the user has asked a question, it determines whether the scene style embedding information corresponding to the home scene has been obtained. If it is determined that the scene style embedding information corresponding to the home scene has been obtained, the acoustic feature prediction model trained based on the text information to be output and the scene style embedding information is called, and the response text content information and the scene style embedding information corresponding to the home scene are input into the acoustic feature prediction model to generate the predicted acoustic features of Lombardspeech in the corresponding home scene, and the response voice data is synthesized and output through the vocoder. If it is determined that the scene style embedding information corresponding to the home scene has not been obtained, the Lombard speech generation model is called, the response text content information is input into the Lombard speech generation model, the predicted acoustic features of Lombard speech in the corresponding home scene are generated, and the response voice data is synthesized and output through the vocoder. The intelligent device broadcasts the synthesized output response voice data, so that it can interact with the user with a voice with the acoustic style in the corresponding home scene. The voice broadcast is Lombard speech with good naturalness and clarity, so as to ensure the smoothness of voice interaction. It can be understood that the above voice interaction process can be repeated according to user needs.

[0194] In order to execute the corresponding steps in the above embodiments and various possible methods, a method for implementing a speech synthesis device is given below. Fig. 27 , Fig. 27 A functional module diagram of a first speech synthesis device 140 provided in an embodiment of the present invention, the first speech synthesis device 140 can be applied to Figure 1The electronic device 100 is shown. It should be noted that the basic principle and technical effect of the first speech synthesis device 140 provided in this embodiment are the same as those of the above-mentioned speech synthesis method embodiment. For the sake of brief description, for the parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above-mentioned speech synthesis method embodiment. The first speech synthesis device 140 includes an information determination module 141 and a response speech data synthesis module 142.

[0195] Among them, the information determination module 141 is used to determine the scene style embedding information corresponding to the home scene, and to determine the predicted acoustic features corresponding to the home scene based on the response text content information and the scene style embedding information; wherein the scene style embedding information represents the scene acoustic style corresponding to the home scene.

[0196] The response voice data synthesis module 142 is used to synthesize the predicted acoustic features to obtain output response voice data.

[0197] Please refer to Fig.28 The embodiment of the present invention also provides an implementation method of the second speech synthesis device 150. The second speech synthesis device 150 includes: a predicted acoustic feature acquisition module 151 and an information synthesis module 152.

[0198] Among them, the predicted acoustic feature acquisition module 151 is used to input the response text content information into the acoustic feature prediction model to obtain the predicted acoustic features corresponding to the home scene.

[0199] The information synthesis module 152 is used to synthesize the predicted acoustic features to obtain output response speech data.

[0200] In order to execute the corresponding steps in the above embodiments and various possible methods, a method for implementing a scene classification model training device is given below. Fig.29 , Fig.29 A functional module diagram of a scene classification model training device 160 provided in an embodiment of the present invention, the scene classification model training device 160 can be applied to Figure 1 The electronic device 100 is shown. It should be noted that the basic principle and technical effect of the scene classification model training device 160 provided in this embodiment are the same as those of the above-mentioned scene classification model training method embodiment. For the sake of brief description, for the parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above-mentioned scene classification model training method embodiment. The scene classification model training device 160 includes an environmental acoustic feature acquisition module 161 and a scene classification network training module 162.

[0201] Among them, the environmental acoustic feature acquisition module 161 is used to obtain environmental acoustic features of various home scenes.

[0202] The scene classification network training module 162 is used to input the environmental acoustic features of the various home scenes into the scene classification network for training, so as to obtain scene type information corresponding to each of the home scenes.

[0203] In order to perform the corresponding steps in the above embodiments and various possible methods, a method for implementing a scene acoustic style extractor training device is given below. Fig.30 , Fig.30 A functional module diagram of a scene acoustic style extractor training device 170 provided in an embodiment of the present invention, the scene acoustic style extractor training device 170 can be applied to Figure 1 The electronic device 100 is shown. It should be noted that the basic principle and technical effect of the scene acoustic style extractor training device 170 provided in this embodiment are the same as those of the above-mentioned scene acoustic style extractor training method embodiment. For the sake of brief description, for the parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above-mentioned scene acoustic style extractor training method embodiment. The scene acoustic style extractor training device 170 includes a noisy environment acoustic feature acquisition module 171 and a scene acoustic style extractor training module 172.

[0204] Among them, the noisy environment acoustic feature acquisition module 171 is used to obtain the acoustic features of the home scene in a noisy acoustic environment.

[0205] The scene acoustic style extractor training module 172 is used to input the acoustic features of the home scene in the noisy acoustic environment into the scene acoustic style extractor to obtain the training style embedding information corresponding to the home scene; wherein the training style embedding information represents the scene acoustic style corresponding to the home scene.

[0206] The scene acoustic style extractor training module 172 is also used to: input the environmental acoustic features of the home scene into the first scene classification model to obtain scene type information corresponding to the home scene; and input the scene type information into the scene acoustic style extractor for fusion.

[0207] When the first scene classification model is a VGG16 network, the scene acoustic style extractor training module 172 is used to input the environmental acoustic features of the home scene into the first scene classification model through the following steps to obtain scene type information corresponding to the home scene:

[0208] Inputting the environmental acoustic features of the home scene into the VGG16 network to obtain a Softmax probability value;

[0209] Determining a first scene type weight corresponding to the Softmax probability value;

[0210] The first scene type weight is used as the scene type information.

[0211] The scene acoustic style extractor includes: a reference encoder, a first attention module and a fully connected layer; the scene acoustic style extractor training module 172 is used to input the acoustic features of the home scene in the noisy acoustic environment into the scene acoustic style extractor through the following steps to obtain the training style embedding information corresponding to the home scene:

[0212] Inputting the acoustic features of the home scene in a noisy acoustic environment into the reference encoder to obtain reference embedding information;

[0213] Inputting the reference embedding information into the first attention module to obtain an attention weight;

[0214] The attention weight and the first scene type weight are input into the fully connected layer for weighting to obtain the training style embedding information.

[0215] The scene acoustic style extractor training module 172 is used to determine the first scene type weight corresponding to the Softmax probability value through the following steps:

[0216] According to the correspondence between the Softmax probability value and the first scene type weight, the first scene type weight corresponding to the Softmax probability value is determined.

[0217] When the first scene classification model is a ResNet network, the scene acoustic style extractor training module 172 is used to input the environmental acoustic features of the home scene into the first scene classification model through the following steps to obtain scene type information corresponding to the home scene:

[0218] Inputting the environmental acoustic features of the home scene into the ResNet network to obtain a label value of the home scene;

[0219] Determining a second scene type weight corresponding to the tag value;

[0220] The second scene type weight is used as the scene type information.

[0221] The scene acoustic style extractor includes: a reference encoder, a first attention module and a fully connected layer; the scene acoustic style extractor training module 172 is used to input the acoustic features of the home scene in the noisy acoustic environment into the scene acoustic style extractor through the following steps to obtain the training style embedding information corresponding to the home scene:

[0222] Inputting the acoustic features of the home scene in a noisy acoustic environment into the reference encoder to obtain reference embedding information;

[0223] Inputting the reference embedding information into the first attention module to obtain an attention weight;

[0224] The attention weight and the second scene type weight are input into the fully connected layer for weighting to obtain the training style embedding information.

[0225] The scene acoustic style extractor training module 172 is used to determine the second scene type weight corresponding to the label value through the following steps:

[0226] According to the correspondence between the tag value and the second scene type weight, the second scene type weight corresponding to the tag value is determined.

[0227] The scene acoustic style extractor training module 172 is also used to train the first scene classification model through the following steps: obtaining the environmental acoustic features of a variety of home scenes; inputting the environmental acoustic features of the various home scenes into the scene classification network for training to obtain scene type information corresponding to each of the home scenes.

[0228] The scene acoustic style extractor includes: a reference encoder, a first attention module and a fully connected layer; the scene acoustic style extractor training module 172 is used to input the acoustic features of the home scene in the noisy acoustic environment into the scene acoustic style extractor through the following steps to obtain the training style embedding information corresponding to the home scene:

[0229] Inputting the acoustic features of the home scene in a noisy acoustic environment into the reference encoder to obtain reference embedding information;

[0230] Inputting the reference embedding information into the first attention module to obtain an attention weight;

[0231] The attention weight is input into the fully connected layer to obtain training style embedding information.

[0232] The scene acoustic style extractor training module 172 is also used to: input the acoustic features of the home scene in a noisy acoustic environment into a second scene classification model to obtain scene type indication information corresponding to the home scene; and perform training feedback on the fully connected layer according to the scene type indication information.

[0233] The scene acoustic style extractor training module 172 is further used to train the second scene classification model through the following steps:

[0234] Obtain the acoustic characteristics of various home scenes in noisy acoustic environments;

[0235] The acoustic features of the multiple home scenes in a noisy acoustic environment are input into a scene classification network for training to obtain scene type indication information corresponding to each of the home scenes.

[0236] The noisy environment acoustic feature acquisition module 171 is used to obtain the acoustic features of the home scene in the noisy acoustic environment through the following steps: collecting Lombard speech in the noisy home scene and performing acoustic feature extraction to obtain the acoustic features of the home scene in the noisy acoustic environment.

[0237] Based on the above, an embodiment of the present invention further provides a computer-readable storage medium, which includes a computer program. When the computer program is executed, the electronic device where the computer-readable storage medium is located is controlled to execute the above-mentioned scene acoustic style extractor training method.

[0238] In order to execute the corresponding steps in the above embodiments and various possible methods, an implementation method of an acoustic feature prediction model training device is given below. Fig.31 , Fig.31 A functional module diagram of an acoustic feature prediction model training device 180 provided in an embodiment of the present invention, the acoustic feature prediction model training device 180 can be applied to Figure 1 The electronic device 100 is shown. It should be noted that the basic principle and technical effect of the acoustic feature prediction model training device 180 provided in this embodiment are the same as those of the acoustic feature prediction model training method embodiment. For the sake of brief description, for the parts not mentioned in this embodiment, reference can be made to the corresponding contents in the acoustic feature prediction model training method embodiment. The acoustic feature prediction model training device 180 includes a data acquisition module 181 and an acoustic feature prediction model training module 182.

[0239] Among them, the data acquisition module 181 is used to obtain the scene style embedding information corresponding to the text information to be output and the home scene.

[0240] The acoustic feature prediction model training module 182 is used to input the text information to be output and the training style embedding information into the acoustic feature prediction network to obtain the predicted features of the sound to be synthesized corresponding to the home scene.

[0241] In order to execute the corresponding steps in the above embodiments and various possible methods, a method for implementing a speech synthesis model training device is given below. Fig.32 , Fig.32 A functional module diagram of a speech synthesis model training device 190 provided in an embodiment of the present invention, the speech synthesis model training device 190 can be applied to Figure 1The electronic device 100 is shown. It should be noted that the basic principle and technical effect of the speech synthesis model training device 190 provided in this embodiment are the same as those of the above-mentioned speech synthesis model training method embodiment. For the sake of brief description, for the parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above-mentioned speech synthesis model training method embodiment. The speech synthesis model training device 190 includes an information acquisition module 191 and a speech data synthesis module 192 to be output.

[0242] Among them, the information acquisition module 191 is used to obtain the scene style embedding information corresponding to the text information to be output and the home scene, and obtain the predicted features of the sound to be synthesized corresponding to the home scene based on the text information to be output and the scene style embedding information.

[0243] The to-be-output voice data synthesis module 192 is used to synthesize the predicted features of the to-be-synthesized sound to obtain the to-be-output voice data.

[0244] In the embodiment of the present invention, the synthesized output response voice data is Lombard speech with good naturalness and clarity, which can ensure the smoothness of voice interaction and enhance the user experience.

[0245] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0246] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0247] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0248] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A scene acoustic style extractor training method, characterized in that: include: Obtain the acoustic characteristics of home scenes in noisy acoustic environments; Inputting the acoustic features of the home scene in the noisy acoustic environment into a scene acoustic style extractor to obtain training style embedding information corresponding to the home scene; wherein the training style embedding information represents the scene acoustic style corresponding to the home scene; Obtaining environmental acoustic characteristics of the home scene; Inputting the environmental acoustic features of the home scene into a first scene classification model to obtain scene type information corresponding to the home scene; The scene type information is input into the scene acoustic style extractor for fusion.

2. The scene acoustic style extractor training method according to claim 1, characterized in that: When the first scene classification model is a VGG16 network, the scene type information is a first scene type weight corresponding to a Softmax probability value output by the VGG16 network.

3. The scene acoustic style extractor training method according to claim 2, characterized in that: The scene acoustic style extractor includes: a reference encoder, a first attention module and a fully connected layer; the step of inputting the acoustic features of the home scene in the noisy acoustic environment into the scene acoustic style extractor to obtain the training style embedding information corresponding to the home scene includes: Inputting the acoustic features of the home scene in a noisy acoustic environment into the reference encoder to obtain reference embedding information; Inputting the reference embedding information into the first attention module to obtain an attention weight; The step of inputting the scene type information into the scene acoustic style extractor for fusion comprises: The attention weight and the first scene type weight are input into the fully connected layer for weighting to obtain the training style embedding information.

4. The scene acoustic style extractor training method according to claim 1, characterized in that: When the first scene classification model is a ResNet network, the scene type information is a second scene type weight corresponding to a label value output by the ResNet network.

5. The scene acoustic style extractor training method according to claim 4, characterized in that: The scene acoustic style extractor includes: a reference encoder, a first attention module and a fully connected layer; the step of inputting the acoustic features of the home scene in the noisy acoustic environment into the scene acoustic style extractor to obtain the training style embedding information corresponding to the home scene includes: Inputting the acoustic features of the home scene in a noisy acoustic environment into the reference encoder to obtain reference embedding information; Inputting the reference embedding information into the first attention module to obtain an attention weight; The step of inputting the scene type information into the scene acoustic style extractor for fusion comprises: The attention weight and the second scene type weight are input into the fully connected layer for weighting to obtain the training style embedding information.

6. The scene acoustic style extractor training method according to claim 1, characterized in that: The scene acoustic style extractor includes: a reference encoder, a first attention module and a fully connected layer; the step of inputting the acoustic features of the home scene in the noisy acoustic environment into the scene acoustic style extractor to obtain the training style embedding information corresponding to the home scene includes: Inputting the acoustic features of the home scene in a noisy acoustic environment into the reference encoder to obtain reference embedding information; Inputting the reference embedding information into the first attention module to obtain an attention weight; The attention weight is input into the fully connected layer to obtain training style embedding information.

7. The scene acoustic style extractor training method according to claim 3, 5 or 6, characterized in that: Also includes: Inputting the acoustic features of the home scene in a noisy acoustic environment into a second scene classification model to obtain scene type indication information corresponding to the home scene; Training feedback is performed on the fully connected layer according to the scene type indication information.

8. The scene acoustic style extractor training method according to claim 3, 5 or 6, characterized in that: The reference encoder includes a convolutional neural network, a bidirectional long short-term memory recurrent neural network and a mapping layer, and the reference embedding information is obtained by the following steps: inputting the acoustic features of the home scene in a noisy acoustic environment into the convolutional neural network, extracting further acoustic features, inputting the further acoustic features into the bidirectional long short-term memory recurrent neural network to obtain acoustic features related to the context, and inputting the acoustic features related to the context into the mapping layer to obtain reference embedding information of fixed length; or, The reference encoder includes a first pre-trained model output layer, a convolutional neural network and a mapping layer, and the reference embedding information is obtained by the following steps: inputting the acoustic features of the home scene in a noisy acoustic environment into the first pre-trained model output layer to obtain a pre-trained vector, inputting the pre-trained vector into the convolutional neural network to extract further acoustic features, and inputting the extracted further acoustic features into the mapping layer to obtain reference embedding information mapped to a predefined dimension; or, The reference encoder includes a second pre-trained model output layer, a bidirectional long short-term memory recurrent neural network and a mapping layer, and the reference embedding information is obtained by the following steps: inputting the acoustic features of the home scene in a noisy acoustic environment into the second pre-trained model output layer to obtain a pre-trained vector, inputting the pre-trained vector into the bidirectional long short-term memory recurrent neural network to obtain further context-related acoustic features, and inputting the context-related acoustic features into the mapping layer to obtain reference embedding information mapped to a predefined dimension.

9. The scene acoustic style extractor training method according to claim 1, characterized in that: The step of obtaining the acoustic characteristics of the home scene in a noisy acoustic environment includes: Lombard speech in noisy home scenes is collected and acoustic features are extracted to obtain the acoustic features of home scenes in noisy acoustic environments.

10. A scene acoustic style extractor training device, characterized in that: include: A noisy environment acoustic feature acquisition module is used to obtain the acoustic features of home scenes in a noisy acoustic environment; An environmental acoustic feature acquisition module, used to obtain the environmental acoustic features of the home scene; A scene acoustic style extractor training module is used to input the acoustic features of the home scene in the noisy acoustic environment into the scene acoustic style extractor to obtain training style embedding information corresponding to the home scene; wherein the training style embedding information represents the scene acoustic style corresponding to the home scene; input the environmental acoustic features of the home scene into a first scene classification model to obtain scene type information corresponding to the home scene; and input the scene type information into the scene acoustic style extractor for fusion.

11. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the scene acoustic style extractor training method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises a computer program, and when the computer program is executed, the electronic device where the computer-readable storage medium is located is controlled to execute the scene acoustic style extractor training method according to any one of claims 1 to 9.

Images