Audio synthesis method, apparatus, device, and medium

Abstract

Embodiments of the present application provide an audio synthesis method, device, equipment and medium, wherein the method comprises: obtaining audio key information used for synthesizing audio data; performing encoding processing on the audio key information to obtain audio attribute features, generating K candidate spectral features according to the audio attribute features and diffusion frequency information; K is a positive integer; obtaining time dimension information and frequency dimension information corresponding to the K candidate spectral features, performing sampling processing on the K candidate spectral features according to the time dimension information and the frequency dimension information to obtain K target spectral features; performing feature fusion processing on the K target spectral features to obtain a fusion spectral feature, and synthesizing the fusion spectral feature into target audio data. By using the embodiments of the present application, the quality of audio synthesis can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to an audio synthesis method and device, equipment and medium. BACKGROUND

[0002] The audio synthesis technology is a technology of automatically generating audio data by learning text information and score information of to-be-synthesized audio data through an audio synthesis model. At present, the technology is widely applied in the fields of virtual anchors, smart homes, intelligent robots and the like. In the extraction of spectrum features used for synthesizing audio, a traditional audio synthesis model often extracts required spectrum features through simple convolution processing. However, the spectrum features extracted through this way are too monotonous, which leads to a great difference between audio data synthesized by the spectrum features and expected audio data, and low audio synthesis quality. SUMMARY

[0003] The embodiments of the present application provide an audio synthesis method, device, equipment and medium, which can improve the efficiency of audio synthesis.

[0004] In one aspect, the embodiments of the present application provide an audio synthesis method, comprising:

[0005] obtaining audio key information used for synthesizing audio data;

[0006] encoding the audio key information to obtain audio attribute features, and generating K candidate spectrum features according to the audio attribute features and diffusion frequency information; K is a positive integer;

[0007] obtaining time dimension information and frequency dimension information corresponding to the K candidate spectrum features, and sampling the K candidate spectrum features according to the time dimension information and the frequency dimension information to obtain K target spectrum features;

[0008] performing feature fusion processing on the K target spectrum features to obtain fusion spectrum features, and synthesizing the fusion spectrum features into target audio data.

[0009] In one aspect, the embodiments of the present application provide an audio synthesis device, comprising:

[0010] a first obtaining module configured to obtain audio key information used for synthesizing audio data;

[0011] a generating module configured to encode the audio key information to obtain audio attribute features, and generate K candidate spectrum features according to the audio attribute features and diffusion frequency information; K is a positive integer;

[0012] The sampling module is configured to acquire time dimension information and frequency dimension information corresponding to the K candidate spectral features, and perform sampling processing on the K candidate spectral features according to the time dimension information and the frequency dimension information, to obtain K target spectral features.

[0013] The synthesis module is configured to perform feature fusion processing on the K target spectral features, to obtain a fused spectral feature, and synthesize the fused spectral feature into the target audio data.

[0014] The audio key information includes a phoneme sequence, a duration sequence, and a pitch sequence, and the generation module includes:

[0015] The concatenation unit is configured to concatenate the phoneme sequence, the duration sequence, and the pitch sequence to obtain an audio input feature, and input the audio input feature into the audio synthesis model.

[0016] The encoding unit is configured to perform encoding processing on the audio input feature according to an encoder in the audio synthesis model, to obtain an audio attribute feature.

[0017] The conversion unit is configured to convert the audio attribute feature into an initial spectral feature M0 by a diffusion decoder of the audio synthesis model.

[0018] The enhancement unit is configured to input the initial spectral feature M0 into the diffusion decoder, perform feature enhancement processing on the initial spectral feature M0 based on the diffusion decoder, and obtain an initial spectral feature M1.

[0019] The first acquisition unit is configured to acquire T initial spectral features when the number of the initial spectral features matches a value T indicated by the diffusion frequency information, and acquire K candidate spectral features from the T initial spectral features; T is an integer greater than K.

[0020] The time dimension information includes a target time dimension of an a-th candidate spectral feature in the K candidate spectral features in the sampling processing, and the frequency dimension information includes b target frequency dimensions of the a-th candidate spectral feature in the sampling processing; a is a positive integer less than or equal to K, and b is an integer greater than 1; and the sampling module includes:

[0021] The combination unit is configured to combine the target time dimension and the b target frequency dimensions to obtain b sampling dimension information groups.

[0022] The up-sampling unit is configured to perform up-sampling processing on the a-th candidate spectral feature according to the b sampling dimension information groups, to obtain b first sub-spectral features corresponding to the a-th candidate spectral feature.

[0023] The downsampling unit is configured to perform downsampling processing on the b first sub-spectrum features according to the initial time dimension and the initial frequency dimension corresponding to the a-th candidate spectrum feature, to obtain b second sub-spectrum features, and to perform summation operation on the b second sub-spectrum features to obtain the a-th target spectrum feature.

[0024] The synthesis module comprises:

[0025] The convolution unit is configured to obtain convolution parameter information corresponding to the K target spectrum features, and to perform convolution processing on the K target spectrum features according to the convolution parameter information to obtain convolution spectrum features corresponding to the K target spectrum features.

[0026] The activation unit is configured to perform activation processing on the convolution spectrum features to obtain fusion spectrum features corresponding to the convolution spectrum features.

[0027] The audio synthesis device further comprises:

[0028] The second obtaining module is configured to obtain sample key information used for synthesizing audio data, and to obtain sample attribute features corresponding to the sample key information through an encoder of the initial synthesis model.

[0029] The third obtaining module is configured to output V initial sample spectrum features corresponding to the sample attribute features through a diffusion decoder of the initial synthesis model; V is an integer greater than K.

[0030] The fourth obtaining module is configured to sequentially obtain an i-th initial sample spectrum feature and a j-th initial sample spectrum feature in V initial sample spectrum features according to a generation time sequence corresponding to the V initial sample spectrum features, and to determine a label spectrum feature in the V initial sample spectrum features; i is a positive integer less than V; j is an integer less than or equal to V and greater than i.

[0031] The determining module is configured to determine a first loss value according to the i-th initial sample spectrum feature and the j-th initial sample spectrum feature, and to determine a second loss value according to the j-th initial sample spectrum feature and the label spectrum feature.

[0032] The correction module is configured to correct network parameters in the initial synthesis model based on the first loss value and the second loss value, and to determine the initial synthesis model containing the corrected network parameters as the audio synthesis model.

[0033] The determining module comprises:

[0034] The second obtaining unit is configured to perform sampling processing on the i-th initial sample spectrum feature according to time dimension information and frequency dimension information corresponding to the i-th initial sample spectrum feature to obtain an i-th target sample spectrum feature, and to obtain a first loss value between the i-th target sample spectrum feature and the j-th initial sample spectrum feature.

[0035] The third acquisition unit is configured to sample the jth initial sample spectral feature according to the time dimension information and the frequency dimension information corresponding to the jth initial sample spectral feature, to obtain a jth target sample spectral feature, to perform feature fusion processing on the ith target sample spectral feature and the jth target sample spectral feature, to obtain a sample fusion spectral feature, and to acquire a second loss value between the sample fusion spectral feature and the label spectral feature.

[0036] The embodiment of the present application provides a computer device, including a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to execute the steps of the method in the aspect of the embodiment of the present application.

[0037] The embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, the computer program includes program instructions, and the program instructions are executed by the processor to execute the steps of the method in the aspect of the embodiment of the present application.

[0038] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional manners of the aspect.

[0039] In the embodiment of the present application, after the audio key information used for synthesizing the audio data is acquired, the audio key information is encoded to obtain the audio attribute feature, so as to convert the audio key information into a vector form, which helps to guarantee the feasibility of audio synthesis. Then, the K candidate spectral features are generated according to the audio attribute feature and the diffusion frequency information, so that the K candidate spectral features have high feature richness. Further, the time dimension information and the frequency dimension information corresponding to the K candidate spectral features are acquired, the K candidate spectral features are sampled according to the time dimension information and the frequency dimension information to obtain K target spectral features, the K target spectral features are subjected to feature fusion processing to obtain a fusion spectral feature, and the fusion spectral feature is synthesized into target audio data. In the embodiment of the present application, the feature extraction of the audio information contained in the time dimension and the frequency dimension is enhanced, the spectral information is deeply expanded, the effectiveness of the feature expression of the fusion spectral feature in the time dimension and the frequency dimension is enhanced, the richness of the obtained fusion spectral feature is high, and therefore the synthesis of the corresponding target audio data by using the fusion spectral feature is more in line with the expected requirements, and the quality of the audio synthesis can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0041] Figure 1 is a structural schematic diagram of a network architecture provided by an embodiment of the present application;

[0042] Figure 2 is an application scenario diagram of audio synthesis provided by an embodiment of the present application;

[0043] Figure 3 is a structural schematic diagram of an audio synthesis model provided by an embodiment of the present application;

[0044] Figure 4 is a flow schematic diagram of an audio synthesis method provided by an embodiment of the present application;

[0045] Figure 5 is a composition schematic diagram of audio key information provided by an embodiment of the present application;

[0046] Figure 6 is a schematic diagram of obtaining a target spectral feature provided by an embodiment of the present application;

[0047] Figure 7 is a schematic diagram of obtaining a fusion spectral feature provided by an embodiment of the present application;

[0048] Figure 8 is a flow schematic diagram of an audio synthesis model training method provided by an embodiment of the present application;

[0049] Figure 9 is a schematic diagram of audio synthesis model training provided by an embodiment of the present application;

[0050] Figure 10 is a structural schematic diagram of an audio synthesis device provided by an embodiment of the present application;

[0051] Figure 11 is a structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0052] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0053] The data processing method provided by the embodiments of the present application can specifically relate to a cloud database in cloud technology. In short, the cloud database can be regarded as an electronic file cabinet (a place for storing electronic files), and a user can perform operations such as adding, querying, updating, and deleting data in the file. The so-called "database" is a collection of data stored together in a certain way, shared by multiple users, with as little redundancy as possible, and independent of application programs. In the embodiments of the present application, after obtaining audio key information used for synthesizing audio data, the computer device can synthesize target audio data based on the audio key information through an audio synthesis model, and store the finally synthesized target audio data in the cloud database.

[0054] Please refer to Figure 1 , Figure 1 is a structural schematic diagram of a network architecture provided by the embodiments of the present application. As Figure 1 indicated, the network architecture can include a server 101, a terminal device 102a, a terminal device 102b, and a terminal device 102c, etc. The network architecture can include one or more servers, and can also include at least one or more terminal devices, and the number of servers and terminal devices will not be limited here. As Figure 1 indicated, the server 101 can be connected with each terminal device through a network, so that the server 101 can perform data interaction with each terminal device through the network connection.

[0055] The server 101 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud database, cloud service, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and basic cloud computing services such as big data and artificial intelligence platform. The terminal cluster (including terminal device 102a, terminal device 102b, terminal device 103c, etc.) can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a palm computer, a mobile internet device (MID), a wearable device (such as a smart watch, a smart bracelet, etc.), a smart computer, a smart car, and the like. The server 101 and each terminal device in the terminal cluster can establish a communication connection, and each terminal device in the terminal cluster can also establish a communication connection. In other words, the server 101 can establish a communication connection with each terminal device in the terminal device 102a, the terminal device 102b, the terminal device 103c, and the like. For example, the terminal device 102a and the server 101 can establish a communication connection. The terminal device 102a and the terminal device 102b can establish a communication connection, and the terminal device 102a and the terminal device 102c can also establish a communication connection. The communication connection is not limited to the connection mode, and can be directly or indirectly connected by wired communication or wireless communication, and the like. The specific application scenario can be determined, and the embodiments of the present application are not limited.

[0056] In the embodiments of the present application, the server 101 and the terminal device (such as the terminal device 102a) can be independently used for synthesizing target audio data, and the server 101 and the terminal device (such as the terminal device 102a) can also be used for synthesizing target audio data. The target audio data can be voice broadcast, song, recitation, drama, speech, and the like. Please see Figure 2 Figure 2 is an application scenario diagram of audio synthesis provided by the embodiments of the present application, taking song synthesis as an example for illustration. It should be noted that Figure 2 The interface shown in the figure is only a schematic, and the layout and display content of the interface can be adjusted according to the needs in actual application, which is not limited here. When the user A needs to perform audio synthesis, the terminal device 102a can open the audio synthesis application 20 (the corresponding server can be the server 101), and the audio synthesis interface 201 is displayed in the audio synthesis application 20. The user A can set the audio key information for synthesizing audio data in the audio synthesis interface 201, which can include phoneme sequence, duration sequence, and pitch sequence. As shown in Figure 2 ​As shown, the audio synthesis interface 201 includes a lyrics input area 202, a brush control 206, an audio key information determination area 30, and a synthesis control 208. In a possible implementation, the user A can input lyrics (for example, the corresponding pinyin of the lyrics) in the lyrics input area 202, and determine the phoneme sequence in the audio key information according to the lyrics. In addition, the user A can also import corresponding text as the phoneme sequence in the audio key information by clicking the file in the menu bar 210. After determining the phoneme sequence, the user A can click the brush control 206 to draw the corresponding grid in the audio key information determination area 30, so as to determine the duration sequence and the pitch sequence in the audio key information. For example, the user A inputs the lyrics "La" in the lyrics input area 202, and the user A wants to set the pitch corresponding to the lyrics "La" as C5 and the duration as 63 ms. The user A can draw the grid 207 at the pitch C5 by clicking the brush control 206, and set the duration of the corresponding lyrics to 63 ms by stretching the grid 207, so as to determine that the corresponding duration is 63 ms. The longer the grid is, the longer the duration corresponding to the lyrics is.

[0057] Optionally, the audio synthesis interface 201 can further include a singer switching control 205 and other parameter setting controls 209. The user A can trigger the singer switching control 205 to switch the current singer Xia A to another singer. The gender, age, and preferred vocal range of each singer can be different, for example, the information of the singer Xia A can be female, 28 years old, and the preferred vocal range C4-E5. The user A can trigger the singer switching control 205 to select a suitable singer according to actual needs. The user A can also add accompaniment, breathing sound, and breathing to the audio through the other parameter setting controls 209, so that the finally synthesized target audio data is closer to the real pronunciation, which helps to improve the naturalness of the target audio data. When all the parameters are set, the user A can trigger the synthesis control 208 to complete the synthesis of the final target audio data.

[0058] It can be understood that the audio synthesis application 20 can integrate an audio synthesis model for synthesizing target audio data. After obtaining the audio key information for synthesizing the audio data, the audio synthesis model is triggered in response to the triggering operation of the synthesis control 208 to synthesize the target audio data.

[0059] Specifically, please refer to Figure 3 , Figure 3 is a structural schematic diagram of an audio synthesis model provided by an embodiment of the present application. As shown in Figure 3As shown, the audio synthesis model can include an encoder 401, a diffusion decoder 402, a spectrum selection component 403, a sampling component 404, a fusion component 405, and a vocoder 406. It should be noted that the specific synthesis process of the target audio data, and the role of each component in the audio synthesis model, will be described in detail below in conjunction with Figure 4 The audio synthesis method shown will not be described here.

[0060] As Figure 2 shown, after the synthesis of the target audio data is completed, a synthesis success prompt message 211 can be displayed in the audio synthesis interface 201, for example, the synthesis success prompt message 211 can be "song synthesis is complete, enjoy your work!", to prompt the user A that the target audio data synthesis is complete. The user A can play the target audio data by triggering the play control 213, so as to determine whether to adjust the synthesis parameters according to the current effect of the target audio data. In addition, the user A can also store the target audio data in the local database by triggering the save control 212, or it can also be stored in the cloud database. It can be understood that through the audio synthesis model provided by the embodiments of the present application, the target audio data can be quickly synthesized, which helps to improve the quality of audio synthesis.

[0061] Further, please refer to Figure 4 , Figure 4 is a flow diagram of an audio synthesis method provided by an embodiment of the present application. It can be understood that the audio synthesis method is executed by a computer device, which can be a terminal device (for example, the terminal device 102a, the terminal device 102b or the terminal device 102c in the embodiment corresponding to Figure 1 , or a server (for example, the server 101 in the embodiment corresponding to Figure 1 ), or a computer program (including program code). As Figure 4 shown, the audio synthesis method can include the following steps S101-S104:

[0062] Step S101: obtaining audio key information for synthesizing audio data.

[0063] The audio key information can be understood as information used for synthesizing audio data. For example, the audio key information can include, but is not limited to, phoneme sequence, duration sequence, and pitch sequence. Of course, in some possible implementation, the audio key information can also include accompaniment, breathing sound, and the like. In the embodiment of the present application, the synthesized target audio data can be voice broadcast, song, recitation, drama, speech, and the like, which are not limited in the embodiment of the present application. Taking a song as an example, the audio key information corresponding to the song can be obtained from the music score corresponding to the song, or can be determined according to the parameters input by the user of the computer device, which are not limited in the embodiment of the present application.

[0064] The phoneme sequence can be understood as a sequence composed of a plurality of phonemes, and can be used to represent pronunciation of the final target audio data. The phoneme can be understood as the smallest unit of speech divided according to the natural properties of speech, and is analyzed according to the pronunciation action in the syllable. For example, the syllable "wang" includes two pronunciation actions "w" and "ang", that is, two phonemes. The duration sequence is used to indicate a sequence composed of the duration of each phoneme in the phoneme sequence during pronunciation. The duration sequence can be used to represent the rhythm of the final target audio data. Since the final synthesized target audio data is output in the form of audio frames, the shortest duration of the audio frame can be determined according to the sampling rate of the audio. For example, the shortest duration of each frame of the audio frame can be set to 32 ms, or 40 ms, or other values, which are not limited in the embodiment of the present application. When the shortest duration of each frame of the audio frame is set to 32 ms, the minimum resolution of the rhythm can be set to 32 ms, that is, the shortest duration is 32 ms. In addition, in order to facilitate the representation of the duration of each phoneme, the duration of each phoneme can be divided into 200 levels or 300 levels in an increasing manner. The number of levels can be determined according to actual needs, which are not limited in the embodiment of the present application. For example, the phoneme sequence includes "w" and "ang", wherein the duration of the phoneme "w" can be set to 63 ms, and the duration of the phoneme "ang" can be set to 267 ms, and the duration sequence is 63 ms and 267 ms. The pitch sequence can be understood as a sequence composed of the height of the sound of each phoneme in the phoneme sequence during pronunciation. The pitch sequence can be used to represent the height of the sound of the final target audio data. Since the nature of the pitch is frequency, the pitch can be divided into 88 levels according to the international pitch hertz spectrum, for example, A3, C4, and the like.

[0065] Please refer to Figure 5 , Figure 5 is a composition diagram of audio key information provided by the embodiment of the present application. The audio key information can include, but is not limited to, phoneme sequence, duration sequence, and pitch sequence. Figure 5It can be seen that taking the Chinese text "凭 (píng)", "什 (shén)", "么 (me)", "要 (yào)", "失 (shī)", "望 (wàng)" as examples for illustration, the text includes 5 syllables "ping", "shen", "me", "yao", "shi", "wang", and each syllable can be exemplarily divided into at least one phoneme. For example, the syllable "wang" can be divided into 2 phonemes "w" and "ang", where the phoneme "w" corresponds to an initial consonant and the phoneme "ang" corresponds to a final sound. Each phoneme forms a phoneme sequence [p, ing, sh, en, m, e, y, ao, sh, i, w, ang], and its corresponding duration sequence can be [63ms, 157ms, 63ms, 157ms, 63ms, 157ms, 63ms, 237ms, 63ms, 197ms, 63ms, 267ms]. Generally speaking, one syllable corresponds to the same pitch, that is, all phonemes in the same syllable correspond to the same pitch. For example, Figure 5 the syllable "yao" in Figure 5 corresponds to the pitch C4, then the phonemes "y" and "ao" both correspond to the pitch C4. Then the pitch sequence corresponding to the text "凭什么要失望 (píng shén me yào shī wàng)" can be [G3, A#3, A#3, C4, C4, A#3 - G#3].

[0066] Step S102: Perform encoding processing on the audio key information to obtain audio attribute features, and generate K candidate spectral features according to the audio attribute features and the diffusion frequency information; K is a positive integer.

[0067] Among them, the audio attribute features can be understood as the features related to the audio information in the audio key information. As described above, the audio key information can include but is not limited to the phoneme sequence, duration sequence, pitch sequence, etc. The definitions of the audio sequence, duration sequence, and pitch sequence can be referred to the previous description and will not be elaborated here. The diffusion frequency information can be understood as Figure 3 the relevant parameters of the diffusion decoder 402 in the audio synthesis model shown, specifically, the diffusion frequency information can include the number of diffusions, and the number of diffusions can be understood as the iteration number of the diffusion decoder 402. The candidate spectral features can be understood as the features obtained after the audio attribute features are subjected to feature enhancement processing. The candidate spectral features can be used to represent the acoustic characteristics in the audio key information, specifically, it can be the Mel spectrum, or other acoustic features such as the energy concentration area representing timbre, formant frequency, formant intensity, bandwidth, duration representing speech rhythm characteristics, fundamental frequency, or average speech power, etc. This application embodiment does not make limitations on this. K is a positive integer, and K can take values of 1, 2, 3,....

[0068] In a possible implementation, the computer device can concatenate the phoneme sequence, the duration sequence, and the pitch sequence to obtain audio input features, and input the audio input features into the audio synthesis model; and then encode the audio input features according to an encoder in the audio synthesis model to obtain audio attribute features.

[0069] The audio input features can be understood as features containing relevant information of the phoneme sequence, the duration sequence, and the pitch sequence. Specifically, the phoneme sequence, the duration sequence, and the pitch sequence can be concatenated to obtain an audio sequence, and then the audio sequence is converted into a vector to obtain the audio attribute features. The audio sequence contains a plurality of elements, each element including a phoneme and a pitch and a duration corresponding to the phoneme, and can be represented as (phoneme, pitch, duration). For example, (i, C4, 197), where i represents a phoneme, C4 represents a pitch, and 197 represents a duration, which can be in ms.

[0070] Further, referring to FIG. 4, Figure 3 After the computer device concatenates the phoneme sequence, the duration sequence, and the pitch sequence in the audio key information to obtain the audio input features, the computer device can input the audio input features into the encoder 401 in the audio synthesis model, and encode the audio input features by using the encoder 401 to obtain the audio attribute features. The encoder 401 is configured to encode the audio input features. The encoder 401 can include a lyric encoder, a duration predictor, and a pitch encoder, which are configured to extract features in the phoneme sequence, the duration sequence, and the pitch sequence in the audio input features simultaneously to obtain the audio attribute features. As can be seen, the encoding of the audio input features by using the encoder can improve the extraction efficiency and accuracy of the audio attribute features, and thus can improve the quality of the audio synthesis.

[0071] In a possible implementation, after obtaining the audio attribute features, the computer device can convert the audio attribute features into initial spectrum features M0 by using a diffusion decoder of the audio synthesis model; input the initial spectrum features M0 into the diffusion decoder, perform feature enhancement processing on the initial spectrum features M0 based on the diffusion decoder to obtain initial spectrum features M1; when the number of the initial spectrum features matches a value T indicated by the diffusion frequency information, obtain T initial spectrum features, and obtain K candidate spectrum features from the T initial spectrum features.

[0072] The initial spectrum feature M0 can be understood as a spectrum feature obtained after the audio attribute feature is converted by the diffusion decoder; the initial spectrum feature M1 is an output result obtained after the initial spectrum feature M0 is taken as the input of the diffusion decoder. The diffusion decoder can be understood as an autoregressive model, which needs to take itself as a regression variable. In other words, the diffusion decoder needs to use the previous variable to describe the linear regression model of the subsequent time random variable. In the embodiment of the application, specifically, the initial spectrum feature M1 depends on the initial spectrum feature M0. In addition, after each iteration, the feature richness of the subsequent output initial spectrum feature is higher than that of the initial spectrum feature as the input, that is, the feature richness of the initial spectrum feature M1 is higher than that of the initial spectrum feature M0, and the purpose of repeating the iteration T times is to make the final Tth initial spectrum feature have a higher feature richness.

[0073] Specifically, please see Figure 3 As Figure 3 shown, after the audio attribute feature is converted into the initial spectrum feature M0 by the diffusion decoder 402, the initial spectrum feature M0 can be taken as the input of the diffusion decoder 402 again, the initial spectrum feature M0 is subjected to feature enhancement by the diffusion decoder 402, and the initial spectrum feature M1 is output, and then the initial spectrum feature M1 is taken as the input of the diffusion decoder 402, the initial spectrum feature M1 is subjected to feature enhancement by the diffusion decoder 402, and the initial spectrum feature M2 is output, and the iteration is repeated in turn until the iteration number reaches the value indicated by the diffusion frequency information, and the result output by the diffusion decoder 402 is saved. For example, the value indicated by the diffusion frequency information, then the iteration is performed T times, and T initial spectrum features are obtained. The feature enhancement process of the diffusion decoder 402 is essentially a Markov chain with fixed parameters, and each iteration adds features to the initial spectrum feature obtained in the previous iteration until the initial spectrum feature that meets the expected requirement is obtained. In the embodiment of the application, it can be considered that when the number of initial spectrum features matches the value T indicated by the diffusion frequency information, it can be considered that the Tth initial spectrum feature meets the expected requirement. In addition, the T initial spectrum features output by the diffusion decoder 402 can be stored in the local database or the cloud database for subsequent use. The structure of the diffusion decoder 402 can include a non-causal WaveNet structure, can include a 1x1 convolutional layer and a plurality of convolutional modules, and in a specific application, the structure of the diffusion decoder can be adjusted according to actual needs.

[0074] After obtaining the T initial spectral features, the T initial spectral features can be screened by the spectral selection component 403 to obtain K candidate spectral features. Since the K candidate spectral features are selected from the T initial spectral features, T is an integer greater than K, and the specific value of T can be K, K+1, …, and so on. K can take values of 1, 2, 3, …, K, K+1, …, and so on. In a possible implementation, K initial spectral features can be randomly selected from the T initial spectral features as the K candidate spectral features; for example, when T is 10 and K is 3, the first initial spectral feature, the second initial spectral feature, and the fifth initial spectral feature can be selected as the K candidate spectral features.

[0075] In another possible implementation, the K candidate spectral features can also be screened according to the similarity in each initial spectral feature in the T initial spectral features. Specifically, after obtaining the T initial spectral features, a clustering algorithm can be used to cluster the T initial spectral features. Optionally, the clustering process is specifically described below by taking K-means as an example.

[0076] The computer device can first perform vector conversion on the T initial spectral features to obtain T initial spectral feature vectors, and select a center vector tx from the T initial spectral feature vectors, where x is a positive integer less than or equal to K; obtain the similarity between each initial spectral feature vector of the T initial spectral feature vectors and the center vector tx, and if the similarity between an initial spectral feature vector Gy in the T initial spectral feature vectors and the center vector tx is the largest, the initial spectral feature vector Gy is added to the to-be-processed cluster Cx to which the center vector tx belongs; then the center vector tx can be updated according to the initial spectral feature vectors contained in the to-be-processed cluster Cx, until the updated center vector tx in the to-be-processed cluster Cx and the center vector tx before the update meet a preset condition, and K initial spectral feature sets are obtained according to the to-be-processed cluster Cx. The preset condition can be understood as a condition preset in advance, and can be, for example, that the error between the updated center vector tx in the to-be-processed cluster Cx and the center vector tx before the update is less than a preset threshold, or that a preset iteration number is reached, and so on. The specific values of the preset threshold and the iteration number can be determined according to actual requirements, and the embodiments of the present application do not limit them. The initial spectral feature vector Gy is one of the T initial spectral feature vectors.

[0077] Optionally, in order to achieve better clustering effect, the computer device can select K initial spectral feature vectors from the T initial spectral feature vectors as initial center vectors. The specific selection method is: randomly select an initial spectral feature vector from the T initial spectral feature vectors as the first center vector t1; then select the initial spectral feature vector farthest from the first center vector t1 from the remaining initial spectral feature vectors (initial spectral feature vectors not selected) of the T initial spectral feature vectors as the second center vector t2; the center point between the first center vector t1 and the second center vector t2 can be calculated (the center point can be the initial spectral feature vector corresponding to the average value between the first center vector t1 and the second center vector t2, such as the initial spectral feature vector closest to the average value), and the initial spectral feature vector farthest from the center point is selected from the remaining initial spectral feature vectors of the T initial spectral feature vectors as the third center vector t3; in this way, until the Kth center vector tK is determined, that is, the K center vectors are determined.

[0078] Further, the computer device can calculate the distance between each initial spectral feature vector in the T initial spectral feature vectors and each center vector (the distance between two initial spectral feature vectors can be measured by the similarity between the two initial spectral feature vectors, the greater the similarity, the shorter the distance, the smaller the similarity, the greater the distance), that is, the distance between each initial spectral feature vector Gy in the T initial spectral feature vectors and each center vector tx, the initial spectral feature vector Gy is divided into the to-be-processed cluster to which the nearest center vector belongs, and K to-be-processed clusters can be obtained initially, each center vector corresponds to a to-be-processed cluster, and each to-be-processed cluster can include multiple initial spectral feature vectors. The number of initial spectral feature vectors included in different to-be-processed clusters can be the same or different.

[0079] The method for calculating the distance between the two initial spectral feature vectors can include, but is not limited to, Eucledian distance, Manhattan distance, Minkowski distance, and cosine similarity. Taking cosine similarity as an example, the distance between each initial spectral feature vector Gy in the T initial spectral feature vectors and each center vector tx can be represented as dis< Gy, tx> = cosin(Gy, tx), where dis< Gy, tx> represents the distance between the initial spectral feature vector Gy and the center vector tx, and cosin is a cosine function. If the distances between each initial spectral feature vector Gy in the T initial spectral feature vectors and the three center vectors (the value of the cluster number K is 3) are 10, 15, and 3, respectively, the initial spectral feature vector Gy can be divided into the third center vector belonging to the to-be-processed cluster C3.

[0080] For the K to-be-processed clusters obtained above, the mean of all initial spectral feature vectors included in each to-be-processed cluster can be calculated, the center vector corresponding to the to-be-processed cluster is updated according to the mean to obtain a new center vector, and then the initial spectral feature vectors included in each to-be-processed cluster are updated according to the distance between each initial spectral feature vector in the T initial spectral feature vectors and each new center vector. The above process is repeatedly performed, and when the initial spectral feature vectors included in each to-be-processed cluster no longer change, that is, the center vectors corresponding to each to-be-processed cluster meet the specified condition, the K to-be-processed clusters at this time can be determined as the final clustering result of the T initial spectral feature vectors. The specified condition can be understood as a pre-set condition, and specifically includes, but is not limited to, reaching a pre-set iteration number, or the center vectors corresponding to each to-be-processed cluster being fixed and unchanged, and the like. The specific value of the iteration number can be determined according to actual needs, and the embodiments of the present application do not limit this. Then, an initial spectral feature vector is randomly selected from each to-be-processed cluster in the K to-be-processed clusters to obtain K initial spectral feature vectors, and then K candidate spectral features are obtained. As can be seen, the spectral feature similarity between each candidate spectral feature in the K candidate spectral features obtained by the above method is small, and therefore, when the K candidate spectral features are subsequently sampled and fused, the weakening or loss of some features in the initial spectral feature that have a greater impact on quality evaluation can be reduced, and the fusion spectral feature used for synthesizing the target audio data finally can have more abundant features.

[0081] Step S103: Obtain time dimension information and frequency dimension information corresponding to the K candidate spectral features, and perform sampling processing on the K candidate spectral features according to the time dimension information and the frequency dimension information, to obtain K target spectral features.

[0082] The time dimension information can be understood as information of the candidate spectral feature in the time dimension. Similarly, the frequency dimension information can be understood as information of the candidate spectral feature in the frequency dimension. The target spectral feature can be understood as a spectral feature obtained by performing sampling processing on the candidate spectral feature. The target spectral feature has more detailed information than the candidate spectral feature.

[0083] Please refer to Figure 3 After obtaining the K candidate spectral features, the candidate spectral features can be sampled by the sampling component 404 in the audio synthesis model based on the time dimension information and the frequency dimension information corresponding to the K candidate spectral features, to obtain K target spectral features. The specific structure of the sampling component 404 can be referred to Figure 6 The role of the sampling component 404 is to make the target spectral feature obtained by sampling processing integrate the features in the time dimension and the frequency dimension, so as to avoid weakening or loss of features in the candidate spectral feature that have a greater impact on audio synthesis in the time dimension and the frequency dimension, and thus improve the utilization rate and expression ability of the target spectral feature.

[0084] To facilitate understanding, the following takes the a-th target spectral feature corresponding to the a-th candidate spectral feature in the K candidate spectral features as an example to introduce the target spectral feature in detail. The acquisition process of other target spectral features can refer to the following processing process. Specifically, taking the shape of the a-th candidate spectral feature as [N, F, T] = [N, 80, T] as an example, the specific description is as follows. N represents the batch size, F represents the frequency dimension, and T represents the time dimension. In the embodiments of the present application, N takes a fixed value, that is, N remains unchanged.

[0085] In the embodiments of the present application, the time dimension information can specifically include a target time dimension of the a-th candidate spectral feature in the K candidate spectral features in the sampling processing, and the frequency dimension information can specifically include b target frequency dimensions of the a-th candidate spectral feature in the sampling processing; a is a positive integer less than or equal to K, and b is an integer greater than 1. Specifically, the computer device can combine the target time dimension and the b target frequency dimensions to obtain b sampling dimension information groups; then perform upsampling processing on the a-th candidate spectral feature according to the b sampling dimension information groups to obtain b first sub-spectral features corresponding to the a-th candidate spectral feature; and then perform downsampling processing on the b first sub-spectral features according to the initial time dimension and the initial frequency dimension of the a-th candidate spectral feature to obtain b second sub-spectral features, and add the b second sub-spectral features to obtain the a-th target spectral feature.

[0086] In the embodiments of the present application, one target time dimension and two or more target frequency dimensions are taken as examples, and in actual applications, multiple target time dimensions can also be set according to actual needs. The sampling dimension information group is obtained by combining the target time dimension and the b target frequency dimensions. The initial time dimension can be understood as the time dimension of the a-th candidate spectral feature before sampling processing; similarly, the initial frequency dimension can be understood as the frequency dimension of the a-th candidate spectral feature before sampling processing. Taking the size (shape) of the a-th candidate spectral feature as [N, F, T] = [N, 80, T] as an example, the initial time dimension is T, and the initial frequency dimension is 80; the target time dimension can be set to 2T; taking b = 3 as an example, the target frequency dimensions can be set to 120, 160 and 240; and the sampling dimension information group can be 、 and .

[0087] In order to better understand the embodiments of the present application, taking b = 3 as an example for illustration. It can be understood that when b is other values, the specific acquisition process of the target spectral feature is similar to the following description.

[0088] Specifically, please refer to Figure 6 , a schematic diagram of acquiring a target spectral feature provided by the embodiments of the present application. It can be understood that Figure 6 The number of upsampling layers and downsampling layers in the sampling component 404 shown is only an example, and in actual applications, it can be adjusted according to needs. For example Figure 6As shown, the sampling component 404 can include three up-sampling layers, namely a first up-sampling layer 501, a second up-sampling layer 502, and a third up-sampling layer 503; and three down-sampling layers, namely a first down-sampling layer 504, a second down-sampling layer 505, and a third down-sampling layer 506. Before the a-th candidate spectral feature is sampled, the target time dimension and the three target frequency dimensions can be combined to obtain three sets of sampling dimension information, for example, the three sets of sampling dimension information can be , and wherein the set of sampling dimension information corresponding to the first up-sampling layer 501 can be , the set of sampling dimension information corresponding to the second up-sampling layer 502 can be , and the set of sampling dimension information corresponding to the third up-sampling layer 503 can be The set of sampling dimension information of the down-sampling layer is associated with the initial time dimension and the initial frequency dimension of the a-th candidate spectral feature, and thus the set of sampling dimension information of the first down-sampling layer 504, the second down-sampling layer 505, and the third down-sampling layer 506 can all be set to [N, 80, T].

[0089] When the a-th candidate spectral feature is up-sampled, the a-th candidate spectral feature can be up-sampled by the first up-sampling layer 501. The first up-sampling layer 501 can specifically include a first deconvolution subnetwork and a first activation subnetwork. The deconvolution parameters of the first deconvolution subnetwork can be: input channel Cin=80, output channel Cout=120, convolution kernel size k=4, stride=2, padding=2. The activation function corresponding to the first activation subnetwork can include but is not limited to the activation functions of leaky_relu, relu, sigmod, tanh, etc. Since the set of sampling dimension information corresponding to the first up-sampling layer 501 is , after the a-th candidate spectral feature is up-sampled by the first up-sampling layer 501, a first sub-spectral feature 1 is obtained, and the shape of the first sub-spectral feature 1 is That is, compared with the a th candidate spectral feature, the frequency dimension of the first sub-spectral feature 1 is expanded from 80 dimensions to 120 dimensions, and the time dimension is expanded from T to 2T, achieving the effect of rich frequency dimension and time dimension features. In order to ensure the quality of the first sub-spectral feature 1, the first sub-spectral feature 1 can be down-sampled by the first down-sampling layer 504 according to the initial time dimension and the initial frequency dimension corresponding to the a th candidate spectral feature. The first down-sampling layer 504 can specifically include a first convolutional subnetwork and a second activation subnetwork. The convolutional parameters of the first convolutional subnetwork can be: input channel Cin=120, output channel Cout=80, convolution kernel size k=7, stride=2, padding=3. The activation function corresponding to the second activation subnetwork can include but is not limited to leaky_relu, relu, sigmod, tanh and the like. The sampling dimension information group corresponding to the first down-sampling layer 504 is [N, 80, T], and after the first sub-spectral feature 1 is down-sampled by the first down-sampling layer 504, the second sub-spectral feature 1 is obtained, and the size (shape) corresponding to the second sub-spectral feature 1 is [N, 80, T]. That is, after down-sampling processing, the second sub-spectral feature 1 obtained has the same frequency dimension and time dimension as the a th candidate spectral feature, so that the second sub-spectral feature 1 obtained can have higher quality while performing feature enhancement.

[0090] Similarly, in order to obtain more rich feature information, when the a th candidate spectral feature is up-sampled, the a th candidate spectral feature can be first up-sampled by the second up-sampling layer 502. The second up-sampling layer 502 can specifically include a second deconvolutional subnetwork and a third activation subnetwork. The deconvolutional parameters of the second deconvolutional subnetwork can be: input channel Cin=80, output channel Cout=160, convolution kernel size k=4, stride=2, padding=2. The activation function corresponding to the third activation subnetwork can include but is not limited to leaky_relu, relu, sigmod, tanh and the like. Since the sampling dimension information group corresponding to the second up-sampling layer 502 is , after the a th candidate spectral feature is up-sampled by the second up-sampling layer 502, the second sub-spectral feature 2 is obtained, and the size (shape) corresponding to the second sub-spectral feature 2 is That is, compared with the a th candidate spectral feature, the frequency dimension of the second sub-spectral feature 2 is expanded from 80 dimensions to 160 dimensions, and the time dimension is expanded from T to 2T, achieving the effect of rich frequency dimension and time dimension features. Further, in order to ensure the quality of the first sub-spectral feature 2, the first sub-spectral feature 2 can be down-sampled by the second down-sampling layer 505 according to the initial time dimension and the initial frequency dimension corresponding to the a th candidate spectral feature. The second down-sampling layer 505 can specifically include a second convolutional sub-network and a fourth activation sub-network. The convolutional parameters of the second convolutional sub-network can be: input channel Cin=160, output channel Cout=80, convolution kernel size k=7, stride=2, padding=3. The activation function corresponding to the fourth activation sub-network can include but is not limited to leaky_relu, relu, sigmod, tanh and the like. The sampling dimension information group corresponding to the second down-sampling layer 505 is [N, 80, T], and after the first sub-spectral feature 2 is down-sampled by the second down-sampling layer 505, the second sub-spectral feature 2 is obtained, and the size (shape) corresponding to the second sub-spectral feature 2 is [N, 80, T]. That is, after down-sampling processing, the second sub-spectral feature 2 obtained has the same frequency dimension and time dimension as the a th candidate spectral feature, so that the second sub-spectral feature 2 obtained can have higher quality while performing feature enhancement.

[0091] In addition, in order to obtain more rich feature information, the a th candidate spectral feature can be first up-sampled by the third up-sampling layer 503 when the a th candidate spectral feature is up-sampled. The third up-sampling layer 503 can specifically include a third deconvolutional sub-network and a fifth activation sub-network. The deconvolutional parameters of the third deconvolutional sub-network can be: input channel Cin=80, output channel Cout=240, convolution kernel size k=4, stride=2, padding=2. The activation function corresponding to the fifth activation sub-network can include but is not limited to leaky_relu, relu, sigmod, tanh and the like. Since the sampling dimension information group corresponding to the third up-sampling layer 503 is , after the a th candidate spectral feature is up-sampled by the third up-sampling layer 503, the first sub-spectral feature 3 is obtained, and the size (shape) corresponding to the first sub-spectral feature 3 is That is, compared with the a th candidate spectral feature, the frequency dimension of the first sub-spectral feature 3 is expanded from 80 dimensions to 240 dimensions, and the time dimension is expanded from T to 2T, achieving the effect of rich frequency dimension and time dimension features. Further, in order to ensure the quality of the first sub-spectral feature 3, the first sub-spectral feature 3 can be down-sampled by the third down-sampling layer 506 according to the initial time dimension and the initial frequency dimension corresponding to the a th candidate spectral feature. The third down-sampling layer 506 can specifically include a third convolutional sub-network and a sixth activation sub-network. The convolutional parameters of the third convolutional sub-network can be: input channel Cin=240, output channel Cout=80, convolution kernel size k=7, stride=2, padding=3. The activation function corresponding to the fourth activation sub-network can include but is not limited to the activation functions of leaky_relu, relu, sigmod, tanh, etc. The sampling dimension information group corresponding to the third down-sampling layer 506 is [N, 80, T], and after the second sub-spectral feature 3 is down-sampled by the third down-sampling layer 506, the second sub-spectral feature 3 is obtained, and the size (shape) corresponding to the second sub-spectral feature 3 is [N, 80, T]. That is, after down-sampling, the second sub-spectral feature 3 obtained has the same frequency dimension and time dimension as the a th candidate spectral feature, so that the second sub-spectral feature 3 obtained can have high quality while being enhanced in features.

[0092] It can be understood that the parameters of the first up-sampling layer 501, the second up-sampling layer 502, the third up-sampling layer 503, the first down-sampling layer 504, the second down-sampling layer 505 and the third down-sampling layer 506 in the embodiments of the present application are only examples, and in specific applications, they can be adjusted accordingly according to the requirements.

[0093] Further, after obtaining the second sub-spectral feature 1, the second sub-spectral feature 2 and the second sub-spectral feature 3, the three second sub-spectral features can be added to obtain the a th target spectral feature. It can be seen that by enhancing the feature extraction of the audio information contained in the time dimension and the frequency dimension of the candidate spectral feature, the spectral information is deeply expanded, so that the richness of the target spectral feature obtained is higher.

[0094] S104: performing feature fusion processing on the K target spectral features to obtain a fusion spectral feature, and combining the fusion spectral feature into a target audio data.

[0095] The fusion spectral feature is the feature finally used to synthesize the target audio data. The target audio data is the final synthesized audio data, which can be output in the form of an audio frame, and can specifically include but is not limited to voice broadcast, song, recitation, drama, speech, etc.

[0096] Please refer to Figure 3 After obtaining the K target spectrum features, the K target spectrum features can be subjected to feature fusion processing by a fusion component 405 in the audio synthesis model, so as to obtain a fused spectrum feature, and finally subjected to Figure 7 The role of the sampling component 404 is to make the target spectrum features obtained by sampling processing fused with features in the time dimension and the frequency dimension, so as to avoid weakening or loss of features in the time dimension and the frequency dimension of the candidate spectrum features that have a greater impact on audio synthesis, and thus improve the utilization rate and expression ability of the target spectrum features.

[0097] Specifically, the computer device can obtain convolution parameter information corresponding to the K target spectrum features, perform convolution processing on the K target spectrum features according to the convolution parameter information, to obtain convolution spectrum features corresponding to the K target spectrum features; and then perform activation processing on the convolution spectrum features, to obtain fused spectrum features corresponding to the convolution spectrum features.

[0098] The convolution parameter information can be understood as parameters for convolution operation on the K target spectrum features, and specifically can include input channels (Cin), output channels (Cout), convolution kernel size, stride, padding, and the like. The convolution spectrum features can be understood as features obtained after convolution operation on the K target spectrum features; and the activation processing can be understood as nonlinear transformation on the convolution spectrum features, so that the processed fused spectrum features have richer expression.

[0099] Specifically, please refer to Figure 7 , Figure 7 is a schematic diagram of obtaining fused spectrum features provided by an embodiment of the present application. It can be understood that Figure 7 The parameters of the convolution layer 601 and the activation layer 602 in the fusion component shown are only examples, and in actual applications, they can be adjusted accordingly according to requirements. For example Figure 7As shown, the fusion component may include a convolutional layer 601 and an activation layer 602. After obtaining K target spectral features, the convolutional parameter information corresponding to the K target spectral features can be obtained first. For example, the convolutional parameter information may be: input channel Cin=3, output channel Cout=1, kernel size k=K, stride=1, and padding="same". Then, based on the above convolutional parameter information, the K target spectral features are convolved through the convolutional layer 601 in the fusion component 405 to obtain convolutional fused features. In this embodiment, the role of the convolutional layer is to perform feature fusion on the K target spectral features obtained by the sampling component 404 in the time and frequency dimensions using a convolutional kernel of size K, so that the obtained convolutional fused features have higher richness. Finally, the convolutional fused features are activated by the activation layer 602 in the sampling component 404 to obtain the target fused spectral features. The activation function corresponding to the activation layer 602 may include, but is not limited to, leaky_relu, relu, sigmoid, tanh, and other activation functions. It can be seen that by performing convolution and activation processing on the K target spectral features, the effectiveness of the fused spectral features used to synthesize the target audio data in terms of both time and frequency dimensions can be enhanced.

[0100] Alternatively, in one possible implementation, the fusion component 405 may further include Figure 7 The transformation layer is not shown in the diagram. Before fusing the K target spectral features, the K target spectral features can be concatenated to obtain concatenated spectral features. Then, the transformation layer of the fusion component 405 performs a dimensionality transformation on the concatenated spectral features to obtain the K target spectral features. It can be understood that the size (shape) of each of the K target spectral features is [N, F, T]. The size (shape) of the concatenated spectral features is... In this embodiment, the size (shape) of the concatenated spectral features is more suitable as input to the fusion component 405 than that of the K target spectral features. Therefore, using the concatenated spectral features obtained by concatenating the K target spectral features as input to the fusion component 405 helps reduce the probability of errors in the fusion component 405. Subsequently, the transformation layer of the fusion component 405 can be used to process the concatenated spectral features with a size (shape) of... The stitching spectrum feature is subjected to a dimension transformation operation to change from three-dimensional data to four-dimensional data, and the stitching spectrum feature is re-converted into K target spectrum features with a shape of [N, F, T]. Then, the K target spectrum features are subjected to a convolution operation and an activation operation. The specific process of the convolution operation and the activation operation can be referred to the foregoing description, and will not be described here.

[0101] Further, after obtaining the fusion spectrum feature, the fusion spectrum feature is converted into target synthesized audio by the vocoder 406. It can be understood that the computer device can use, but is not limited to, a griffin-lim vocoder, a hifi-gan vocoder, a pwg vocoder, and the like to convert and synthesize the fusion spectrum feature to obtain the target synthesized audio.

[0102] In the embodiment of the present application, after obtaining the audio key information used for synthesizing the audio data, the audio key information is first subjected to an encoding process to obtain the audio attribute feature, so as to convert the audio key information into a vector form, which helps to guarantee the feasibility of audio synthesis. Then, according to the audio attribute feature and the diffusion frequency information, K candidate spectrum features are generated, so that the K candidate spectrum features have a high feature richness. Further, the time dimension information and the frequency dimension information corresponding to the K candidate spectrum features are obtained, and the K candidate spectrum features are subjected to a sampling process according to the time dimension information and the frequency dimension information to obtain K target spectrum features. The K target spectrum features are subjected to feature fusion processing to obtain a fusion spectrum feature, and the fusion spectrum feature is synthesized into target audio data. In the embodiment of the present application, by enhancing the feature extraction of the audio information contained in the time dimension and the frequency dimension, the spectrum information is deeply expanded, so as to enhance the effectiveness of the feature expression of the fusion spectrum feature in the time dimension and the frequency dimension. Therefore, the fusion spectrum feature obtained has a high richness, and the corresponding target audio data synthesized by using the fusion spectrum feature is more in line with the expected requirements, which can improve the quality of audio synthesis.

[0103] Before applying the audio synthesis model to an audio synthesis scene (for example, the process of synthesizing target audio data from audio key information), the audio synthesis model needs to be trained in advance. The training process can be referred to the foregoing description. Figure 8 , Figure 8 is a flowchart of an audio synthesis model training method provided by an embodiment of the present application. It can be understood that the audio synthesis model training method is executed by a computer device, which can be a terminal device (for example, the terminal device 102a, the terminal device 102b, or the terminal device 102c in the corresponding embodiment of the present application) or a server (for example, the server 104 in the corresponding embodiment of the present application). Figure 1 It can be understood that the audio synthesis model training method is executed by a computer device, which can be a terminal device (for example, the terminal device 102a, the terminal device 102b, or the terminal device 102c in the corresponding embodiment of the present application) or a server (for example, the server 104 in the corresponding embodiment of the present application). Figure 1The server 101 in the corresponding embodiment) or a computer program (including program code). As shown in the figure, the audio synthesis model training method can include the following steps S201-S205: Figure 8

[0104] Step S201: Obtain sample key information for synthesizing audio data, and obtain sample attribute features corresponding to the sample key information through an encoder of an initial synthesis model.

[0105] The sample key information is a sample used to train the initial synthesis model. Specifically, the sample key information can include a sample phoneme sequence, a sample duration sequence, and a sample pitch sequence, etc. The initial synthesis model can refer to a network model before the audio synthesis model is trained. The sample attribute features can be understood as features representing the audio information in the sample key information.

[0106] Specifically, the computer device can obtain the sample key information for training in a local database or a cloud database, and then obtain the sample attribute features corresponding to the sample key information through the encoder of the initial synthesis model.

[0107] Step S202: Output V initial sample spectrum features corresponding to the sample attribute features through a diffusion decoder of the initial synthesis model; V is an integer greater than K.

[0108] The initial sample spectrum features can be understood as spectrum features obtained by feature enhancement on the sample attribute features. V can be determined according to diffusion frequency information of the diffusion decoder, V is an integer greater than K, and V is an integer greater than T, that is, the number of iterations in the training stage is greater than the number of iterations in the application stage. When the number of initial sample spectrum features matches the value V indicated by the diffusion frequency information, V initial sample spectrum features are obtained. The first initial sample spectrum feature is the input of the second initial sample spectrum feature, and after obtaining the second initial sample spectrum feature, the second initial sample spectrum feature can be input to the diffusion decoder again, and the initial sample spectrum feature is enhanced through the diffusion decoder to output the third initial sample spectrum feature, and the iteration is repeated in turn until V initial sample spectrum features are obtained, so that the initial sample spectrum features are enhanced.

[0109] Step S203: According to the generation time sequence corresponding to the V initial sample spectrum features, the i-th initial sample spectrum feature and the j-th initial sample spectrum feature are sequentially obtained in the V initial sample spectrum features, and the label spectrum feature is determined in the V initial sample spectrum features; i is a positive integer less than V; j is an integer less than or equal to V and greater than i.

[0110] ​The generation time sequence of the i-th initial sample spectral feature is earlier than the generation time sequence of the j-th initial sample spectral feature. For example, the generation time of the i-th initial sample spectral feature is t0, and the generation time of the j-th initial sample spectral feature can be t1 or t2, and so on. The label spectral feature can be any one of the V initial sample spectral features, for example, the V-th initial sample spectral feature, or the V-1-th initial sample spectral feature, and so on, and the generation time sequence of the label spectral feature is later than the generation time sequence of the j-th initial sample spectral feature.

[0111] Specifically, after obtaining the V initial sample spectral features, the computer device can sequentially obtain the i-th initial sample spectral feature and the j-th initial sample spectral feature in the V initial sample spectral features according to the generation time sequence corresponding to the V initial sample spectral features through the spectral selection component of the initial synthesis model, and determine the label spectral feature. i is a positive integer less than V, and i can take values of 1, 2, 3, …, V-1; j is an integer less than or equal to V and greater than i, and i can take values of i+1, …, V-1.

[0112] Step S204: determining a first loss value according to the i-th initial sample spectral feature and the j-th initial sample spectral feature, and determining a second loss value according to the j-th initial sample spectral feature and the label spectral feature.

[0113] The first loss value can be understood as the difference between the i-th target sample spectral feature corresponding to the i-th initial sample spectral feature and the j-th initial sample spectral feature. The i-th target sample spectral feature is a spectral feature obtained after sampling processing of the i-th initial sample spectral feature. The loss function used to calculate the first loss value can be any one of L1_los, L2_loss, and the like. The second loss value can be understood as the difference between the sample fusion spectral feature and the label spectral feature, wherein the sample fusion spectral feature is a feature obtained by performing feature fusion on the i-th target sample spectral feature corresponding to the i-th initial sample spectral feature and the j-th target sample spectral feature corresponding to the j-th initial sample spectral feature. The j-th target sample spectral feature is a spectral feature obtained after sampling processing of the j-th initial sample spectral feature. The loss function used to calculate the second loss value can be any one of L1_los, L2_loss, and the like. The first loss value and the second loss value can be used to determine whether the network parameters (such as learning rate, convolution parameter information, etc.) of the initial synthesis model need to be adjusted.

[0114] Specifically, the computer device can sample the i-th initial sample spectral feature according to the time dimension information and the frequency dimension information corresponding to the i-th initial sample spectral feature to obtain an i-th target sample spectral feature. The specific definition of the time dimension information and the frequency dimension information and the specific sampling process can be referred to the foregoing description, and will not be described here. After obtaining the i-th target sample spectral feature, the j-th initial sample spectral feature can be taken as a constraint of the i-th target sample spectral feature to obtain a first loss value between the two. The main role of this process is to help the initial synthesis model quickly learn the mapping relationship from the feature generation time corresponding to the i-th initial sample spectral feature to the feature generation time corresponding to the j-th initial sample spectral feature, so that the feature information of the i-th target sample spectral feature gradually approaches the V-th initial sample spectral feature, thereby improving the quality and efficiency of audio synthesis. Then, the computer device can sample the j-th initial sample spectral feature according to the time dimension information and the frequency dimension information corresponding to the j-th initial sample spectral feature to obtain a j-th target sample spectral feature, and perform feature fusion processing on the i-th target sample spectral feature and the j-th target sample spectral feature to obtain a sample fusion spectral feature. The specific definition of the time dimension information and the frequency dimension information and the specific sampling and fusion processes can be referred to the foregoing description, and will not be described here. After obtaining the sample fusion spectral feature, the label spectral feature can be taken as a constraint of the sample fusion spectral feature to obtain a second loss value between the two. The main role of this process is to help the initial synthesis model quickly learn the mapping relationship from the feature generation time corresponding to the j-th initial sample spectral feature to the feature generation time corresponding to the label spectral feature, so that the feature information of the j-th target sample spectral feature is closer to the V-th initial sample spectral feature, thereby improving the quality of audio synthesis while improving the efficiency of audio synthesis.

[0115] Step S205: Based on the first loss value and the second loss value, the network parameters in the initial synthesis model are corrected, and the initial synthesis model containing the corrected network parameters is determined as the audio synthesis model.

[0116] The network parameters in the initial synthesis model can include, but are not limited to, a learning rate, convolution parameter information, and the like. Specifically, the network parameters of the initial synthesis model can be modified if the preset condition is met by comparing the first loss value with the first loss threshold and comparing the second loss value with the second loss threshold. For example, if the first loss value is greater than the first loss threshold, it can be considered that the first loss value does not reach convergence; or if the second loss value is greater than the second loss threshold, it can be considered that the second loss value does not reach convergence; at this time, the network parameters of the initial synthesis model need to be adjusted, and the initial synthesis model continues to be iteratively trained until the first loss value is less than the first loss threshold and the second loss value is less than the second loss threshold, at which time it can be considered that the first loss value and the second loss value converge, thereby determining that the initial synthesis model obtained at this time converges, and the initial synthesis model containing the modified network parameters is determined as the audio synthesis model. The audio synthesis model obtained in this way has a high accuracy. The first loss threshold and the second loss threshold are parameters set in advance, and their specific values can be determined according to actual application scenarios, which are not limited by the embodiments of the present application.

[0117] Please refer to Figure 9 , Figure 9 is a schematic diagram of training an audio synthesis model provided by an embodiment of the present application. As shown in Figure 9 , the initial synthesis model can include an encoder 701, a diffusion decoder 702, a spectrum selection component 703, a first sampling component 704, a second sampling component 705, a third sampling component 706, a fusion component 707, and a vocoder 708. It can be understood that Figure 9 the number of sampling components shown is only an example, and in specific applications, it can be adjusted according to actual needs. As Figure 9As shown, the sample key information can be input into the initial synthesis model, the sample key information is encoded by the encoder 701 in the initial synthesis model to obtain the sample attribute feature; the V initial sample spectrum features corresponding to the sample attribute feature are output through the diffusion decoder 702; and the initial sample spectrum feature V0, the initial sample spectrum feature V1, the initial sample spectrum feature V2 and the label spectrum feature are selected from the V initial sample spectrum features through the spectrum selection component 703. Wherein, the generation time sequence of the initial sample spectrum feature V1 is earlier than that of the initial sample spectrum feature V2 and later than that of the initial sample spectrum feature V0; the generation time sequence of the label spectrum feature is later than that of the initial sample spectrum feature V2. Taking V=10 as an example, the initial sample spectrum feature V0 can be the initial sample spectrum feature output by the diffusion decoder for the first time; the initial sample spectrum feature V1 can be the initial sample spectrum feature output by the diffusion decoder for the second time; the initial sample spectrum feature V2 can be the initial sample spectrum feature output by the diffusion decoder for the fifth time; and the label spectrum feature can be the initial sample spectrum feature output by the diffusion decoder for the tenth time.

[0118] Further, the initial sample spectrum feature V0 can be sampled by the first sampling component 704 according to the time dimension information and the frequency dimension information corresponding to the initial sample spectrum feature V0 to obtain the target sample spectrum feature V0; then, the initial sample spectrum feature V1 can be taken as a constraint of the target sample spectrum feature V0 to obtain the first loss value 1 between them. The main role of this process is to help the initial synthesis model quickly learn the mapping relationship from the feature generation time corresponding to the initial sample spectrum feature V0 to the feature generation time corresponding to the initial sample spectrum feature V1, so that the feature information of the target sample spectrum feature V0 gradually approaches the Vth initial sample spectrum feature, thereby improving the quality and efficiency of audio synthesis. Similarly, the initial sample spectrum feature V1 can be sampled by the second sampling component 705 according to the time dimension information and the frequency dimension information corresponding to the initial sample spectrum feature V1 to obtain the target sample spectrum feature V1; then, the initial sample spectrum feature V2 can be taken as a constraint of the target sample spectrum feature V1 to obtain the first loss value 2 between them. The main role of this process is to help the initial synthesis model quickly learn the mapping relationship from the feature generation time corresponding to the initial sample spectrum feature V1 to the feature generation time corresponding to the initial sample spectrum feature V2, so that the feature information of the target sample spectrum feature V1 gradually approaches the Vth initial sample spectrum feature, thereby improving the quality and efficiency of audio synthesis.

[0119] Further, the initial sample spectrum feature V2 can be sampled by the third sampling component 706 according to the time dimension information and the frequency dimension information corresponding to the initial sample spectrum feature V2, to obtain a target sample spectrum feature V2; and the i-th target sample spectrum feature and the j-th target sample spectrum feature are fused to obtain a sample fusion spectrum feature. The specific definition of the time dimension information and the frequency dimension information, and the specific sampling and fusion processes can be referred to the foregoing description, and will not be described here. After obtaining the sample fusion spectrum feature, the label spectrum feature can be used as a constraint of the sample fusion spectrum feature to obtain a second loss value between them. The main role of this process is to help the initial synthesis model quickly learn the mapping relationship from the feature generation moment corresponding to the j-th initial sample spectrum feature to the feature generation moment corresponding to the label spectrum feature, so that the feature information of the j-th target sample spectrum feature is closer to the V-th initial sample spectrum feature, thereby improving the accuracy and efficiency of audio synthesis. Then, the computer device can correct the network parameters in the initial synthesis model based on the first loss value 1, the second loss value 2 and the second loss value, and determine the initial synthesis model containing the corrected network parameters as the audio synthesis model, so that the synthesized audio output by the final vocoder 708 has good synthesis quality.

[0120] In the embodiments of the present application, the network parameters in the initial synthesis model are corrected based on the first loss value and the second loss value, so that the target sample fusion feature after sampling processing of the j-th target sample spectrum feature and the sample fusion spectrum feature are closer to the label spectrum feature, and thus the audio synthesis model finally obtained can have good synthesis quality. In the process of synthesizing target audio data, by selecting the spectrum selection component in the audio synthesis model to select K candidate spectrum features less than V as the input of the sampling component, and through sampling processing and fusion processing, the fusion spectrum feature enhances the effectiveness of feature expression in the time dimension and the spectrum dimension, so that the audio synthesis effect of the target audio data synthesized by using the fusion spectrum feature is better than that of the audio data synthesized by using the label spectrum feature. In addition, when synthesizing the target audio data by using the fusion spectrum feature, V times of iteration is not required, which significantly reduces the iteration time, improves the processing speed of audio synthesis, and thus improves the efficiency of audio synthesis.

[0121] Please refer to Figure 10 , Figure 10 is a structural schematic diagram of an audio synthesis device provided by the embodiments of the present application. As Figure 10 shown, the audio synthesis device 1 can include a first acquisition module 11, a generation module 12, a sampling module 13 and a synthesis module 14, and the detailed description of each module is as follows:

[0122] The first obtaining module is configured to obtain audio key information used for synthesizing audio data.

[0123] The generating module is configured to perform encoding processing on the audio key information to obtain audio attribute features, and generate K candidate spectral features according to the audio attribute features and diffusion frequency information, where K is a positive integer.

[0124] The sampling module is configured to obtain time dimension information and frequency dimension information corresponding to the K candidate spectral features, and perform sampling processing on the K candidate spectral features according to the time dimension information and the frequency dimension information to obtain K target spectral features.

[0125] The synthesizing module is configured to perform feature fusion processing on the K target spectral features to obtain a fusion spectral feature, and synthesize the fusion spectral feature into target audio data.

[0126] The specific function implementation manners of the first obtaining module 11, the generating module 12, the sampling module 13, and the synthesizing module 14 can be referred to steps S101-S104 in the corresponding embodiments, which will not be described here again. Figure 4

[0127] In one or more embodiments, the audio key information includes a phoneme sequence, a duration sequence, and a pitch sequence, and the generating module 12 can include a splicing unit 121, an encoding unit 122, a conversion unit 123, an enhancement unit 124, and a first obtaining unit 125, where:

[0128] The splicing unit 121 is configured to splice the phoneme sequence, the duration sequence, and the pitch sequence to obtain audio input features, and input the audio input features to an audio synthesis model.

[0129] The encoding unit 122 is configured to perform encoding processing on the audio input features according to an encoder in the audio synthesis model to obtain audio attribute features.

[0130] The conversion unit 123 is configured to convert the audio attribute features to initial spectral features M0 through a diffusion decoder of the audio synthesis model.

[0131] The enhancement unit 124 is configured to input the initial spectral features M0 to the diffusion decoder, perform feature enhancement processing on the initial spectral features M0 based on the diffusion decoder to obtain initial spectral features M1.

[0132] The first obtaining unit 125 is configured to, when the number of the initial spectral features matches a value T indicated by the diffusion frequency information, obtain T initial spectral features, and obtain K candidate spectral features from the T initial spectral features, where T is an integer greater than K.

[0133] ​The specific function implementation manners of the splicing unit 121, the encoding unit 122, the conversion unit 123, the enhancement unit 124 and the first acquisition unit 125 can be referred to the corresponding embodiments. Figure 4 The step S102 in the corresponding embodiment will not be repeated here.

[0134] In one or more embodiments, the time dimension information includes a target time dimension of an a-th candidate spectral feature in K candidate spectral features in the sampling processing, and the frequency dimension information includes b target frequency dimensions of the a-th candidate spectral feature in the sampling processing; a is a positive integer less than or equal to K, b is an integer greater than 1, and the sampling module 13 can include a combination unit 131, an up-sampling unit 132 and a down-sampling unit 133, wherein:

[0135] The combination unit 131 is configured to combine the target time dimension and the b target frequency dimensions to obtain b sampling dimension information groups.

[0136] The up-sampling unit 132 is configured to perform up-sampling processing on the a-th candidate spectral feature according to the b sampling dimension information groups to obtain b first sub-spectral features corresponding to the a-th candidate spectral feature.

[0137] The down-sampling unit 133 is configured to perform down-sampling processing on the b first sub-spectral features according to an initial time dimension and an initial frequency dimension corresponding to the a-th candidate spectral feature to obtain b second sub-spectral features, and perform addition operation on the b second sub-spectral features to obtain an a-th target spectral feature.

[0138] The specific function implementation manners of the combination unit 131, the up-sampling unit 132 and the down-sampling unit 133 can be referred to the corresponding embodiments. Figure 4 The step S103 in the corresponding embodiment will not be repeated here.

[0139] In one or more embodiments, the synthesis module 14 can include a convolution unit 141 and an activation unit 142, wherein:

[0140] The convolution unit 141 is configured to obtain convolution parameter information corresponding to the K target spectral features, and perform convolution processing on the K target spectral features according to the convolution parameter information to obtain convolution spectral features corresponding to the K target spectral features.

[0141] The activation unit 142 is configured to perform activation processing on the convolution spectral features to obtain fusion spectral features corresponding to the convolution spectral features.

[0142] The specific function implementation manners of the convolution unit 141 and the activation unit 142 can be referred to the corresponding embodiments. Figure 4 The step S104 in the corresponding embodiment will not be repeated here.

[0143] In one or more embodiments, the audio synthesis device 1 can further comprise a second obtaining module 15, a third obtaining module 16, a fourth obtaining module 17, a determining module 18 and a revising module 19, wherein:

[0144] The second obtaining module 15 is configured to obtain sample key information for synthesizing audio data, and obtain sample attribute features corresponding to the sample key information through an encoder of an initial synthesis model.

[0145] The third obtaining module 16 is configured to output V initial sample spectral features corresponding to the sample attribute features through a diffusion decoder of the initial synthesis model; V is an integer greater than K.

[0146] The fourth obtaining module 17 is configured to obtain an i-th initial sample spectral feature and a j-th initial sample spectral feature in the V initial sample spectral features in a generation time sequence corresponding to the V initial sample spectral features, and determine a label spectral feature in the V initial sample spectral features; i is a positive integer less than V; j is an integer less than or equal to V and greater than i.

[0147] The determining module 18 is configured to determine a first loss value according to the i-th initial sample spectral feature and the j-th initial sample spectral feature, and determine a second loss value according to the j-th initial sample spectral feature and the label spectral feature.

[0148] The revising module 19 is configured to revise network parameters in the initial synthesis model based on the first loss value and the second loss value, and determine the initial synthesis model containing the revised network parameters as an audio synthesis model.

[0149] The specific function implementation manners of the second obtaining module 15, the third obtaining module 16, the fourth obtaining module 17, the determining module 18 and the revising module 19 can refer to the step S104 in the corresponding embodiments, which will not be repeated here. Figure 4

[0150] In one or more embodiments, the determining module 18 can comprise a second obtaining unit 181 and a third obtaining unit 182, wherein:

[0151] The second obtaining unit 181 is configured to sample process the i-th initial sample spectral feature according to time dimension information and frequency dimension information corresponding to the i-th initial sample spectral feature to obtain an i-th target sample spectral feature, and obtain a first loss value between the i-th target sample spectral feature and the j-th initial sample spectral feature.

[0152] ​The third obtaining unit 182 is configured to sample the jth initial sample spectral feature according to the time dimension information and the frequency dimension information corresponding to the jth initial sample spectral feature, to obtain a jth target sample spectral feature, to perform feature fusion processing on the ith target sample spectral feature and the jth target sample spectral feature, to obtain a sample fusion spectral feature, and to obtain a second loss value between the sample fusion spectral feature and the label spectral feature.

[0153] The specific function implementation manners of the second obtaining unit 181 and the third obtaining unit 182 can be referred to the step S104 in the embodiment Figure 4 The step S104 in the corresponding embodiment, which will not be repeated here.

[0154] In the embodiment of the present application, after obtaining the audio key information used for synthesizing the audio data, the audio key information is first encoded to obtain the audio attribute feature, so as to convert the audio key information into a vector form, which helps to guarantee the feasibility of audio synthesis. Then, according to the audio attribute feature and the diffusion frequency information, K candidate spectral features are generated, so that the K candidate spectral features have high feature richness. Furthermore, the time dimension information and the frequency dimension information corresponding to the K candidate spectral features are obtained, the K candidate spectral features are sampled according to the time dimension information and the frequency dimension information to obtain K target spectral features, the K target spectral features are subjected to feature fusion processing to obtain a fusion spectral feature, and the fusion spectral feature is synthesized into target audio data. In the embodiment of the present application, by enhancing the feature extraction of the audio information contained in the time dimension and the frequency dimension, the spectral information is deeply expanded, so that the effectiveness of the feature expression of the fusion spectral feature in the time dimension and the frequency dimension is enhanced, and the richness of the obtained fusion spectral feature is high. Therefore, the synthesis of the corresponding target audio data by using the fusion spectral feature is more in line with the expected requirements, and the quality of the audio synthesis can be improved.

[0155] Please refer to Figure 11 , Figure 11 is a structural schematic diagram of a computer device provided in the embodiment of the present application. As shown in Figure 11As shown, the computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include a user interface 1003 and one or more communication buses 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. Optionally, the network interface 1004 may include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as one or more disk drives. Optionally, the memory 1005 may also be one or more storage devices located remotely from the processor 1001. Figure 11 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0156] In such Figure 11 In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0157] Obtain key audio information used for synthesizing audio data;

[0158] The key audio information is encoded to obtain audio attribute features. Based on the audio attribute features and the diffusion frequency information, K candidate spectral features are generated; K is a positive integer.

[0159] Obtain the time dimension information and frequency dimension information corresponding to K candidate spectral features. Based on the time dimension information and frequency dimension information, sample the K candidate spectral features to obtain K target spectral features.

[0160] Feature fusion processing is performed on K target spectral features to obtain fused spectral features, and the fused spectral features are then synthesized into target audio data.

[0161] It should be understood that the computer device 1000 described in the embodiments of this application can execute the foregoing text. Figure 4 The description of the audio synthesis method in the corresponding embodiments can also be performed as described above. Figure 10 The description of the audio synthesis apparatus 1 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0162] In addition, it should be noted that the embodiments of the present application also provide a computer readable storage medium, and the computer readable storage medium stores the computer program executed by the audio synthesis device 1 mentioned above, and the computer program includes program instructions, which can execute the above-mentioned method when the processor executes the program instructions. Figure 4 The description of the audio synthesis method in the corresponding embodiments will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated here. For technical details not disclosed in the computer readable storage medium embodiments involved in the present application, please refer to the description of the method embodiments of the present application. As an example, the program instructions can be deployed on one computing device for execution, or on multiple computing devices located in one place for execution, or on multiple computing devices distributed in multiple places and interconnected through a communication network for execution. The multiple computing devices distributed in multiple places and interconnected through a communication network can constitute a blockchain system.

[0163] In addition, it should be noted that the embodiments of the present application also provide a computer program product or computer program, which can include computer instructions that can be stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor can execute the computer instructions to make the computer device execute the above-mentioned method. Figure 4 The description of the audio synthesis method in the corresponding embodiments will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated here. For technical details not disclosed in the computer program product or computer program embodiments involved in the present application, please refer to the description of the method embodiments of the present application.

[0164] It should be noted that for the above-mentioned various method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the action order described, because according to the present application, some steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0165] The steps in the method embodiments of the present application can be adjusted, combined and deleted according to actual needs.

[0166] The modules in the device embodiments of the present application can be combined, divided and deleted according to actual needs.

[0167] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiment methods can be included. The storage medium can be a magnetic disc, an optical disc, a Read-Only Memory (ROM) or a Random Access Memory (RAM) and the like.

[0168] The above disclosure is merely preferred embodiments of the present application and is not intended to limit the scope of the present application. Therefore, equivalent changes made within the scope of the claims of the present application are still within the scope of the present application.

Claims

1. An audio synthesis method, characterized in that, include: Obtain key audio information used for synthesizing audio data; The key audio information is encoded to obtain audio attribute features, which are then converted into initial spectral features M0 by the diffusion decoder of the audio synthesis model. The initial spectral feature M0 is input into the diffusion decoder, and the initial spectral feature M0 is subjected to feature enhancement processing based on the diffusion decoder to obtain the initial spectral feature M1. When the number of initial spectral features matches the value T indicated by the diffusion frequency information, T initial spectral features are obtained, and K candidate spectral features are obtained from the T initial spectral features; K is a positive integer, and T is an integer greater than K; Obtain the time dimension information and frequency dimension information corresponding to the K candidate spectral features, and perform sampling processing on the K candidate spectral features according to the time dimension information and the frequency dimension information to obtain K target spectral features; The K target spectral features are subjected to feature fusion processing to obtain fused spectral features, and the fused spectral features are synthesized into target audio data.

2. The method according to claim 1, characterized in that, The key audio information includes phoneme sequences, duration sequences, and pitch sequences; The process of encoding the key audio information to obtain audio attribute features includes: The phoneme sequence, the duration sequence, and the pitch sequence are concatenated to obtain audio input features, which are then input into an audio synthesis model. The audio input features are encoded using the encoder in the audio synthesis model to obtain audio attribute features.

3. The method according to claim 1, characterized in that, The time dimension information includes the target time dimension of the a-th candidate spectral feature among the K candidate spectral features in the sampling process, and the frequency dimension information includes b target frequency dimensions of the a-th candidate spectral feature in the sampling process; a is a positive integer less than or equal to K, and b is an integer greater than 1; The step of sampling and processing the K candidate spectral features based on the time dimension information and the frequency dimension information to obtain K target spectral features includes: The target time dimension and the b target frequency dimensions are combined to obtain b sampling dimension information groups; Based on the b sampling dimension information groups, the a-th candidate spectral feature is upsampled to obtain b first sub-spectral features corresponding to the a-th candidate spectral feature; Based on the initial time dimension and initial frequency dimension corresponding to the a-th candidate spectral feature, the b first sub-spectral features are downsampled to obtain b second sub-spectral features. The b second sub-spectral features are then summed to obtain the a-th target spectral feature.

4. The method according to claim 1, characterized in that, The feature fusion processing of the K target spectral features to obtain fused spectral features includes: Obtain the convolution parameter information corresponding to the K target spectral features, and perform convolution processing on the K target spectral features according to the convolution parameter information to obtain the convolution spectral features corresponding to the K target spectral features; The convolutional spectral features are activated to obtain the fused spectral features corresponding to the convolutional spectral features.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Obtain key sample information for synthesizing audio data, and obtain sample attribute features corresponding to the key sample information through the encoder of the initial synthesis model; The diffusion decoder of the initial synthesis model outputs V initial sample spectral features corresponding to the sample attribute features; V is an integer greater than K. According to the generation time order corresponding to the V initial sample spectral features, the i-th initial sample spectral feature and the j-th initial sample spectral feature are obtained sequentially from the V initial sample spectral features, and the label spectral feature is determined from the V initial sample spectral features; i is a positive integer less than V; j is an integer less than or equal to V and greater than i; Based on the spectral features of the i-th initial sample and the spectral features of the j-th initial sample, a first loss value is determined; based on the spectral features of the j-th initial sample and the label spectral features, a second loss value is determined. Based on the first loss value and the second loss value, the network parameters in the initial synthesis model are corrected, and the initial synthesis model containing the corrected network parameters is determined as the audio synthesis model.

6. The method according to claim 5, characterized in that, The step of determining a first loss value based on the spectral features of the i-th initial sample and the spectral features of the j-th initial sample, and determining a second loss value based on the spectral features of the j-th initial sample and the label spectral features, includes: Based on the time dimension information and frequency dimension information corresponding to the spectral features of the i-th initial sample, the spectral features of the i-th initial sample are sampled to obtain the spectral features of the i-th target sample, and the first loss value between the spectral features of the i-th target sample and the spectral features of the j-th initial sample is obtained. Based on the time dimension information and frequency dimension information corresponding to the j-th initial sample spectral feature, the j-th initial sample spectral feature is sampled to obtain the j-th target sample spectral feature. The i-th target sample spectral feature and the j-th target sample spectral feature are then fused to obtain the sample fused spectral feature. Finally, a second loss value is obtained between the sample fused spectral feature and the label spectral feature.

7. An audio synthesis device, characterized in that, include: The acquisition module is used to acquire key audio information for synthesizing audio data; The generation module is used to encode the key audio information to obtain audio attribute features, and then convert the audio attribute features into initial spectral features M0 through the diffusion decoder of the audio synthesis model. The generation module is further configured to input the initial spectral feature M0 into the diffusion decoder, and perform feature enhancement processing on the initial spectral feature M0 based on the diffusion decoder to obtain the initial spectral feature M1; The generation module is further configured to obtain T initial spectral features when the number of initial spectral features matches the value T indicated by the diffusion frequency information, and to obtain K candidate spectral features from the T initial spectral features; where K is a positive integer and T is an integer greater than K. The sampling module is used to acquire the time dimension information and frequency dimension information corresponding to the K candidate spectral features, and to perform sampling processing on the K candidate spectral features according to the time dimension information and the frequency dimension information to obtain K target spectral features; The synthesis module is used to perform feature fusion processing on the K target spectral features to obtain fused spectral features, and to synthesize the fused spectral features into target audio data.

8. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program, which, when executed by the processor, performs the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, perform the steps of the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a processor, implement the method according to any one of claims 1 to 6.

Images