Audio data processing method, device, equipment and storage medium

Abstract

The present application discloses an audio data processing method, apparatus, device and storage medium, which relates to machine learning technology related to artificial intelligence, wherein the method comprises: obtaining at least two sample audio data, and sample audio data Y of the at least two sample audio data i Related object information; for sample audio data Y i Perform feature extraction to obtain sample audio data Y i The text feature information and music score feature information of the sample audio data Y i The text feature information and music score feature information are normalized to obtain the sample audio data Y i Sample audio feature information; using sample audio data Y i Sample audio feature information and sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain the target audio synthesis model. The present application can effectively improve the pronunciation stability of audio data and the quality of audio data.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology in artificial intelligence, and in particular to an audio data processing method, device, equipment and storage medium. Background Art

[0002] With the development of Internet technology, the demand for audio data is also growing. However, traditional audio data requires manual conversion of text information and music score information into audio data, which is inefficient. Therefore, audio synthesis technology came into being. Audio synthesis technology automatically generates audio data by learning the text feature information and music score feature information of the synthesized audio data through an audio synthesis model, which can improve the efficiency of audio data generation. Therefore, this technology is widely used in pan-intelligent dubbing, virtual anchors, smart homes, smart robots and related intelligent fields. However, due to the large difference in the distribution of the text feature information and music score feature information of the audio data to be synthesized, the pronunciation stability of the generated audio data is relatively poor, which in turn leads to poor quality of the audio data. Summary of the invention

[0003] The technical problem to be solved by the embodiments of the present application is to provide an audio data processing method, device, equipment and storage medium, which can effectively improve the pronunciation stability of audio data and the quality of audio data.

[0004] An embodiment of the present application provides an audio data processing method, including:

[0005] Acquire at least two sample audio data and sample audio data Y of the at least two sample audio data i Associated object information; the sample audio data Y i The associated object information is used to indicate the sample audio data Y i Belongs to the sample object P n , the at least two sample audio data belong to at least two sample objects, i is a positive integer less than or equal to M, M is the number of sample audio data in the at least two sample audio data, n is a positive integer less than or equal to Q, Q is the number of objects in the at least two sample objects;

[0006] For the sample audio data Y i Perform feature extraction to obtain the sample audio data Y i Text feature information and music score feature information;

[0007] For the sample audio data Y i The text feature information and the music score feature information are normalized to obtain the sample audio data Y i Sample audio feature information;

[0008] Using the sample audio data Y i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain a target audio synthesis model; the target audio synthesis model is used to synthesize the target audio data of the target object.

[0009] An embodiment of the present application provides an audio data processing device, including:

[0010] an acquisition module, used to acquire at least two sample audio data and sample audio data Y of the at least two sample audio data i Associated object information; the sample audio data Y i The associated object information is used to indicate the sample audio data Y i The sample object P n , the at least two sample audio data belong to at least two sample objects, i is less than or equal to M, M is the number of sample audio data in the at least two sample audio data, n is less than or equal to Q, Q is the number of objects in the at least two sample objects;

[0011] Extraction module, used for the sample audio data Y i Perform feature extraction to obtain the sample audio data Y i Text feature information and music score feature information;

[0012] A processing module is used to process the sample audio data Y i The text feature information and the music score feature information are normalized to obtain the sample audio data Y i Sample audio feature information;

[0013] An adjustment module is used to adopt the sample audio data Y i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain a target audio synthesis model; the target audio synthesis model is used to synthesize the target audio data of the target object.

[0014] On one hand, the present application provides a computer device, including: a processor and a memory;

[0015] The memory is used to store a computer program, and the processor is used to call the computer program to perform the following steps:

[0016] Acquire at least two sample audio data and sample audio data Y of the at least two sample audio data iAssociated object information; the sample audio data Y i The associated object information is used to indicate the sample audio data Y i Belongs to the sample object P n , the at least two sample audio data belong to at least two sample objects, i is a positive integer less than or equal to M, M is the number of sample audio data in the at least two sample audio data, n is a positive integer less than or equal to Q, Q is the number of objects in the at least two sample objects;

[0017] For the sample audio data Y i Perform feature extraction to obtain the sample audio data Y i Text feature information and music score feature information;

[0018] For the sample audio data Y i The text feature information and the music score feature information are normalized to obtain the sample audio data Y i Sample audio feature information;

[0019] Using the sample audio data Y i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain a target audio synthesis model; the target audio synthesis model is used to synthesize the target audio data of the target object.

[0020] In one aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the following steps are performed:

[0021] Acquire at least two sample audio data and sample audio data Y of the at least two sample audio data i Associated object information; the sample audio data Y i The associated object information is used to indicate the sample audio data Y i Belongs to the sample object P n , the at least two sample audio data belong to at least two sample objects, i is a positive integer less than or equal to M, M is the number of sample audio data in the at least two sample audio data, n is a positive integer less than or equal to Q, Q is the number of objects in the at least two sample objects;

[0022] For the sample audio data Y i Perform feature extraction to obtain the sample audio data Y i Text feature information and music score feature information;

[0023] For the sample audio data Y i The text feature information and the music score feature information are normalized to obtain the sample audio data Y i Sample audio feature information;

[0024] Using the sample audio data Y i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain a target audio synthesis model; the target audio synthesis model is used to synthesize the target audio data of the target object.

[0025] In the present application, by obtaining at least two sample audio data, the text feature information and the music score feature information of each sample audio data in the at least two sample audio data are extracted. Since there are differences in the distribution of the text feature information and the music score feature information of the sample audio data, it is easy to cause the pronunciation of the synthesized audio data to be unstable; therefore, by normalizing the text feature information and the music score feature information of each sample audio data to obtain the audio feature information of each sample audio data, it is beneficial to reduce the distribution differences of the text feature and the music score feature information of the sample audio data, and then improve the pronunciation stability of the synthesized audio data. Further, the above-mentioned sample audio data Y can be used i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain the target audio synthesis model; at least two sample audio data belong to at least two sample objects, that is, the candidate audio model is trained by using sample audio data of multiple sample objects to improve the diversity of the training corpus, which is beneficial to improve the robustness of the target audio synthesis model and avoid instability problems in high notes, low notes and sustained notes. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 It is a schematic diagram of the architecture of an audio data processing system provided by the present application;

[0028] Figure 2 This is a schematic diagram of a scenario of data interaction between various devices in an audio data processing system provided by the present application;

[0029] Figure 3 This is a schematic diagram of a scenario of data interaction between various devices in an audio data processing system provided by the present application;

[0030] Figure 4 This is the process intention of an audio data processing method provided by this application;

[0031] Figure 5 It is a schematic diagram of the composition of an acoustic model, a duration model, and a candidate audio synthesis model provided by the present application;

[0032] Figure 6 This is a schematic diagram of a scenario for performing frequency domain transformation on candidate audio feature information provided by the present application;

[0033] Figure 7 This is the process intention of an audio data processing method provided by this application;

[0034] Figure 8 is a schematic diagram of a scenario for adjusting a candidate audio synthesis model provided by the present application;

[0035] Fig. 9 is a structural diagram of an audio data processing device provided in an embodiment of the present application;

[0036] Fig.10 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] At present, in the process of synthesizing audio data, the audio synthesis model is directly used to learn the text feature information and the music score feature information of the audio data to be synthesized to generate audio data. Due to the large distribution difference between the text feature information and the music score feature information of the audio data to be synthesized, the pronunciation stability of the generated audio data is relatively poor, and then the quality of the audio data is poor. Based on this, the present application uses machine learning technology related to artificial intelligence to extract features from at least two sample audio data, obtain text feature information and music score feature information of each sample audio data in the at least two sample audio data, and obtain audio feature information of each sample audio data by normalizing the text feature information and the music score feature information of each sample audio data, which can reduce the distribution difference between the text feature and the music score feature of each sample audio data. Then, the candidate audio synthesis model can be adjusted by using the sample audio feature information of each sample audio data and the object information associated with the each sample audio data to obtain the target audio synthesis model; by reducing the distribution differences of the text features and the music score features of each sample audio data, it is beneficial to improve the pronunciation stability of the audio data generated by the target audio synthesis model; at the same time, by using the sample audio data of multiple sample users to adjust the candidate audio synthesis model, it is beneficial to improve the robustness of the target audio synthesis model and avoid instability problems in the high notes, low notes and sustained notes.

[0039] Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0040] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0041] Among them, the above-mentioned machine learning (ML) is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0042] In order to facilitate a clearer understanding of the present application, an audio data processing system for implementing the audio data processing method of the present application is first introduced. Figure 1 As shown, the audio data processing system includes Figure 1 As shown, the audio data processing system includes a server and a terminal.

[0043] The terminal may refer to a user-oriented device, which may include an audio data application platform (i.e., an audio data application program) for playing audio data; the audio data platform here may refer to an audio website platform (such as a forum, a post bar), a social application platform, a shopping application platform, a content interaction platform (such as an audio playback application platform), etc. The server may refer to a device for providing audio data background services, which may be used to generate audio data according to text feature information and music score feature information, and upload the audio data to the audio data platform for users to play the audio data on the audio data platform.

[0044] Among them, the server can be an independent physical server, or a server cluster or distributed system composed of at least two physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms and other basic cloud computing services. The terminal can be a smart phone, a tablet computer, a laptop, a desktop computer, a smart speaker, a screen speaker, a smart watch, a smart TV, etc., but is not limited to this. Each terminal and server can be directly or indirectly connected by wired or wireless communication. At the same time, the number of terminals and servers can be one or at least two, and this application is not limited here. Among them, the business scenarios applicable to the above-mentioned audio data processing data system can specifically include: intelligent robots, intelligent dubbing, song singing, virtual anchors, virtual teaching assistants, AI customer service, TTS (Text To Speech, text to speech) cloud services, etc., and the business scenarios applicable to the audio data processing system will not be listed one by one here.

[0045] The audio data involved in this application may refer to digitized sound, and the audio data includes text feature information and music score feature information. The text feature information of the audio data refers to the specific content of the sound. The music score feature information of the audio data refers to the pronunciation information of the specific content of the sound. When the audio data is the singing data of a song, the pronunciation information may include phonemes, phoneme types, notes, note values, etc.; when the audio data is not the singing data of a song, the pronunciation information may include phonemes, phoneme types, etc. It is understandable that the specific content of the audio data is related to the business scenario. For example, in a song singing scenario, the audio data refers to the singing data of the song sung by the object; for another example, in a virtual teaching assistant scenario, the audio data may be the voice data of the content explained by the object; for another example, in an intelligent dubbing scenario, the audio data may refer to the dubbing of a film or television work by the object, and so on.

[0046] For further understanding, please refer to Figure 2 and Figure 3 , is a schematic diagram of a data interaction scenario provided by an embodiment of the present application. Figure 2 and Figure 3 In the present invention, the data exchange process is explained by taking the business scenario of song singing as an example. In this business scenario, audio data can be called singing data. The data interaction process includes the training process of candidate audio synthesis models and the process of generating target singing data.

[0047] like Figure 2 As shown, the training process of the candidate audio synthesis model includes the following steps 1-3:

[0048] 1. The server obtains text feature information and music score feature information of each sample singing data. The server device can obtain at least two sample singing data and object information associated with each sample singing data from the terminal. Furthermore, the server can convert each sample singing data into a language, obtain lyrics information of each sample singing data, and determine the lyrics information of each sample singing data as the text feature information of each sample singing data; then, acoustic analysis is performed on each sample singing data to obtain music score feature information of each sample singing data.

[0049] It is understandable that the sample singing data Y in the at least two sample singing data i The associated object information is used to indicate the sample singing data Y i The sample object P n , the at least two sample singing data belong to at least two sample objects. i is less than or equal to M, M is the number of sample singing data in the at least two sample singing data, and n is less than or equal to Q, Q is the number of objects in the at least two sample objects. That is to say, at least two sample singing data are obtained by singing by multiple sample objects, and the sample object may refer to any user who has published singing data in the audio data application platform. Therefore, the at least two sample singing data may specifically refer to the singing data of the songs sung by the user in a historical time period (such as the past week or the past month). In one embodiment, the songs corresponding to each sample singing data may be the same. For example, the at least two sample singing data are all singing data obtained by different user objects singing song A. In another embodiment, the songs corresponding to each sample singing data may be different. For example, the sample singing data Y i For the sample object P n The singing data obtained by singing song A, the sample singing data Y i+1 For the sample object P n+1 The singing data obtained by singing song A; this is conducive to improving the diversity of sample singing data, and then improving the robustness of the audio synthesis model.

[0050] 2. The server generates audio feature information of each sample singing data. In one embodiment, the server may first perform splicing processing on the feature information, and then perform normalization processing on the feature information to obtain the audio feature information of each sample singing data; specifically, the server may generate the audio feature information of each sample singing data Y i The text feature information and the music score feature information are spliced ​​to obtain the spliced ​​feature information, and the spliced ​​feature information is normalized to obtain the sample singing data Y iIn another embodiment, the server may first normalize the feature information, and then concatenate the feature information to obtain the audio feature information of each sample singing data; specifically, the server may perform normalization on the sample singing data Y. i The text feature information of the sample singing data Y is normalized to obtain the normalized text feature information. i The music score feature information is normalized to obtain the normalized music score feature information; then, the normalized text feature information and the normalized music score feature information are concatenated to obtain the sample singing data Y i The audio feature information of each sample singing data can be obtained by analogy. It should be noted that normalization processing refers to adjusting the text feature information and the music score feature information to reduce the distribution difference between the adjusted text feature information and the music score feature information; for example, adjusting the text feature information and the music score feature information to the same data interval, such as [0,1].

[0051] 3. The server can use the audio feature information of each sample singing data and the object information of each sample object to train the candidate audio synthesis model. i Get the object information of the sample object P n Specifically, according to the sample singing data Y i The object information of the timbre database is queried to see whether the sample object P is included. n If the sample object P is found in the timbre database, n The candidate timbre feature information is used as the sample object P n The timbre characteristic information of the sample object P n The timbre feature information is used to reflect the singing style of the sample object. n The server can perform the singing data Y according to the candidate timbre feature information of the sample i The object information of the sample object P is determined from at least two sample singing data. n The sample singing data belongs to the sample object P n The sample singing data is intercepted from the sample singing data segment, and the timbre feature is extracted from the sample singing data segment to obtain the sample object P n The timbre characteristic information of the sample object P n The timbre characteristic information is added to the timbre database, which is conducive to the subsequent rapid acquisition of the sample object Pn The timbre characteristic information of the sample object can be improved.

[0052] Furthermore, the server can use the candidate audio synthesis model to synthesize the sample object P n The timbre characteristic information and the sample singing data Y i The audio feature information is predicted to obtain the sample object P n The predicted singing data is the synthesized singing data. n The predicted singing data and the sample singing data Y i The similarity between the two can be used to determine the prediction error of the candidate audio synthesis model based on the similarity, and the prediction error is used to reflect the accuracy of the audio synthesis of the candidate audio synthesis model. If the prediction error of the candidate audio synthesis model is in a converged state, the candidate audio synthesis model can be used as the target audio synthesis model; if the prediction error of the candidate audio synthesis model is not in a converged state, the candidate audio synthesis model is adjusted according to the prediction error to obtain an adjusted candidate audio synthesis model, and the adjusted candidate audio synthesis model is determined as the target audio synthesis model.

[0053] like Figure 3 As shown, the process of generating target singing data includes the following steps 4-6:

[0054] 4. The server can obtain the reference singing data segment of the target object, as well as the text feature information and music score feature information of the target singing data to be synthesized. The server can obtain the historical singing data belonging to the target object, intercept the singing data segment from the historical singing data, and determine the intercepted singing data segment as the reference singing data segment; the target object can refer to a user or a virtual user (such as a robot, smart speaker, etc.). The target singing data refers to the singing data that needs to be synthesized. The target singing data can specifically refer to the singing data of a song. The lyrics of the song can be called the text feature information of the target singing data, and the music score of the song can be called the music score feature information of the target singing data.

[0055] 5. The server can obtain the audio feature information of the target singing data. In one embodiment, the server can first perform splicing processing on the feature information, and then perform normalization processing on the feature information to obtain the audio feature information of the target singing data; specifically, the server can perform splicing processing on the text feature information and the music score feature information of the target singing data to obtain the spliced ​​feature information, and perform normalization processing on the spliced ​​feature information to obtain the audio feature information of the target singing data. In another embodiment, the server can first perform normalization processing on the feature information, and then perform splicing processing on the feature information to obtain the audio feature information of the target singing data; specifically, the server can perform normalization processing on the text feature information of the target singing data to obtain the normalized text feature information, and perform normalization processing on the music score feature information of the target singing data to obtain the normalized music score feature information; then, the normalized text feature information and the normalized music score feature information are spliced ​​to obtain the audio feature information of the target singing data.

[0056] 6. The server uses the target audio synthesis model to generate target singing data. The server can use the target audio synthesis model to synthesize the timbre feature information of the target object and the audio feature information of the target singing data to obtain the target singing data belonging to the target object. After the server obtains the target singing data, it can send the target singing data to the terminal belonging to the target object, and the terminal belonging to the target object can play the audio data.

[0057] In summary, in the above-mentioned training process of the candidate audio synthesis model, by normalizing the text feature information and the music score feature information of the sample singing data, the distribution difference of the text feature information and the music score feature information of the sample singing data can be reduced, and then it is beneficial to improve the pronunciation stability of the singing data generated by the target audio synthesis model. And by using the sample singing data of multiple sample users to train the candidate audio model, it is beneficial to improve the robustness of the target audio synthesis model and avoid the problem of instability in the high notes, low notes and sustained notes. In the above-mentioned process of generating the target singing data, by normalizing the text feature information and the music score feature information of the target singing data, the distribution difference of the text feature information and the music score feature information of the target singing data can be reduced, and then it is beneficial to improve the pronunciation stability of the synthesized target singing data. And by synthesizing the target singing data fragment of the target object, it is possible to use a small amount of reference singing data to achieve the timbre customization of the target object, which can reduce the cost of synthesized singing data and improve the quality of synthesized singing data. That is to say, the target audio synthesis model can customize the timbre of the target object based on the target object's existing singing data fragments, and automatically synthesize singing data close to the target object's singing style, which can give the target object a more comprehensive singing ability. At the same time, the target audio synthesis model can be used to cultivate virtual idols, providing fans with entertainment value of singing and dancing anytime and anywhere.

[0058] It should be noted that the above-mentioned process of training the candidate audio synthesis model and generating the target singing data can be Figure 1 can be executed by the server in Figure 1 The terminal executes the training process of the candidate audio synthesis model and the process of generating the target singing data, which can be referred to Figure 2 The server executes the training process of the candidate audio synthesis model and the process of generating the target singing data, and the repeated parts will not be repeated.

[0059] In particular, when the terminal and the server jointly execute the training process of the candidate audio synthesis model and the process of generating the target singing data, the server and the terminal respectively execute different steps in the training process of the candidate audio synthesis model and the process of generating the target singing data. By synthesizing singing data through a distributed system composed of the terminal and the server, the singing data processing pressure of each device can be effectively reduced, and the efficiency of generating singing data can be improved. For example, the server can execute the training process of the candidate audio synthesis model. When the server completes the training of the candidate audio synthesis model, the target audio synthesis model can be obtained, and the target audio synthesis model can be sent to Figure 1Any terminal in. The terminal can execute the process of generating target singing data according to the target audio synthesis model.

[0060] For further information, see Figure 4 , is a flowchart of an audio data processing method provided in an embodiment of the present application. Figure 4 As shown, the method can be executed by a computer device, which can refer to Figure 1 Alternatively, the computer device may refer to Figure 1 The server in, or the computer device includes Figure 1 The terminal and server in the method can be Figure 1 The terminal and the server in the embodiment are executed together. The audio data processing method may include the following steps S101 to S104:

[0061] S101, obtaining at least two sample audio data, and sample audio data Y of the at least two sample audio data i Related object information; the above sample audio data Y i The associated object information is used to indicate the sample audio data Y i Belongs to the sample object P n , the above-mentioned at least two sample audio data belong to at least two sample objects, i is a positive integer less than or equal to M, M is the number of sample audio data in the above-mentioned at least two sample audio data, n is a positive integer less than or equal to Q, Q is the number of objects in the above-mentioned at least two sample objects.

[0062] In the present application, in one embodiment, the computer device can obtain at least two sample audio data from the terminal, and the sample audio data Y of the at least two sample audio data. i In another embodiment, the computer device can obtain at least two sample audio data and the sample audio data Y of the at least two sample audio data from the Internet. i Associated object information. Since the multiple sample audio data belong to at least two sample objects, that is, by obtaining the sample audio data of multiple sample objects, the diversity of the sample audio data can be effectively improved (that is, the diversity of the training corpus is improved), which is beneficial to improving the robustness of the target audio synthesis model.

[0063] S102: the sample audio data Y i Perform feature extraction to obtain the above sample audio data Y i The text feature information and music score feature information.

[0064] In the present application, the computer device can process the sample audio data Y iPerform language conversion processing to obtain the sample audio data Y i For example, the computer device may use a language model to analyze the sample audio data Y i Perform language recognition to obtain the above sample audio data Y i Further, the computer device can perform acoustic analysis on the sample audio data to obtain the sample audio data Y i The music score feature information.

[0065] S103, the sample audio data Y i The text feature information and the music score feature information are normalized to obtain the above-mentioned sample audio data Y i Sample audio feature information.

[0066] In this application, since the above sample audio data Y i There are differences in the distribution of the score feature information and the text feature information, which may easily lead to unstable pronunciation of the synthesized audio data. Based on this, the computer device can perform the above sample audio data Y i The text feature information and the music score feature information are normalized to obtain the above-mentioned sample audio data Y i The sample audio feature information can reduce the above sample audio data Y i The distribution difference between the music score feature information and the text feature information is conducive to improving the pronunciation stability of the synthesized audio data.

[0067] It should be noted that the computer device can use the following method a or method b to process the sample audio data Y i The text feature information and the music score feature information are normalized to obtain the above-mentioned sample audio data Y i Method a: the computer device may first normalize the feature information and then concatenate the feature information to obtain the sample audio data Y i Method b: The computer device may first perform concatenation processing on the feature information, and then perform normalization processing on the feature information to obtain the above-mentioned sample audio data Y i Sample audio feature information.

[0068] Optionally, when the computer device uses method a to process the sample audio data Y i When normalizing the text feature information and the music score feature information, the computer device can respectively perform normalization processing on the sample audio data Y iThe text feature information and the music score characteristic information are normalized to obtain the normalized text feature information and the normalized music score characteristic information; the normalized text feature information and the normalized music score characteristic information are concatenated to obtain the sample audio data Y i Sample audio feature information.

[0069] Optionally, when the computer device uses method b to process the sample audio data Y i When normalizing the text feature information and the music score characteristic information, the above step S103 may include the following steps s11 and s12:

[0070] s11, for the above sample audio data Y i The text feature information and the music score feature information are spliced ​​together to obtain the spliced ​​feature information.

[0071] s12, normalize the feature information after the above splicing process to obtain the above sample audio data Y i Sample audio feature information.

[0072] In steps s11 to s12, the computer device may perform the above-mentioned sample audio data Y according to the correspondence between the text feature information and the music score feature information. i The text feature information and the music score feature information are spliced ​​to obtain the spliced ​​feature information; for example, the text feature information includes multiple words, and the music score feature information corresponding to each word is spliced ​​to the position adjacent to each word to obtain the spliced ​​feature information. Further, the computer device can normalize the spliced ​​feature information to obtain the sample audio data Y i The sample audio feature information can reduce the above sample audio data Y i The distribution difference between the music score feature information and the text feature information is conducive to improving the pronunciation stability of the synthesized audio data.

[0073] It should be noted that the present application includes an acoustic model, a duration model, and a candidate audio synthesis model, and each step in the present application can be performed by a corresponding model. Figure 5As shown, the acoustic model is a model for generating audio feature information of audio data; the acoustic model may include a normalization layer, a residual encoder, an attention encoder, a gradient reversal layer, an object classification layer, an attention decoder, a smoothing layer, and the like; the normalization layer may be used to normalize the text feature information and the score feature information of the audio data; the residual encoder is used to obtain the timbre feature coding value of the object, that is, a representation of the identity of the object (i.e., the singing style), and the attention encoder is used to obtain the relational feature coding value reflecting the contextual relationship of the audio data. The gradient reversal layer is used to multiply the gradient error of the object classification layer by a reverse coefficient (i.e., a negative number) to achieve the reverse effect on the encoder in the acoustic model; the object classification layer is used to determine the category of the sample object, and the smoothing layer is used to smooth the energy feature information of the sample audio data. The duration model is used to predict the frame length of the pronunciation unit of the audio data; the candidate audio synthesis model is used to synthesize the audio data, and may specifically refer to the vocoder model. Optionally, after the computer device normalizes the feature information, it may encode the feature information to obtain the sample audio data Y i Specifically, the above step s12 may include the following steps s21 to s23:

[0074] s21, normalize the feature information after the above splicing process to obtain the above sample audio data Y i The candidate sample audio feature information.

[0075] s22, for the above sample audio data Y i The candidate sample audio feature information is encoded to obtain the above sample audio data Y i The audio encoding value.

[0076] s23, the sample audio data Y i The audio encoding value is determined as the above sample audio data Y i Sample audio feature information.

[0077] In steps s21 to s23, the computer device may normalize the feature information after the splicing process to obtain the sample audio data Y i Then, for the above sample audio data Y i The candidate sample audio feature information is encoded to obtain the above sample audio data Y i The audio code value of the sample audio data Y i The audio coding value may refer to the value used to reflect the sample audio data Y i The key feature information may include a code value reflecting the sample audio data Y iThe feature information of the contextual relationship and the feature information reflecting the sample audio data Y i Further, the sample audio data Y i The audio encoding value is determined as the above sample audio data Y i By analyzing the above sample audio data Y i The candidate sample audio feature information is encoded and processed, which is conducive to extracting the above sample audio data Y i key feature information.

[0078] Optionally, the sample audio data Y i The candidate sample audio feature information belongs to the time domain signal. Since the time domain signal is relatively complex, it is difficult to obtain the above sample audio data Y i The frequency domain signal is relatively simple and can provide more abundant information, making it easy to obtain the above sample audio data Y i Therefore, the computer device can analyze the above sample audio data Y i The candidate sample audio feature information is processed in the frequency domain to obtain the above sample audio data Y i key feature information.

[0079] Specifically, the above step s21 may include the following steps s31 to s33:

[0080] s31, for the above sample audio data Y i The candidate sample audio feature information is transformed into the frequency domain to obtain the above sample audio data Y i The frequency domain feature information.

[0081] s32, according to the above sample audio data Y i The frequency domain feature information generates the above sample audio data Y i energy characteristic information.

[0082] s33, for the above sample audio data Y i The energy characteristic information is encoded to obtain the above sample audio data Y i The audio encoding value.

[0083] In steps s31 to s33, if Figure 6 As shown, the computer device can process the sample audio data Y i The candidate sample audio feature information is transformed into the frequency domain to obtain the above sample audio data Y i The frequency domain feature information of the sample audio data Y i The frequency domain feature information is used to reflect the above sample audio data Y iThe amplitude parameter and frequency parameter of the sample audio data Y i The loudness of the sample audio data Y is a parameter that reflects the strength of the sound. i The larger the vibration amplitude, the louder the sound, and the stronger the sound; conversely, the sample audio data Y i The smaller the vibration amplitude, the smaller the loudness and the weaker the sound. The frequency parameter refers to the frequency parameter used to reflect the sample audio data Y i The pitch of the sound is the parameter used to reflect the pitch of the sound. The sample audio data Y i The higher the frequency, the higher the pitch; conversely, the higher the frequency of the sample audio data Y i If the frequency is low, the tone is also low. Further, the computer device can i The frequency domain feature information generates the above sample audio data Y i The energy spectrum curve is used to reflect the sample audio data Y i The relationship between the frequency parameter and the energy parameter can be obtained from the energy spectrum curve. i energy characteristic information; for the above sample audio data Y i The energy characteristic information is encoded to obtain the above sample audio data Y i The audio code value of the above sample audio data Y i The frequency domain transformation of the candidate sample audio feature information can reduce the acquisition of sample audio data Y i Optionally, since the human ear can only perceive a limited number of frequencies, the audio data corresponding to the frequencies that the human ear cannot perceive is usually called noise. Therefore, the computer device can process the sample audio data Y i Specifically, the above step s33 may include the following steps s41 to s43:

[0084] s41, for the above sample audio data Y i The energy characteristic information is filtered to obtain the above sample audio data Y i The effective energy characteristic information.

[0085] s42, for the above sample audio data Y i The effective energy characteristic information is discretized to obtain the above sample audio data Y i Discrete energy feature information.

[0086] s43, for the above sample audio data Y i The discrete energy feature information is encoded to obtain the above sample audio data Y i The audio encoding value.

[0087] In steps s41 to s43, the computer device can generate a filter based on the auditory characteristics of the human ear, and use the filter to filter the sample audio data Y i The energy characteristic information is filtered to obtain the above sample audio data Y i The effective energy characteristic information, that is, the energy characteristic information whose frequency parameters belong to the filter is regarded as the effective energy characteristic information; the energy characteristic information whose frequency parameters belong to the filter is regarded as the invalid energy characteristic information. i The effective energy characteristic information is discretized to obtain the above sample audio data Y i The discrete energy feature information of the above sample audio data Y i The discrete energy feature information is encoded to obtain the above sample audio data Y i The audio code value of the above sample audio data Y i The energy characteristic information of the sample audio data Y can be filtered to avoid noise interference and improve the quality of the sample audio data Y. i The accuracy of the audio feature information can be improved, and at the same time, the subsequent invalid noise processing can be avoided, which can save the processing resources of the computer equipment. i The candidate sample audio feature information is subjected to frequency domain transformation, filtering, and discretization, and the final sample audio data Y i The audio coding value may specifically refer to the value used to reflect the above-mentioned sample audio data Y i The Bark Frequency Cepstrum Coefficient (BFCC) feature encoding value is a Bark-level cepstrum coefficient. By dividing the audio frequency band at the Bark level, a set of cepstrum features closer to the human hearing sensitivity is extracted. The first dimension of the encoding value based on the BFCC feature represents energy. Compared with the encoding value based on the Mel spectrum, the encoding value based on the BFCC feature has explicit energy. In the case of single-sentence synthesis, the energy difference between sentences is not obvious, so that the synthesized audio data can sound more natural and pleasant.

[0088] Optionally, the computer device may use an attention encoding method and a residual encoding method to encode the sample audio data Y i The discrete energy characteristic information is encoded; specifically, the above step s43 may include the following steps s51 to s53:

[0089] s51, for the above sample audio data Y i The discrete energy feature information is residually encoded to obtain the above sample audio data Y iThe timbre feature encoding value of .

[0090] s52, for the above sample audio data Y i The discrete energy feature information is encoded with attention to obtain the sample audio data Y i The relation feature encoding value of the contextual relationship.

[0091] s53, the above sample audio data Y i The timbre feature coding value and the above relationship feature coding value are concatenated to obtain the above sample audio data Y i The audio encoding value.

[0092] In steps s51 to s53, if Figure 5 As shown, the computer device can use a residual encoder (ResidualEncoder) to encode the sample audio data Y i The discrete energy feature information is residually encoded to obtain the above sample audio data Y i The timbre feature encoding value, that is, the above sample audio data Y i The timbre feature encoding value can be used to reflect the sample object P n Then, the self-attention encoder is used to encode the above sample audio data Y i The discrete energy feature information is encoded with attention to obtain the sample audio data Y i The relational feature encoding value of the contextual relationship; for example, the attention encoder may refer to an encoder composed of the self-attention mechanism in the translation model Transformer. Further, the above sample audio data Y i The timbre feature coding value and the above relationship feature coding value are concatenated to obtain the above sample audio data Y i By getting the sample audio data Y i The relationship feature encoding values ​​and timbre feature values ​​are conducive to providing more effective training corpus for training candidate audio synthesis models, which can ensure the robustness of the target audio synthesis model.

[0093] Optionally, the computer device may process the sample audio data Y i Specifically, the above step s51 may include the following steps s61 to s63:

[0094] s61, for the above sample audio data Y i The discrete energy feature information is residually encoded to obtain the above sample audio data Y i The candidate timbre feature encoding value of .

[0095] s62, according to the above sample audio data Yi The object information associated with the above sample audio data Y i The candidate timbre feature coding value is compensated to obtain the above-mentioned sample audio data Y i The compensation timbre characteristic encoding value.

[0096] s63, the above sample audio data Y i The compensation timbre characteristic coding value is determined as the above sample audio data Y i The timbre feature encoding value of .

[0097] In steps s61 to s63, the computer device may process the sample audio data Y i The discrete energy feature information is residually encoded to obtain the above sample audio data Y i The candidate timbre feature coding value of the above sample audio data Y i The candidate timbre feature coding value may refer to a dynamic feature used to reflect the acoustics of the sample object, that is, the candidate timbre feature coding value changes with the change of the sample audio data. In order to realize multi-person audio synthesis, the sample audio data Y i The object information associated with the above sample audio data Y i The candidate timbre feature coding value is compensated to obtain the above-mentioned sample audio data Y i The compensated timbre characteristic encoding value of the sample audio data Y i The associated object information refers to the static features used to reflect the sample object. i The compensation timbre characteristic coding value is determined as the above sample audio data Y i The timbre feature encoding value of the above sample audio data Y i The timbre feature encoding value can be used to reflect the dynamic timbre characteristics and static characteristics of the sample object, which is conducive to realizing multi-person audio synthesis and providing richer training expectations for the subsequent training process of candidate audio synthesis models.

[0098] Optionally, the above relationship feature encoding value uses an attention encoder to the above sample audio data Y i The discrete energy feature information is obtained by attention encoding, and the above method also includes the following steps s71 to s74:

[0099] s71, using the gradient inversion layer to convert the above sample audio data Y i The relational feature encoding values ​​are passed to the object classification layer.

[0100] s72, using the above object classification layer to classify the above sample audio data Y i The relationship feature encoding value is used to identify the sample object Pn Category.

[0101] s73. According to the above sample object P n The category of determines the gradient error of the above attention encoder.

[0102] s74. Use the reverse coefficient to weight the gradient error of the above-mentioned attention encoder to obtain the encoding error of the above-mentioned attention encoder, and adjust the above-mentioned attention encoder according to the above-mentioned encoding error.

[0103] In steps s71 to s74, if Figure 5 As shown, the above self-attention encoder and residual encoder are both encoders of the acoustic model. In order to make the encoder of the acoustic model independent of the sample object, a gradient reversal layer is added to the output of the encoder of the acoustic model. An object classification layer (i.e., object classification layer) is connected after the gradient reversal layer. By minimizing the cross-entropy object classification loss function, the classification ability of the object classification layer is optimized. At the same time, due to the existence of the gradient reversal layer, the encoder of the acoustic model is also independent of the sample object. The characteristic that the encoder of the acoustic model is independent of the sample object can greatly improve the diversity of the training corpus, thereby improving the stability and robustness of the audio data of multi-person audio synthesis. Specifically, the computer device can use the gradient reversal layer to convert the above sample audio data Y i The relationship feature encoding value is passed to the object classification layer, and then the object classification layer is used to classify the sample audio data Y i The relationship feature encoding value is used to identify the sample object P n The category of the sample object P n The category can refer to the sample object P n Further, according to the above sample object P n The category determines the gradient error of the above attention encoder, that is, obtains the above sample object P n The difference between the category of the sample object and the category of the adjacent sample object is determined according to the difference, and the gradient error of the attention encoder is determined. Further, the gradient error of the attention encoder is weighted by a reverse coefficient (i.e., a negative number) to obtain the encoding error of the attention encoder, and the attention encoder is adjusted according to the encoding error. By adding a gradient reversal layer and an object classification layer, the reverse effect on the encoder is achieved; the encoder can be made independent of the sample object, the diversity of the training corpus is increased, and thus the stability and robustness of the audio data of multi-person audio synthesis are improved.

[0104] S104, using the above sample audio data Y i The sample audio feature information and the sample audio data Y iThe candidate audio synthesis model is adjusted based on the associated object information to obtain the target audio synthesis model; the above target audio synthesis model is used to synthesize the target audio data of the target object.

[0105] In the present application, the candidate audio synthesis model may refer to an audio synthesis model with relatively low audio synthesis accuracy. Therefore, in order to improve the audio synthesis accuracy of the candidate audio synthesis model, the computer device may train the candidate audio synthesis model. Specifically, the computer device may use the above sample audio data Y i The sample audio feature information and the sample audio data Y i The associated object information is the training corpus, the candidate audio synthesis model is trained (i.e., adjusted), and the trained candidate audio synthesis model is determined as the target audio synthesis model. By using a variety of training corpora to train the candidate audio synthesis model, it is beneficial to improve the robustness of the trained target audio synthesis model and the stability of the synthesized audio.

[0106] In the present application, by obtaining at least two sample audio data, the text feature information and the music score feature information of each sample audio data in the at least two sample audio data are extracted. Since there are differences in the distribution of the text feature information and the music score feature information of the sample audio data, it is easy to cause the pronunciation of the synthesized audio data to be unstable; therefore, by normalizing the text feature information and the music score feature information of each sample audio data to obtain the audio feature information of each sample audio data, it is beneficial to reduce the distribution differences of the text feature and the music score feature information of the sample audio data, and then improve the pronunciation stability of the synthesized audio data. Further, the above-mentioned sample audio data Y can be used i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain the target audio synthesis model; at least two sample audio data belong to at least two sample objects, that is, the candidate audio model is trained by using sample audio data of multiple sample objects to improve the diversity of the training corpus, which is beneficial to improve the robustness of the target audio synthesis model and avoid instability problems in high notes, low notes and sustained notes.

[0107] For further information, see Figure 7 , is a flowchart of an audio data processing method provided in an embodiment of the present application. Figure 7 As shown, the method can be executed by a computer device, which can refer to Figure 1 Alternatively, the computer device may refer to Figure 1 The server in, or the computer device includes Figure 1 The terminal and server in the method can be Figure 1The terminal and the server in the embodiment are executed together. The method may at least include the following S201-S208:

[0108] S201, obtaining at least two sample audio data, and sample audio data Y of the at least two sample audio data i Related object information; the above sample audio data Y i The associated object information is used to indicate the sample audio data Y i Belongs to the sample object P n , the above-mentioned at least two sample audio data belong to at least two sample objects, i is a positive integer less than or equal to M, M is the number of sample audio data in the above-mentioned at least two sample audio data, n is a positive integer less than or equal to Q, Q is the number of objects in the above-mentioned at least two sample objects.

[0109] S202: the sample audio data Y i Perform feature extraction to obtain the above sample audio data Y i The text feature information and music score feature information.

[0110] S203, the sample audio data Y i The text feature information and the music score feature information are normalized to obtain the above-mentioned sample audio data Y i Sample audio feature information.

[0111] S204, using the above sample audio data Y i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain the target audio synthesis model; the above target audio synthesis model is used to synthesize the target audio data of the target object.

[0112] It should be noted that the explanation of step S201 in this application can refer to Figure 3 For the explanation of step S101, please refer to the explanation of step S202. Figure 3 For the explanation of step S102, the explanation of step S203 can be referred to Figure 3 For the explanation of step S103, the explanation of step S204 can be referred to. Figure 3 The explanation of step S104 is given in the above description, and the repeated parts are not repeated here.

[0113] Optionally, the above step S204 may include the following steps s81 to s84:

[0114] s81. According to the above sample audio data Y i Associated object information, obtain the above sample object Pn The timbre characteristic information of

[0115] s82, using the candidate audio synthesis model to synthesize the sample audio data Y i The sample audio feature information and the timbre feature information of each sample object are predicted to obtain the sample object P n The predicted audio data.

[0116] s83. According to the above sample object P n The predicted audio data and the sample audio data Y i , adjust the above candidate audio synthesis model to obtain an adjusted candidate audio synthesis model.

[0117] s84. Determine the adjusted candidate audio synthesis model as the target audio synthesis model.

[0118] In steps s81 to s84, in order to customize the timbre for the sample object, the computer device can i The associated object information is queried in the timbre database to see whether the sample object P is included. n candidate timbre feature information; if the sample object P is found in the timbre database n The candidate timbre feature information is determined as the sample object P n If the sample object P is not found in the timbre database n The candidate timbre feature information can be obtained based on the sample object P n The sample audio data generates a sample object P n The timbre feature information of the sample object P n The timbre feature information is added to the timbre database to avoid regenerating the sample object P next time. n Furthermore, the candidate audio synthesis model can be used to synthesize the sample audio data Y i The sample audio feature information and the timbre feature information of each sample object are predicted to obtain the sample object P n If the above sample object P n The predicted audio data and sample audio data Y i The difference is relatively large, indicating that the audio synthesis accuracy of the candidate audio synthesis model is relatively low; if the above sample object P n The predicted audio data and sample audio data Y i The difference is relatively small, indicating that the audio synthesis accuracy of the candidate audio synthesis model is relatively high. nThe predicted audio data and the sample audio data Y i , adjust the candidate audio synthesis model to obtain an adjusted candidate audio synthesis model. The adjusted candidate audio synthesis model is determined as the target audio synthesis model. i The sample audio feature information and the timbre feature information of each sample object are used to train the candidate audio synthesis model, which is conducive to customizing the timbre for the object, and can be used to synthesize audio data for multiple people, thereby improving the robustness of the candidate audio synthesis model.

[0119] Optionally, if the sample object P is not found in the timbre database n The computer device can generate the sample object P n Specifically, the above step s81 may include the following steps s91 to s93:

[0120] s91. According to the above sample audio data Y i The associated object information is determined to belong to the above sample object P n Sample audio data of .

[0121] s92, belonging to the above sample object P n Extract sample audio fragments from the sample audio data.

[0122] s93, extract the timbre features of the sample audio clip to obtain the sample object P n The timbre characteristic information of

[0123] In steps s91 to s93, if the sample object P is not found in the timbre database, n The computer device can use the candidate timbre feature information of the sample audio data Y i The associated object information is determined from at least two sample audio data to belong to the sample object P n Then, belonging to the above sample object P n Randomly extract a sample audio segment from the sample audio data; extract the timbre features of the sample audio segment to obtain the sample object P n The timbre characteristic information of the sample object is determined by a small amount of sample audio data, and the timbre of the sample object is customized, which can save costs and improve the efficiency of generating the timbre characteristic information of the sample object.

[0124] Optionally, the above step s82 may include the following steps s94 to s95:

[0125] s94. According to the above sample audio data Y iThe sample audio feature information is used to determine the sample audio data Y i The frame length of the pronunciation unit.

[0126] s95. Use the candidate audio synthesis model to synthesize the sample audio data Y i The sample audio feature information, the sample audio data Y i The frame length of the pronunciation unit and the sample object P n The timbre feature information is predicted to obtain the sample object P n The predicted audio data.

[0127] In steps s94-s95, the computer device may use the time length model to analyze the sample audio data Y i The sample audio feature information is predicted to obtain the above sample audio data Y i The frame length of the pronunciation unit, that is, the above sample audio data Y i The frame length of the pronunciation unit refers to the above sample audio data Y i Further, the candidate audio synthesis model can be used to synthesize the above sample audio data Y i The sample audio feature information, the sample audio data Y i The frame length of the pronunciation unit and the sample object P n The timbre feature information is predicted to obtain the sample object P n By obtaining the above sample audio data Y i The frame length of the pronunciation unit is conducive to making the audio data synthesized for the sample object (ie, the predicted audio data) closer to the singing style of the sample object.

[0128] It should be noted that the sample audio data Y i The sample audio feature information may refer to the sample audio data Y i The text feature information and the music score feature information are normalized and residual encoded; this is conducive to improving the prediction of the sample audio data Y by the time length model i The accuracy of the frame length of the pronunciation unit can further make the audio data synthesized by the candidate audio synthesis model closer to the singing style of the sample object, thereby achieving personalized customization.

[0129] It should be noted that the candidate audio synthesis model may refer to a 24KHz LPC (Linear Predictive Coding) Net vocoder, WaveRNN (waveform neural network model), etc., and this application does not limit this. The 24KHz LPCNet is used to synthesize 24KHz audio data, divides the 24Khz audio data into 24 frequency bands, and uses 24-order LPC coefficients to predict the audio data waveform. The 24KHz LPCNet vocoder has a more stable and higher-definition sound quality than WaveRNN with the same amount of computation.

[0130] Optionally, the above step s83 may include the following steps s96 to s98:

[0131] s96. Get the sample object P n The predicted audio data and sample audio data Y i The similarity between .

[0132] s97. Determine the prediction error of the above-mentioned candidate audio synthesis model based on the above-mentioned similarity.

[0133] s98. If the prediction error of the above-mentioned candidate audio synthesis model is not in a convergence state, the above-mentioned candidate audio synthesis model is adjusted according to the prediction error of the above-mentioned candidate audio synthesis model to obtain an adjusted candidate audio synthesis model.

[0134] In steps s96 to s98, the computer device may n The predicted audio data and sample audio data Y i Compare to determine the sample object P n The predicted audio data and sample audio data Y i The greater the similarity, the closer the synthesized predicted audio data is to the sample audio data Y. i The smaller the difference between them, the lower the prediction error of the candidate audio synthesis model; on the contrary, if the similarity is smaller, it means that the synthesized predicted audio data is similar to the sample audio data Y. iThe difference between them is relatively large, and the prediction error of the candidate audio synthesis model is relatively high. Therefore, the computer device can determine the prediction error of the candidate audio synthesis model according to the above similarity. If the prediction error of the candidate audio synthesis model is in a convergence state, it indicates that the prediction error of the candidate audio synthesis model is relatively small, and the candidate audio synthesis model can be determined as the target audio synthesis model. If the prediction error of the candidate audio synthesis model is not in a convergence state, it indicates that the prediction error of the candidate audio synthesis model is relatively large; therefore, the candidate audio synthesis model is adjusted according to the prediction error of the candidate audio synthesis model to obtain the adjusted candidate audio synthesis model. By adjusting the candidate audio synthesis model according to the prediction error of the candidate audio synthesis model, it is beneficial to improve the accuracy of the synthesized audio of the target candidate audio synthesis model, so that the synthesized audio data is closer to the singing style of the target object.

[0135] It should be noted that, in one embodiment, determining the prediction error of the candidate audio synthesis model according to the similarity may specifically refer to: accumulating the similarity corresponding to each sample audio data in the at least two sample audio data to obtain a total similarity, and determining the prediction error of the candidate audio synthesis model according to the total similarity. Figure 8 As shown, the computer device can input the audio feature information of the first sample audio data and the timbre feature information of the first sample object to which the first sample audio data belongs into the candidate audio synthesis model, generate the predicted audio data of the first sample object (i.e., synthesized audio data), obtain the similarity between the first sample audio data and the predicted audio data of the first sample object, and calculate the similarity corresponding to the other sample audio data in at least two audio data in this type. Then, the similarity of each sample audio data is accumulated to obtain the total similarity, and the prediction error of the candidate audio synthesis model is determined according to the total similarity. If the prediction error of the candidate audio synthesis model is less than the error threshold, it is determined that the prediction error of the candidate audio synthesis model is in a convergence state, and the candidate audio synthesis model is determined as the target audio synthesis model. If the prediction error of the candidate audio synthesis model is greater than or equal to the error threshold, it is determined that the prediction error of the candidate audio synthesis model is not in a convergence state, and the candidate audio synthesis model is adjusted according to the prediction error of the candidate audio synthesis model to obtain the adjusted candidate audio synthesis model, and the adjusted candidate audio synthesis model is determined as the target audio synthesis model.

[0136] S205: Acquire a reference audio segment of the target object, as well as text feature information and music score feature information of the target audio data to be synthesized.

[0137] S206: Generate timbre feature information of the target object according to the reference audio segment of the target object.

[0138] S207: Normalize the text feature information and the music score feature information of the target audio data to obtain the audio feature information of the target audio data.

[0139] S208. Use the target audio synthesis model to synthesize the audio feature information of the target audio data and the timbre feature information of the target object to obtain target audio data belonging to the target object.

[0140] In steps S205 to S208, after obtaining the target audio synthesis model, the computer device can use the target audio synthesis model to synthesize audio data for the target object. Specifically, the computer device can obtain a reference audio segment of the target object, as well as text feature information and music score feature information of the target audio data to be synthesized, perform timbre extraction on the reference audio segment of the target object, and obtain the timbre feature information of the target object; the timbre information of the target object refers to information used to reflect the pronunciation style of the target object, and the target object may refer to a user or a virtual user. Further, the text feature information and music score feature information of the above-mentioned target audio data are normalized to obtain the audio feature information of the above-mentioned target audio data, and the audio feature information of the above-mentioned target audio data and the timbre feature information of the above-mentioned target audio synthesis model are synthesized to obtain the target audio data belonging to the above-mentioned target object. By normalizing the text feature information and music score feature information of the target audio data, the distribution difference of the text feature information and the music score feature information of the target audio data can be reduced, thereby facilitating the pronunciation stability of the synthesized target audio data. By synthesizing the target audio data through the reference audio clips of the target object, the timbre of the target object can be customized using a small amount of reference audio data, which can reduce the cost of synthesized audio data and improve the quality of synthesized audio data.

[0141] In the present application, by normalizing the text feature information and the music score feature information of the target audio data, the distribution difference of the text feature information and the music score feature information of the target audio data can be reduced, thereby facilitating the pronunciation stability of the synthesized target audio data. By synthesizing the target audio data through the reference audio clip of the target object, the timbre customization of the target object can be realized by using a small amount of reference audio data, which can reduce the cost of synthesized audio data and improve the quality of synthesized audio data.

[0142] See also Fig. 9, is a schematic diagram of the structure of an audio data processing device 1 provided in an embodiment of the present application. The above-mentioned audio data processing device 1 may be a computer program (including program code) running in a computer device, for example, the audio data processing device 1 is an application software; the device may be used to execute the corresponding steps in the method provided in an embodiment of the present application. Fig. 9 As shown, the audio data processing device 1 may include: an acquisition module 901 , an extraction module 902 , a processing module 903 , an adjustment module 904 and a synthesis module 905 .

[0143] an acquisition module, used to acquire at least two sample audio data and sample audio data Y of the at least two sample audio data i Associated object information; the sample audio data Y i The associated object information is used to indicate the sample audio data Y i The sample object P n , the at least two sample audio data belong to at least two sample objects, i is less than or equal to M, M is the number of sample audio data in the at least two sample audio data, n is less than or equal to Q, Q is the number of objects in the at least two sample objects;

[0144] Extraction module, used for the sample audio data Y i Perform feature extraction to obtain the sample audio data Y i Text feature information and music score feature information;

[0145] A processing module is used to process the sample audio data Y i The text feature information and the music score feature information are normalized to obtain the sample audio data Y i Sample audio feature information;

[0146] An adjustment module is used to adopt the sample audio data Y i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain a target audio synthesis model; the target audio synthesis model is used to synthesize the target audio data of the target object.

[0147] Optionally, the processing module processes the sample audio data Y i The text feature information and the music score feature information are normalized to obtain the sample audio data Y i Sample audio feature information, including:

[0148] For the sample audio data Y iThe text feature information and the music score feature information are spliced ​​together to obtain the spliced ​​feature information;

[0149] The feature information after the splicing process is normalized to obtain the sample audio data Y i Sample audio feature information.

[0150] Optionally, the processing module normalizes the feature information after the splicing process to obtain the sample audio data Y i Sample audio feature information, including:

[0151] The feature information after the splicing process is normalized to obtain the sample audio data Y i Audio feature information of candidate samples;

[0152] For the sample audio data Y i The candidate sample audio feature information is encoded to obtain the sample audio data Y i The audio code value of

[0153] The sample audio data Y i The audio code value is determined as the sample audio data Y i Sample audio feature information.

[0154] Optionally, the processing module processes the sample audio data Y i The candidate sample audio feature information is encoded to obtain the sample audio data Y i The audio encoding value includes:

[0155] For the sample audio data Y i The candidate sample audio feature information is transformed into the frequency domain to obtain the sample audio data Y i Frequency domain feature information;

[0156] According to the sample audio data Y i The frequency domain feature information generates the sample audio data Y i Energy characteristic information;

[0157] For the sample audio data Y i The energy characteristic information is encoded to obtain the sample audio data Y i The audio encoding value.

[0158] Optionally, the processing module processes the sample audio data Y i The energy characteristic information is encoded to obtain the sample audio data Y i The audio encoding value includes:

[0159] For the sample audio data Y i The energy characteristic information is filtered to obtain the sample audio data Y i The effective energy characteristic information;

[0160] For the sample audio data Y i The effective energy characteristic information is discretized to obtain the sample audio data Y i Discrete energy characteristic information;

[0161] For the sample audio data Y i The discrete energy feature information is encoded to obtain the sample audio data Y i The audio encoding value.

[0162] Optionally, the processing module processes the sample audio data Y i The discrete energy feature information is encoded to obtain the sample audio data Y i The audio encoding value includes:

[0163] For the sample audio data Y i The discrete energy feature information is residually encoded to obtain the sample audio data Y i The timbre feature encoding value of

[0164] For the sample audio data Y i The discrete energy feature information is encoded with attention to obtain the information used to reflect the sample audio data Y i The relational feature encoding value of the contextual relationship;

[0165] The sample audio data Y i The timbre feature coding value and the relationship feature coding value are concatenated to obtain the sample audio data Y i The audio encoding value.

[0166] Optionally, the processing module processes the sample audio data Y i The discrete energy feature information is residually encoded to obtain the sample audio data Y i The timbre feature encoding value includes:

[0167] For the sample audio data Y i The discrete energy feature information is residually encoded to obtain the sample audio data Y i The candidate timbre feature encoding value of ;

[0168] According to the sample audio data Y i The object information associated with the sample audio data Y iThe candidate timbre feature coding value is compensated to obtain the sample audio data Y i The compensation timbre characteristic coding value of

[0169] The sample audio data Y i The compensated timbre characteristic encoding value is determined as the sample audio data Y i The timbre feature encoding value of .

[0170] Optionally, the processing module relation feature encoding value is to use an attention encoder to the sample audio data Y i The discrete energy feature information is obtained by performing attention encoding, and the method further includes:

[0171] The sample audio data Y is transformed into i The relational feature encoding value is passed to the object classification layer;

[0172] The object classification layer is used to classify the sample audio data Y i The relationship feature encoding value is identified to obtain the sample object P n Category of;

[0173] According to the sample object P n The category determines the gradient error of the attention encoder;

[0174] The gradient error of the attention encoder is weighted by a reverse coefficient to obtain an encoding error of the attention encoder, and the attention encoder is adjusted according to the encoding error.

[0175] Optionally, the adjustment module uses the sample audio data Y i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain the target audio synthesis model, including:

[0176] According to the sample audio data Y i Associated object information, obtain the sample object P n The timbre characteristic information;

[0177] The sample audio data Y is synthesized using a candidate audio synthesis model i The sample audio feature information and the timbre feature information of each sample object are predicted to obtain the sample object P n The predicted audio data;

[0178] According to the sample object P n The predicted audio data and the sample audio data Y i, adjusting the candidate audio synthesis model to obtain an adjusted candidate audio synthesis model;

[0179] The adjusted candidate audio synthesis model is determined as the target audio synthesis model.

[0180] Optionally, the adjustment module is based on the sample audio data Y i Associated object information, obtain the sample object P n The timbre characteristics of the

[0181] According to the sample audio data Y i The associated object information is determined to belong to the sample object P n Sample audio data of

[0182] Belongs to the sample object P n Extracting a sample audio segment from the sample audio data;

[0183] The timbre feature of the sample audio clip is extracted to obtain the sample object P n The timbre characteristic information of

[0184] Optionally, the adjustment module uses a candidate audio synthesis model to adjust the sample audio data Y i The sample audio feature information and the timbre feature information of each sample object are predicted to obtain the sample object P n The predicted audio data includes:

[0185] According to the sample audio data Y i The sample audio feature information is used to determine the sample audio data Y i The frame length of the pronunciation unit;

[0186] The sample audio data Y is synthesized using a candidate audio synthesis model i The sample audio feature information, the sample audio data Y i The frame length of the pronunciation unit and the sample object P n The timbre feature information is predicted to obtain the sample object P n The predicted audio data.

[0187] Optionally, the adjustment module is based on the sample object P n The predicted audio data and the sample audio data Y i , adjusting the candidate audio synthesis model to obtain an adjusted candidate audio synthesis model, including:

[0188] Get the sample object P n The predicted audio data and sample audio data Yi The similarity between

[0189] Determining a prediction error of the candidate audio synthesis model according to the similarity;

[0190] If the prediction error of the candidate audio synthesis model is not in a converged state, the candidate audio synthesis model is adjusted according to the prediction error of the candidate audio synthesis model to obtain an adjusted candidate audio synthesis model.

[0191] Optionally, the acquisition module is further used to acquire a reference audio segment of the target object, as well as text feature information and music score feature information of the target audio data to be synthesized;

[0192] Optionally, the extraction module is further used to generate the timbre feature information of the target object according to the reference audio segment of the target object;

[0193] Optionally, the processing module is further used to perform normalization processing on the text characteristic information and the music score characteristic information of the target audio data to obtain the audio characteristic information of the target audio data;

[0194] Optionally, the device further comprises:

[0195] The synthesis module is used to synthesize the audio feature information of the target audio data and the timbre feature information of the target object by using the target audio synthesis model to obtain the target audio data belonging to the target object.

[0196] According to one embodiment of the present application, Figure 4 The steps involved in the audio data processing method shown can be represented by Fig. 9 The various modules in the audio data processing device shown are executed. For example, Figure 4 The step S101 shown in FIG. 1 can be performed by Fig. 9 The acquisition module 901 in is executed, Figure 4 The step S102 shown in FIG. 1 can be performed by Fig. 9 The extraction module 902 is executed; Figure 4 The step S103 shown in FIG. 1 can be performed by Fig. 9 The inspection processing block 903 is executed; Figure 4 The step S104 shown in FIG. 1 can be performed by Fig. 9 The adjustment module 904 in is used for execution.

[0197] According to one embodiment of the present application, Fig. 9The various modules in the audio data processing device shown can be separately or all combined into one or several units to constitute, or one (some) of the units can be further split into at least two functionally smaller sub-units, and the same operation can be achieved without affecting the realization of the technical effects of the embodiments of the present application. The above modules are divided based on logical functions. In practical applications, the functions of one module can also be implemented by at least two units, or the functions of at least two modules can be implemented by one unit. In other embodiments of the present application, the audio data processing device can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of at least two units.

[0198] According to one embodiment of the present application, the program can be executed by running on a general-purpose computer device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 4 A computer program (including program code) for each step involved in the corresponding method shown in Fig. 9 The audio data processing device shown in the embodiment of the present application is implemented by the audio data processing method. The above computer program can be recorded on a computer-readable recording medium, for example, and loaded into the above computing device through the computer-readable recording medium and run therein.

[0199] In the present application, by obtaining at least two sample audio data, the text feature information and the music score feature information of each sample audio data in the at least two sample audio data are extracted. Since there are differences in the distribution of the text feature information and the music score feature information of the sample audio data, it is easy to cause the pronunciation of the synthesized audio data to be unstable; therefore, by normalizing the text feature information and the music score feature information of each sample audio data to obtain the audio feature information of each sample audio data, it is beneficial to reduce the distribution differences of the text feature and the music score feature information of the sample audio data, and then improve the pronunciation stability of the synthesized audio data. Further, the above-mentioned sample audio data Y can be used i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain the target audio synthesis model; at least two sample audio data belong to at least two sample objects, that is, the candidate audio model is trained by using sample audio data of multiple sample objects to improve the diversity of the training corpus, which is beneficial to improve the robustness of the target audio synthesis model and avoid instability problems in high notes, low notes and sustained notes.

[0200] See also Fig.10 , is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.10 As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the computer device 1000 may also include: an object interface 1003 and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The object interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the object interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (such as W I -F I The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The memory 1005 may also be at least one storage device located away from the processor 1001. Fig.10 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, an object interface module, and a device control application program.

[0201] exist Fig.10 In the computer device 1000 shown, the network interface 1004 can provide a network communication function; the object interface 1003 is mainly used to provide an input interface for the object; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0202] Acquire at least two sample audio data and sample audio data Y of the at least two sample audio data i Associated object information; the sample audio data Y i The associated object information is used to indicate the sample audio data Y i Belongs to the sample object P n , the at least two sample audio data belong to at least two sample objects, i is a positive integer less than or equal to M, M is the number of sample audio data in the at least two sample audio data, n is a positive integer less than or equal to Q, Q is the number of objects in the at least two sample objects;

[0203] For the sample audio data Y i Perform feature extraction to obtain the sample audio data Y i Text feature information and music score feature information;

[0204] For the sample audio data Y i The text feature information and the music score feature information are normalized to obtain the sample audio data Yi Sample audio feature information;

[0205] Using the sample audio data Y i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain a target audio synthesis model; the target audio synthesis model is used to synthesize the target audio data of the target object.

[0206] Optionally, the processor 1001 may be used to call a device control application stored in the memory 1005 to implement the sample audio data Y i The text feature information and the music score feature information are normalized to obtain the sample audio data Y i Sample audio feature information, including:

[0207] For the sample audio data Y i The text feature information and the music score feature information are spliced ​​together to obtain the spliced ​​feature information;

[0208] The feature information after the splicing process is normalized to obtain the sample audio data Y i Sample audio feature information.

[0209] Optionally, the processor 1001 may be used to call the device control application stored in the memory 1005 to normalize the feature information after the splicing process to obtain the sample audio data Y i Sample audio feature information, including:

[0210] The feature information after the splicing process is normalized to obtain the sample audio data Y i Audio feature information of candidate samples;

[0211] For the sample audio data Y i The candidate sample audio feature information is encoded to obtain the sample audio data Y i The audio code value of

[0212] The sample audio data Y i The audio code value is determined as the sample audio data Y i Sample audio feature information.

[0213] Optionally, the processor 1001 may be used to call a device control application stored in the memory 1005 to implement the sample audio data Y i The candidate sample audio feature information is encoded to obtain the sample audio data Y iThe audio encoding value includes:

[0214] For the sample audio data Y i The candidate sample audio feature information is transformed into the frequency domain to obtain the sample audio data Y i Frequency domain feature information;

[0215] According to the sample audio data Y i The frequency domain feature information generates the sample audio data Y i Energy characteristic information;

[0216] For the sample audio data Y i The energy characteristic information is encoded to obtain the sample audio data Y i The audio encoding value.

[0217] Optionally, the processor 1001 may be used to call a device control application stored in the memory 1005 to implement the sample audio data Y i The energy characteristic information is encoded to obtain the sample audio data Y i The audio encoding value includes:

[0218] For the sample audio data Y i The energy characteristic information is filtered to obtain the sample audio data Y i The effective energy characteristic information;

[0219] For the sample audio data Y i The effective energy characteristic information is discretized to obtain the sample audio data Y i Discrete energy characteristic information;

[0220] For the sample audio data Y i The discrete energy feature information is encoded to obtain the sample audio data Y i The audio encoding value.

[0221] Optionally, the processor 1001 may be used to call a device control application stored in the memory 1005 to implement the sample audio data Y i The discrete energy feature information is encoded to obtain the sample audio data Y i The audio encoding value includes:

[0222] For the sample audio data Y i The discrete energy feature information is residually encoded to obtain the sample audio data Y i The timbre feature encoding value of

[0223] For the sample audio data Y iThe discrete energy feature information is encoded with attention to obtain the information used to reflect the sample audio data Y i The relational feature encoding value of the contextual relationship;

[0224] The sample audio data Y i The timbre feature coding value and the relationship feature coding value are concatenated to obtain the sample audio data Y i The audio encoding value.

[0225] Optionally, the processor 1001 may be used to call a device control application stored in the memory 1005 to implement the sample audio data Y i The discrete energy feature information is residually encoded to obtain the sample audio data Y i The timbre feature encoding value includes:

[0226] For the sample audio data Y i The discrete energy feature information is residually encoded to obtain the sample audio data Y i The candidate timbre feature encoding value of ;

[0227] According to the sample audio data Y i The object information associated with the sample audio data Y i The candidate timbre feature coding value is compensated to obtain the sample audio data Y i The compensation timbre characteristic coding value of

[0228] The sample audio data Y i The compensated timbre characteristic encoding value is determined as the sample audio data Y i The timbre feature encoding value of .

[0229] Optionally, the relationship feature encoding value is obtained by using an attention encoder to encode the sample audio data Y i The processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0230] The sample audio data Y is transformed into i The relational feature encoding value is passed to the object classification layer;

[0231] The object classification layer is used to classify the sample audio data Y i The relationship feature encoding value is identified to obtain the sample object P n Category of;

[0232] According to the sample object P n The category determines the gradient error of the attention encoder;

[0233] The gradient error of the attention encoder is weighted by a reverse coefficient to obtain an encoding error of the attention encoder, and the attention encoder is adjusted according to the encoding error.

[0234] Optionally, the processor 1001 may be used to call a device control application stored in the memory 1005 to implement the sample audio data Y i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain the target audio synthesis model, including:

[0235] According to the sample audio data Y i Associated object information, obtain the sample object P n The timbre characteristic information;

[0236] The sample audio data Y is synthesized using a candidate audio synthesis model i The sample audio feature information and the timbre feature information of each sample object are predicted to obtain the sample object P n The predicted audio data;

[0237] According to the sample object P n The predicted audio data and the sample audio data Y i , adjusting the candidate audio synthesis model to obtain an adjusted candidate audio synthesis model;

[0238] The adjusted candidate audio synthesis model is determined as the target audio synthesis model.

[0239] Optionally, the processor 1001 may be used to call a device control application stored in the memory 1005 to implement the control according to the sample audio data Y i Associated object information, obtain the sample object P n The timbre characteristics of the

[0240] According to the sample audio data Y i The associated object information is determined to belong to the sample object P n Sample audio data of

[0241] Belongs to the sample object P n Extracting a sample audio segment from the sample audio data;

[0242] The timbre feature of the sample audio clip is extracted to obtain the sample object P n The timbre characteristic information of

[0243] Optionally, the processor 1001 may be used to call a device control application stored in the memory 1005 to implement the candidate audio synthesis model for the sample audio data Y. i The sample audio feature information and the timbre feature information of each sample object are predicted to obtain the sample object P n The predicted audio data includes:

[0244] According to the sample audio data Y i The sample audio feature information is used to determine the sample audio data Y i The frame length of the pronunciation unit;

[0245] The sample audio data Y is synthesized using a candidate audio synthesis model i The sample audio feature information, the sample audio data Y i The frame length of the pronunciation unit and the sample object P n The timbre feature information is predicted to obtain the sample object P n The predicted audio data.

[0246] Optionally, the processor 1001 may be configured to call a device control application stored in the memory 1005 to implement the control according to the sample object P n The predicted audio data and the sample audio data Y i , adjusting the candidate audio synthesis model to obtain an adjusted candidate audio synthesis model, including:

[0247] Get the sample object P n The predicted audio data and sample audio data Y i The similarity between

[0248] Determining a prediction error of the candidate audio synthesis model according to the similarity;

[0249] If the prediction error of the candidate audio synthesis model is not in a converged state, the candidate audio synthesis model is adjusted according to the prediction error of the candidate audio synthesis model to obtain an adjusted candidate audio synthesis model.

[0250] Optionally, the processor 1001 may be configured to call a device control application stored in the memory 1005 to implement:

[0251] Acquire a reference audio clip of the target object, as well as text feature information and music score feature information of the target audio data to be synthesized;

[0252] generating timbre characteristic information of the target object according to a reference audio segment of the target object;

[0253] Normalizing the text feature information and the music score feature information of the target audio data to obtain audio feature information of the target audio data;

[0254] The target audio synthesis model is used to synthesize the audio feature information of the target audio data and the timbre feature information of the target object to obtain the target audio data belonging to the target object.

[0255] In the present application, by obtaining at least two sample audio data, the text feature information and the music score feature information of each sample audio data in the at least two sample audio data are extracted. Since there are differences in the distribution of the text feature information and the music score feature information of the sample audio data, it is easy to cause the pronunciation of the synthesized audio data to be unstable; therefore, by normalizing the text feature information and the music score feature information of each sample audio data to obtain the audio feature information of each sample audio data, it is beneficial to reduce the distribution differences of the text feature and the music score feature information of the sample audio data, and then improve the pronunciation stability of the synthesized audio data. Further, the above-mentioned sample audio data Y can be used i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain the target audio synthesis model; at least two sample audio data belong to at least two sample objects, that is, the candidate audio model is trained by using sample audio data of multiple sample objects to improve the diversity of the training corpus, which is beneficial to improve the robustness of the target audio synthesis model and avoid instability problems in high notes, low notes and sustained notes.

[0256] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 4 And the previous article Figure 7 The description of the above audio data processing method in the corresponding embodiment can also be performed as described above. Fig. 9 The description of the audio data processing device in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0257] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the audio data processing device mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, the above-mentioned Figure 4 and Figure 7The description of the above-mentioned audio data processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in the application, please refer to the description of the method embodiment of the application.

[0258] As an example, the above program instructions may be deployed on a computer device for execution, or deployed on at least two computer devices located at one location for execution, or executed on at least two computer devices distributed at at least two locations and interconnected through a communication network. At least two computer devices distributed at at least two locations and interconnected through a communication network may constitute a blockchain network.

[0259] The computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0260] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0261] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0262] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0263] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An audio data processing method, characterized in that: include: Acquire at least two sample audio data and sample audio data Y of the at least two sample audio data i Associated object information; the sample audio data Y i The associated object information is used to indicate the sample audio data Y i Belongs to the sample object P n , the at least two sample audio data belong to at least two sample objects, i is a positive integer less than or equal to M, M is the number of sample audio data in the at least two sample audio data, n is a positive integer less than or equal to Q, Q is the number of objects in the at least two sample objects; For the sample audio data Y i Perform feature extraction to obtain the sample audio data Y i Text feature information and music score feature information; According to the sample audio data Y i The text feature information and the music score feature information are used to determine the sample audio data Y i Discrete energy characteristic information; For the sample audio data Y i The discrete energy feature information is residually encoded to obtain the sample audio data Y i The timbre feature encoding value of For the sample audio data Y i The discrete energy feature information is encoded with attention to obtain the information used to reflect the sample audio data Y i The relational feature encoding value of the contextual relationship; The sample audio data Y i The timbre feature coding value and the relationship feature coding value are concatenated to obtain the sample audio data Y i The audio code value of The sample audio data Y i The audio code value is determined as the sample audio data Y i Sample audio feature information; Using the sample audio data Y i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain a target audio synthesis model; the target audio synthesis model is used to synthesize the target audio data of the target object.

2. The method according to claim 1, characterized in that According to the sample audio data Y i The text feature information and the music score feature information are used to determine the sample audio data Y i Discrete energy characteristic information, including: For the sample audio data Y i The text feature information and the music score feature information are spliced ​​together to obtain the spliced ​​feature information; The feature information after the splicing process is normalized to obtain the sample audio data Y i Audio feature information of candidate samples; According to the sample audio data Y i The candidate sample audio feature information is determined to determine the sample audio data Y i Discrete energy feature information.

3. The method according to claim 2, characterized in that The sample audio data Y i The candidate sample audio feature information is determined to determine the sample audio data Y i Discrete energy characteristic information, including: For the sample audio data Y i The candidate sample audio feature information is transformed into the frequency domain to obtain the sample audio data Y i Frequency domain feature information; According to the sample audio data Y i The frequency domain feature information generates the sample audio data Y i Energy characteristic information; According to the sample audio data Y i energy characteristic information, determining the sample audio data Y i Discrete energy feature information.

4. The method according to claim 3, characterized in that According to the sample audio data Y i energy characteristic information, determining the sample audio data Y i Discrete energy characteristic information, including: For the sample audio data Y i The energy characteristic information is filtered to obtain the sample audio data Y i The effective energy characteristic information; For the sample audio data Y i The effective energy characteristic information is discretized to obtain the sample audio data Y i Discrete energy feature information.

5. The method according to claim 1, characterized in that The sample audio data Y i The discrete energy feature information is residually encoded to obtain the sample audio data Y i The timbre feature encoding value includes: For the sample audio data Y i The discrete energy feature information is residually encoded to obtain the sample audio data Y i The candidate timbre feature encoding value of ; According to the sample audio data Y i The object information associated with the sample audio data Y i The candidate timbre feature coding value is compensated to obtain the sample audio data Y i The compensation timbre characteristic coding value of The sample audio data Y i The compensated timbre characteristic encoding value is determined as the sample audio data Y i The timbre feature encoding value of .

6. The method according to claim 1, characterized in that The relationship feature encoding value is obtained by using an attention encoder to encode the sample audio data Y i The discrete energy feature information is obtained by performing attention encoding, and the method further includes: The sample audio data Y is transformed into i The relational feature encoding value is passed to the object classification layer; The object classification layer is used to classify the sample audio data Y i The relationship feature encoding value is identified to obtain the sample object P n Category of; According to the sample object P n The category determines the gradient error of the attention encoder; The gradient error of the attention encoder is weighted by a reverse coefficient to obtain an encoding error of the attention encoder, and the attention encoder is adjusted according to the encoding error.

7. The method according to any one of claims 1 to 6, characterized in that: The sample audio data Y is used i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain the target audio synthesis model, including: According to the sample audio data Y i Associated object information, obtain the sample object P n The timbre characteristic information; The sample audio data Y is synthesized using a candidate audio synthesis model i The sample audio feature information and the timbre feature information of each sample object are predicted to obtain the sample object P n The predicted audio data; According to the sample object P n The predicted audio data and the sample audio data Y i , adjusting the candidate audio synthesis model to obtain an adjusted candidate audio synthesis model; The adjusted candidate audio synthesis model is determined as the target audio synthesis model.

8. The method according to claim 7, characterized in that According to the sample audio data Y i Associated object information, obtain the sample object P n The timbre characteristics of the According to the sample audio data Y i The associated object information is determined to belong to the sample object P n Sample audio data of Belongs to the sample object P n Extracting a sample audio segment from the sample audio data; The timbre feature of the sample audio clip is extracted to obtain the sample object P n The timbre characteristic information of 9. The method according to claim 7, characterized in that The candidate audio synthesis model is used to synthesize the sample audio data Y i The sample audio feature information and the timbre feature information of each sample object are predicted to obtain the sample object P n The predicted audio data includes: According to the sample audio data Y i The sample audio feature information is used to determine the sample audio data Y i The frame length of the pronunciation unit; The sample audio data Y is synthesized using a candidate audio synthesis model i The sample audio feature information, the sample audio data Y i The frame length of the pronunciation unit and the sample object P n The timbre feature information is predicted to obtain the sample object P n The predicted audio data.

10. The method according to claim 7, characterized in that The sample object P n The predicted audio data and the sample audio data Y i , adjusting the candidate audio synthesis model to obtain an adjusted candidate audio synthesis model, including: Get the sample object P n The predicted audio data and sample audio data Y i The similarity between Determining a prediction error of the candidate audio synthesis model according to the similarity; If the prediction error of the candidate audio synthesis model is not in a converged state, the candidate audio synthesis model is adjusted according to the prediction error of the candidate audio synthesis model to obtain an adjusted candidate audio synthesis model.

11. The method according to claim 1, characterized in that The method further comprises: Acquire a reference audio clip of the target object, as well as text feature information and music score feature information of the target audio data to be synthesized; generating timbre characteristic information of the target object according to a reference audio segment of the target object; Normalizing the text feature information and the music score feature information of the target audio data to obtain audio feature information of the target audio data; The target audio synthesis model is used to synthesize the audio feature information of the target audio data and the timbre feature information of the target object to obtain the target audio data belonging to the target object.

12. An audio data processing device, characterized in that: include: an acquisition module, used to acquire at least two sample audio data and sample audio data Y of the at least two sample audio data i Associated object information; the sample audio data Y i The associated object information is used to indicate the sample audio data Y i The sample object P n , the at least two sample audio data belong to at least two sample objects, i is less than or equal to M, M is the number of sample audio data in the at least two sample audio data, n is less than or equal to Q, Q is the number of objects in the at least two sample objects; Extraction module, used for the sample audio data Y i Perform feature extraction to obtain the sample audio data Y i Text feature information and music score feature information; A processing module is used to process the sample audio data Y i The text feature information and the music score feature information are used to determine the sample audio data Y i Discrete energy characteristic information; For the sample audio data Y i The discrete energy feature information is residually encoded to obtain the sample audio data Y i The timbre feature encoding value of For the sample audio data Y i The discrete energy feature information is encoded with attention to obtain the information used to reflect the sample audio data Y i The relational feature encoding value of the contextual relationship; The sample audio data Y i The timbre feature coding value and the relationship feature coding value are concatenated to obtain the sample audio data Y i The audio code value of the sample audio data Y i The audio code value is determined as the sample audio data Y i Sample audio feature information; An adjustment module is used to adopt the sample audio data Y i The sample audio feature information and the sample audio data Y i The candidate audio synthesis model is adjusted based on the associated object information to obtain a target audio synthesis model; the target audio synthesis model is used to synthesize the target audio data of the target object.

13. A computer device, characterized in that: include: Processor and memory; The processor is connected to a memory; the memory is used to store program codes, and the processor is used to call the program codes to execute the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 11.

Images