Timbre extraction, model training method and device, equipment, medium and program

Abstract

The embodiment of the application discloses a timbre extraction method and device, model training method and device, equipment, medium and program, and belongs to the field of audio and video. The method comprises the following steps: inputting M initial audios into a first timbre extraction model to obtain first timbre features. The first timbre features and N first media information are input into a first audio synthesis model to obtain N first synthesized audios. If the N first synthesized audios meet a first convergence condition, the first timbre features are determined as the timbre features of a first object. The first timbre extraction model is obtained by adjusting a second timbre extraction model according to evaluation results of a user on N second synthesized audios. The embodiment of the application adjusts the timbre extraction model according to the evaluation results of the user, which is equivalent to customizing the timbre extraction model meeting the evaluation standard of the user, and ensures that the finally determined timbre features meet the actual needs of the user.

Description

Technical Field

[0001] This application relates to the field of audio and video technology, and in particular to a method, apparatus, device, medium and program for timbre extraction and model training. Background Technology

[0002] Timbre extraction technology is a technique that processes audio using computers and signal processing methods to obtain timbre characteristics. These timbre characteristics can be applied to subsequent short video production, synthesized musical instruments, intelligent robot dialogue, pet alarm clocks, and pet humming, among other fields.

[0003] In related technologies, when extracting timbre from audio, the audio is typically used as input to a timbre extraction model to obtain the timbre features output by the model. Then, these timbre features, along with media information such as lyrics, text, and sheet music to be synthesized, are used as input to an audio synthesis model to obtain the synthesized audio output by the model.

[0004] However, for audio in low-resource scenarios such as animal audio and musical instrument audio, there is only a small amount of audio data and a lack of corresponding text information. Therefore, the timbre features extracted by the timbre extraction model are difficult to meet the actual needs of users. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and program for timbre extraction and model training, which can solve the problem that the timbre features extracted by related technologies are difficult to meet the actual needs of users. The technical solution is as follows:

[0006] Firstly, a timbre extraction method is provided. In this method, M initial audio samples are input into a first timbre extraction model to obtain a first timbre feature. The initial audio samples are acquired from a first object, which is an object whose speech is unrecognizable. M is an integer greater than or equal to 1. The first timbre feature and N first media information are input into a first audio synthesis model to obtain N first synthesized audio samples, where N is an integer greater than or equal to 1. If the N first synthesized audio samples satisfy a first convergence condition, then the first timbre feature is determined as the timbre feature of the first object.

[0007] The first timbre extraction model is obtained by adjusting the second timbre extraction model based on user evaluation results of N second synthesized audios. The N second synthesized audios are obtained by synthesizing the second timbre features with the N first media information respectively. The second timbre features are obtained by inputting the M initial audios into the second timbre extraction model.

[0008] Since M is an integer greater than or equal to 1, the user terminal can input one initial audio file into the first timbre extraction model to obtain the first timbre feature output by the model, or it can input multiple initial audio files into the first timbre extraction model to obtain the same first timbre feature. In other words, the user terminal can extract the first timbre feature from one initial audio file or from multiple initial audio files. Typically, to ensure a more accurate extraction of the first timbre feature, the user terminal inputs multiple initial audio files into the first timbre extraction model. That is, M is an integer greater than 1, for example, M is any value between 3 and 10.

[0009] It should be noted that the first object is an object whose speech cannot be recognized, such as a pet or a musical instrument, which are objects whose pronunciation cannot be identified based on their sounds. The network structure of the first timbre extraction model is a neural network; of course, it may also be other network structures, which are not limited in this embodiment.

[0010] In some embodiments, the user's evaluation results for the N second synthesized audios include the quality evaluation results corresponding to each of the N second synthesized audios; alternatively, the user's evaluation results for the N second synthesized audios may include not only the quality evaluation results corresponding to each of the N second synthesized audios but also the category evaluation results corresponding to each of the N second synthesized audios. Since the user's evaluation results for the N second synthesized audios differ, the way the user terminal adjusts the second timbre extraction model also differs. Therefore, the following will describe two separate cases.

[0011] The first scenario, The user's evaluation results for the N second synthesized audios include the quality evaluation results corresponding to each of the N second synthesized audios. The first timbre extraction model is obtained by adjusting the second timbre extraction model based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios, the N third synthesized audios, and the N second synthesized audios. The N second synthesized audios are obtained by the user terminal inputting the second timbre features and N first media information into the second audio synthesis model. The quality prediction results corresponding to the N third synthesized audios and the N second synthesized audios are obtained by the user terminal inputting the N second synthesized audios into the second timbre extraction model to obtain N third timbre features, and then inputting these N third timbre features into the second audio synthesis model.

[0012] In other words, the user terminal inputs M initial audio samples into the second timbre extraction model to obtain second timbre features. The user terminal then inputs the second timbre features and N pieces of first media information into the second audio synthesis model to obtain N second synthesized audio samples. The user terminal displays an audio quality evaluation interface, which includes the N second synthesized audio samples, and obtains the user's quality evaluation result for these N second synthesized audio samples based on this interface. Next, the user terminal inputs the N second synthesized audio samples into the second timbre extraction model to obtain N third timbre features corresponding to each of the N second synthesized audio samples. These N third timbre features are then input into the second audio synthesis model to obtain N third synthesized audio samples corresponding to each of the N third timbre features, as well as quality prediction results corresponding to each of the N second synthesized audio samples. Based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audio samples, the N third synthesized audio samples, and the N second synthesized audio samples, the user terminal adjusts the second timbre extraction model to obtain the first timbre extraction model.

[0013] It should be noted that the above describes inputting N second-synthesized audio samples into a second timbre extraction model to obtain N third-synthesized timbre features, and then inputting these N third-synthesized timbre features into a second audio synthesis model to obtain N third-synthesized audio samples, along with the quality prediction results corresponding to each of the N second-synthesized audio samples. In practical applications, the second timbre extraction model and the second audio synthesis model can also be treated as a whole. In this case, the N second-synthesized audio samples can be directly input into this unified model to obtain N third-synthesized audio samples, along with the quality prediction results corresponding to each of the N second-synthesized audio samples.

[0014] Both the quality prediction results and the quality evaluation results include quality results in K dimensions. The quality results in K dimensions include audio quality scores and / or audio pair quality comparison results. The audio pair quality comparison results are determined by comparing the quality between two audios in the audio pair containing the corresponding synthesized audio. The audio pair includes the corresponding synthesized audio and another audio. K is an integer greater than or equal to 1.

[0015] The second scenario,The user's evaluation results for the N second synthesized audios include not only the quality evaluation results corresponding to the N second synthesized audios, but also the category evaluation results corresponding to the N second synthesized audios. The second audio synthesis model, given the input of the second timbre features and N first media information, also receives N reference category information corresponding one-to-one with the N first media information. The first timbre extraction model is obtained by adjusting the second timbre extraction model based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios, N third synthesized audios, and N second synthesized audios, as well as the N reference category information and the category evaluation results corresponding to the N second synthesized audios.

[0016] In other words, the user terminal inputs M initial audio samples into the second timbre extraction model to obtain second timbre features. The user terminal then inputs the second timbre features, N pieces of first media information, and N reference category information corresponding to each of the N pieces of first media information into the second audio synthesis model to obtain N second synthesized audio samples. The user terminal displays an audio quality evaluation interface, which includes the N second synthesized audio samples. Based on this interface, the user obtains the quality evaluation results and category evaluation results for the N second synthesized audio samples. Next, the user terminal inputs the N second synthesized audio samples into the second timbre extraction model to obtain N third timbre features corresponding to each of the N second synthesized audio samples. These N third timbre features are then input into the second audio synthesis model to obtain N third synthesized audio samples corresponding to each of the N third timbre features, as well as the quality prediction results corresponding to each of the N second synthesized audio samples. The user terminal adjusts the second timbre extraction model based on the quality prediction results and quality evaluation results corresponding to N second synthesized audios, N third synthesized audios, and N second synthesized audios, as well as the N reference category information and the category evaluation results corresponding to the N second synthesized audios, to obtain the first timbre extraction model.

[0017] Both the reference category information and the category evaluation results include L-dimensional categories, which include audio emotion categories and / or audio scene categories, where L is an integer greater than or equal to 1.

[0018] In this scenario, the N second-synthesized audio tracks already contain audio emotion categories and / or audio scene categories. The audio quality evaluation interface also includes category evaluation boxes for these N second-synthesized audio tracks. While the user's terminal plays the corresponding second-synthesized audio track, the user can determine the category evaluation result for that track and enter the corresponding category evaluation result in the category evaluation box. That is, the user enters the audio emotion category and / or audio scene category corresponding to that second-synthesized audio track in the category evaluation box.

[0019] The audio emotion categories include happiness, sadness, and crying. The audio scene categories include unfamiliar environments, touching, and feeding.

[0020] In some embodiments, the first audio synthesis model and the second audio synthesis model in the above two cases are the same model, or the first audio synthesis model and the second audio synthesis model are different models. Similar to the first timbre extraction model, in the first case, the first audio synthesis model is obtained by adjusting the second audio synthesis model based on the quality prediction results and quality evaluation results corresponding to N second synthesized audios, N third synthesized audios, and N second synthesized audios respectively. In the second case, the first audio synthesis model is obtained by adjusting the second audio synthesis model based on the quality prediction results and quality evaluation results corresponding to N second synthesized audios, N third synthesized audios, and N second synthesized audios respectively, as well as N reference category information and the category evaluation results corresponding to the N second synthesized audios respectively.

[0021] The first convergence condition is met for the N first synthesized audio files, including: the average audio quality score of the N first synthesized audio files reaches a score threshold; or, the user terminal detects a user-triggered stop operation during the playback of the N first synthesized audio files. That is, after the user terminal obtains the user's audio quality scores for the N first synthesized audio files, it determines the average audio quality score of the N first synthesized audio files. When this average score reaches the score threshold, the N first synthesized audio files are determined to meet the first convergence condition. Alternatively, during the playback of the N first synthesized audio files through the audio quality evaluation interface on the user terminal, if a user-triggered stop operation is detected, the N first synthesized audio files are determined to meet the first convergence condition, meaning the user can manually stop the aforementioned loop process.

[0022] If the N first synthesized audio samples do not meet the first convergence condition, the user terminal obtains the user's evaluation results on the N first synthesized audio samples. Based on the user's evaluation results on the N first synthesized audio samples, the first timbre extraction model is adjusted to obtain the third timbre extraction model. If the third timbre extraction model meets the second convergence condition, M initial audio samples are input into the third timbre extraction model to obtain the timbre features output by the third timbre extraction model. The timbre features output by the third timbre extraction model are determined as the timbre features of the first object.

[0023] The third timbre extraction model satisfies the second convergence condition by having an iteration count greater than or equal to an iteration count threshold. This iteration count threshold can be pre-set; it can be a specified number of iterations or a maximum number of iterations, and can be set according to different needs. This embodiment of the application does not limit this setting.

[0024] Similar to the evaluation results of the N second synthesized audios mentioned above, the user's evaluation results of the N first synthesized audios include the quality evaluation results corresponding to each of the N first synthesized audios. Alternatively, the user's evaluation results of the N first synthesized audios may include not only the quality evaluation results corresponding to each of the N first synthesized audios but also the category evaluation results corresponding to each of the N first synthesized audios. The way the user terminal adjusts the first timbre extraction model differs depending on the user's evaluation results of the N first synthesized audios. Therefore, the following two scenarios will be explained separately.

[0025] In the first scenario, the user's evaluation results for N first-synthesized audio files include the quality evaluation results corresponding to each of the N first-synthesized audio files. The user terminal inputs the N first-synthesized audio files into a first timbre extraction model to obtain N sixth-timbre features. These N sixth-timbre features are then input into a first audio synthesis model to obtain N fifth-synthesized audio files and the quality prediction results corresponding to the N first-synthesized audio files. Based on the quality prediction results and quality evaluation results corresponding to the N first-synthesized audio files, the N fifth-synthesized audio files, and the N first-synthesized audio files, the first timbre extraction model is adjusted to obtain a third timbre extraction model.

[0026] The quality prediction and quality evaluation results corresponding to the N first synthesized audios are similar to those corresponding to the N second synthesized audios. Furthermore, the process of adjusting the first timbre extraction model by the user terminal is similar to the process of adjusting the second timbre extraction model. Please refer to the previous description for details, which will not be repeated here.

[0027] In the second scenario, the user's evaluation of the N first synthesized audio files includes not only the quality evaluation results corresponding to each of the N first synthesized audio files, but also the category evaluation results corresponding to each of the N first synthesized audio files. The first audio synthesis model, given the input of first timbre features and N first media information, also receives N reference category information corresponding one-to-one with the N first media information. In this case, the user terminal inputs the N first synthesized audio files into the first timbre extraction model to obtain N sixth timbre features. These N sixth timbre features are then input into the first audio synthesis model to obtain N fifth synthesized audio files and the quality prediction results corresponding to each of the N first synthesized audio files. Based on the quality prediction results and quality evaluation results corresponding to the N first synthesized audio files, the N fifth synthesized audio files, and the N first synthesized audio files, as well as the N reference category information and the category evaluation results corresponding to each of the N first synthesized audio files, the first timbre extraction model is adjusted to obtain the third timbre extraction model.

[0028] The category evaluation results corresponding to the N first synthesized audios are similar to those corresponding to the N second synthesized audios. Furthermore, the process of adjusting the first timbre extraction model by the user terminal is similar to the process of adjusting the second timbre extraction model. Please refer to the previous description for details, which will not be repeated here.

[0029] It should be noted that the first and second convergence conditions are not limited to the conditions described above. The first and second convergence conditions may also be other conditions.

[0030] If the N first synthesized audio samples satisfy the first convergence condition, it indicates that the reliability of the first timbre feature obtained based on the first timbre extraction model is high. Therefore, the first timbre feature is directly determined as the timbre feature of the first object. If the third timbre extraction model satisfies the second convergence condition, it indicates that the third timbre extraction model has undergone a large number of iterations, and the timbre feature extracted by the third timbre extraction model is relatively reliable. To improve efficiency, the timbre feature extracted by the third timbre extraction model is directly determined as the timbre feature of the first object. If the N first synthesized audio samples do not satisfy the first convergence condition and the third timbre extraction model does not satisfy the second convergence condition, the first audio synthesis model is readjusted according to the above method to obtain the third audio synthesis model. Then, the above method is re-executed using the third timbre extraction model and the third audio synthesis model.

[0031] As can be seen from the above method, the embodiments of this application determine the timbre features of the first object by adjusting the timbre extraction model and audio synthesis model stored in the user terminal through a cyclic iterative process. Furthermore, if N first synthesized audio samples fail to meet the first convergence condition and the third timbre extraction model fails to meet the second convergence condition, the cyclic iterative process continues. Thus, for audio in low-resource scenarios such as pet audio and musical instrument audio, the user's subjective preferences are incorporated to ensure that the final extracted timbre features meet the user's needs.

[0032] It should be noted that the timbre extraction model and audio synthesis model initially stored on the user terminal are synchronized with the server. However, for audio in low-resource scenarios, a large amount of data needs to be manually labeled, and the cost of sample labeling is high. Therefore, the user terminal can also send N second-synthesized audios and the user's evaluation results on the N second-synthesized audios to the server, so that the server can use the received N second-synthesized audios and the user's evaluation results on the N second-synthesized audios as sample data to train the timbre extraction model and audio synthesis model.

[0033] Based on the above description, the timbre extraction method provided in this application embodiment can be applied to various scenarios, taking short video production as an example. After the user terminal determines the first timbre feature as the timbre feature of the first object, it can also determine the synthesized audio required for short video production. That is, M initial audios and M initial videos are input into the audio-visual feature extraction model to obtain the first audio-visual feature, where the M initial videos correspond one-to-one with the M initial audios. Based on the first audio-visual feature, a fourth timbre feature and second media information are obtained from the database. The fourth timbre feature is the timbre feature corresponding to the second object, which is different from the first object. The first timbre feature and the fourth timbre feature are fused to obtain the fifth timbre feature. The fifth timbre feature and the second media information are input into the first audio synthesis model to obtain the fourth synthesized audio. The fourth synthesized audio is the synthesized audio required for short video production.

[0034] The database stores the correspondence between audio / video features, timbre features, and media information. In some embodiments, the user terminal retrieves the fourth timbre feature and the second media information from the database in the following manner: The user terminal retrieves audio / video features that match the first audio / video feature from the correspondence between the audio / video features, timbre features, and media information to obtain one or more candidate audio / video features. Based on the one or more candidate audio / video features, the second audio / video feature is determined. The user terminal identifies the timbre feature corresponding to the second audio / video feature in the correspondence as the fourth timbre feature and the media information corresponding to the second audio / video feature in the correspondence as the second media information.

[0035] There are several ways for the user terminal to fuse the first timbre feature and the fourth timbre feature. For example, the user terminal displays a feature fusion interface, obtains the fusion ratio of the first timbre feature and the fourth timbre feature through this interface, and fuses the first timbre feature and the fourth timbre feature based on this fusion ratio to obtain the fifth timbre feature.

[0036] After determining the first timbre feature as the timbre feature of the first object, it indicates that the reliability of the output result of the first audio synthesis model is relatively high. Therefore, the user terminal directly inputs the fifth timbre feature and the second media information into the first audio synthesis model to obtain the fourth synthesized audio output by the first audio synthesis model. At this time, the quality of the fourth synthesized audio is also relatively good.

[0037] Secondly, a model training method is provided. In this method, T sample audios and their corresponding sample annotation results are obtained, where T is an integer greater than or equal to 1. Based on the T sample audios and their corresponding sample annotation results, an initial timbre extraction model and an initial audio synthesis model are jointly trained to obtain a trained timbre extraction model and a trained audio synthesis model.

[0038] The sample audio includes audio from low-resource scenarios such as pet audio, animal audio, and musical instrument audio. The annotation results for the T sample audios include the quality annotation results for each of the T sample audios, or, the annotation results for the T sample audios include not only the quality annotation results for each of the T sample audios, but also the category annotation results for each of the T sample audios.

[0039] When the sample annotation results include quality annotation results, the quality annotation results include K dimensions of quality results. These K dimensions of quality results include the audio quality score of the sample audio and / or the audio pair quality comparison results. The audio pair quality comparison results are determined by comparing the quality of two sample audios in a sample audio pair to which the corresponding sample audio belongs. A sample audio pair includes the corresponding sample audio and one other audio. When the sample annotation results include category annotation results, the category annotation results include L dimensions of categories. These L dimensions of categories include the audio sentiment category and / or audio scene category of the sample audio.

[0040] Based on the above description, the sample annotation results include quality annotation results, or, in addition to quality annotation results, also include category annotation results. When the sample annotation results differ, the server will perform different joint training methods on the initial timbre extraction model and the initial audio synthesis model based on the sample annotation results corresponding to the T sample audio samples and the T sample audio samples respectively. Therefore, the following will describe these two cases separately.

[0041] In the first case, the sample labeling results include the quality labeling results.

[0042] In this case, the server jointly trains the initial timbre extraction model and the initial audio synthesis model using the following two implementation methods.

[0043] In the first implementation, the server inputs T sample audios into an initial timbre extraction model to obtain T first-sample timbre features. These T first-sample timbre features are then input into an initial audio synthesis model to obtain T first-sample synthesized audios, with each of the T first-sample synthesized audios corresponding one-to-one with the T sample audios. The server jointly trains the initial timbre extraction model and the initial audio synthesis model based on the T sample audios and the T first-sample synthesized audios. The server uses the converged initial timbre extraction model as the trained timbre extraction model and modifies the network structure of the converged initial audio synthesis model to obtain a modified audio synthesis model. Then, the server inputs the T sample audios into the trained timbre extraction model to obtain T second-sample timbre features. These T second-sample timbre features are then input into the modified audio synthesis model to obtain the quality prediction results corresponding to the T second-sample synthesized audios and the T sample audios, respectively. The server trains the modified audio synthesis model based on T sample audios, T second sample synthesized audios, and the quality prediction and quality labeling results corresponding to the T sample audios, to obtain the trained audio synthesis model.

[0044] In pitch-based scenarios such as humming, during the process of inputting T first-sample timbre features into the initial audio synthesis model, pitch features can also be input into the initial audio synthesis model. That is, by inputting T first-sample timbre features and pitch features into the initial audio synthesis model, T first-sample synthesized audio samples are obtained. Here, the pitch feature is either randomly selected by the server or selected during manual annotation.

[0045] In the first implementation, the initial audio synthesis model does not have quality prediction capabilities; that is, its network structure does not include a quality prediction branch. Therefore, when jointly training the initial timbre extraction model and the initial audio synthesis model, it is only necessary to determine the first sample loss value based on T sample audios and T first sample synthesized audios, and then adjust the initial timbre extraction model and the initial audio synthesis model based on the first sample loss value. After the initial timbre extraction model and the initial audio synthesis model converge, the server modifies the network structure of the converged audio synthesis model to add a quality prediction branch, meaning the modified audio synthesis model now possesses quality prediction capabilities.

[0046] In the second implementation, the server inputs T sample audios into an initial timbre extraction model to obtain T first sample timbre features. These T first sample timbre features are then input into an initial audio synthesis model to obtain T first sample synthesized audios and the corresponding quality prediction results for the T sample audios. Based on the T sample audios, the T first sample synthesized audios, and the corresponding quality prediction and quality annotation results, the server jointly trains the initial timbre extraction model and the initial audio synthesis model.

[0047] In the second implementation, the initial audio synthesis model has the function of quality prediction. That is, the network structure of the initial audio synthesis model includes a quality prediction branch. Therefore, after the server inputs the timbre features of the first sample into the initial audio synthesis model, it can obtain the quality prediction results corresponding to the T first sample synthesized audio and the T sample audio respectively.

[0048] In the second scenario, the sample labeling results include not only quality labeling results but also category labeling results.

[0049] In this case, the server jointly trains the initial timbre extraction model and the initial audio synthesis model using the following two implementation methods.

[0050] In the first implementation, the server inputs T sample audios into an initial timbre extraction model to obtain T first-sample timbre features. These T first-sample timbre features are then input into an initial audio synthesis model to obtain T first-sample synthesized audios, which correspond one-to-one with the T sample audios. The server jointly trains the initial timbre extraction model and the initial audio synthesis model based on the T sample audios and the T first-sample synthesized audios. The server uses the converged initial timbre extraction model as the trained timbre extraction model and modifies the network structure of the converged initial audio synthesis model to obtain a modified audio synthesis model. Then, the server inputs the T sample audios into the trained timbre extraction model to obtain T second-sample timbre features. The server inputs the category labeling results corresponding to the T second-sample timbre features and the T sample audios into the modified audio synthesis model to obtain the quality prediction results and category prediction results corresponding to the T second-sample synthesized audios and the T sample audios, respectively. The server trains the modified audio synthesis model based on the quality prediction and quality labeling results of T sample audios, T second sample synthesized audios, and T sample audios respectively, as well as the category prediction and category labeling results of T sample audios respectively, to obtain the trained audio synthesis model.

[0051] In the first implementation, the initial audio synthesis model lacks quality and category prediction capabilities. That is, its network structure does not include branches for quality and category prediction. Therefore, when jointly training the initial timbre extraction model and the initial audio synthesis model, it is only necessary to determine the first sample loss value based on T sample audio files and T first sample synthesized audio files, and then adjust the initial timbre extraction model and the initial audio synthesis model based on this first sample loss value. After the initial timbre extraction model and the initial audio synthesis model converge, the server modifies the network structure of the converged audio synthesis model to add branches for quality and category prediction. In other words, the modified audio synthesis model possesses both quality and category prediction capabilities.

[0052] In the second implementation, the server inputs T sample audios into an initial timbre extraction model to obtain T first sample timbre features. The server then inputs the T first sample timbre features and the corresponding category labels of the T sample audios into an initial audio synthesis model to obtain T first sample synthesized audios, as well as the corresponding quality prediction and category prediction results for the T sample audios. Based on the T sample audios, the T first sample synthesized audios, the corresponding quality prediction and quality labeling results, and the corresponding category prediction and category labeling results, the server jointly trains the initial timbre extraction model and the initial audio synthesis model to obtain a trained timbre extraction model and a trained audio synthesis model.

[0053] In the second implementation, the initial audio synthesis model has the functions of quality prediction and category prediction. That is, the network structure of the initial audio synthesis model includes a quality prediction branch and a category prediction branch. Therefore, after the server inputs the timbre features of the T first samples and the category labeling results corresponding to the T sample audios into the initial audio synthesis model, it can obtain the synthesized audios of the T first samples, as well as the quality prediction results and category prediction results corresponding to the T sample audios.

[0054] It should be noted that the server can also receive N second-synthesized audio files sent by the user terminal, as well as the user's evaluation results for these N second-synthesized audio files. The server uses these N second-synthesized audio files as N sample audio files and the user's evaluation results as sample annotation results for these N sample audio files. Based on these N second-synthesized audio files and the user's evaluation results, the server adjusts the trained timbre extraction model and audio synthesis model. The method by which the server adjusts the trained timbre extraction model and audio synthesis model based on the N second-synthesized audio files and the user's evaluation results is the same as the method described above for adjusting the initial timbre extraction model and initial audio synthesis model based on T sample audio files and their sample annotation results; therefore, it will not be repeated here.

[0055] Thirdly, a timbre extraction device is provided, which has the function of implementing the timbre extraction method described in the first aspect above. The device includes at least one module for implementing the timbre extraction method provided in the first aspect above.

[0056] Fourthly, a model training apparatus is provided, the apparatus having the function of implementing the model training method behavior described in the second aspect above. The apparatus includes at least one module for implementing the model training method provided in the first aspect above.

[0057] Fifthly, a computer device is provided, comprising a processor and a memory, the memory being used to store a computer program for executing the timbre extraction method provided in the first aspect. The processor is configured to execute the computer program stored in the memory to implement the timbre extraction method described in the first aspect.

[0058] Optionally, the computer device may further include a communication bus for establishing a connection between the processor and the memory.

[0059] In a sixth aspect, a computer device is provided, the computer device including a processor and a memory, the memory being used to store a computer program for executing the model training method provided in the second aspect above. The processor is configured to execute the computer program stored in the memory to implement the model training method described in the second aspect above.

[0060] Optionally, the computer device may further include a communication bus for establishing a connection between the processor and the memory.

[0061] In a seventh aspect, a computer-readable storage medium is provided, wherein the storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the timbre extraction method described in the first aspect or the steps of the model training method described in the second aspect.

[0062] Eighthly, a computer program product comprising instructions is provided, which, when executed on a computer, causes the computer to perform the steps of the timbre extraction method described in the first aspect or the steps of the model training method described in the second aspect. Alternatively, a computer program is provided that, when executed on a computer, causes the computer to perform the steps of the timbre extraction method described in the first aspect or the steps of the model training method described in the second aspect.

[0063] The technical effects achieved by the third to eighth aspects mentioned above are similar to those achieved by the corresponding technical means in the first or second aspects, and will not be repeated here.

[0064] The technical solutions provided in this application can bring at least the following beneficial effects:

[0065] In this embodiment of the application, during the process of determining the timbre features of the first object, the timbre extraction model is adjusted based on the user's evaluation results of the synthesized audio. This is equivalent to customizing a timbre extraction model that meets the user's own evaluation standards and wishes for each user. In this way, it can be ensured that the timbre features of the first object determined in the end meet the user's actual needs. That is, in low-resource scenarios such as pet audio, animal audio, and musical instrument audio, adjusting the timbre extraction model based on the user's evaluation results of the synthesized audio can ensure that the extracted timbre features meet the user's actual needs. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0067] Figure 2 This is a schematic diagram of the structure of a user terminal provided in an embodiment of this application;

[0068] Figure 3 This is a flowchart of a timbre extraction method provided in an embodiment of this application;

[0069] Figure 4 This is a schematic diagram illustrating the adjustment of a second timbre extraction model according to an embodiment of this application;

[0070] Figure 5 This is a schematic diagram illustrating another adjustment to the second timbre extraction model provided in an embodiment of this application;

[0071] Figure 6 This is a flowchart of a model training method provided in an embodiment of this application;

[0072] Figure 7 This is a schematic diagram illustrating the joint training of an initial timbre extraction model and an initial audio synthesis model, provided in an embodiment of this application.

[0073] Figure 8 This is a schematic diagram illustrating the training of a modified audio synthesis model according to an embodiment of this application;

[0074] Figure 9 This is a schematic diagram illustrating another method for training the modified audio synthesis model, as provided in an embodiment of this application.

[0075] Figure 10 This is a schematic diagram of the structure of a timbre extraction device provided in an embodiment of this application;

[0076] Figure 11 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application;

[0077] Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0078] Figure 13 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0079] Figure 14 This is a schematic diagram of the structure of another terminal device provided in an embodiment of this application. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0081] Before providing a detailed explanation of the timbre extraction method provided in the embodiments of this application, the application scenarios and implementation environment provided in the embodiments of this application will be introduced first.

[0082] The timbre extraction method provided in this application can be applied to various scenarios, such as short video production, pet humming, pet music composition, pet alarm clock, and smart assistant.

[0083] One method is to use a pet's voice to dub a video. Specifically, the process involves extracting the pet's vocal characteristics from its audio, then inputting these characteristics, along with lyrics, text, and sheet music, into an audio synthesis model to produce synthesized audio. This synthesized audio is then used to dub a video, thus creating a short video.

[0084] Pet humming utilizes pet audio to extract the pet's vocal characteristics. These characteristics, along with lyrics, text, and sheet music, are then input into an audio synthesis model to produce synthesized audio, achieving the effect of humming a song using the pet's vocal timbre. Optionally, in pet humming scenarios, the vocal characteristics of musical instruments can also be extracted from instrument audio and incorporated into the synthesized audio process to enhance the humming effect.

[0085] Similar scenarios include pet music composition, pet alarm clocks, and smart assistants, which also require extracting the pet's vocal characteristics from its audio data and then implementing corresponding functions based on those characteristics.

[0086] However, in low-resource scenarios such as pet audio, animal audio, and musical instrument audio, there is only a small amount of audio data and a lack of corresponding text information. After extracting the timbre from the audio in low-resource scenarios using traditional timbre extraction methods, the resulting timbre features may not meet the actual needs of users. Therefore, this application provides a timbre extraction method. The timbre features extracted according to the timbre extraction method provided in this application can meet the actual needs of users.

[0087] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an implementation environment according to an embodiment of this application. The implementation environment includes a user terminal 101 and a server 102. The user terminal 101 and the server 102 are communicatively connected. This communication connection can be wired or wireless, and this embodiment of the application does not limit this.

[0088] Server 102 jointly trains the initial timbre extraction model and the initial audio synthesis model to obtain trained timbre extraction models and trained audio synthesis models. Then, server 102 synchronizes the trained timbre extraction model and trained audio synthesis model to user terminal 101.

[0089] After the user terminal 101 obtains the trained timbre extraction model and the trained audio synthesis model, the user terminal 101 uses the trained timbre extraction model to extract the timbre features of the initial audio, and uses the trained audio synthesis model to synthesize the extracted timbre features and media information to obtain synthesized audio. Then, the user's evaluation of the synthesized audio is obtained, and the trained timbre extraction model and audio synthesis model are adjusted based on the user's evaluation. The timbre features of the initial audio are then extracted again using the aforementioned method until the timbre features satisfying the user are obtained.

[0090] Optionally, after obtaining the timbre features that satisfy the user, the user terminal 101 can also recommend text, sheet music, other timbre features, etc. to the user for the above-mentioned scenario, allowing the user to select the required information and then fuse it with the timbre features to obtain the audio for the corresponding scenario.

[0091] Optionally, user terminal 101 can also send the synthesized audio and the user's evaluation results on the synthesized audio to server 102. Server 102 retrains the trained timbre extraction model and audio synthesis model based on the synthesized audio and the user's evaluation results on the synthesized audio, and continues to synchronize the trained timbre extraction model and audio synthesis model to user terminal 101 so that user terminal 101 can update its local timbre extraction model and audio synthesis model.

[0092] Please refer to Figure 2 The user terminal 101 includes an input module, a timbre determination module, a recommendation module, and an output module. The input module acquires initial audio and initial video. The timbre determination module extracts timbre features from the initial audio using a timbre extraction model, synthesizes these features and media information using an audio synthesis model, and presents the synthesized audio to the user for evaluation. Based on the user's evaluation, the timbre extraction model is adjusted, and the adjusted model is used to re-extract timbre from the initial audio. This process is repeated multiple times to obtain timbre features that satisfy the user. The recommendation module recommends lyrics, text, sheet music, and other media information to the user based on the initial audio and video. It can also recommend other timbre features for the user to choose from. Furthermore, the recommendation module fuses the timbre features finally determined by the timbre determination module with other timbre features selected by the user to obtain a fused timbre feature. The output module is used to synthesize the fused timbre features with the media information selected by the user to obtain synthesized audio, thereby realizing multiple functions such as pet humming, pet composing music, pet alarm clock, and intelligent robot dialogue.

[0093] Among them, the user terminal 101 is any electronic product that can interact with the user through one or more means such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as personal computer (PC), mobile phone, smartphone, personal digital assistant (PDA), pocket PC (PPC), tablet computer, smart TV, etc.

[0094] Server 102 can be a standalone server, a server cluster or distributed system composed of multiple physical servers, a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, or a cloud computing service center.

[0095] Those skilled in the art should understand that the user terminal 101 and server 102 described above are merely examples. Other existing or future terminals or servers that are applicable to the embodiments of this application should also be included within the scope of protection of the embodiments of this application, and are hereby incorporated by reference.

[0096] It should be noted that the implementation environment and application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of the implementation environment and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0097] The timbre extraction method provided in the embodiments of this application will be explained in detail below.

[0098] Figure 3 This is a flowchart of a timbre extraction method provided in an embodiment of this application. This method is applied in a user terminal; please refer to [the documentation / reference]. Figure 3 The method includes the following steps.

[0099] Step 301: The user terminal inputs M initial audios into the first timbre extraction model to obtain the first timbre features. The initial audios are obtained by collecting the first object, which is an object that cannot be recognized by speech. M is an integer greater than or equal to 1.

[0100] Since M is an integer greater than or equal to 1, the user terminal can input one initial audio file into the first timbre extraction model to obtain the first timbre feature output by the model, or it can input multiple initial audio files into the first timbre extraction model to obtain the same first timbre feature. In other words, the user terminal can extract the first timbre feature from one initial audio file or from multiple initial audio files. Typically, to ensure a more accurate extraction of the first timbre feature, the user terminal inputs multiple initial audio files into the first timbre extraction model. That is, M is an integer greater than 1, for example, M is any value between 3 and 10.

[0101] The initial audio is either the audio currently recorded by the user through the user terminal, or it is an audio file currently input by the user on the user terminal that was pre-recorded or obtained in advance through other means. Of course, the initial audio may also be obtained through other means, and this application embodiment does not limit this.

[0102] As an example, a user terminal displays a first user interface, which includes recording options. In response to a user's first action regarding the recording options, audio is captured from a first object to obtain an initial audio recording. Multiple initial audio recordings can be obtained through repeated iterations. That is, the user terminal displays a first user interface, and the user records initial audio by operating the recording options on the first user interface; multiple initial audio recordings can be obtained after multiple operations.

[0103] The first operation includes actions such as the user clicking the record option, the user long-pressing the record option, etc.

[0104] As another example, the user terminal displays a second user interface including input options. In response to a second user action, a third user interface is displayed, which includes multiple stored audio files. In response to a third user action, the user selects one or more audio files from these multiple audio files as the initial audio input. That is, the user terminal displays a second user interface, and after the user operates on the input options on the second user interface, the user terminal displays a third user interface showing multiple audio files, from which the user can select one or more audio files as the initial audio input.

[0105] The second operation includes actions such as the user clicking on an input option or sliding an input option. The third operation includes actions such as the user clicking on a selected audio file or dragging a selected audio file.

[0106] It should be noted that the first object is an object whose speech cannot be recognized, such as a pet or a musical instrument, which are objects whose pronunciation cannot be identified based on their sounds. The network structure of the first timbre extraction model is a neural network; of course, it may also be other network structures, which are not limited in this embodiment.

[0107] Step 302: The user terminal inputs the first timbre feature and N first media information into the first audio synthesis model to obtain N first synthesized audios, where N is an integer greater than or equal to 1.

[0108] The first audio synthesis model is used to synthesize a first timbre feature with N pieces of first media information to obtain N first synthesized audio files, each corresponding one-to-one with the N pieces of first media information. The network structure of the first audio synthesis model is a neural network; however, other network structures are also possible, such as an encoder-decoder structure. Depending on the network structure of the first audio synthesis model, the way the user terminal inputs the first timbre feature and the N pieces of first media information into the first audio synthesis model will also differ. Furthermore, the first media information can include lyrics, text features, musical scores, pitch features, etc.

[0109] For example, taking the network structure of the first audio synthesis model as an encoder-decoder structure, and the first media information including text features and pitch features as an example, the user terminal inputs the text features included in the N first media information into the encoding layer of the first audio synthesis model, and inputs the first timbre feature into any other encoding layer. Alternatively, the user terminal concatenates the first timbre feature and the text features included in the N first media information, inputs them together into the encoding layer of the first audio synthesis model, and inputs the pitch features included in the N first media information into the decoding layer of the first audio synthesis model.

[0110] It should be noted that the user terminal can concatenate the first timbre feature and the text features included in the N first media information in various ways. For example, the user terminal can directly add the first timbre feature and the text features included in the N first media information, or the user terminal can concatenate the one-dimensional vector containing the first timbre feature and the one-dimensional vector containing the text features included in the N first media information to obtain a single one-dimensional vector. That is, the user terminal can horizontally concatenate the one-dimensional vector containing the first timbre feature and the one-dimensional vector containing the text features included in the N first media information to obtain a single one-dimensional vector.

[0111] There are several ways to determine the N first media information items. For example, a user terminal sends M initial audio files to a server. The server selects N first media information items from a media information library based on the received M initial audio files and recommends them to the user terminal. The user terminal receives and displays these N first media information items. Alternatively, the server sends all media information from its media information library to the user terminal. When the user terminal receives all media information from the server, it displays all media information. When the user terminal detects a user selection operation, it uses the selected media information as the N first media information items. In other words, the user selects N media information items from the media information library as their first media information items based on their actual needs.

[0112] In some embodiments, regardless of whether the user terminal receives N first media messages or all media messages in the media information library, the media messages can be displayed in the form of a pop-up window or a floating window when the user terminal receives the media messages. Of course, other methods can also be used to display the media messages, and this application embodiment does not limit the display method of the media messages.

[0113] Step 303: If N first synthesized audios satisfy the first convergence condition, then the user terminal determines the first timbre feature as the timbre feature of the first object.

[0114] The first timbre extraction model is obtained by adjusting the second timbre extraction model based on the user's evaluation results of N second synthesized audios. The N second synthesized audios are obtained by the user terminal synthesizing the second timbre features with N first media information respectively. The second timbre features are obtained by the user terminal inputting M initial audios into the second timbre extraction model.

[0115] In some embodiments, the user's evaluation results for the N second synthesized audios include the quality evaluation results corresponding to each of the N second synthesized audios; alternatively, the user's evaluation results for the N second synthesized audios may include not only the quality evaluation results corresponding to each of the N second synthesized audios but also the category evaluation results corresponding to each of the N second synthesized audios. Since the user's evaluation results for the N second synthesized audios differ, the way the user terminal adjusts the second timbre extraction model also differs. Therefore, the following will describe two separate cases.

[0116] The first scenario, The user's evaluation results for the N second synthesized audios include the quality evaluation results corresponding to each of the N second synthesized audios. The first timbre extraction model is obtained by adjusting the second timbre extraction model based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios, the N third synthesized audios, and the N second synthesized audios. The N second synthesized audios are obtained by the user terminal inputting the second timbre features and N first media information into the second audio synthesis model. The quality prediction results corresponding to the N third synthesized audios and the N second synthesized audios are obtained by the user terminal inputting the N second synthesized audios into the second timbre extraction model to obtain N third timbre features, and then inputting these N third timbre features into the second audio synthesis model.

[0117] In other words, the user terminal inputs M initial audio samples into the second timbre extraction model to obtain second timbre features. The user terminal then inputs the second timbre features and N pieces of first media information into the second audio synthesis model to obtain N second synthesized audio samples. The user terminal displays an audio quality evaluation interface, which includes the N second synthesized audio samples, and obtains the user's quality evaluation result for these N second synthesized audio samples based on this interface. Next, the user terminal inputs the N second synthesized audio samples into the second timbre extraction model to obtain N third timbre features corresponding to each of the N second synthesized audio samples. These N third timbre features are then input into the second audio synthesis model to obtain N third synthesized audio samples corresponding to each of the N third timbre features, as well as quality prediction results corresponding to each of the N second synthesized audio samples. Based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audio samples, the N third synthesized audio samples, and the N second synthesized audio samples, the user terminal adjusts the second timbre extraction model to obtain the first timbre extraction model.

[0118] It should be noted that the above describes inputting N second-synthesized audio samples into a second timbre extraction model to obtain N third-synthesized timbre features, and then inputting these N third-synthesized timbre features into a second audio synthesis model to obtain N third-synthesized audio samples, along with the quality prediction results corresponding to each of the N second-synthesized audio samples. In practical applications, the second timbre extraction model and the second audio synthesis model can also be treated as a whole. In this case, the N second-synthesized audio samples can be directly input into this unified model to obtain N third-synthesized audio samples, along with the quality prediction results corresponding to each of the N second-synthesized audio samples.

[0119] Both the quality prediction results and the quality evaluation results include quality results in K dimensions. The quality results in K dimensions include audio quality scores and / or audio pair quality comparison results. The audio pair quality comparison results are determined by comparing the quality between two audios in the audio pair containing the corresponding synthesized audio. The audio pair includes the corresponding synthesized audio and another audio. K is an integer greater than or equal to 1.

[0120] With K dimensions of quality results including audio quality scores, the audio quality evaluation interface includes N playback options for the second synthesized audio and a quality rating box. After a user clicks on a playback option for a second synthesized audio, the user's terminal plays the corresponding second synthesized audio. The user can then input the corresponding audio quality rating in the quality rating box, effectively rating the quality of the second synthesized audio.

[0121] With K dimensions of quality results including audio pair quality comparison results, the audio quality evaluation interface includes playback options for two audio items within each of the N second-synthesized audio pairs, as well as a quality comparison box. Users can compare the quality of one second-synthesized audio item and the other audio item within its pair by clicking their playback options, and then input the corresponding audio pair quality comparison result in the audio comparison box.

[0122] As an example, in the audio pair containing the second synthesized audio, each of the two audio tracks corresponds to a quality comparison box. The user can enter a first specific value in the quality comparison box corresponding to each of the two audio tracks in the audio pair to determine the quality comparison result of the two audio tracks. For example, the first specific value is 1 or 0. If the first specific value entered in the quality comparison box corresponding to one of the audio tracks in the audio pair is 1, then that audio track is determined to be the audio track with better quality in the audio pair. If the first specific value entered in the quality comparison box corresponding to one of the audio tracks in the audio pair is 0, then that audio track is determined to be the audio track with poorer quality in the audio pair.

[0123] It should be noted that the first specific value can be represented not only by 1 or 0, but also by other means, and the embodiments of this application do not limit this.

[0124] For any audio pair containing any of the N second-synthesized audio tracks, one audio track is that specific second-synthesized audio track, and the other audio track could be another second-synthesized audio track from the N second-synthesized audio tracks, or it could be another audio track randomly selected by the user terminal from the audio library. In either case, the user terminal can present the audio pair in the audio quality evaluation interface for the user to compare quality. Furthermore, when obtaining the quality prediction results for the N second-synthesized audio tracks, the user terminal needs to input not only the N second-synthesized audio tracks into the second timbre extraction model, but also the other audio tracks in the audio pairs containing the N second-synthesized audio tracks into the second audio synthesis model to obtain the quality comparison results for the audio pairs containing the N second-synthesized audio tracks.

[0125] The user terminal can display the audio quality evaluation interface via a pop-up window or a floating window. Of course, other methods can also be used to display the audio quality evaluation interface; this application embodiment does not limit the display method of the audio quality evaluation interface.

[0126] The process by which the user terminal adjusts the second timbre extraction model based on the quality prediction results and quality evaluation results corresponding to N second synthesized audios, N third synthesized audios, and N second synthesized audios respectively includes: determining a first loss value based on the N second synthesized audios and the N third synthesized audios; determining a second loss value based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios respectively; adding the first loss value and the second loss value to obtain a joint loss value; and adjusting the second timbre extraction model based on the joint loss value to obtain a first timbre extraction model.

[0127] As an example, the user terminal determines the first loss value based on N second synthesized audios and N third synthesized audios according to the following formula (1).

[0128]

[0129] In formula (1) above, Loss1 is the first loss value, and A i For the i-th second synthesized audio among the N second synthesized audios, A' i For the i-th third synthesized audio among the N third synthesized audios, Loss(A) i ,A' i ) represents the loss value between the i-th second synthesized audio and the i-th third synthesized audio.

[0130] The loss value between the second and third synthesized audio can be determined by any loss function, such as the TTS loss function, but this application embodiment does not limit this.

[0131] As an example, when the evaluation results of the second synthesized audio include audio quality scores and audio pair quality comparison results, the user terminal determines the second loss value according to the following formula (2) based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios respectively.

[0132]

[0133] In formula (2) above, Loss2 is the second loss value, and B i B' is the audio quality score in the quality prediction result corresponding to the i-th second synthesized audio among the N second synthesized audios. i Loss(B) is the audio quality score in the quality evaluation result corresponding to the i-th second synthesized audio among the N second synthesized audios. i ,B' i ) represents the loss value between the quality prediction result and the audio quality score in the quality evaluation result corresponding to the i-th second synthesized audio. C iC' represents the quality comparison result of the audio pair in the quality prediction result corresponding to the i-th second synthesized audio among the N second synthesized audios. i For the quality prediction results of the audio pairs corresponding to the i-th second synthesized audio among the N second synthesized audios, Loss(C) is the quality comparison result. i ,C' i Let α be the loss value between the quality prediction result and the audio pair quality comparison result in the quality evaluation result corresponding to the i-th second synthesized audio. α and β are different weights.

[0134] The loss value between audio quality scores can be determined by any loss function, such as the CE loss function. The loss value between audio pair quality comparison results can be determined by any loss function, such as the BCE loss function. This application does not limit this.

[0135] The process by which the user terminal adjusts the second timbre extraction model based on the joint loss value is actually the process of adjusting the network parameters in the second timbre extraction model. That is, the user terminal adjusts the network parameters in the second timbre extraction model based on the joint loss value to obtain the first timbre extraction model. The implementation process of the user terminal adjusting the network parameters of the model based on the loss value can refer to related technologies, and this application embodiment does not limit it in this way.

[0136] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating how a user terminal adjusts a second timbre extraction model, as provided in an embodiment of this application. Figure 4 In the first step, the user terminal inputs M initial audio samples into a second timbre extraction model to obtain second timbre features. These second timbre features, along with N pieces of first media information, are then input into a second audio synthesis model to obtain N second synthesized audio samples. The user evaluates the quality of these N second synthesized audio samples to obtain quality evaluation results, which include quality scores for the second synthesized audio samples and quality comparisons between the audio pairs containing the second synthesized audio samples. Next, the N second synthesized audio samples are input into the second timbre extraction model to obtain N third timbre features. These N third timbre features are then input into the second audio synthesis model to obtain N third synthesized audio samples and corresponding quality prediction results for the N second synthesized audio samples. Finally, based on the quality prediction results and quality evaluation results for the N second synthesized audio samples, the N third synthesized audio samples, and the N second synthesized audio samples, the user terminal adjusts the second timbre extraction model to obtain a first timbre extraction model.

[0137] The second scenario,The user's evaluation results for the N second synthesized audios include not only the quality evaluation results corresponding to the N second synthesized audios, but also the category evaluation results corresponding to the N second synthesized audios. The second audio synthesis model, given the input of the second timbre features and N first media information, also receives N reference category information corresponding one-to-one with the N first media information. The first timbre extraction model is obtained by adjusting the second timbre extraction model based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios, N third synthesized audios, and N second synthesized audios, as well as the N reference category information and the category evaluation results corresponding to the N second synthesized audios.

[0138] In other words, the user terminal inputs M initial audio samples into the second timbre extraction model to obtain second timbre features. The user terminal then inputs the second timbre features, N pieces of first media information, and N reference category information corresponding to each of the N pieces of first media information into the second audio synthesis model to obtain N second synthesized audio samples. The user terminal displays an audio quality evaluation interface, which includes the N second synthesized audio samples. Based on this interface, the user obtains the quality evaluation results and category evaluation results for the N second synthesized audio samples. Next, the user terminal inputs the N second synthesized audio samples into the second timbre extraction model to obtain N third timbre features corresponding to each of the N second synthesized audio samples. These N third timbre features are then input into the second audio synthesis model to obtain N third synthesized audio samples corresponding to each of the N third timbre features, as well as the quality prediction results corresponding to each of the N second synthesized audio samples. The user terminal adjusts the second timbre extraction model based on the quality prediction results and quality evaluation results corresponding to N second synthesized audios, N third synthesized audios, and N second synthesized audios, as well as the N reference category information and the category evaluation results corresponding to the N second synthesized audios, to obtain the first timbre extraction model.

[0139] Both the reference category information and the category evaluation results include L-dimensional categories, which include audio emotion categories and / or audio scene categories, where L is an integer greater than or equal to 1.

[0140] In this scenario, the N second-synthesized audio tracks already contain audio emotion categories and / or audio scene categories. The audio quality evaluation interface also includes category evaluation boxes for these N second-synthesized audio tracks. While the user's terminal plays the corresponding second-synthesized audio track, the user can determine the category evaluation result for that track and enter the corresponding category evaluation result in the category evaluation box. That is, the user enters the audio emotion category and / or audio scene category corresponding to that second-synthesized audio track in the category evaluation box.

[0141] The audio emotion categories include happiness, sadness, and crying. The audio scene categories include unfamiliar environments, touching, and feeding.

[0142] As an example, the category evaluation result corresponding to the second synthesized audio can be represented by a second specific numerical value. For instance, if the second specific numerical value entered in the category evaluation box for a second synthesized audio is 0-0, then the emotional category of the second synthesized audio is determined to be happy. If the second specific numerical value entered in the category evaluation box for a second synthesized audio is 0-1, then the emotional category of the second synthesized audio is determined to be sad. If the second specific numerical value entered in the category evaluation box for a second synthesized audio is 0-2, then the emotional category of the second synthesized audio is determined to be crying. Or, if the second specific numerical value entered in the category evaluation box for a second synthesized audio is 1-0, then the scene category of the second synthesized audio is determined to be an unfamiliar environment. If the second specific numerical value entered in the category evaluation box for a second synthesized audio is 1-1, then the scene category of the second synthesized audio is determined to be stroking. If the second specific numerical value entered in the category evaluation box for a second synthesized audio is 1-2, then the scene category of the second synthesized audio is determined to be feeding.

[0143] It should be noted that the second specific value can be represented not only in the above manner, but also in other ways, and the embodiments of this application do not limit this.

[0144] The process by which the user terminal adjusts the second timbre extraction model based on N second synthesized audios, N third synthesized audios, and the quality prediction and evaluation results corresponding to the N second synthesized audios, as well as the N reference category information and the category evaluation results corresponding to the N second synthesized audios, includes: determining a first loss value based on the N second synthesized audios and the N third synthesized audios; determining a second loss value based on the quality prediction and evaluation results corresponding to the N second synthesized audios; determining a third loss value based on the N reference category information and the category evaluation results corresponding to the N second synthesized audios; adding the first loss value, the second loss value, and the third loss value to obtain a joint loss value; and adjusting the second timbre extraction model based on the joint loss value to obtain a first timbre extraction model.

[0145] As an example, the user terminal determines the third loss value based on the N reference category information and the category evaluation results corresponding to the N second synthesized audios according to the following formula (3).

[0146]

[0147] In formula (3) above, Loss3 is the third loss value, and L iFor the i-th reference category information among the N reference category information, L' i For the category evaluation result corresponding to the i-th second synthesized audio among the N second synthesized audios, Loss(L i ,L' i ) represents the loss value between the reference category information and the category evaluation result corresponding to the i-th second synthesized audio.

[0148] The loss value between the second and third synthesized audio can be determined by any loss function, and this application embodiment does not limit this. Furthermore, other details in the second case are described in the relevant description in the first case above, and will not be repeated here.

[0149] For example, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating another user terminal adjusting the second timbre extraction model according to an embodiment of this application. Figure 5 In the first step, the user terminal inputs M initial audio samples into a second timbre extraction model to obtain second timbre features. The second timbre features, N pieces of first media information, and N reference category information corresponding to the N pieces of first media information are then input into a second audio synthesis model to obtain N second synthesized audio samples. The user evaluates the quality and category of the N second synthesized audio samples to obtain quality evaluation results and category evaluation results for each sample. The quality evaluation results include a quality score for the second synthesized audio sample and a quality comparison result of the audio pair containing the second synthesized audio sample; the category evaluation results include the category of the second synthesized audio sample. Next, the N second synthesized audio samples are input into the second timbre extraction model to obtain N third timbre features. These N third timbre features are then input into the second audio synthesis model to obtain quality prediction results for the N third synthesized audio samples and the N second synthesized audio samples. Finally, the user terminal adjusts the second timbre extraction model based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios, the N third synthesized audios, and the N second synthesized audios, as well as the N reference category information and the category evaluation results corresponding to the N second synthesized audios, to obtain the first timbre extraction model.

[0150] Based on the description of the first scenario above, the first timbre extraction model is obtained by adjusting the second timbre extraction model based on the quality prediction results and quality evaluation results corresponding to N second-synthesized audio files, N third-synthesized audio files, and N second-synthesized audio files respectively. Based on the description of the second scenario above, the first timbre extraction model is obtained by adjusting the second timbre extraction model based on the quality prediction results and quality evaluation results corresponding to N second-synthesized audio files, N third-synthesized audio files, and N second-synthesized audio files respectively, as well as N reference category information and the category evaluation results corresponding to the N second-synthesized audio files respectively.

[0151] In some embodiments, the first audio synthesis model and the second audio synthesis model in the above two cases are the same model, or the first audio synthesis model and the second audio synthesis model are different models. Similar to the first timbre extraction model, in the first case, the first audio synthesis model is obtained by adjusting the second audio synthesis model based on the quality prediction results and quality evaluation results corresponding to N second synthesized audios, N third synthesized audios, and N second synthesized audios respectively. In the second case, the first audio synthesis model is obtained by adjusting the second audio synthesis model based on the quality prediction results and quality evaluation results corresponding to N second synthesized audios, N third synthesized audios, and N second synthesized audios respectively, as well as N reference category information and the category evaluation results corresponding to the N second synthesized audios respectively.

[0152] The process of adjusting the second audio synthesis model by the user terminal is similar to the process of adjusting the second timbre extraction model. Please refer to the previous description for details, which will not be repeated here.

[0153] The first convergence condition is met for the N first synthesized audio files, including: the average audio quality score of the N first synthesized audio files reaches a score threshold; or, the user terminal detects a user-triggered stop operation during the playback of the N first synthesized audio files. That is, after the user terminal obtains the user's audio quality scores for the N first synthesized audio files, it determines the average audio quality score of the N first synthesized audio files. When this average score reaches the score threshold, the N first synthesized audio files are determined to meet the first convergence condition. Alternatively, during the playback of the N first synthesized audio files through the audio quality evaluation interface on the user terminal, if a user-triggered stop operation is detected, the N first synthesized audio files are determined to meet the first convergence condition, meaning the user can manually stop the aforementioned loop process.

[0154] It should be noted that the score threshold is preset, and it can be adjusted according to different needs. Furthermore, as an example, the audio quality evaluation interface includes a stop button, which users can touch to trigger a stop operation. Of course, users can also trigger the stop operation in other ways.

[0155] If the N first synthesized audio samples do not meet the first convergence condition, the user terminal obtains the user's evaluation results on the N first synthesized audio samples. Based on the user's evaluation results on the N first synthesized audio samples, the first timbre extraction model is adjusted to obtain the third timbre extraction model. If the third timbre extraction model meets the second convergence condition, M initial audio samples are input into the third timbre extraction model to obtain the timbre features output by the third timbre extraction model. The timbre features output by the third timbre extraction model are determined as the timbre features of the first object.

[0156] The third timbre extraction model satisfies the second convergence condition by having an iteration count greater than or equal to an iteration count threshold. This iteration count threshold can be pre-set; it can be a specified number of iterations or a maximum number of iterations, and can be set according to different needs. This embodiment of the application does not limit this setting.

[0157] Similar to the evaluation results of the N second synthesized audios mentioned above, the user's evaluation results of the N first synthesized audios include the quality evaluation results corresponding to each of the N first synthesized audios. Alternatively, the user's evaluation results of the N first synthesized audios may include not only the quality evaluation results corresponding to each of the N first synthesized audios but also the category evaluation results corresponding to each of the N first synthesized audios. The way the user terminal adjusts the first timbre extraction model differs depending on the user's evaluation results of the N first synthesized audios. Therefore, the following two scenarios will be explained separately.

[0158] In the first scenario, the user's evaluation results for N first-synthesized audio files include the quality evaluation results corresponding to each of the N first-synthesized audio files. The user terminal inputs the N first-synthesized audio files into a first timbre extraction model to obtain N sixth-timbre features. These N sixth-timbre features are then input into a first audio synthesis model to obtain N fifth-synthesized audio files and the quality prediction results corresponding to the N first-synthesized audio files. Based on the quality prediction results and quality evaluation results corresponding to the N first-synthesized audio files, the N fifth-synthesized audio files, and the N first-synthesized audio files, the first timbre extraction model is adjusted to obtain a third timbre extraction model.

[0159] The quality prediction and quality evaluation results corresponding to the N first synthesized audios are similar to those corresponding to the N second synthesized audios. Furthermore, the process of adjusting the first timbre extraction model by the user terminal is similar to the process of adjusting the second timbre extraction model. Please refer to the previous description for details, which will not be repeated here.

[0160] In the second scenario, the user's evaluation of the N first synthesized audio files includes not only the quality evaluation results corresponding to each of the N first synthesized audio files, but also the category evaluation results corresponding to each of the N first synthesized audio files. The first audio synthesis model, given the input of first timbre features and N first media information, also receives N reference category information corresponding one-to-one with the N first media information. In this case, the user terminal inputs the N first synthesized audio files into the first timbre extraction model to obtain N sixth timbre features. These N sixth timbre features are then input into the first audio synthesis model to obtain N fifth synthesized audio files and the quality prediction results corresponding to each of the N first synthesized audio files. Based on the quality prediction results and quality evaluation results corresponding to the N first synthesized audio files, the N fifth synthesized audio files, and the N first synthesized audio files, as well as the N reference category information and the category evaluation results corresponding to each of the N first synthesized audio files, the first timbre extraction model is adjusted to obtain the third timbre extraction model.

[0161] The category evaluation results corresponding to the N first synthesized audios are similar to those corresponding to the N second synthesized audios. Furthermore, the process of adjusting the first timbre extraction model by the user terminal is similar to the process of adjusting the second timbre extraction model. Please refer to the previous description for details, which will not be repeated here.

[0162] It should be noted that the first and second convergence conditions are not limited to the conditions described above. The first and second convergence conditions may also be other conditions.

[0163] If the N first synthesized audio samples satisfy the first convergence condition, it indicates that the reliability of the first timbre feature obtained based on the first timbre extraction model is high. Therefore, the first timbre feature is directly determined as the timbre feature of the first object. If the third timbre extraction model satisfies the second convergence condition, it indicates that the third timbre extraction model has undergone a large number of iterations, and the timbre feature extracted by the third timbre extraction model is relatively reliable. To improve efficiency, the timbre feature extracted by the third timbre extraction model is directly determined as the timbre feature of the first object. If the N first synthesized audio samples do not satisfy the first convergence condition and the third timbre extraction model does not satisfy the second convergence condition, the first audio synthesis model is readjusted according to the above method to obtain the third audio synthesis model. Then, the above method is re-executed using the third timbre extraction model and the third audio synthesis model.

[0164] As can be seen from the above method, the embodiments of this application determine the timbre features of the first object by adjusting the timbre extraction model and audio synthesis model stored in the user terminal through a cyclic iterative process. Furthermore, if N first synthesized audio samples fail to meet the first convergence condition and the third timbre extraction model fails to meet the second convergence condition, the cyclic iterative process continues. Thus, for audio in low-resource scenarios such as pet audio and musical instrument audio, the user's subjective preferences are incorporated to ensure that the final extracted timbre features meet the user's needs.

[0165] It should be noted that, in order to reduce the number of times the user terminal performs model training, the user terminal uses a larger learning rate to adjust the network parameters in the model compared to the server. Typically, gradient descent is used to adjust the network parameters, with one gradient corresponding to one learning rate. Using a larger learning rate on the user terminal can improve the model's convergence speed.

[0166] In addition, the timbre extraction model and audio synthesis model initially stored on the user terminal are synchronized with the server. However, for audio in low-resource scenarios, a lot of data needs to be manually labeled, and the cost of sample labeling is high. Therefore, the user terminal can also send N second synthesized audios and the user's evaluation results on the N second synthesized audios to the server, so that the server can use the received N second synthesized audios and the user's evaluation results on the N second synthesized audios as sample data to train the timbre extraction model and audio synthesis model.

[0167] Based on the above description, the timbre extraction method provided in this application embodiment can be applied to a variety of scenarios, taking the short video production scenario as an example. After the user terminal determines the first timbre feature as the timbre feature of the first object, it can also determine the synthesized audio required for short video production through the following steps (1)-(4).

[0168] (1) Input M initial audios and M initial videos into the audio and video feature extraction model to obtain the first audio and video features. The M initial videos correspond one-to-one with the M initial audios.

[0169] The audio and video feature extraction model is pre-trained, and its network structure can be a neural network or any other network structure.

[0170] (2) Based on the first audio and video features, obtain the fourth timbre feature and the second media information from the database. The fourth timbre feature is the timbre feature corresponding to the second object, and the second object is different from the first object.

[0171] The database stores the correspondence between audio / video features, timbre features, and media information. In some embodiments, the user terminal retrieves the fourth timbre feature and the second media information from the database in the following manner: The user terminal retrieves audio / video features that match the first audio / video feature from the correspondence between the audio / video features, timbre features, and media information to obtain one or more candidate audio / video features. Based on the one or more candidate audio / video features, the second audio / video feature is determined. The user terminal identifies the timbre feature corresponding to the second audio / video feature in the correspondence as the fourth timbre feature and the media information corresponding to the second audio / video feature in the correspondence as the second media information.

[0172] When a user terminal determines one or more candidate audio / video features, it determines the distance between each audio / video feature and the first audio / video feature to obtain the distances corresponding to each of the multiple audio / video features. These multiple audio / video features are those from the correspondence between audio / video features, timbre features, and media information stored in the database. One or more audio / video features are selected as one or more candidate audio / video features in ascending order of distance.

[0173] There are several ways for a user terminal to select one or more audio / video features from a plurality of audio / video features in ascending order of distance. For example, the user terminal may select S audio / video features from the plurality of features in ascending order of distance, where S is an integer greater than or equal to 1. Alternatively, the user terminal may select audio / video features whose distance is greater than a distance threshold from the plurality of audio / video features. This distance threshold is preset and can be adjusted according to different needs.

[0174] There are several ways a user terminal can determine a second audio / video feature based on one or more candidate audio / video features. For example, the user terminal may randomly select one audio / video feature from the one or more candidate features as the second audio / video feature. Alternatively, the user terminal may display the one or more candidate audio / video features, and when the user terminal detects a selection operation by the user, it may use the candidate audio / video feature selected by the selection operation as the second audio / video feature. This selection operation instructs the user to select an audio / video feature from the one or more candidate features according to their actual needs.

[0175] (3) The first timbre feature and the fourth timbre feature are fused together to obtain the fifth timbre feature.

[0176] There are several ways for the user terminal to fuse the first timbre feature and the fourth timbre feature. For example, the user terminal displays a feature fusion interface, obtains the fusion ratio of the first timbre feature and the fourth timbre feature through this interface, and fuses the first timbre feature and the fourth timbre feature based on this fusion ratio to obtain the fifth timbre feature.

[0177] As an example, the feature fusion interface includes a progress bar that the user can adjust by clicking, swiping, or dragging to determine the fusion ratio of the first and fourth timbre features. The user terminal then performs a weighted sum of the first and fourth timbre features according to the fusion ratio to obtain the fifth timbre feature.

[0178] As another example, the feature fusion interface includes two input windows, corresponding to a first timbre feature and a fourth timbre feature, respectively. In this case, the user can input the fusion coefficient corresponding to the first timbre feature and the fusion coefficient corresponding to the fourth timbre feature in the two input windows, respectively. The user terminal multiplies the first timbre feature by its corresponding fusion coefficient to obtain a first value, multiplies the fourth timbre feature by its corresponding fusion coefficient to obtain a second value, and adds the first and second values ​​to obtain a fifth timbre feature.

[0179] (4) Input the fifth timbre feature and the second media information into the first audio synthesis model to obtain the fourth synthesized audio.

[0180] After determining the first timbre feature as the timbre feature of the first object, it indicates that the reliability of the output result of the first audio synthesis model is relatively high. Therefore, the user terminal directly inputs the fifth timbre feature and the second media information into the first audio synthesis model to obtain the fourth synthesized audio output by the first audio synthesis model. At this time, the quality of the fourth synthesized audio is also relatively good.

[0181] In this embodiment, during the process of determining the timbre features of the first object, the user terminal adjusts the timbre extraction model and the audio synthesis model based on the user's evaluation results of the synthesized audio. This is equivalent to customizing the timbre extraction model and audio synthesis model for each user to match their own evaluation standards and preferences. This ensures that the final determined timbre features of the first object meet the user's actual needs. Specifically, in low-resource scenarios such as pet audio, animal audio, and musical instrument audio, adjusting the timbre extraction model and audio synthesis model based on the user's evaluation results of the synthesized audio ensures that the extracted timbre features meet the user's actual needs. Furthermore, the user terminal can determine N synthesized audios based on the timbre features and N pieces of first media information. When N is greater than 1, multiple synthesized audios better reflect the user's evaluation standards and preferences. Moreover, by considering multiple dimensions such as audio quality scoring, audio quality comparison results, and category evaluation results, the timbre extraction model and audio synthesis model can achieve better training results and faster convergence speed, thereby improving the speed of determining the timbre features of the first object.

[0182] Furthermore, in this embodiment, the user terminal sends N second synthesized audio files and the user's evaluation results for these N second synthesized audio files to the server. The server uses the user's evaluation results for the N second synthesized audio files as sample data to retrain the trained timbre extraction model and the trained audio synthesis model. This solves the problems of insufficient server sample data and high sample annotation costs when the server jointly trains the timbre extraction model and the audio synthesis model. Moreover, when determining the synthesized audio files needed for short video production, audio and video features are determined by combining audio and video, and then audio and video features are recommended based on the distance between these features. This solves the problem of difficulty in recommending audio in low-resource scenarios such as pet audio due to limited information.

[0183] Based on the above description, the timbre extraction model and audio synthesis model initially stored in the user terminal are synchronized with the server. Before the server synchronization, the untrained initial timbre extraction model and initial audio synthesis model can be jointly trained. After that, the trained timbre extraction model and audio synthesis model are synchronized to the user terminal. Figure 6 This is a flowchart illustrating a model training method provided in an embodiment of this application. This method is applied in a server; please refer to [the documentation / reference]. Figure 6 The method includes the following steps.

[0184] Step 601: The server obtains the T sample audios and the sample annotation results corresponding to the T sample audios, where T is an integer greater than or equal to 1.

[0185] The sample audio includes audio from low-resource scenarios such as pet audio, animal audio, and musical instrument audio. The annotation results for the T sample audios include the quality annotation results for each of the T sample audios, or, the annotation results for the T sample audios include not only the quality annotation results for each of the T sample audios, but also the category annotation results for each of the T sample audios.

[0186] When the sample annotation results include quality annotation results, the quality annotation results include K dimensions of quality results. These K dimensions of quality results include the audio quality score of the sample audio and / or the audio pair quality comparison results. The audio pair quality comparison results are determined by comparing the quality of two sample audios in a sample audio pair to which the corresponding sample audio belongs. A sample audio pair includes the corresponding sample audio and one other audio. When the sample annotation results include category annotation results, the category annotation results include L dimensions of categories. These L dimensions of categories include the audio sentiment category and / or audio scene category of the sample audio.

[0187] Step 602: Based on the T sample audios and the sample annotation results corresponding to the T sample audios, the server jointly trains the initial timbre extraction model and the initial audio synthesis model to obtain the trained timbre extraction model and the trained audio synthesis model.

[0188] Based on the above description, the sample annotation results include quality annotation results, or, in addition to quality annotation results, also include category annotation results. When the sample annotation results differ, the server will perform different joint training methods on the initial timbre extraction model and the initial audio synthesis model based on the sample annotation results corresponding to the T sample audio samples and the T sample audio samples respectively. Therefore, the following will describe these two cases separately.

[0189] In the first case, the sample labeling results include the quality labeling results.

[0190] In this case, the server jointly trains the initial timbre extraction model and the initial audio synthesis model using the following two implementation methods.

[0191] In the first implementation, the server inputs T sample audios into an initial timbre extraction model to obtain T first-sample timbre features. These T first-sample timbre features are then input into an initial audio synthesis model to obtain T first-sample synthesized audios, with each of the T first-sample synthesized audios corresponding one-to-one with the T sample audios. The server jointly trains the initial timbre extraction model and the initial audio synthesis model based on the T sample audios and the T first-sample synthesized audios. The server uses the converged initial timbre extraction model as the trained timbre extraction model and modifies the network structure of the converged initial audio synthesis model to obtain a modified audio synthesis model. Then, the server inputs the T sample audios into the trained timbre extraction model to obtain T second-sample timbre features. These T second-sample timbre features are then input into the modified audio synthesis model to obtain the quality prediction results corresponding to the T second-sample synthesized audios and the T sample audios, respectively. The server trains the modified audio synthesis model based on T sample audios, T second sample synthesized audios, and the quality prediction and quality labeling results corresponding to the T sample audios, to obtain the trained audio synthesis model.

[0192] In pitch-based scenarios such as humming, during the process of inputting T first-sample timbre features into the initial audio synthesis model, pitch features can also be input into the initial audio synthesis model. That is, by inputting T first-sample timbre features and pitch features into the initial audio synthesis model, T first-sample synthesized audio samples are obtained. Here, the pitch feature is either randomly selected by the server or selected during manual annotation.

[0193] The process of jointly training the initial timbre extraction model and the initial audio synthesis model based on T sample audio and T first sample synthesized audio includes: determining the first sample loss value based on T sample audio and T first sample synthesized audio, and adjusting the initial timbre extraction model and the initial audio synthesis model based on the first sample loss value.

[0194] The method by which the server determines the first sample loss value based on T sample audio and T first sample synthesized audio is similar to the method by which the user terminal determines the first loss value based on N second synthesized audio and N third synthesized audio in the above embodiment. Please refer to the relevant content in the above embodiment for details, which will not be repeated here.

[0195] The process by which the server adjusts the initial timbre extraction model and the initial audio synthesis model based on the first sample loss value is actually a process of adjusting the network parameters in the initial timbre extraction model and the initial audio synthesis model. That is, the server adjusts the network parameters in the initial timbre extraction model and the network parameters in the initial audio synthesis model based on the first sample loss value. The implementation process of the server adjusting the network parameters of the model based on the loss value can refer to related technologies, and this application embodiment does not limit it.

[0196] In the first implementation, the initial audio synthesis model does not have quality prediction capabilities; that is, its network structure does not include a quality prediction branch. Therefore, when jointly training the initial timbre extraction model and the initial audio synthesis model, it is only necessary to determine the first sample loss value based on T sample audios and T first sample synthesized audios, and then adjust the initial timbre extraction model and the initial audio synthesis model based on the first sample loss value. After the initial timbre extraction model and the initial audio synthesis model converge, the server modifies the network structure of the converged audio synthesis model to add a quality prediction branch, meaning the modified audio synthesis model now possesses quality prediction capabilities.

[0197] At this point, the process of training the modified audio synthesis model based on T sample audios, T second sample synthesized audios, and the quality prediction and quality labeling results corresponding to the T sample audios includes: determining the second sample loss value based on the T sample audios and T second sample synthesized audios; determining the third sample loss value based on the quality prediction and quality labeling results corresponding to the T sample audios; adding the second sample loss value and the third sample loss value to obtain the sample joint loss value; and adjusting the modified audio synthesis model based on the sample joint loss value to obtain the trained audio synthesis model.

[0198] The method by which the server determines the second sample loss value based on T sample audio and T second sample synthesized audio is similar to the method by which the user terminal determines the first loss value based on N second synthesized audio and N third synthesized audio in the above embodiment. Please refer to the relevant content in the above embodiment for details, which will not be repeated here.

[0199] The way the server determines the third sample loss value based on the quality prediction results and quality annotation results corresponding to T sample audios is similar to the way the user terminal determines the second loss value based on the quality prediction results and quality evaluation results corresponding to N second synthesized audios in the above embodiment. Please refer to the relevant content in the above embodiment for details, which will not be repeated here.

[0200] The process by which the server adjusts the modified audio synthesis model based on the joint loss value of the sample is actually the process of adjusting the network parameters in the modified audio synthesis model. That is, the server adjusts the network parameters in the modified audio synthesis model based on the joint loss value of the sample. The implementation process of the server adjusting the network parameters of the model based on the loss value can refer to relevant technologies, and this application embodiment does not limit it.

[0201] For example, please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating a server jointly training an initial timbre extraction model and an initial audio synthesis model, as provided in an embodiment of this application. Figure 7 In the initial audio synthesis model, the server inputs T sample audios into the initial timbre extraction model to obtain T first sample timbre features. These T first sample timbre features are then input into the initial audio synthesis model to obtain T first sample synthesized audios. A first sample loss value is determined based on the T sample audios and the T first sample synthesized audios. Based on this first sample loss value, the initial timbre extraction model and the initial audio synthesis model are jointly trained. Subsequently, the server uses the converged initial timbre extraction model as the trained timbre extraction model and modifies the network structure of the converged initial audio synthesis model to obtain the modified audio synthesis model.

[0202] Please refer to Figure 8 , Figure 8 This is a schematic diagram illustrating how a server trains a modified audio synthesis model, as provided in an embodiment of this application. Figure 8 In this process, the server inputs T sample audios into a trained timbre extraction model to obtain T second sample timbre features. These T second sample timbre features are then input into a modified audio synthesis model to obtain T second sample synthesized audios and the corresponding quality prediction results for the T sample audios. Based on the T sample audios and the T second sample synthesized audios, a second sample loss value is determined. Based on the quality prediction results and quality annotation results corresponding to the T sample audios, a third sample loss value is determined. Based on the second sample loss value and the third sample loss value, the modified audio synthesis model is trained to obtain a trained audio synthesis model.

[0203] In the second implementation, the server inputs T sample audios into an initial timbre extraction model to obtain T first sample timbre features. These T first sample timbre features are then input into an initial audio synthesis model to obtain T first sample synthesized audios and the corresponding quality prediction results for the T sample audios. Based on the T sample audios, the T first sample synthesized audios, and the corresponding quality prediction and quality annotation results, the server jointly trains the initial timbre extraction model and the initial audio synthesis model.

[0204] In the second implementation, the initial audio synthesis model has the function of quality prediction. That is, the network structure of the initial audio synthesis model includes a quality prediction branch. Therefore, after the server inputs the timbre features of the first sample into the initial audio synthesis model, it can obtain the quality prediction results corresponding to the T first sample synthesized audio and the T sample audio respectively.

[0205] The process of jointly training the initial timbre extraction model and the initial audio synthesis model based on T sample audios, T first sample synthesized audios, and the quality prediction and quality annotation results corresponding to the T sample audios includes: the server determines the first sample loss value based on the T sample audios and T first sample synthesized audios, determines the third sample loss value based on the quality prediction and quality annotation results corresponding to the T sample audios, adds the first sample loss value and the third sample loss value to obtain the sample joint loss value, and adjusts the initial timbre extraction model and the initial audio synthesis model based on the sample joint loss value to obtain the trained timbre extraction model and the trained audio synthesis model.

[0206] The method by which the server determines the first sample loss value based on T sample audio and T first sample synthesized audio is similar to the method by which the user terminal determines the first loss value based on N second synthesized audio and N third synthesized audio in the above embodiment. Please refer to the relevant content in the above embodiment for details, which will not be repeated here.

[0207] The process by which the server adjusts the initial timbre extraction model and the initial audio synthesis model based on the joint loss value of the sample is actually a process of adjusting the network parameters in the initial timbre extraction model and the initial audio synthesis model. That is, the server adjusts the network parameters in the initial timbre extraction model and the network parameters in the initial audio synthesis model based on the joint loss value of the sample. The implementation process of the server adjusting the network parameters of the model based on the loss value can refer to related technologies, and this application embodiment does not limit it.

[0208] In the second scenario, the sample labeling results include not only quality labeling results but also category labeling results.

[0209] In this case, the server jointly trains the initial timbre extraction model and the initial audio synthesis model using the following two implementation methods.

[0210] In the first implementation, the server inputs T sample audios into an initial timbre extraction model to obtain T first-sample timbre features. These T first-sample timbre features are then input into an initial audio synthesis model to obtain T first-sample synthesized audios, which correspond one-to-one with the T sample audios. The server jointly trains the initial timbre extraction model and the initial audio synthesis model based on the T sample audios and the T first-sample synthesized audios. The server uses the converged initial timbre extraction model as the trained timbre extraction model and modifies the network structure of the converged initial audio synthesis model to obtain a modified audio synthesis model. Then, the server inputs the T sample audios into the trained timbre extraction model to obtain T second-sample timbre features. The server inputs the category labeling results corresponding to the T second-sample timbre features and the T sample audios into the modified audio synthesis model to obtain the quality prediction results and category prediction results corresponding to the T second-sample synthesized audios and the T sample audios, respectively. The server trains the modified audio synthesis model based on the quality prediction and quality labeling results of T sample audios, T second sample synthesized audios, and T sample audios respectively, as well as the category prediction and category labeling results of T sample audios respectively, to obtain the trained audio synthesis model.

[0211] The method for jointly training the initial timbre extraction model and the initial audio synthesis model based on T sample audio and T first sample synthesized audio is described in the first case above, and will not be repeated here.

[0212] In the first implementation, the initial audio synthesis model lacks quality and category prediction capabilities. That is, its network structure does not include branches for quality and category prediction. Therefore, when jointly training the initial timbre extraction model and the initial audio synthesis model, it is only necessary to determine the first sample loss value based on T sample audio files and T first sample synthesized audio files, and then adjust the initial timbre extraction model and the initial audio synthesis model based on this first sample loss value. After the initial timbre extraction model and the initial audio synthesis model converge, the server modifies the network structure of the converged audio synthesis model to add branches for quality and category prediction. In other words, the modified audio synthesis model possesses both quality and category prediction capabilities.

[0213] At this point, the process of training the modified audio synthesis model based on T sample audios, T second sample synthesized audios, the quality prediction results and quality annotation results corresponding to the T sample audios, and the category prediction results and category annotation results corresponding to the T sample audios includes: determining the second sample loss value based on the T sample audios and T second sample synthesized audios; determining the third sample loss value based on the quality prediction results and quality annotation results corresponding to the T sample audios; determining the fourth sample loss value based on the category prediction results and category annotation results corresponding to the T sample audios; adding the second sample loss value, the third sample loss value, and the fourth sample loss value to obtain the sample joint loss value; and adjusting the modified audio synthesis model based on this sample joint loss value to obtain the trained audio synthesis model.

[0214] The method by which the server determines the second and third sample loss values ​​refers to the relevant content in the first case above. The method by which the server determines the fourth sample loss value based on the category prediction results and category labeling results corresponding to the T sample audios is similar to the method by which the user terminal determines the third loss value based on the category evaluation results corresponding to N reference category information and N second synthesized audios in the above embodiment. For details, please refer to the relevant content in the above embodiment, which will not be repeated here.

[0215] For example, please refer to Figure 9 , Figure 9 This is a schematic diagram illustrating another method by which a server trains a modified audio synthesis model, as provided in an embodiment of this application. Figure 9 In this process, the server inputs T sample audios into a trained timbre extraction model to obtain T second sample timbre features. These T second sample timbre features and the corresponding category labeling results of the T sample audios are then input into a modified audio synthesis model to obtain T second sample synthesized audios and the corresponding quality prediction and category prediction results of the T sample audios. Based on the T sample audios and the T second sample synthesized audios, a second sample loss value is determined. Based on the quality prediction and quality labeling results of the T sample audios respectively, a third sample loss value is determined. Based on the corresponding category prediction and category labeling results of the T sample audios respectively, a fourth sample loss value is determined. Based on the second sample loss value, the third sample loss value, and the fourth sample loss value, the modified audio synthesis model is trained to obtain a trained audio synthesis model.

[0216] In the second implementation, the server inputs T sample audios into an initial timbre extraction model to obtain T first sample timbre features. The server then inputs the T first sample timbre features and the corresponding category labels of the T sample audios into an initial audio synthesis model to obtain T first sample synthesized audios, as well as the corresponding quality prediction and category prediction results for the T sample audios. Based on the T sample audios, the T first sample synthesized audios, the corresponding quality prediction and quality labeling results, and the corresponding category prediction and category labeling results, the server jointly trains the initial timbre extraction model and the initial audio synthesis model to obtain a trained timbre extraction model and a trained audio synthesis model.

[0217] In the second implementation, the initial audio synthesis model has the functions of quality prediction and category prediction. That is, the network structure of the initial audio synthesis model includes a quality prediction branch and a category prediction branch. Therefore, after the server inputs the timbre features of the T first samples and the category labeling results corresponding to the T sample audios into the initial audio synthesis model, it can obtain the synthesized audios of the T first samples, as well as the quality prediction results and category prediction results corresponding to the T sample audios.

[0218] The process of training the modified audio synthesis model based on T sample audios, T first sample synthesized audios, the quality prediction results and quality annotation results corresponding to the T sample audios, and the category prediction results and category annotation results corresponding to the T sample audios includes: determining the first sample loss value based on the T sample audios and T first sample synthesized audios; determining the third sample loss value based on the quality prediction results and quality annotation results corresponding to the T sample audios; determining the fourth sample loss value based on the category prediction results and category annotation results corresponding to the T sample audios; adding the first sample loss value, the third sample loss value, and the fourth sample loss value to obtain the sample joint loss value; and adjusting the modified audio synthesis model based on the sample joint loss value to obtain the trained audio synthesis model.

[0219] It should be noted that the server can also receive N second-synthesized audio files sent by the user terminal, as well as the user's evaluation results for these N second-synthesized audio files. The server uses these N second-synthesized audio files as N sample audio files and the user's evaluation results as sample annotation results for these N sample audio files. Based on these N second-synthesized audio files and the user's evaluation results, the server adjusts the trained timbre extraction model and audio synthesis model. The method by which the server adjusts the trained timbre extraction model and audio synthesis model based on the N second-synthesized audio files and the user's evaluation results is the same as the method described above for adjusting the initial timbre extraction model and initial audio synthesis model based on T sample audio files and their sample annotation results; therefore, it will not be repeated here.

[0220] It should be noted that the server can synchronize the trained timbre extraction model and audio synthesis model to the user terminal. It can also distill the trained timbre extraction model and audio synthesis model and synchronize the distilled model to the user terminal. Furthermore, whenever the user terminal determines the timbre characteristics of a first object, it requests the server to synchronize the timbre extraction model and audio synthesis model to the user terminal. Alternatively, the server can periodically send synchronization requests to the user terminal, and when the user terminal agrees to synchronization, the server will synchronize the timbre extraction model and audio synthesis model to the user terminal.

[0221] In this embodiment, the user terminal sends N second synthesized audio samples and the user's evaluation results for the N second synthesized audio samples to the server. The server uses the user's evaluation results for the N second synthesized audio samples as sample data to retrain the trained timbre extraction model and the trained audio synthesis model. This solves the problems of insufficient server sample data and high sample annotation costs when the server jointly trains the timbre extraction model and the audio synthesis model.

[0222] Figure 10 This is a schematic diagram of a timbre extraction device provided in an embodiment of this application. The device can be implemented as part or all of a user terminal by software, hardware, or a combination of both. The user terminal can be... Figure 1 The user terminal shown. See also Figure 10 The device includes: a timbre feature extraction module 1001, a first audio synthesis module 1002, and a timbre feature determination module 1003.

[0223] The timbre feature extraction module 1001 is used to input M initial audios into the first timbre extraction model to obtain the first timbre feature. The initial audios are obtained by collecting the first object, which is an object that cannot be recognized by speech. M is an integer greater than or equal to 1.

[0224] The first audio synthesis module 1002 is used to input the first timbre feature and N first media information into the first audio synthesis model to obtain N first synthesized audios, where N is an integer greater than or equal to 1;

[0225] The timbre feature determination module 1003 is used to determine the first timbre feature as the timbre feature of the first object if N first synthesized audios satisfy the first convergence condition;

[0226] The first timbre extraction model is obtained by adjusting the second timbre extraction model based on user evaluation results of N second synthesized audios. The N second synthesized audios are obtained by synthesizing the second timbre features with N first media information respectively. The second timbre features are obtained by inputting M initial audios into the second timbre extraction model.

[0227] Optionally, the user's evaluation results for the N second synthesized audios include the quality evaluation results corresponding to each of the N second synthesized audios. The first timbre extraction model is obtained by adjusting the second timbre extraction model based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios, the N third synthesized audios, and the N second synthesized audios respectively. The N second synthesized audios are obtained by inputting the second timbre features and N first media information into the second audio synthesis model. The quality prediction results corresponding to the N third synthesized audios and the N second synthesized audios are obtained by inputting the N second synthesized audios into the second timbre extraction model to obtain N third timbre features, and then inputting the N third timbre features into the second audio synthesis model.

[0228] Optionally, the user's evaluation results for the N second synthesized audios also include the category evaluation results corresponding to the N second synthesized audios. When the second audio synthesis model is input with second timbre features and N first media information, it is also input with N reference category information corresponding one-to-one with the N first media information. The first timbre extraction model is obtained by adjusting the second timbre extraction model based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios, the N third synthesized audios, and the N second synthesized audios, as well as the N reference category information and the category evaluation results corresponding to the N second synthesized audios.

[0229] Optionally, both the quality prediction result and the quality evaluation result include quality results in K dimensions. The quality results in K dimensions include audio quality scores and / or audio pair quality comparison results. The audio pair quality comparison results are determined by comparing the quality between two audios in the audio pair containing the corresponding synthesized audio. The audio pair includes the corresponding synthesized audio and another audio. K is an integer greater than or equal to 1.

[0230] Optionally, the reference category information and category evaluation results both include L-dimensional categories, where L-dimensional categories include audio emotion categories and / or audio scene categories, and L is an integer greater than or equal to 1.

[0231] Optionally, the first audio synthesis model is obtained by adjusting the second audio synthesis model based on the quality prediction results and quality evaluation results corresponding to N second synthesized audios, N third synthesized audios, and N second synthesized audios, respectively.

[0232] Optionally, the user's evaluation results for the N second synthesized audios also include the category evaluation results corresponding to the N second synthesized audios. When the second audio synthesis model is input with second timbre features and N first media information, it is also input with N reference category information corresponding one-to-one with the N first media information. The first audio synthesis model is obtained by adjusting the second audio synthesis model based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios, the N third synthesized audios, and the N second synthesized audios, as well as the N reference category information and the category evaluation results corresponding to the N second synthesized audios.

[0233] Optionally, the device further includes:

[0234] The audio and video feature extraction model is used to input M initial audios and M initial videos into the audio and video feature extraction model to obtain the first audio and video features. The M initial videos correspond one-to-one with the M initial audios.

[0235] The information acquisition module is used to acquire the fourth timbre feature and the second media information from the database based on the first audio and video features. The fourth timbre feature is the timbre feature corresponding to the second object, which is different from the first object.

[0236] The feature fusion module is used to fuse the first timbre feature and the fourth timbre feature to obtain the fifth timbre feature;

[0237] The second audio synthesis module is used to input the fifth timbre feature and the second media information into the first audio synthesis model to obtain the fourth synthesized audio.

[0238] Optionally, the database is used to store the correspondence between audio and video features, timbre features, and media information;

[0239] The information acquisition module is specifically used for:

[0240] Obtain audio and video features that match the first audio and video features from the correspondence to obtain one or more candidate audio and video features;

[0241] Based on one or more candidate audio and video features, determine the second audio and video features;

[0242] The timbre feature corresponding to the second audio-visual feature in the correspondence is determined as the fourth timbre feature, and the media information corresponding to the second audio-visual feature in the correspondence is determined as the second media information.

[0243] Optionally, the device further includes:

[0244] The sending module is used to send N second-synthesized audio files, along with the user's evaluation results for the N second-synthesized audio files, to the server.

[0245] In this embodiment, during the process of determining the timbre features of the first object, the user terminal adjusts the timbre extraction model and the audio synthesis model based on the user's evaluation results of the synthesized audio. This is equivalent to customizing the timbre extraction model and audio synthesis model for each user to match their own evaluation standards and preferences. This ensures that the final determined timbre features of the first object meet the user's actual needs. Specifically, in low-resource scenarios such as pet audio, animal audio, and musical instrument audio, adjusting the timbre extraction model and audio synthesis model based on the user's evaluation results of the synthesized audio ensures that the extracted timbre features meet the user's actual needs. Furthermore, the user terminal can determine N synthesized audios based on the timbre features and N pieces of first media information. When N is greater than 1, multiple synthesized audios better reflect the user's evaluation standards and preferences. Moreover, by considering multiple dimensions such as audio quality scoring, audio quality comparison results, and category evaluation results, the timbre extraction model and audio synthesis model can achieve better training results and faster convergence speed, thereby improving the speed of determining the timbre features of the first object.

[0246] It should be noted that the timbre extraction device provided in the above embodiments is only illustrated by the division of the above functional modules when extracting timbre features. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the timbre extraction device and the timbre extraction method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0247] Figure 11 This is a schematic diagram of a model training device provided in an embodiment of this application. The device can be implemented as part or all of a server, which can be software, hardware, or a combination of both. Figure 1 The server shown. See also Figure 11 The device includes a sample acquisition module 1101 and a model training module 1102.

[0248] The sample acquisition module 1101 is used to acquire T sample audios and the sample annotation results corresponding to the T sample audios respectively, where T is an integer greater than or equal to 1;

[0249] The model training module 1102 is used to jointly train the initial timbre extraction model and the initial audio synthesis model based on the sample annotation results corresponding to the T sample audios and the T sample audios respectively, so as to obtain the trained timbre extraction model and the trained audio synthesis model.

[0250] Optionally, the sample labeling results include quality labeling results;

[0251] Model training module 1102 includes:

[0252] The first timbre feature extraction submodule is used to input T sample audios into the initial timbre extraction model to obtain T first sample timbre features, and input the T first sample timbre features into the initial audio synthesis model to obtain T first sample synthesized audios;

[0253] The first joint training submodule is used to jointly train the initial timbre extraction model and the initial audio synthesis model based on T sample audio and T first sample synthesized audio.

[0254] The model modification submodule is used to modify the network structure of the converged initial timbre extraction model as a trained timbre extraction model to obtain the modified audio synthesis model.

[0255] The second timbre feature extraction submodule is used to input T sample audios into the trained timbre extraction model to obtain T second sample timbre features, and input the T second sample timbre features into the modified audio synthesis model to obtain the quality prediction results corresponding to the T second sample synthesized audios and the T sample audios respectively.

[0256] The model training submodule is used to train the modified audio synthesis model based on T sample audios, T second sample synthesized audios, and the quality prediction and quality labeling results corresponding to the T sample audios, so as to obtain the trained audio synthesis model.

[0257] Optionally, the sample annotation results may also include category annotation results;

[0258] The second timbre feature extraction submodule is specifically used for:

[0259] T sample audios are input into a trained timbre extraction model to obtain T second sample timbre features. The T second sample timbre features and the category labeling results corresponding to the T sample audios are input into a modified audio synthesis model to obtain T second sample synthesized audios, and the quality prediction results and category prediction results corresponding to the T sample audios.

[0260] The model training submodule is specifically used for:

[0261] The modified audio synthesis model is trained based on the quality prediction and quality labeling results corresponding to T sample audio, T second sample synthesized audio, and T sample audio, as well as the category prediction and category labeling results corresponding to T sample audio.

[0262] Optionally, the sample labeling results include quality labeling results;

[0263] Model training module 1102 includes:

[0264] The third timbre feature extraction submodule is used to input T sample audios into the initial timbre extraction model to obtain T first sample timbre features, and input the T first sample timbre features into the initial audio synthesis model to obtain the quality prediction results corresponding to the T first sample synthesized audios and the T sample audios respectively.

[0265] The second joint training submodule is used to jointly train the initial timbre extraction model and the initial audio synthesis model based on the quality prediction results and quality annotation results corresponding to T sample audios, T first sample synthesized audios, and T sample audios respectively.

[0266] Optionally, the sample annotation results may also include category annotation results;

[0267] The third timbre feature extraction submodule is specifically used for:

[0268] T sample audios are input into the initial timbre extraction model to obtain T first sample timbre features. The T first sample timbre features and the category labeling results corresponding to the T sample audios are input into the initial audio synthesis model to obtain T first sample synthesized audios, as well as the quality prediction results and category prediction results corresponding to the T sample audios.

[0269] The second joint training submodule is specifically used for:

[0270] Based on the quality prediction results and quality labeling results corresponding to T sample audios, T first sample synthesized audios, and T sample audios respectively, as well as the category prediction results and category labeling results corresponding to T sample audios respectively, the initial timbre extraction model and the initial audio synthesis model are jointly trained.

[0271] Optionally, the device further includes:

[0272] The receiving module is used to receive N second synthesized audios sent by the user terminal, as well as the user's evaluation results on the N second synthesized audios, where N is an integer greater than or equal to 1;

[0273] The model tuning module is used to adjust the trained timbre extraction model and audio synthesis model based on N second synthesized audio samples and user evaluations of the N second synthesized audio samples.

[0274] In this embodiment, the user terminal sends N second synthesized audio samples and the user's evaluation results for the N second synthesized audio samples to the server. The server uses the user's evaluation results for the N second synthesized audio samples as sample data to retrain the trained timbre extraction model and the trained audio synthesis model. This solves the problems of insufficient server sample data and high sample annotation costs when the server jointly trains the timbre extraction model and the audio synthesis model.

[0275] It should be noted that the model training device provided in the above embodiments is only illustrated by the division of the above functional modules during model training. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the model training device and the model training method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0276] Please refer to Figure 12 , Figure 12 This is a schematic diagram of a computer device according to an embodiment of this application. The computer device may be the user terminal or server described above. The computer device includes at least one processor 1201, a communication bus 1202, a memory 1203, and at least one communication interface 1204.

[0277] The processor 1201 can be a general-purpose central processing unit (CPU), graphics processing unit (GPU), network processor (NP), microprocessor, or one or more integrated circuits for implementing the solutions of this application, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0278] The communication bus 1202 is used to transmit information between the aforementioned components. The communication bus 1202 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.

[0279] The memory 1203 may be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compressed optical disc, a laser disc, a digital versatile optical disc, a Blu-ray disc, etc.), a magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but not limited thereto. The memory 1203 may exist independently and be connected to the processor 1201 via a communication bus 1202. The memory 1203 may also be integrated with the processor 1201.

[0280] Communication interface 1204 uses any transceiver-like device for communicating with other devices or communication networks. Communication interface 1204 includes a wired communication interface and may also include a wireless communication interface. The wired communication interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface may be a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0281] In a specific implementation, as one example, the processor 1201 may include one or more CPUs, such as Figure 12 CPU0 and CPU1 are shown in the diagram.

[0282] In a specific implementation, as one example, a computer device may include multiple processors, such as... Figure 12 The processors 1201 and 1205 are shown. Each of these processors may be a single-core processor or a multi-core processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0283] In a specific implementation, as one embodiment, the computer device may further include an output device 1206 and an input device 1207. The output device 1206 communicates with the processor 1201 and can display information in various ways. For example, the output device 1206 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 1207 communicates with the processor 1201 and can receive user input in various ways. For example, the input device 1207 may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0284] In some embodiments, memory 1203 is used to store program code 1210 for executing the scheme of this application, and processor 1201 can execute program code 1210 stored in memory 1203. The program code 1210 may include one or more software modules, and the computer device can implement the method provided in the above embodiments through processor 1201 and program code 1210 in memory 1203.

[0285] Please refer to Figure 13 , Figure 13This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. The terminal device can be the user terminal described above. The terminal device includes a sensor unit 1110, a computing unit 1120, a storage unit 1140, and an interaction unit 1130.

[0286] Sensor unit 1110 typically includes a vision sensor (such as a camera), a depth sensor, an IMU, a laser sensor, etc.

[0287] The computing unit 1120 typically includes a CPU, GPU, cache, registers, etc., and is mainly used to run the operating system;

[0288] Storage unit 1140 mainly includes memory and external storage, and is mainly used for reading and writing local and temporary data.

[0289] The interaction unit 1130 mainly includes a display screen, touchpad, speaker, microphone, etc., and is mainly used to interact with the user, obtain input, and implement the presentation algorithm effect, etc.

[0290] For ease of understanding, the structure of a terminal device 100 provided in an embodiment of this application will be illustrated below. See also Figure 14 , Figure 14 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application.

[0291] like Figure 14 As shown, the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, etc.

[0292] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0293] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. Processor 110 can execute computer programs to implement any of the methods described in the embodiments of this application.

[0294] The controller can serve as the central nervous system and command center of the terminal device 100. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0295] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory, avoiding repeated accesses, reducing the waiting time of the processor 110, and thus improving the efficiency of the system.

[0296] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I1C) interface, an inter-integrated circuit sound (I1S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0297] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may also adopt different interface connection methods or a combination of multiple interface connection methods as described in the above embodiments.

[0298] The charging management module 140 is used to receive charging input from the charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130.

[0299] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.

[0300] The wireless communication function of the terminal device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0301] In some feasible implementations, the terminal device 100 can use wireless communication functions to communicate with other devices. For example, the terminal device 100 can communicate with a second electronic device, establish a screen mirroring connection with the second electronic device, and output screen mirroring data to the second electronic device. The screen mirroring data output by the terminal device 100 can be audio or video data.

[0302] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0303] The mobile communication module 150 can provide solutions for wireless communication, including 1G / 3G / 4G / 5G, applied to the terminal device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 2. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0304] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0305] The wireless communication module 160 can provide solutions for wireless communication applications on the terminal device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 1, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0306] In some embodiments, antenna 1 of terminal device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling terminal device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0307] Terminal device 100 implements display functions through a GPU, display screen 194, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0308] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, terminal device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0309] In some feasible implementations, the display screen 194 can be used to display various interfaces of the system output of the terminal device 100.

[0310] Terminal device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0311] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0312] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the terminal device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0313] A digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals.

[0314] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. Thus, terminal device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG1, MPEG3, MPEG4, etc.

[0315] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in terminal devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0316] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0317] The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the terminal device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as the indoor positioning method in this embodiment), etc. The data storage area may store data created during the use of the terminal device 100 (such as audio data, phonebook, etc.). Furthermore, the internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0318] Terminal device 100 can implement audio functions through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor, such as music playback and recording. In some feasible implementations, audio module 170 can be used to play the sound corresponding to the video. For example, when display screen 194 displays the video playback screen, audio module 170 outputs the sound of the video playback.

[0319] The audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal.

[0320] The loudspeaker 170A, also known as a "loudspeaker", is used to convert audio electrical signals into sound signals.

[0321] The receiver 170B, also known as the "earpiece", is used to convert audio electrical signals into sound signals.

[0322] The microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals.

[0323] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0324] Pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, pressure sensor 180A may be disposed on display screen 194. Gyroscope sensor 180B can be used to determine the motion posture of terminal device 100. Barometric pressure sensor 180C is used to measure barometric pressure.

[0325] The accelerometer 180E can detect the magnitude of acceleration of the terminal device 100 in various directions (including three-axis or six-axis). When the terminal device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the attitude of the terminal device and can be applied to applications such as landscape / portrait switching and pedometers.

[0326] Distance sensor 180F is used to measure distance.

[0327] The 180L ambient light sensor is used to detect ambient light intensity.

[0328] The fingerprint sensor 180H is used to collect fingerprints.

[0329] The 180J temperature sensor is used to detect temperature.

[0330] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of terminal device 100, in a different position than display screen 194.

[0331] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Terminal device 100 can receive button input and generate key signal inputs related to user settings and function control of terminal device 100.

[0332] Motor 191 can generate vibration alerts.

[0333] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0334] The SIM card interface 195 is used to connect the SIM card.

[0335] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0336] It should be understood that "multiple" as mentioned herein refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., do not necessarily imply that they are different.

[0337] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for extracting timbre, characterized in that, The method includes: M initial audio samples are input into the first timbre extraction model to obtain the first timbre feature. The initial audio samples are obtained by collecting data from a first object, which is an object that cannot be recognized by speech. M is an integer greater than or equal to 1. The first timbre feature and N first media information are input into the first audio synthesis model to obtain N first synthesized audios, where N is an integer greater than or equal to 1; If the N first synthesized audios satisfy the first convergence condition, then the first timbre feature is determined as the timbre feature of the first object; The first timbre extraction model is obtained by adjusting the second timbre extraction model based on N second synthesized audios, N third synthesized audios, the quality prediction results corresponding to the N second synthesized audios respectively, and the user's evaluation results on the N second synthesized audios, including the quality evaluation results corresponding to the N second synthesized audios respectively. The N second synthesized audios are obtained by inputting the second timbre features and the N first media information into the second audio synthesis model respectively. The quality prediction results corresponding to the N third synthesized audios and the N second synthesized audios are obtained by inputting the N second synthesized audios into the second timbre extraction model to obtain N third timbre features, and then inputting the N third timbre features into the second audio synthesis model. The second timbre features are obtained by inputting the M initial audios into the second timbre extraction model. The second timbre extraction model is a trained timbre extraction model, or is obtained by adjusting the trained timbre extraction model. The second audio synthesis model is a trained audio synthesis model, or is obtained by adjusting the trained audio synthesis model.

2. The method as described in claim 1, characterized in that, The user's evaluation results regarding the N second synthesized audios also include the category evaluation results corresponding to the N second synthesized audios. When the second audio synthesis model is input with the second timbre feature and the N first media information, it is also input with N reference category information corresponding one-to-one with the N first media information. The first timbre extraction model is obtained by adjusting the second timbre extraction model based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios, the N third synthesized audios, and the N second synthesized audios, as well as the category evaluation results corresponding to the N reference category information and the N second synthesized audios.

3. The method as described in claim 1 or 2, characterized in that, Both the quality prediction result and the quality evaluation result include quality results in K dimensions. The quality results in K dimensions include audio quality scores and / or audio pair quality comparison results. The audio pair quality comparison results are determined by comparing the quality between two audios in an audio pair containing the corresponding synthesized audio. The audio pair includes the corresponding synthesized audio and another audio. K is an integer greater than or equal to 1.

4. The method as described in claim 2, characterized in that, Both the reference category information and the category evaluation results include L-dimensional categories, which include audio emotion categories and / or audio scene categories, where L is an integer greater than or equal to 1.

5. The method according to any one of claims 1-4, characterized in that, The first audio synthesis model is obtained by adjusting the second audio synthesis model based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios, the N third synthesized audios, and the N second synthesized audios, respectively.

6. The method as described in claim 5, characterized in that, The user's evaluation results regarding the N second synthesized audios also include the category evaluation results corresponding to the N second synthesized audios respectively. When the second audio synthesis model is input with the second timbre feature and the N first media information, it is also input with N reference category information corresponding one-to-one with the N first media information. The first audio synthesis model is obtained by adjusting the second audio synthesis model based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios, the N third synthesized audios, and the N second synthesized audios respectively, as well as the N reference category information and the category evaluation results corresponding to the N second synthesized audios respectively.

7. The method according to any one of claims 1-6, characterized in that, After determining the first timbre feature as the timbre feature of the first object, the method further includes: The M initial audios and M initial videos are input into the audio-visual feature extraction model to obtain the first audio-visual features, wherein the M initial videos correspond one-to-one with the M initial audios; Based on the first audio and video features, a fourth timbre feature and second media information are obtained from the database. The fourth timbre feature is the timbre feature corresponding to the second object, which is different from the first object. The first timbre feature and the fourth timbre feature are fused together to obtain the fifth timbre feature; The fifth timbre feature and the second media information are input into the first audio synthesis model to obtain the fourth synthesized audio.

8. The method as described in claim 7, characterized in that, The database is used to store the correspondence between audio and video features, timbre features and media information; The step of obtaining the fourth timbre feature and the second media information from the database based on the first audio and video features includes: From the correspondence, obtain audio and video features that match the first audio and video features to obtain one or more candidate audio and video features; Based on the one or more candidate audio and video features, a second audio and video feature is determined; The timbre feature corresponding to the second audio-visual feature in the correspondence is determined as the fourth timbre feature, and the media information corresponding to the second audio-visual feature in the correspondence is determined as the second media information.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: The N second synthesized audios, along with the user's evaluation results regarding the N second synthesized audios, are sent to the server.

10. A model training method, characterized in that, The method includes: Obtain T sample audios and the sample annotation results corresponding to the T sample audios, where T is an integer greater than or equal to 1; Based on the T sample audios and the sample annotation results corresponding to the T sample audios, the initial timbre extraction model and the initial audio synthesis model are jointly trained to obtain the trained timbre extraction model and the trained audio synthesis model as described in any one of claims 1-9.

11. The method as described in claim 10, characterized in that, The sample labeling results include quality labeling results; The joint training of the initial timbre extraction model and the initial audio synthesis model based on the T sample audio samples and their corresponding sample annotation results includes: The T sample audios are input into the initial timbre extraction model to obtain T first sample timbre features, and the T first sample timbre features are input into the initial audio synthesis model to obtain T first sample synthesized audios; Based on the T sample audios and the T first sample synthesized audios, the initial timbre extraction model and the initial audio synthesis model are jointly trained; The converged initial timbre extraction model is used as the trained timbre extraction model. The network structure of the converged initial audio synthesis model is modified to obtain the modified audio synthesis model. The T sample audios are input into the trained timbre extraction model to obtain T second sample timbre features. The T second sample timbre features are then input into the modified audio synthesis model to obtain T second sample synthesized audios and the quality prediction results corresponding to the T sample audios, respectively. Based on the T sample audios, the T second sample synthesized audios, and the quality prediction results and quality labeling results corresponding to the T sample audios, the modified audio synthesis model is trained to obtain the trained audio synthesis model.

12. The method as described in claim 11, characterized in that, The sample labeling results also include category labeling results; The process of inputting the T sample audios into the trained timbre extraction model to obtain T second sample timbre features, and then inputting the T second sample timbre features into the modified audio synthesis model to obtain T second sample synthesized audios and the quality prediction results corresponding to the T sample audios, includes: The T sample audios are input into the trained timbre extraction model to obtain the T second sample timbre features. The T second sample timbre features and the category labeling results corresponding to the T sample audios are input into the modified audio synthesis model to obtain the T second sample synthesized audios, the quality prediction results and category prediction results corresponding to the T sample audios. The process of training the modified audio synthesis model based on the T sample audios, the T second sample synthesized audios, and the quality prediction and quality labeling results corresponding to the T sample audios includes: The modified audio synthesis model is trained based on the T sample audios, the T second sample synthesized audios, the quality prediction results and quality labeling results corresponding to the T sample audios, and the category prediction results and category labeling results corresponding to the T sample audios.

13. The method as described in claim 10, characterized in that, The sample labeling results include quality labeling results; The joint training of the initial timbre extraction model and the initial audio synthesis model based on the T sample audio samples and their corresponding sample annotation results includes: The T sample audios are input into the initial timbre extraction model to obtain T first sample timbre features. The T first sample timbre features are then input into the initial audio synthesis model to obtain T first sample synthesized audios and the quality prediction results corresponding to the T sample audios respectively. Based on the T sample audios, the T first sample synthesized audios, and the quality prediction and quality labeling results corresponding to the T sample audios, the initial timbre extraction model and the initial audio synthesis model are jointly trained.

14. The method as described in claim 13, characterized in that, The sample labeling results also include category labeling results; The step of inputting the T sample audios into the initial timbre extraction model to obtain T first sample timbre features, and inputting the T first sample timbre features into the initial audio synthesis model to obtain T first sample synthesized audios and the quality prediction results corresponding to the T sample audios respectively, includes: The T sample audios are input into the initial timbre extraction model to obtain the T first sample timbre features. The T first sample timbre features and the category labeling results corresponding to the T sample audios are input into the initial audio synthesis model to obtain the T first sample synthesized audios, as well as the quality prediction results and category prediction results corresponding to the T sample audios. The joint training of the initial timbre extraction model and the initial audio synthesis model based on the T sample audios, the T first sample synthesized audios, and the quality prediction and quality labeling results corresponding to the T sample audios respectively includes: Based on the T sample audios, the T first sample synthesized audios, the quality prediction results and quality labeling results corresponding to the T sample audios, and the category prediction results and category labeling results corresponding to the T sample audios, the initial timbre extraction model and the initial audio synthesis model are jointly trained.

15. The method according to any one of claims 10-14, characterized in that, The method further includes: Receive N second synthesized audios sent by the user terminal, and the user's evaluation results on the N second synthesized audios, where N is an integer greater than or equal to 1; Based on the N second synthesized audio samples and the user's evaluation results regarding the N second synthesized audio samples, the trained timbre extraction model and audio synthesis model are adjusted.

16. A timbre extraction device, characterized in that, The device includes: The timbre feature extraction module is used to input M initial audios into the first timbre extraction model to obtain the first timbre feature. The initial audios are obtained by collecting data from a first object, which is an object that cannot be recognized by speech. M is an integer greater than or equal to 1. The first audio synthesis module is used to input the first timbre feature and N first media information into the first audio synthesis model to obtain N first synthesized audios, where N is an integer greater than or equal to 1; The timbre feature determination module is used to determine the first timbre feature as the timbre feature of the first object if the N first synthesized audios satisfy the first convergence condition. The first timbre extraction model is obtained by adjusting the second timbre extraction model based on N second synthesized audios, N third synthesized audios, the quality prediction results corresponding to the N second synthesized audios respectively, and the user's evaluation results on the N second synthesized audios, including the quality evaluation results corresponding to the N second synthesized audios respectively. The N second synthesized audios are obtained by inputting the second timbre features and the N first media information into the second audio synthesis model respectively. The quality prediction results corresponding to the N third synthesized audios and the N second synthesized audios are obtained by inputting the N second synthesized audios into the second timbre extraction model to obtain N third timbre features, and then inputting the N third timbre features into the second audio synthesis model. The second timbre features are obtained by inputting the M initial audios into the second timbre extraction model. The second timbre extraction model is a trained timbre extraction model, or is obtained by adjusting the trained timbre extraction model. The second audio synthesis model is a trained audio synthesis model, or is obtained by adjusting the trained audio synthesis model.

17. The apparatus as claimed in claim 16, characterized in that, The user's evaluation results regarding the N second synthesized audios also include the category evaluation results corresponding to the N second synthesized audios. When the second audio synthesis model is input with the second timbre feature and the N first media information, it is also input with N reference category information corresponding one-to-one with the N first media information. The first timbre extraction model is obtained by adjusting the second timbre extraction model based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios, the N third synthesized audios, and the N second synthesized audios, as well as the category evaluation results corresponding to the N reference category information and the N second synthesized audios.

18. The apparatus as claimed in claim 16 or 17, characterized in that, Both the quality prediction result and the quality evaluation result include quality results in K dimensions. The quality results in K dimensions include audio quality scores and / or audio pair quality comparison results. The audio pair quality comparison results are determined by comparing the quality between two audios in an audio pair containing the corresponding synthesized audio. The audio pair includes the corresponding synthesized audio and another audio. K is an integer greater than or equal to 1.

19. The apparatus as claimed in claim 17, characterized in that, Both the reference category information and the category evaluation results include L-dimensional categories, which include audio emotion categories and / or audio scene categories, where L is an integer greater than or equal to 1.

20. The apparatus according to any one of claims 16-19, characterized in that, The first audio synthesis model is obtained by adjusting the second audio synthesis model based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios, the N third synthesized audios, and the N second synthesized audios, respectively.

21. The apparatus as claimed in claim 20, characterized in that, The user's evaluation results regarding the N second synthesized audios also include the category evaluation results corresponding to the N second synthesized audios respectively. When the second audio synthesis model is input with the second timbre feature and the N first media information, it is also input with N reference category information corresponding one-to-one with the N first media information. The first audio synthesis model is obtained by adjusting the second audio synthesis model based on the quality prediction results and quality evaluation results corresponding to the N second synthesized audios, the N third synthesized audios, and the N second synthesized audios respectively, as well as the N reference category information and the category evaluation results corresponding to the N second synthesized audios respectively.

22. The apparatus according to any one of claims 16-21, characterized in that, The device further includes: An audio-visual feature extraction model is used to input the M initial audios and M initial videos into the audio-visual feature extraction model to obtain the first audio-visual features, wherein the M initial videos correspond one-to-one with the M initial audios; The information acquisition module is used to acquire a fourth timbre feature and second media information from the database based on the first audio and video features, wherein the fourth timbre feature is the timbre feature corresponding to the second object, and the second object is different from the first object; The feature fusion module is used to fuse the first timbre feature and the fourth timbre feature to obtain the fifth timbre feature; The second audio synthesis module is used to input the fifth timbre feature and the second media information into the first audio synthesis model to obtain the fourth synthesized audio.

23. The apparatus as claimed in claim 22, characterized in that, The database is used to store the correspondence between audio and video features, timbre features and media information; The information acquisition module is specifically used for: From the correspondence, obtain audio and video features that match the first audio and video features to obtain one or more candidate audio and video features; Based on the one or more candidate audio and video features, a second audio and video feature is determined; The timbre feature corresponding to the second audio-visual feature in the correspondence is determined as the fourth timbre feature, and the media information corresponding to the second audio-visual feature in the correspondence is determined as the second media information.

24. The apparatus according to any one of claims 16-23, characterized in that, The device further includes: The sending module is used to send the N second synthesized audios and the user's evaluation results on the N second synthesized audios to the server.

25. A model training device, characterized in that, The device includes: The sample acquisition module is used to acquire T sample audios and the sample annotation results corresponding to the T sample audios, where T is an integer greater than or equal to 1; The model training module is used to jointly train the initial timbre extraction model and the initial audio synthesis model based on the T sample audios and the sample annotation results corresponding to the T sample audios, so as to obtain the trained timbre extraction model and the trained audio synthesis model as described in any one of claims 1-9.

26. The apparatus as claimed in claim 25, characterized in that, The sample labeling results include quality labeling results; The model training module includes: The first timbre feature extraction submodule is used to input the T sample audios into the initial timbre extraction model to obtain T first sample timbre features, and input the T first sample timbre features into the initial audio synthesis model to obtain T first sample synthesized audios; The first joint training submodule is used to jointly train the initial timbre extraction model and the initial audio synthesis model based on the T sample audios and the T first sample synthesized audios. The model modification submodule is used to modify the network structure of the converged initial timbre extraction model as a trained timbre extraction model to obtain the modified audio synthesis model. The second timbre feature extraction submodule is used to input the T sample audios into the trained timbre extraction model to obtain T second sample timbre features, and input the T second sample timbre features into the modified audio synthesis model to obtain T second sample synthesized audios and the quality prediction results corresponding to the T sample audios respectively. The model training submodule is used to train the modified audio synthesis model based on the T sample audios, the T second sample synthesized audios, and the quality prediction results and quality labeling results corresponding to the T sample audios, so as to obtain the trained audio synthesis model.

27. The apparatus as claimed in claim 26, characterized in that, The sample labeling results also include category labeling results; The second timbre feature extraction submodule is specifically used for: The T sample audios are input into the trained timbre extraction model to obtain the T second sample timbre features. The T second sample timbre features and the category labeling results corresponding to the T sample audios are input into the modified audio synthesis model to obtain the T second sample synthesized audios, the quality prediction results and category prediction results corresponding to the T sample audios. The model training submodule is specifically used for: The modified audio synthesis model is trained based on the T sample audios, the T second sample synthesized audios, the quality prediction results and quality labeling results corresponding to the T sample audios, and the category prediction results and category labeling results corresponding to the T sample audios.

28. The apparatus as claimed in claim 25, characterized in that, The sample labeling results include quality labeling results; The model training module includes: The third timbre feature extraction submodule is used to input the T sample audios into the initial timbre extraction model to obtain T first sample timbre features, and input the T first sample timbre features into the initial audio synthesis model to obtain T first sample synthesized audios and the quality prediction results corresponding to the T sample audios respectively. The second joint training submodule is used to jointly train the initial timbre extraction model and the initial audio synthesis model based on the T sample audios, the T first sample synthesized audios, and the quality prediction results and quality annotation results corresponding to the T sample audios respectively.

29. The apparatus as claimed in claim 28, characterized in that, The sample labeling results also include category labeling results; The third timbre feature extraction submodule is specifically used for: The T sample audios are input into the initial timbre extraction model to obtain the T first sample timbre features. The T first sample timbre features and the category labeling results corresponding to the T sample audios are input into the initial audio synthesis model to obtain the T first sample synthesized audios, as well as the quality prediction results and category prediction results corresponding to the T sample audios. The second joint training submodule is specifically used for: Based on the T sample audios, the T first sample synthesized audios, the quality prediction results and quality labeling results corresponding to the T sample audios, and the category prediction results and category labeling results corresponding to the T sample audios, the initial timbre extraction model and the initial audio synthesis model are jointly trained.

30. The apparatus according to any one of claims 25-29, characterized in that, The device further includes: The receiving module is used to receive N second synthesized audios sent by the user terminal, and the user's evaluation results on the N second synthesized audios, where N is an integer greater than or equal to 1; The model adjustment module is used to adjust the trained timbre extraction model and audio synthesis model based on the N second synthesized audios and the user's evaluation results on the N second synthesized audios.

31. A computer device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor being configured to execute the computer program stored in the memory to implement the steps of the method according to any one of claims 1-15.

32. A computer-readable storage medium, characterized in that, The storage medium stores instructions that, when executed on the computer, cause the computer to perform the steps of the method described in any one of claims 1-15.

33. A computer program, characterized in that, The computer program includes instructions that, when executed on the computer, cause the computer to perform the steps of the method according to any one of claims 1-15.

Images