Tone cloning method and device

Abstract

The embodiment of the invention provides a timbre cloning method. The method comprises the following steps: acquiring a timbre reference audio; loading a timbre cloning engine in response to the timbre cloning request; the timbre cloning request comprises a target text; generating a plurality of timbre cloning tasks through a timbre cloning engine according to the target text; and sending the plurality of timbre cloning tasks and the timbre reference audio to a server through a timbre cloning engine to obtain a timbre cloning result. According to the technical scheme of the embodiment of the invention, the timbre cloning task is efficiently managed and distributed through the timbre cloning engine, so that the efficiency and stability of the timbre cloning process are improved, a user can obtain a high-quality and rapid timbre cloning result, and the use experience of the user is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of timbre cloning, and in particular to a timbre cloning method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of machine learning and artificial intelligence technology, voice cloning technology has made significant breakthroughs in the field of speech synthesis. Voice cloning applications developed through voice cloning technology can capture the voice characteristics of the speaker, including intonation, rhythm, pitch and emotional expression, making the generated speech almost identical to the original voice. However, due to the disordered allocation of voice cloning tasks, the current voice cloning applications have low efficiency in generating voice cloning results and poor user experience.

[0003] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for timbre cloning to solve or alleviate one or more of the technical problems mentioned above.

[0005] One aspect of an embodiment of the present application provides a method for timbre cloning, the method comprising: Get the timbre reference audio; In response to a voice cloning request, loading a voice cloning engine; the voice cloning request includes a target text; According to the target text, generating a plurality of timbre cloning tasks by the timbre cloning engine; The timbre reference audio is sent to a server, and the server is notified of a plurality of timbre cloning tasks in batches through the timbre cloning engine, so as to obtain timbre cloning results corresponding to the target text from the server.

[0006] Optionally, the timbre cloning engine includes: A task generation service, used for generating a plurality of voice cloning tasks according to the target text; A task scheduling service, used for sending a plurality of the timbre cloning tasks to the server; A result acquisition service, used to acquire the timbre cloning result generated by the server; A sound download service, used to download the sound cloning result to a client; and / or The data analysis service is used to collect feedback information to adjust the parameters of the model used by the server to generate the timbre cloning result.

[0007] Optionally, sending a plurality of the timbre cloning tasks to the server includes: Based on preset rules, adjusting configuration parameters of the timbre cloning engine; the configuration parameters are used to determine the number of timbre cloning tasks sent to the server each time; According to the configuration parameters, a preset number of the timbre cloning tasks are sent to the server through the task scheduling service.

[0008] Optionally, based on preset rules, configuration parameters of the timbre cloning engine are set, including: Obtain the current load status of the server; According to the current load condition, the configuration parameters are adjusted to adapt to the current processing capability of the server.

[0009] Optionally, the method further comprises: When the plurality of timbre cloning tasks are completed, obtaining the timbre cloning result from the server through the result obtaining service; Downloading the timbre cloning results corresponding to the plurality of timbre cloning tasks to a buffer area through the timbre download service; According to the timbre cloning result in the buffer area and the plurality of timbre cloning tasks, task progress is obtained.

[0010] Optionally, the method further comprises: Identify text content snippets within the target content; Searching the cache for a target timbre cloning result having the same content as the text content segment; When the target timbre cloning result is found, returning the target timbre cloning result in the buffer area to the client; If the target timbre cloning result is not found, a corresponding timbre cloning task is created for the text content segment.

[0011] Optionally, the timbre cloning engine is formed based on a cross-platform framework.

[0012] Another aspect of the embodiments of the present application provides a device for cloning a tone color, the device comprising: An acquisition module is used to obtain timbre reference audio; A loading module, used for loading a timbre cloning engine in response to a timbre cloning request; the timbre cloning request includes a target text; A generating module, used for generating a plurality of timbre cloning tasks through the timbre cloning engine according to the target text; The sending module is used to send the timbre reference audio to the server and notify the server of multiple timbre cloning tasks in batches through the timbre cloning engine, so as to obtain the timbre cloning results corresponding to the target text from the server.

[0013] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0014] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0015] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0016] The above technical solution adopted in the embodiment of the present application may have the following advantages: the timbre cloning task is efficiently managed and distributed through the timbre cloning engine, and the generation and sending process of the timbre cloning task is optimized. Thus, the efficiency and stability of the timbre cloning process are improved, so that users can obtain high-quality and fast timbre cloning results, and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0018] Figure 1 The flowchart of the method for timbre cloning according to the first embodiment of the present application is schematically shown; Figure 2 The newly added flow chart of the timbre cloning method according to the first embodiment of the present application is schematically shown; Figure 3 Schematically shows Figure 2 Flow chart of sub-steps of step S200; Figure 4 Another newly added flow chart of the method for timbre cloning according to the first embodiment of the present application is schematically shown; Figure 5 A further newly added flow chart of the timbre cloning method according to the first embodiment of the present application is schematically shown; Figure 6 A diagram schematically shows an application example of the timbre cloning method according to the first embodiment of the present application; Figure 7 Another application example diagram of the timbre cloning method according to the first embodiment of the present application is schematically shown; Figure 8 A block diagram of a timbre cloning device according to the second embodiment of the present application is schematically shown; and Fig. 9 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0020] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0021] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0022] First, the following terms are explained: Voice cloning: A technical process that simulates and replicates specific sound characteristics. Based on reference audio, it analyzes the sound spectrum, timbre characteristics and vocalization pattern, and generates voice output similar to the original sound through a deep learning model. It is widely used in speech synthesis, virtual anchors, game sound effects, and personalized voice assistants, allowing users to experience more realistic and personalized voice interaction.

[0023] Task scheduling: The process of allocating tasks to appropriate computing resources based on task requirements, resource availability, and system operating status. The core goal is to optimize resource utilization, shorten task completion time, and ensure successful task completion.

[0024] Cross-platform framework: refers to a software development tool that runs on multiple platforms (such as iOS, Android, Web, etc.) by writing code once, thereby reducing duplication of development work. By sharing business logic code and providing specific adaptation implementations for each platform, developers can improve development efficiency and reduce maintenance costs while providing consistent functions and user experience on different platforms. Cross-platform frameworks include KMP (Kotlin Multiplatform Mobile), Flutter, ReactNative, etc.

[0025] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided by the embodiments of the present application, the relevant technologies are described below: The inventors understand that in the field of client (mobile phones, tablets, etc.) timbre cloning, separate development is required for different platforms, resulting in a long development cycle and great difficulty. At the same time, timbre cloning lacks an effective caching mechanism, which easily leads to repeated requests and increases the burden on the server. And there are deficiencies in data collection and analysis, making it difficult to provide effective data support for the iteration of the timbre cloning large model. In addition, due to the disordered allocation of timbre cloning tasks, the server load is too high or the resource utilization rate of the server is low, resulting in delays in the execution of timbre cloning tasks, low efficiency in generating timbre cloning results, and poor user experience.

[0026] To this end, the embodiment of the present application provides a technical solution for voice cloning. In this technical solution, the voice cloning task is efficiently managed and distributed through a voice cloning engine configured with a task scheduling service, thereby optimizing the generation and sending process of the voice cloning task. This improves the efficiency and stability of the voice cloning process, allowing users to obtain high-quality and fast voice cloning results, and improving the user experience. See below for details.

[0027] Embodiment 1 Figure 1 The flowchart of the timbre cloning method according to the first embodiment of the present application is schematically shown.

[0028] like Figure 1 As shown, the timbre cloning method may include steps S100 to S106, wherein: Step S100, obtaining timbre reference audio; Step S102, in response to a timbre cloning request, loading a timbre cloning engine; the timbre cloning request includes a target text; Step S104, generating a plurality of timbre cloning tasks through the timbre cloning engine according to the target text; Step S106, sending the plurality of timbre cloning tasks and the timbre reference audio to a server to obtain a timbre cloning result.

[0029] This embodiment is executed on the client side, and the voice cloning task is efficiently managed and distributed through the voice cloning engine, which optimizes the generation and sending process of the voice cloning task. Thus, the efficiency and stability of the voice cloning process are improved, so that users can obtain high-quality and fast voice cloning results, and improve the user experience.

[0030] The following combination Figure 1 , each step in steps S100~S106 and other optional steps are explained in detail.

[0031] Step S100 , get the timbre reference audio.

[0032] By obtaining timbre reference audio, the user's voice characteristics, including intonation, rhythm, emotion, and vocal style, are captured, providing a data foundation for subsequent timbre cloning tasks.

[0033] In some embodiments, the timbre reference audio may be audio recorded by the user. The user records real-time audio through the recording function and uploads it as reference audio. The timbre reference audio may also be the voice of other people obtained by the user through online resources. The user may connect to the online audio database through the application interface to search and download timbre reference audio that meets the user's needs. The timbre reference audio may also be a preset timbre template that the user can use after selecting it. For example, a specific timbre such as "baritone", "soprano", "uncle voice" or "loli voice" may be selected as the timbre reference audio. It should be noted that the timbre reference audio is audio recorded by the user himself or authorized.

[0034] In some embodiments, the timbre reference audio may be subjected to noise reduction processing to reduce interference from external noise and improve audio quality. For example, a user records audio in a noisy environment, such as talking on the street. The recorded audio contains obvious background noise, such as traffic noise, passers-by, etc. At this point, a deep learning model may be used to reduce the noise of the audio, eliminate background traffic noise and human voice interference, and leave a clear voice signal.

[0035] Step S102 , in response to a voice cloning request, loading a voice cloning engine; the voice cloning request includes a target text.

[0036] The sound cloning engine is equipped with various services such as task scheduling services.

[0037] In some embodiments, after receiving a voice cloning request, the mobile terminal creates a voice cloning engine according to preset configuration model parameters and caching strategies. The caching strategy is used to indicate how to cache the voice cloning results on the client to reduce repeated network requests and computing burdens. For example, when the target text of the voice cloning task and the locally cached clone result are the same text, the locally cached result can be used directly without re-requesting the server for processing to improve the response speed. The caching strategy can also manage the cache according to the priority of the voice cloning task. Frequently requested or user-important voice cloning results can be cached first, while low-priority or occasionally requested tasks use a shorter cache time or are not cached.

[0038] In some embodiments, the voice cloning engine may adjust the voice cloning tasks sent to the server according to the task scheduling service to ensure that the voice cloning tasks sent to the server can be efficiently executed on the server.

[0039] In some embodiments, the timbre cloning engine can be loaded by automatic loading. Specifically, when a timbre cloning request is received, the client can automatically check whether the timbre cloning engine has been loaded. If not, the timbre cloning engine is automatically loaded to ensure the efficiency of request processing without human intervention. The timbre cloning engine can also be loaded in a multi-threaded parallel manner, and the timbre cloning engine runs in a multi-threaded environment. When multiple timbre cloning requests are initiated, a thread is assigned to each request, and the timbre cloning engine is loaded independently. In this way, it can be applicable to high-concurrency scenarios and can improve the efficiency of processing multiple requests simultaneously.

[0040] In some embodiments, the voice cloning engine can distribute voice cloning tasks to different processing units (such as different servers, computing nodes, etc.) according to the load of the server. Through load balancing, it can avoid overloading of a single node and improve the efficiency and stability of voice cloning. In addition, voice cloning tasks may fail due to network problems, server failures, etc. The voice cloning engine can set a task retry mechanism to automatically detect failed tasks and retry according to the set rules to ensure that voice cloning tasks can be executed correctly.

[0041] Step S104 , according to the target text, multiple timbre cloning tasks are generated by the timbre cloning engine.

[0042] The target text includes the speech content required by the server to generate the voice cloning results. The server can generate the corresponding voice cloning results based on the target text. The target text may contain extra spaces, special symbols or grammatical errors, so the target text can be preprocessed to clean up useless information. For longer target texts, the voice cloning engine can split the target text into multiple sentences, and each sentence can correspond to a voice cloning task, so as to facilitate voice synthesis sentence by sentence. The target text can be segmented based on sentences, paragraphs or specific pause symbols to ensure that the target text is split reasonably to improve the fluency of voice cloning.

[0043] In some embodiments, the timbre cloning engine can split the target text into multiple timbre cloning tasks according to the length or content complexity of the target text. For example, the target text is "Today is a sunny day, Xiao Ming and Xiao Hong went to climb the Great Wall, and they had a lot of fun", and the timbre cloning engine can split it into multiple timbre cloning tasks, such as: "Today is a sunny day", "Xiao Ming and Xiao Hong went to climb the Great Wall", "They had a lot of fun". Each sentence corresponds to a timbre cloning task. By splitting the target text, the efficiency of parallel processing can be improved and the generation time of timbre cloning results can be reduced. At the same time, the split timbre cloning tasks are more concise and easy to handle, which reduces the burden on the server and improves the response speed.

[0044] In some embodiments, the voice cloning engine can also generate different voice cloning tasks based on the semantic and emotional content in the target text. For example, for the target text "I am very happy to see you", the voice cloning engine can analyze its positive emotions and generate multiple voice cloning tasks to meet the needs of different emotional expressions. For example, for the emotional expression of "very happy", high-pitched and pleasant voice features can be used to identify it in the voice cloning task. Through the combination of semantics and emotions, the voice cloning task can more accurately describe the voice output that meets the target voice characteristics, and enhance the naturalness and emotional expression of the voice cloning results.

[0045] Step S106 , sending the timbre reference audio to a server and notifying the server of a plurality of the timbre cloning tasks in batches through the timbre cloning engine, so as to obtain the timbre cloning results corresponding to the target text from the server.

[0046] The timbre cloning task is used to instruct the remote server to perform a cloning operation, and is also called a remote timbre cloning task.

[0047] In some embodiments, the voice cloning engine notifies (e.g., sends) the server side with multiple voice cloning tasks in batches, and can send them to the server side via HTTP or POST requests. The number of tasks in each batch can be determined according to the load of the server side or the network. After the voice cloning engine sends the voice cloning task to the server side, it can periodically request the server side to return task progress or status update information. For example, the voice cloning engine can monitor whether the voice cloning task is completed, whether there is an error, etc., and take appropriate measures (such as resending failed tasks) based on the feedback.

[0048] In some embodiments, in order to ensure data security during the timbre cloning process, the timbre reference audio and task information may be encrypted during transmission, for example, an encryption protocol (such as HTTPS, SSL / TLS) may be used.

[0049] By configuring a voice cloning engine that generates and manages multiple tasks, the client can send multiple voice cloning tasks corresponding to the target text to the server in batches in an orderly manner. This improves the efficiency and stability of the voice cloning process, allowing users to obtain high-quality and fast voice cloning results, and improves the user experience.

[0050] In an optional embodiment, the timbre cloning engine includes: a task generation service, a task scheduling service, a result acquisition service, a timbre download service and / or a data analysis service.

[0051] A task generation service, used for generating a plurality of timbre cloning tasks according to the target text; A task scheduling service, used for sending a plurality of the timbre cloning tasks to the server; A result acquisition service, used to acquire the timbre cloning result generated by the server; A sound download service, used to download the sound cloning result to a client; and / or The data analysis service is used to collect feedback information to adjust the parameters of the model used by the server to generate the timbre cloning result.

[0052] By way of example, the task generation service can split the target text input or uploaded by the user into multiple voice cloning tasks that can be processed in parallel. Subsequently, the task scheduling service distributes several voice cloning tasks and voice reference audio to the server to ensure balanced task load and efficient execution. After the server completes the voice cloning task, the result acquisition service receives the voice cloning results from the server and merges all the results into complete audio data. The voice download service transmits and stores the generated voice cloning results to the client for the user to audition or save. At the same time, the data analysis service collects user feedback information and sends it to the server model training module to optimize the parameters of the voice cloning algorithm model to continuously improve the quality of voice cloning.

[0053] In this embodiment, through the efficient coordination of various services of the voice cloning engine, a closed-loop processing of the entire process of voice cloning requests from task decomposition, scheduling execution to result feedback is achieved, providing users with a high-quality voice cloning experience.

[0054] In an optional embodiment, if Figure 2 As shown, sending a plurality of the timbre cloning tasks to the server includes: Step S200: adjusting the configuration parameters of the timbre cloning engine based on preset rules; the configuration parameters are used to determine the number of timbre cloning tasks sent to the server each time.

[0055] Step S202: sending a preset number of the tone cloning tasks to the server through the task scheduling service according to the configuration parameters.

[0056] The configuration parameters include the number of timbre cloning tasks sent in each batch. The number of timbre cloning tasks sent to the server each time can be determined by dynamically adjusting the "batch task threshold".

[0057] In some embodiments, before the voice cloning task is sent, the current network bandwidth status can be detected and the configuration parameters can be adjusted according to the network bandwidth status. If the bandwidth is sufficient, the number of voice cloning tasks sent each time is increased. If the bandwidth is limited, the sending amount is reduced to ensure stable and reliable task transmission and avoid network congestion. For example, when the network bandwidth is detected to be 100Mbps, 10 tasks can be sent at a time by adjusting the configuration parameters; if the bandwidth drops to 20Mbps, 2~3 tasks are sent at a time to ensure smooth transmission of the tasks.

[0058] In some embodiments, the number of tasks sent each time can also be dynamically adjusted by monitoring the current processing load of the server. If the server resources are idle, more tasks are sent for batch processing; otherwise, the number of tasks is reduced to avoid resource overload. For example, the CPU and memory utilization of the server node can be queried in real time. When the node idle rate is higher than 70%, 15 tasks are sent each time; when the idle rate drops below 30%, only 3 to 5 tasks are sent.

[0059] In this embodiment, by adjusting the configuration parameters of the voice cloning engine based on preset rules, the voice cloning engine can dynamically allocate the number of voice cloning tasks sent to the server. Thus, reasonable allocation and orderly sending of voice cloning tasks are achieved, thereby improving task transmission efficiency and ensuring the stability and performance of the voice cloning engine under various network and resource conditions.

[0060] In an optional embodiment, if Figure 3 As shown, step S200 includes: Step 300, obtaining the current load status of the server.

[0061] Step 302: adjust the configuration parameters according to the current load condition to adapt to the current processing capability of the server.

[0062] In some embodiments, the current load of the server may include CPU usage, memory usage, network bandwidth usage and / or disk I / O status. CPU usage can reflect whether the computing resources of the server are sufficient. When the CPU usage is too high, the processing capacity of the server may be affected, resulting in response delays or task processing failures. The memory usage determines whether the server can efficiently handle multiple concurrent tasks. If the network bandwidth usage of the server is too high, the data transmission speed will be limited, which may cause delays in the transmission of the sound cloning task. Disk I / O refers to the exchange of data between the storage device and the memory. If the I / O operation is too slow or the disk space is insufficient, it may affect the processing speed of the task.

[0063] In some embodiments, the server load data can be collected regularly by periodically calling relevant system tools and APIs (such as Linux top, free, iostat and other commands) or by introducing a monitoring framework (such as Prometheus, Zabbix, etc.). Push mechanisms (such as WebSocket, gRPC, etc.) can be used to achieve real-time monitoring and transmit the server load data to the client in real time. Then, by calculating indicators such as CPU usage, memory usage, and bandwidth utilization, the current load status of the server is comprehensively judged. For example, the CPU usage of the server is 80%, the memory usage is 90%, and the network bandwidth usage is close to 100%. In this case, the client's timbre cloning engine will adjust the task scheduling strategy according to these indicators, reduce the number of tasks sent to the server each time, or postpone the sending of tasks to avoid overloading the server.

[0064] In this embodiment, by obtaining the load of the server, the client can flexibly adjust the configuration parameters of the voice cloning engine. The voice cloning engine can dynamically allocate the number of voice cloning tasks sent to the server according to the configuration parameters, avoid overloading the server, improve the success rate and response speed of the voice cloning tasks, and further improve the user experience.

[0065] In an optional embodiment, if Figure 4 As shown, the method also includes: Step S400: when the plurality of timbre cloning tasks are completed, the timbre cloning result is obtained from the server through the result obtaining service.

[0066] Step S402: Downloading the timbre cloning results corresponding to the plurality of timbre cloning tasks to a buffer area through the timbre download service.

[0067] Step S404: acquiring task progress according to the timbre cloning result in the buffer and the plurality of timbre cloning tasks.

[0068] In some embodiments, according to the configuration parameters of the voice cloning engine, the voice cloning engine sends several voice cloning tasks to the server each time. Before obtaining the voice cloning results, it is possible to confirm whether several voice cloning tasks have been completed through task status monitoring. For example, the processing status of each voice cloning task is checked, and the processing status may include successful completion, failure, etc. Specifically, the processing status of the voice cloning task can be obtained through an asynchronous callback or polling mechanism.

[0069] In some embodiments, the downloaded sound cloning results are first stored in a cache area of ​​the client, which can be a temporary storage space. The sound download service can improve the reliability of downloading through the breakpoint resume function. Specifically, if the network is interrupted or other problems occur during the download of the sound cloning results, the download can be restarted from the last interruption point without starting from the beginning, so as to reduce resource waste and time cost.

[0070] In some embodiments, the voice cloning engine can determine the status of each voice cloning task through the data in the buffer area. For example, if a voice cloning task has been completed and the voice cloning result has been successfully downloaded, the task is marked as "completed". If a voice cloning task has not been sent to the server for processing or the server has not returned the voice cloning result, the task is marked as "pending" or "processing". The progress information can be dynamically updated as the voice cloning task proceeds, so that the user can clearly obtain the progress of the voice cloning task. The voice cloning engine can update the progress in real time by continuously monitoring the execution status of the task and the changes in the buffer area, which can be based on time intervals or event triggers. For example, after each voice cloning result is downloaded, the progress will be updated accordingly. Exemplarily, if the voice cloning request initiated by the client includes 5 voice cloning tasks, the voice download service has successfully downloaded the voice cloning results of 3 voice cloning tasks, the status of these 3 voice cloning tasks is "completed", and the other 2 voice cloning tasks are waiting for the download of voice cloning results, then the status of the voice cloning task is "processing". At this point, the timbre cloning engine can check the current timbre cloning task status through the cache area, summarize the progress information, and display "60% completed". When the timbre cloning results of the remaining two timbre cloning tasks are returned from the server and downloaded to the client, the progress will be updated to "all completed" and fed back to the user through the user interface.

[0071] In this embodiment, by downloading the timbre cloning results corresponding to the timbre cloning tasks to the buffer area, and obtaining the task progress according to the timbre cloning results and task status in the buffer area, the user can clearly obtain the current status of each timbre cloning task. Thus, the execution transparency of the timbre cloning tasks can be improved, the user's sense of control over the timbre cloning process can be enhanced, and the user's experience can be further improved.

[0072] In an optional embodiment, if Figure 5 As shown, the method also includes: Step S500, determining a text content segment in the target content; Step S502, searching the cache area for a target timbre cloning result having the same content as the text content segment; Step S504, when the target timbre cloning result is found, the target timbre cloning result in the buffer area is returned to the client; Step S506: if the target timbre cloning result is not found, create a corresponding timbre cloning task for the text content segment.

[0073] In some embodiments, by checking whether the text content of the text content segment is the same as the timbre cloning result in the buffer area, if there is already a timbre cloning result consistent with the text content of the text content segment in the buffer area, the timbre cloning result can be directly reused. When checking whether the text content of the text content segment is the same as the timbre cloning result in the buffer area, the timbre cloning result can be located by accurate matching of the text content. Exemplarily, the text content segment is "Hello, how are you?", if the timbre cloning task of the text content has been processed before, and the timbre cloning result of the timbre cloning task is also cached in the buffer area. Then, the timbre cloning result corresponding to the text content segment can be directly retrieved from the buffer area, without repeatedly sending the timbre cloning task to the server and making the server calculate the timbre cloning result, thereby improving the response speed and computing efficiency.

[0074] In this embodiment, by reusing the timbre cloning results in the buffer area, the server is prevented from repeatedly calculating the timbre cloning task of the same text content, thereby effectively reducing the overall processing time of the timbre cloning and the computing load of the server.

[0075] In an optional embodiment, the timbre cloning engine is formed based on a cross-platform framework.

[0076] In the process of developing a timbre cloning engine, it is often necessary to develop separately for different platforms (iOS, Android, etc.). The development languages, APIs, and UI frameworks of different platforms are different, resulting in a large amount of business logic code that needs to be repeatedly written for different development languages ​​of different platforms, resulting in a long development cycle and high difficulty. In addition, since different developers may have deviations in their understanding and implementation of requirements, there may be inconsistent implementation details of the same function on different platforms, making it difficult to maintain consistency in functions and user experience, reducing the user experience. Therefore, the timbre cloning engine of this embodiment is formed based on a cross-platform framework.

[0077] In some embodiments, the cross-platform framework may be a KMP framework. The KMP framework enables developers to reuse core logic codes between different platforms by sharing business logic codes, thereby significantly improving development efficiency and reducing maintenance costs. At the same time, KMP may allow specific implementation codes to be written for different platforms to meet the specific functional requirements of each platform. Exemplarily, the shared timbre cloning business logic code (such as task generation service, task scheduling service, etc.) may be written in Kotlin language. The shared code is compiled into a .aar library file for the Android platform and a .framework library file for the iOS platform by the compiler. Afterwards, these library files are introduced on each platform and combined with the platform-specific implementation code to complete the deployment and loading of the timbre cloning engine. Through the KMP framework, the core business logic code of the timbre cloning engine only needs to be developed once and can be loaded on both Android and iOS platforms, thereby improving development efficiency and code reuse rate.

[0078] In this embodiment, by loading the timbre cloning engine formed based on the cross-platform framework, sharing the core business logic code and writing specific implementations for each platform, the code reuse rate can be significantly improved, and the repeated development work between different platforms can be reduced, thereby shortening the development cycle and reducing development costs. In addition, the cross-platform framework ensures the consistency of functions between different platforms, avoiding the problem of inconsistent implementation details caused by the developer's understanding of the requirements, thereby improving the consistency and overall quality of the user experience.

[0079] In order to make this application easier to understand, the following Figure 6-7 An exemplary application is provided.

[0080] S11, the client obtains a timbre reference audio list input by the user, where the timbre reference audio list includes a plurality of timbre reference audios.

[0081] S12, the client creates a timbre cloning engine (including configuration parameters and cache strategies) in response to the timbre cloning request initiated by the user.

[0082] S13, the sound cloning engine distributes tasks.

[0083] Specifically, the audio in the timbre reference audio list is deduplicated through the timbre distributor, and the same audio is cloned only once. Then, a task pool is created and a task listening thread is started.

[0084] S14, the timbre cloning engine divides the timbre cloning tasks into batches for concurrent task processing.

[0085] Take one of the sound cloning tasks A as an example: S15: Check whether the local cache (cache area) has the corresponding tone audio. Go to step S18.

[0086] If yes, return the corresponding timbre audio in the local cache.

[0087] If not, create a timbre cloning task (ie, remote cloning task) A and send it to the server, and wait for the server to return the corresponding cloning result.

[0088] The server can generate a timbre cloning result through the VALL-E algorithm, and obtain the timbre cloning result through the result acquisition service.

[0089] S16: If the server clones successfully, the corresponding clone result (cloned timbre / timbre data / timbre file) is downloaded to the local computer and cached locally, and then the process goes to step S18.

[0090] S17: If the server-side cloning fails, the error information is recorded and the process proceeds to step S18.

[0091] S18: Update task progress.

[0092] S19: Notify the task result of the tone cloning task A.

[0093] S20: Merge all task results (clone results).

[0094] S21, returns the merged final result set to the client.

[0095] In this exemplary application, the voice cloning engine is used to efficiently manage and distribute voice cloning tasks, optimizing the generation and sending process of voice cloning tasks. Thus, the efficiency and stability of the voice cloning process are improved, allowing users to obtain high-quality and fast voice cloning results, improving the user experience.

[0096] Embodiment 2 Figure 8 The block diagram of the voice cloning device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 8 As shown, the device 1600 may include: an acquisition module 1610, a loading module 1620, a generation module 1630, and a sending module 1640, wherein: An acquisition module 1610 is used to acquire a timbre reference audio; A loading module 1620 is used to load a timbre cloning engine in response to a timbre cloning request; the timbre cloning request includes a target text; A generating module 1630, configured to generate a plurality of timbre cloning tasks through the timbre cloning engine according to the target text; The sending module 1640 is used to send the timbre reference audio to the server and notify the server of the plurality of timbre cloning tasks in batches through the timbre cloning engine, so as to obtain the timbre cloning results corresponding to the target text from the server.

[0097] In an optional embodiment, the timbre cloning engine includes: A task generation service, used for generating a plurality of voice cloning tasks according to the target text; A task scheduling service, used for sending a plurality of the timbre cloning tasks to the server; A result acquisition service, used to acquire the timbre cloning result generated by the server; A sound download service, used to download the sound cloning result to a client; and / or The data analysis service is used to collect feedback information to adjust the parameters of the model used by the server to generate the timbre cloning result.

[0098] In an optional embodiment, sending a plurality of the timbre cloning tasks to the server includes: Based on preset rules, adjusting configuration parameters of the timbre cloning engine; the configuration parameters are used to determine the number of timbre cloning tasks sent to the server each time; According to the configuration parameters, a preset number of the timbre cloning tasks are sent to the server through the task scheduling service.

[0099] In an optional embodiment, the configuration parameters of the timbre cloning engine are set based on a preset rule, including: Obtain the current load status of the server; According to the current load condition, the configuration parameters are adjusted to adapt to the current processing capability of the server.

[0100] In an optional embodiment, the device 1600 further includes a task progress acquisition module, which is used to: When the plurality of timbre cloning tasks are completed, obtaining the timbre cloning result from the server through the result obtaining service; Downloading the timbre cloning results corresponding to the plurality of timbre cloning tasks to a buffer area through the timbre download service; According to the timbre cloning result in the buffer area and the plurality of timbre cloning tasks, task progress is obtained.

[0101] In an optional embodiment, the apparatus 1600 further includes a returning module, configured to: Identify text content snippets within the target content; Searching the cache area for a target timbre cloning result having the same content as the text content segment; When the target timbre cloning result is found, returning the target timbre cloning result in the buffer area to the client; If the target timbre cloning result is not found, a corresponding timbre cloning task is created for the text content segment.

[0102] In an optional embodiment, the timbre cloning engine is formed based on a cross-platform framework.

[0103] Embodiment 3 Fig. 9 The hardware architecture diagram of a computer device 10000 suitable for implementing the voice cloning method according to the third embodiment of the present application is schematically shown. In some embodiments, the computer device 10000 can be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. Fig. 9 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10010 may be an internal storage service of the computer device 10000, such as a hard disk or a memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage service of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed in the computer device 10000, such as the program code of the voice cloning method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0104] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0105] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0106] It should be pointed out that Fig. 9 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0107] In this embodiment, the tone cloning method stored in the memory 10010 can also be divided into one or more program services and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0108] Embodiment 4 The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the tone cloning method in the embodiment are implemented.

[0109] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the sound cloning method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0110] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0111] Obviously, those skilled in the art should understand that the various services or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, and optionally, they can be implemented by program codes executable by computer devices, so that they can be stored in a storage device and executed by a computer device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit services, or multiple services or steps therein can be made into a single integrated circuit service for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0112] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for timbre cloning, characterized in that: For a client, the method comprises: Get the timbre reference audio; In response to a voice cloning request, loading a voice cloning engine; the voice cloning request includes a target text; According to the target text, generating a plurality of timbre cloning tasks by the timbre cloning engine; The timbre reference audio is sent to a server, and the server is notified of a plurality of timbre cloning tasks in batches through the timbre cloning engine, so as to obtain timbre cloning results corresponding to the target text from the server.

2. The method according to claim 1, characterized in that The timbre cloning engine comprises: A task generation service, used for generating a plurality of timbre cloning tasks according to the target text; A task scheduling service, used for sending a plurality of the timbre cloning tasks to the server; A result acquisition service, used to acquire the timbre cloning result generated by the server; A sound download service, used to download the sound cloning result to a client; and / or The data analysis service is used to collect feedback information to adjust the parameters of the model used by the server to generate the timbre cloning result.

3. The method according to claim 2, characterized in that Sending a plurality of the timbre cloning tasks to the server includes: Based on preset rules, adjusting configuration parameters of the timbre cloning engine; the configuration parameters are used to determine the number of timbre cloning tasks sent to the server each time; According to the configuration parameters, a preset number of the timbre cloning tasks are sent to the server through the task scheduling service.

4. The method according to claim 3, characterized in that Based on the preset rules, set the configuration parameters of the sound cloning engine, including: Obtain the current load status of the server; According to the current load condition, the configuration parameters are adjusted to adapt to the current processing capability of the server.

5. The method according to any one of claims 3 to 4, characterized in that: The method further comprises: When the plurality of timbre cloning tasks are completed, obtaining the timbre cloning result from the server through the result obtaining service; Downloading the timbre cloning results corresponding to the plurality of timbre cloning tasks to a buffer area through the timbre download service; According to the timbre cloning result in the buffer area and the plurality of timbre cloning tasks, task progress is obtained.

6. The method according to claim 5, characterized in that The method further comprises: Identify text content snippets within the target content; Searching the cache for a target timbre cloning result having the same content as the text content segment; When the target timbre cloning result is found, returning the target timbre cloning result in the buffer area to the client; If the target timbre cloning result is not found, a corresponding timbre cloning task is created for the text content segment.

7. The method according to any one of claims 1 to 4, characterized in that: The timbre cloning engine is formed based on a cross-platform framework.

8. A timbre cloning device, characterized in that: The device comprises: An acquisition module is used to obtain timbre reference audio; A loading module, used for loading a timbre cloning engine in response to a timbre cloning request; the timbre cloning request includes a target text; A generating module, used for generating a plurality of timbre cloning tasks through the timbre cloning engine according to the target text; The sending module is used to send the timbre reference audio to the server and notify the server of multiple timbre cloning tasks in batches through the timbre cloning engine, so as to obtain the timbre cloning results corresponding to the target text from the server.

9. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 7 are implemented.

Images