Role dubbing method and device, electronic equipment and storage medium

Abstract

The invention provides a role dubbing method and device, electronic equipment and a storage medium, and relates to the technical field of voice processing. According to the method, a role and a dubbing person corresponding to the role are determined, so that a target dubbing timbre matched with to-be-dubbed text information can be determined based on an identifier of the dubbing person and an emotion type corresponding to the to-be-dubbed text information of the role, and voice synthesis processing is performed on the to-be-dubbed text information according to the target dubbing timbre. The text-to-voice conversion processing is automatically realized, the dubbing audio corresponding to the to-be-dubbed text information is obtained, the dubbing efficiency is effectively improved, and the cost is relatively low. Moreover, through the determined dubbing person, each candidate timbre of the dubbing person can be obtained, and according to the emotion type corresponding to the to-be-dubbed text information, a target dubbing timbre having a good matching effect with the to-be-dubbed text information can be determined from each candidate timbre, so that the finally obtained dubbing audio effect is better, and the user experience is improved. And the character is better matched.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a method, device, electronic device and storage medium for character dubbing. Background Art

[0002] Game character dubbing is the dubbing of the characters in the game. No matter whether it is the protagonist or the characters in the plot, they all need to use dialogue to promote the development of the game. Therefore, the purpose of game character dubbing is to make the game characters more heartfelt, allowing players to quickly immerse themselves in the game, and making the game plot develop more naturally.

[0003] At present, real-person dubbing is often used to dub game characters. This method requires dubbing personnel to dub each line, which is huge in both economic and time costs. Summary of the invention

[0004] The purpose of the present application is to provide a method, device, electronic device and storage medium for character dubbing, so as to improve dubbing efficiency and reduce dubbing costs.

[0005] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for dubbing a character, comprising:

[0007] Determine the target roles and target voice actors;

[0008] Determine the text information to be dubbed corresponding to the target character;

[0009] Determining a target dubbing timbre according to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed;

[0010] The text information to be dubbed is converted according to the target dubbing timbre to generate initial dubbing audio.

[0011] In a second aspect, the embodiment of the present application further provides a character dubbing device, including: a determination module, a processing module;

[0012] The determination module is used to determine the target role and the target dubbing person;

[0013] The determination module is used to determine the text information to be dubbed corresponding to the target character;

[0014] The determination module is used to determine the target dubbing timbre according to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed;

[0015] The processing module is used to convert the text information to be dubbed according to the target dubbing timbre to generate initial dubbing audio.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the character dubbing method provided in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the character dubbing method provided in the first aspect is executed.

[0018] The beneficial effects of this application are:

[0019] The present application provides a method, device, electronic device and storage medium for dubbing a role. By determining the role and the dubbing person corresponding to the role, the target dubbing timbre that matches the text information to be dubbed can be determined based on the identification of the dubbing person and the emotional type corresponding to the text information to be dubbed of the role, so as to perform speech synthesis processing on the text information to be dubbed according to the target dubbing timbre, automatically realize the text-to-speech conversion processing, and obtain the dubbing audio corresponding to the text information to be dubbed, effectively improving the dubbing efficiency and at a low cost. Moreover, through the determined dubbing person, the candidate timbre of the dubbing person can be obtained, and according to the emotional type corresponding to the text information to be dubbed, the target dubbing timbre that matches the text information to be dubbed better can be determined from the candidate timbre, so that the final dubbing audio effect is better, more matching with the role itself, and making the role more realistic.

[0020] In addition, through the character dubbing editing interface provided in this embodiment, the generated dubbing audio can be further optimized and adjusted, and multi-dimensional dubbing modification operations are provided, so that the generated dubbing audio is controllable, thereby improving the accuracy of the generated dubbing audio through dubbing optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A schematic diagram of a process for dubbing a character provided in an embodiment of the present application;

[0023] Figure 2 A flowchart of another method for dubbing a character provided in an embodiment of the present application;

[0024] Figure 3 A flowchart of another method for dubbing a character provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of a character dubbing editing interface provided in an embodiment of the present application;

[0026] Figure 5 A flowchart of another method for dubbing a character provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of another character dubbing editing interface provided in an embodiment of the present application;

[0028] Figure 7 A flowchart of another method for dubbing a character provided in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of a character dubbing device provided in an embodiment of the present application;

[0030] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0032] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0033] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0034] Whether it is a film or TV work or a game scene, it involves dubbing the characters to make them more heartfelt. Especially for the dubbing of game characters, whether it is the protagonist in the game or the characters in the plot, they all need to use dialogue to promote the development of the game, so the purpose of dubbing game characters is to make the game characters more heartfelt, so that players can quickly get into the game, and make the game plot develop more naturally.

[0035] In real-life game development, it is very expensive to hire real people to dub the game. Moreover, the efficiency of real-person dubbing is also very low. Usually, only a small amount of dubbing can be recorded in a day. When the plot framework and game process need to be quickly built in the early stage of the game, it is difficult to choose real-person dubbing in terms of time and money costs. In many games, there are a large number of unimportant NPC (non-player character) roles. If professional voice actors are hired to record, the cost will be further increased.

[0036] With the development of speech synthesis technology, the current speech synthesis technology can already achieve highly expressive and natural synthesis performance. The game voice dubbing editing tool that combines speech synthesis technology with game dubbing will greatly reduce the time and money costs in the game development process.

[0037] Based on this, this solution proposes a character dubbing method, which can determine the target dubbing timbre that matches the text information to be dubbed based on the identification of the dubbing personnel matched by the user for the character, combined with the emotion type corresponding to the text information to be dubbed of the character, and then generate the target dubbing audio corresponding to the text information to be dubbed using speech synthesis technology according to the target dubbing timbre. The character dubbing can be performed by this method, which can greatly improve the dubbing efficiency and is low in cost.

[0038] Figure 1The following is a flow chart of a method for dubbing a character provided in an embodiment of the present application; the execution subject of the method may be a terminal device or a server interacting with the terminal device. A character dubbing editing interface may be displayed on the terminal device to provide a dubbing editing entry for the user. Figure 1 As shown, the method may include:

[0039] S101. Determine the target role and target voice actor.

[0040] Optionally, the user can upload the script to be dubbed through the script import entrance of the character dubbing editing interface, or input the script to be dubbed by himself, and the script to be dubbed may include a large number of lines. In some embodiments, each line in the script is matched with a corresponding role. After the script is imported, each line and the role information corresponding to each line can be displayed in the character dubbing editing interface. In other embodiments, each line in the script does not have a pre-matched role. After the script is imported, the user can create corresponding roles for different lines, and display each line and the role information corresponding to each line in the character dubbing editing interface.

[0041] Optionally, the user can determine the target role from various roles, determine the target dubbing artist corresponding to the target role from various candidate dubbing artists, and establish a binding relationship between the target role and the target dubbing artist, so that the target dubbing artist can dub the lines associated with the target role.

[0042] In one implementation, a user can identify a target character from among the characters that match the displayed lines by clicking on the character's logo, for example, by clicking on the character's avatar or the character's name, etc. A list of dubbing artists can be displayed on the character dubbing editing interface, and the user can lock the target dubbing artist from the dubbing artist list and click on the target dubbing artist's logo to identify the target dubbing artist.

[0043] In another implementation, a search entry may be provided in the character dubbing editing interface, and the user may input the target character's ID to determine the target character from all characters, and similarly, input the target dubbing artist's ID to determine the target dubbing artist from all candidate dubbing artists. Of course, according to the dubbing effect expected by the target character, a character feature tag matching the expected dubbing effect may be input to search for the target dubbing artist who has the expected dubbing effect.

[0044] Currently, the determination methods are not limited to the two methods provided above.

[0045] S102: Determine text information to be dubbed corresponding to the target character.

[0046] As described above, no matter the lines in the script are pre-associated with roles or roles are created for the lines later, each line in the script has a role associated with it, that is, there is a correspondence between the role and the lines. Then, after the target role is determined, the lines corresponding to the target role can be determined according to the correspondence between the role and the lines. The lines mentioned here refer to the text information to be dubbed. Among them, the text information to be dubbed corresponding to a role may include multiple pieces.

[0047] S103: Determine a target dubbing timbre according to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed.

[0048] In this implementation, dubbing of any text information to be dubbed of a character is taken as an example. In practice, any text information to be dubbed can be dubbed with reference to this.

[0049] Optionally, since each voice actor has different timbres under different emotion types, that is, one voice actor may correspond to multiple timbres, different emotion types may correspond to a specific timbre, and the character may also have different emotions when saying each line. In order to improve the authenticity of the character's dubbing and make it more in line with the context, the target dubbing timbre can be determined based on the emotion type corresponding to the voice actor and the text information to be dubbed.

[0050] Among them, according to the determined target dubbing person, the candidate timbres of the target dubbing person can be determined, and according to the emotion type corresponding to the text information to be dubbed, the target dubbing timbre matching the text information to be dubbed can be determined from the candidate timbres.

[0051] S104: Convert the text information to be dubbed according to the target dubbing timbre to generate initial dubbing audio.

[0052] Optionally, based on the determined target dubbing timbre, speech synthesis technology may be used to perform text-to-speech conversion processing on the text information to be dubbed, and convert the text information to be dubbed into initial dubbing audio after dubbing with the target dubbing timbre.

[0053] In summary, the character dubbing method provided in this embodiment can determine the character and the dubbing personnel corresponding to the character, so as to determine the target dubbing timbre that matches the text information to be dubbed based on the dubbing personnel and the emotional type corresponding to the text information to be dubbed of the character, so as to perform speech synthesis processing on the text information to be dubbed according to the target dubbing timbre, automatically realize the text-to-speech conversion processing, and obtain the dubbing audio corresponding to the text information to be dubbed, which effectively improves the dubbing efficiency and has low cost. Moreover, through the determined dubbing personnel, the candidate timbres of the dubbing personnel can be obtained, and according to the emotional type corresponding to the text information to be dubbed, the target dubbing timbre that matches the text information to be dubbed better can be determined from the candidate timbres, so that the final dubbing audio effect is better, more matched with the character itself, and the character is more realistic.

[0054] Figure 2 A flow chart of another method for dubbing a role provided in an embodiment of the present application; Optionally, in step S103, determining the target dubbing timbre according to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed may include:

[0055] S201, determining at least one timbre corresponding to the target dubbing person according to the target dubbing person and the pre-constructed mapping relationship between each candidate dubbing person and at least one timbre corresponding to each candidate dubbing person; each timbre corresponds to an emotion type.

[0056] Generally, the correspondence between the dubbing staff, the emotion type and the timbre can be pre-built and stored. For example, each candidate dubbing staff is asked to read (record) a predetermined text with a given different emotion type, and the audio of each candidate dubbing staff under different emotion types is generated, and the audio includes the timbre information.

[0057] Based on this, a mapping relationship between each candidate voice actor and each timbre corresponding to each candidate voice actor can be constructed, and a mapping relationship between each timbre of each candidate voice actor and an emotion type can also be constructed.

[0058] Then, based on the determined target and personnel, the timbres corresponding to the target dubbing personnel can be determined.

[0059] Timbre refers to the fact that different sounds always have unique characteristics in terms of waveform, and different objects have different vibration characteristics. The emotional types here can include various natural emotions such as happiness, sadness, anger, etc.

[0060] S202: Determine the dubbing timbre of the target dubbing person under the emotion type according to the emotion type corresponding to the text information to be dubbed and the pre-constructed correspondence between the timbres corresponding to the emotion types of the candidate dubbing persons.

[0061] The context that a character may face when speaking each line of dialogue is different, and the type of emotion is also different. To improve the dubbing effect, for each independent piece of text information to be dubbed, the voice color corresponding to the emotion type can be determined from the various voice colors of the target dubbing personnel according to the emotion type corresponding to the text information to be dubbed.

[0062] S203. Determine the dubbing voice color of the target dubbing personnel under the emotion type as the target dubbing voice color.

[0063] Optionally, the voice color of the target dubbing personnel determined above under the emotion type corresponding to the text information to be dubbed can be determined as the target dubbing voice color.

[0064] Then, when using this target dubbing voice color to dub the text information to be dubbed, the resulting dubbing audio is more realistic in terms of emotion, the dubbing effect is more natural and smooth, and it is more in line with the character itself.

[0065] Figure 3 It is a schematic flowchart of another character dubbing method provided by an embodiment of the present application; optionally, in step S104, converting the text information to be dubbed according to the target dubbing voice color to generate an initial dubbing audio may include:

[0066] S301. Convert the text information to be dubbed into a phoneme sequence.

[0067] In some embodiments, when converting text information into audio, the input text information to be dubbed can be first converted into a phoneme sequence. A phoneme is the smallest speech unit divided according to the natural attributes of speech. Analyzing based on the pronunciation actions in a syllable, one action constitutes one phoneme. Phonemes are divided into two major categories: vowels and consonants. For example, the Chinese syllable "ā" has only one phoneme, "ài" has two phonemes, "dài" has three phonemes, etc. Converting the text information to be dubbed into a phoneme sequence is to facilitate computer audio processing, be able to accurately analyze each pronunciation in each text, and improve the text-to-speech conversion effect.

[0068] S302. According to the target dubbing voice color, perform text-to-speech conversion processing on the phoneme sequence to generate the initial dubbing audio corresponding to the text information to be dubbed, and the initial dubbing audio has the target dubbing voice color.

[0069] Here, traditional speech synthesis technology is used for text-to-speech conversion. When performing speech synthesis, it is synthesized according to the target dubbing voice color, so as to convert the text information to be dubbed into an initial dubbing audio with the target dubbing voice color through processing. That is, the audio content of the initial dubbing audio is the text information to be dubbed, and the initial dubbing audio has the target dubbing voice color.

[0070] Optionally, in step S104, after converting the text information to be dubbed according to the target dubbing tone color to generate the initial dubbing audio, the method may further include: in response to a dubbing modification instruction for a target character in the input text information to be dubbed, modifying the dubbing of the target character in the initial dubbing audio to generate an optimized dubbing audio corresponding to the text information to be dubbed; the dubbing modification instruction includes at least one of the following: pronunciation modification instruction, tone insertion instruction, pitch modification instruction, and duration modification instruction.

[0071] In this embodiment, some dubbing modification methods are also provided. The initial dubbing audio generated by the above method may not meet the user's expectations. The user can further perform audio modification operations through the character dubbing editing interface to obtain the ideal dubbing audio.

[0072] Figure 4 It is a schematic diagram of a character dubbing editing interface provided by an embodiment of the present application. As Figure 4 shown, taking a text information to be dubbed as a unit, each text information to be dubbed is independently displayed. First, text regular processing can be performed on the text information to be dubbed, and the text information to be dubbed is divided into units of characters. Each character is an independent individual and can be modified for dubbing separately.

[0073] The purpose of text regular processing is to regularize the symbols that do not pronounce in the text information to be dubbed, such as ellipsis, book titles, etc. These do not pronounce separately in pronunciation, so they can be regularized.

[0074] As Figure 4 shown, at least one dubbing modification operation option can be displayed below each text information to be dubbed, including but not limited to: pronunciation modification, inserting a voice event on the left, inserting a voice event on the right, pitch modification, duration modification, etc.

[0075] In addition, each text information to be dubbed can also correspond to a playback control. By clicking the playback control, the initial dubbing audio corresponding to the text information to be dubbed can be played. After playing, the user can select the characters with unsatisfactory pronunciation for separate modification.

[0076] For example Figure 4 the text information to be dubbed in is: The weather is really nice today! If the user is not satisfied with the dubbing audio of "nice", the user can select "nice" and select at least one operation option from at least one dubbing modification operation option displayed below, and then input a dubbing modification instruction for the target character "nice".

[0077] When the selected dubbing modification operation option is pronunciation modification, the corresponding input dubbing modification instruction is pronunciation modification.

[0078] It is worth noting here that the tone insertion instruction may include: insert voice events on the left and insert voice events on the right. The tone insertion instruction is used to insert tone particles on the left or right side of the character, for example: wow, yeah, hey.

[0079] Pronunciation modification is performed on characters with multiple pronunciations. When generating the initial dubbing audio in the early stage, there may be pronunciation errors for some polyphones. Through pronunciation modification, they can be modified to accurate pronunciations to avoid ambiguity.

[0080] Pitch modification can modify the pitch of pronunciation, while length modification can modify the length of pronunciation.

[0081] The above dubbing modification instructions can be based on Figure 4 Enter and modify the dubbing modification operation options displayed below each text information to be dubbed.

[0082] Of course, in actual applications, the dubbing modification operation options that can be provided are not limited to the ones listed in the figure or text.

[0083] Optionally, in step S104, after the text information to be dubbed is converted according to the target dubbing timbre to generate the initial dubbing audio, it includes: in response to the adjustment operation of the emotional intensity and the audio speech speed of the input initial dubbing audio, generating an optimized dubbing audio corresponding to the text information to be dubbed.

[0084] In other embodiments, in addition to the above-mentioned dubbing modification method, an overall modification of the dubbing audio is also provided, such as Figure 4 As shown, emotion controls, emotion intensity controls, and speech speed controls can also be displayed on the right side of the interface.

[0085] The pronunciation emotion of the generated initial dubbing audio may be adjusted in response to an emotion adjustment operation input by the user through the emotion control, where emotion refers to the above-mentioned emotion type.

[0086] Similarly, based on the emotion intensity control, the emotion intensity of the initial dubbing audio can be adjusted by sliding the control in the horizontal direction. Based on the speech speed control, the pronunciation speech speed of the initial dubbing audio can be adjusted by sliding the control in the horizontal direction.

[0087] Of course, the above-mentioned emotion controls, emotion intensity controls, and speech speed controls can also be used to make individual adjustments to individual characters, which can be set flexibly without any special restrictions.

[0088] Alternatively, if Figure 4As shown, the character dubbing editing interface may also include an export audio control. After the user has adjusted the audio as described above and obtained a satisfactory dubbing audio, the produced audio can be exported through the export audio control.

[0089] Figure 5 A flow chart of another method for dubbing a role provided in an embodiment of the present application; optionally, in the above steps, after generating the optimized dubbing audio corresponding to the text information to be dubbed, the method may further include:

[0090] S501: If the optimized dubbing audio does not meet the dubbing requirement, generate voice information corresponding to the text information to be dubbed according to the target pitch requirement.

[0091] Figure 6 A schematic diagram of another character dubbing editing interface provided in an embodiment of the present application, such as Figure 6 As shown, the interface may include a recording function and a voice changing function.

[0092] In some embodiments, if the optimized dubbing audio obtained after the above-mentioned individual character pronunciation modification and overall audio modification still does not meet the user's needs, it can be further optimized and adjusted using the method of this embodiment.

[0093] Optionally, by triggering the recording start control, the user can record the text information to be dubbed in the required tone to generate voice information. Alternatively, a professional dubbing person can record the text information to be dubbed to generate voice information. The text information to be dubbed here refers to the text information corresponding to the optimized dubbing audio that the user is not satisfied with. It also refers to the original script corresponding to the optimized dubbing audio.

[0094] S502: Process the voice information according to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed, and generate a target dubbing audio corresponding to the text information to be dubbed.

[0095] Optionally, the generated voice information can be uploaded through an upload control, and then the voice processing can be triggered by generating a voice change control to convert the voice information into the timbre of the dubbing person corresponding to the role, thereby obtaining the target dubbing audio. Among them, the voice content and pitch of the target dubbing audio remain unchanged from those of the voice information, but the timbre is adjusted to the timbre matched by the text information to be dubbed corresponding to the voice information. That is, the voice information is converted into a target dubbing audio having the target dubbing timbre, using the target dubbing person matched with the target role and the target dubbing timbre determined by the emotion type corresponding to the text information to be dubbed.

[0096] Usually, after the aforementioned individual character modifications and modifications to emotions, etc., the optimized dubbing audio obtained can basically meet the user's expectations, and only in some cases there will be unsatisfactory effects. For individual dubbings with unsatisfactory effects, they can be processed separately through the voice processing method of this embodiment, and the data processing volume is still relatively small, which will not cause processing pressure on the system.

[0097] Figure 7 A flowchart of another method for dubbing a role provided in an embodiment of the present application; Optionally, in step S502, the voice information is processed according to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed, and the target dubbing audio corresponding to the text information to be dubbed is generated, which may include:

[0098] S701, determining a target dubbing timbre according to a target dubbing actor, an emotion type corresponding to text information to be dubbed, a mapping relationship between pre-constructed identifiers of each candidate dubbing actor and at least one timbre corresponding to each candidate dubbing actor, and a corresponding relationship between each timbre corresponding to each candidate dubbing actor and an emotion type.

[0099] The implementation method of this step can refer to steps S201-S203, and the target dubbing timbre can be determined first. The reason why the dubbing staff matching the role is still used here is to ensure that the same role is dubbed by the same dubbing staff to ensure the accuracy of the dubbing.

[0100] S702: Process the voice information according to the target dubbing timbre, adjust the original timbre of the voice information to the target dubbing timbre, and generate the target dubbing audio.

[0101] Optionally, the original timbre of the voice information may be adjusted according to the target dubbing timbre to convert the voice information into a voice with the target dubbing timbre, thereby obtaining the target dubbing audio. Before and after the conversion, the content and intonation of the voice remain unchanged, and the intonation is the intonation that satisfies the user.

[0102] Optionally, in step S101, determining the target role and the target dubbing artist may include: determining the target role from the roles to be dubbed; determining the target dubbing artist matching the target role based on the label information of each candidate dubbing artist; the label information includes: personality, age, gender, and timbre.

[0103] In combination with the relevant embodiments of the above step S102, the target role may be determined by selecting the target role through the identification of each target role displayed on the interface by clicking to confirm, or by inputting the search keyword of the target role to determine the target role.

[0104] In some embodiments, when selecting a matching voice actor for each role, the selection may be made based on the tag information of the voice actor.

[0105] The target dubbing actor may be determined from among the candidate dubbing actors according to the dubbing effect expected by the target role or the character characteristics expected by the target role.

[0106] The required dubbing timbre and other related information can be determined based on the desired dubbing effect, and the timbre is usually related to the character's gender, personality, voice type, etc. Therefore, based on the label information of the candidate dubbing actors, a target dubbing actor who can produce a timbre that matches the dubbing effect expected by the target character can be determined from among the candidate dubbing actors.

[0107] Exemplarily, the labels of the voice actor may include but are not limited to: personality, age, gender, timbre, for example: male, sunny personality, gentle voice, etc.

[0108] In summary, the character dubbing method provided in this embodiment can determine the character and the dubbing personnel corresponding to the character, so as to determine the target dubbing timbre that matches the text information to be dubbed based on the identification of the dubbing personnel and the emotional type corresponding to the text information to be dubbed of the character, so as to perform speech synthesis processing on the text information to be dubbed according to the target dubbing timbre, automatically realize the text-to-speech conversion processing, and obtain the dubbing audio corresponding to the text information to be dubbed, which effectively improves the dubbing efficiency and has low cost. Moreover, through the determined dubbing personnel, the candidate timbres of the dubbing personnel can be obtained, and according to the emotional type corresponding to the text information to be dubbed, the target dubbing timbre that matches the text information to be dubbed better can be determined from the candidate timbres, so that the final dubbing audio effect is better, more matched with the character itself, and the character is more realistic.

[0109] In addition, through the character dubbing editing interface provided in this embodiment, the generated dubbing audio can be further optimized and adjusted, and multi-dimensional dubbing modification operations are provided, so that the generated dubbing audio is controllable, thereby improving the accuracy of the generated dubbing audio through dubbing optimization.

[0110] The following describes the devices, equipment, storage media, etc. used to execute the character dubbing method provided in this application. The specific implementation process and technical effects are described above and will not be repeated below.

[0111] Figure 8This is a schematic diagram of a character dubbing device provided in an embodiment of the present application. The functions implemented by the character dubbing device correspond to the steps performed by the above method. The device can be understood as the above terminal device or server, or the processor of the server, or can also be understood as a component independent of the above server or processor that implements the functions of the present application under the control of the server, such as Figure 8 As shown, the device may include: a determination module 810, a processing module 820;

[0112] A determination module 810 is used to determine a target role and a target dubbing person;

[0113] A determination module 810 is used to determine a target dubbing timbre according to a target dubbing person and an emotion type corresponding to the text information to be dubbed;

[0114] A determination module 810 is used to determine a target dubbing timbre according to a target dubbing person and an emotion type corresponding to the text information to be dubbed;

[0115] The processing module 820 is used to convert the text information to be dubbed according to the target dubbing timbre to generate initial dubbing audio.

[0116] Optionally, the determination module 810 is specifically used to determine at least one timbre corresponding to the target dubbing person according to the target dubbing person and the pre-constructed mapping relationship between each candidate dubbing person and at least one timbre corresponding to each candidate dubbing person; each timbre corresponds to one emotion type;

[0117] Determine the dubbing timbre of the target dubbing person under the emotional type according to the emotional type corresponding to the text information to be dubbed and the pre-established correspondence between the timbre and the emotional type corresponding to each candidate dubbing person;

[0118] The dubbing timbre of the target dubbing person under the emotion type is determined as the target dubbing timbre.

[0119] Optionally, the processing module 820 is specifically used to convert the text information to be dubbed into a phoneme sequence;

[0120] According to the target dubbing timbre, the phoneme sequence is converted from text to speech to generate an initial dubbing audio corresponding to the text information to be dubbed, and the initial dubbing audio has the target dubbing timbre.

[0121] Optionally, it also includes: an optimization module;

[0122] The optimization module is used to respond to the dubbing modification instruction of the target character in the input text information to be dubbed, modify the dubbing of the target character in the initial dubbing audio, and generate the optimized dubbing audio corresponding to the text information to be dubbed; the dubbing modification instruction includes at least one of the following: pronunciation modification instruction, tone insertion instruction, pitch modification instruction, and sound length modification instruction.

[0123] Optionally, the optimization module is further used to generate optimized dubbing audio corresponding to the text information to be dubbed in response to the adjustment operation of the emotional intensity and audio speech rate of the input initial dubbing audio.

[0124] Optionally, the optimization module is further configured to generate voice information corresponding to the text information to be dubbed according to the target pitch requirement if the optimized dubbing audio does not meet the dubbing requirement;

[0125] According to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed, the voice information is processed to generate the target dubbing audio corresponding to the text information to be dubbed.

[0126] Optionally, the optimization module is specifically used to determine the target dubbing timbre according to the target dubbing person, the emotion type corresponding to the text information to be dubbed, the pre-constructed mapping relationship between each candidate dubbing person and at least one timbre corresponding to each candidate dubbing person, and the corresponding relationship between each timbre corresponding to each candidate dubbing person and the emotion type;

[0127] The voice information is processed according to the target dubbing timbre, the original timbre of the voice information is adjusted to the target dubbing timbre, and the target dubbing audio is generated.

[0128] Optionally, the determination module 810 is specifically used to determine the target role from the roles to be dubbed;

[0129] According to the label information of each candidate voice actor, determine the target voice actor who matches the target role; the label information includes: personality, age, gender, and timbre.

[0130] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital singnal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0131] The above modules can be connected or communicated with each other via a wired connection or a wireless connection. The wired connection may include a metal cable, an optical cable, a hybrid cable, etc., or any combination thereof. The wireless connection may include a connection in the form of a LAN, a WAN, Bluetooth, a ZigBee, or NFC, or any combination thereof. Two or more modules may be combined into a single module, and any one module may be divided into two or more units. Those skilled in the art will clearly understand that for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application.

[0132] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application includes: a processor 801, a storage medium 802 and a bus 803, wherein the storage medium 802 stores machine-readable instructions executable by the processor 801. When the electronic device runs a character dubbing method in the embodiment, the processor 801 communicates with the storage medium 802 through the bus 803, and the processor 801 executes the machine-readable instructions to perform the following steps:

[0133] Determine the target roles and target voice actors;

[0134] Determine the text information to be dubbed corresponding to the target role;

[0135] Determine the target dubbing timbre according to the target dubbing personnel and the emotional type corresponding to the text information to be dubbed;

[0136] The text information to be dubbed is converted according to the target dubbing timbre to generate an initial dubbing audio.

[0137] In a feasible implementation manner, when the processor 801 determines the target dubbing timbre according to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed, it is specifically used to:

[0138] Determine at least one timbre corresponding to the target dubbing person according to the mapping relationship between the target dubbing person and each candidate dubbing person and at least one timbre corresponding to each candidate dubbing person; each timbre corresponds to one emotion type;

[0139] Determine the dubbing timbre of the target dubbing person under the emotional type according to the emotional type corresponding to the text information to be dubbed and the pre-established correspondence between the timbre and the emotional type corresponding to each candidate dubbing person;

[0140] The dubbing timbre of the target dubbing person under the emotion type is determined as the target dubbing timbre.

[0141] In a feasible implementation, when the processor 801 converts the text information to be dubbed according to the target dubbing timbre to generate the initial dubbing audio, it is specifically used to: convert the text information to be dubbed into a phoneme sequence;

[0142] According to the target dubbing timbre, the phoneme sequence is converted from text to speech to generate an initial dubbing audio corresponding to the text information to be dubbed, and the initial dubbing audio has the target dubbing timbre.

[0143] In a feasible implementation scheme, after the processor 801 converts the text information to be dubbed according to the target dubbing timbre and generates the initial dubbing audio, it is also used to: respond to the dubbing modification instruction of the target character in the input text information to be dubbed, modify the dubbing of the target character in the initial dubbing audio, and generate the optimized dubbing audio corresponding to the text information to be dubbed; the dubbing modification instruction includes at least one of the following: pronunciation modification instruction, tone insertion instruction, pitch modification instruction, and sound length modification instruction.

[0144] In a feasible implementation scheme, after the processor 801 converts the text information to be dubbed according to the target dubbing timbre and generates the initial dubbing audio, it is also used to: generate optimized dubbing audio corresponding to the text information to be dubbed in response to the adjustment operation of the emotional intensity and audio speech speed of the input initial dubbing audio.

[0145] In a feasible implementation manner, after executing the generation of the optimized dubbing audio corresponding to the text information to be dubbed, the processor 801 is further configured to: if the optimized dubbing audio does not meet the dubbing requirement, generate the voice information corresponding to the text information to be dubbed according to the target pitch requirement;

[0146] According to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed, the voice information is processed to generate the target dubbing audio corresponding to the text information to be dubbed.

[0147] In a feasible implementation scheme, when the processor 801 processes the speech information according to the target dubbing person and the emotion type corresponding to the text information to be dubbed, and generates the target dubbing audio corresponding to the text information to be dubbed, it is specifically used to: determine the target dubbing timbre according to the target dubbing person, the emotion type corresponding to the text information to be dubbed, the mapping relationship between the pre-constructed identification of each candidate dubbing person and at least one timbre corresponding to each candidate dubbing person, and the corresponding relationship between each timbre corresponding to each candidate dubbing person and the emotion type;

[0148] The voice information is processed according to the target dubbing timbre, the original timbre of the voice information is adjusted to the target dubbing timbre, and the target dubbing audio is generated.

[0149] In a feasible implementation scheme, when the processor 801 determines the target role and the target dubbing person, it is specifically used to: determine the target role from the roles to be dubbed; determine the target dubbing person who matches the target role based on the label information of each candidate dubbing person; the label information includes: personality, age, gender, and timbre.

[0150] Through the above method, when the graphical user interface in the terminal device is a character dubbing editing interface, the server can determine the target character and the target dubbing person corresponding to the target character based on the user's operation, so that the target dubbing timbre that matches the text information to be dubbed can be determined based on the target dubbing person and the emotional type corresponding to the text information to be dubbed of the character, so as to perform speech synthesis processing on the text information to be dubbed according to the target dubbing timbre, automatically realize the text-to-speech conversion processing, and obtain the dubbing audio corresponding to the text information to be dubbed, effectively improving the dubbing efficiency and at a low cost. Moreover, through the target dubbing person, the candidate timbre of the target dubbing person can be obtained, and according to the emotional type corresponding to the text information to be dubbed, the target dubbing timbre that matches the text information to be dubbed better can be determined from each candidate timbre, so that the final dubbing audio effect is better, more matched with the character itself, and the character is more real.

[0151] Among them, the storage medium 802 stores program code, and when the program code is executed by the processor 801, the processor 801 executes various steps in the character dubbing method according to various exemplary embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0152] Processor 801 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DigitalSignal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware processor to be executed, or a combination of hardware and software modules in the processor can be executed.

[0153] Storage medium 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. The memory is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The storage medium 802 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0154] Optionally, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor performs the following steps:

[0155] Determine the target roles and target voice actors;

[0156] Determine the text information to be dubbed corresponding to the target role;

[0157] Determine the target dubbing timbre according to the target dubbing personnel and the emotional type corresponding to the text information to be dubbed;

[0158] The text information to be dubbed is converted according to the target dubbing timbre to generate an initial dubbing audio.

[0159] In a feasible implementation scheme, when the processor determines the target dubbing timbre according to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed, it is specifically used to:

[0160] Determine at least one timbre corresponding to the target dubbing person according to the mapping relationship between the target dubbing person and each candidate dubbing person and at least one timbre corresponding to each candidate dubbing person; each timbre corresponds to one emotion type;

[0161] Determine the dubbing timbre of the target dubbing person under the emotional type according to the emotional type corresponding to the text information to be dubbed and the pre-established correspondence between the timbre and the emotional type corresponding to each candidate dubbing person;

[0162] The dubbing timbre of the target dubbing person under the emotion type is determined as the target dubbing timbre.

[0163] In a feasible implementation, when the processor converts the text information to be dubbed according to the target dubbing timbre to generate the initial dubbing audio, it is specifically used to: convert the text information to be dubbed into a phoneme sequence;

[0164] According to the target dubbing timbre, the phoneme sequence is converted from text to speech to generate an initial dubbing audio corresponding to the text information to be dubbed, and the initial dubbing audio has the target dubbing timbre.

[0165] In a feasible implementation scheme, after the processor converts the text information to be dubbed according to the target dubbing timbre and generates the initial dubbing audio, it is also used to: respond to the dubbing modification instruction of the target character in the input text information to be dubbed, modify the dubbing of the target character in the initial dubbing audio, and generate the optimized dubbing audio corresponding to the text information to be dubbed; the dubbing modification instruction includes at least one of the following: pronunciation modification instruction, tone insertion instruction, pitch modification instruction, and sound length modification instruction.

[0166] In a feasible implementation scheme, after the processor converts the text information to be dubbed according to the target dubbing timbre to generate the initial dubbing audio, it is also used to: generate optimized dubbing audio corresponding to the text information to be dubbed in response to the adjustment operation of the emotional intensity and audio speech speed of the input initial dubbing audio.

[0167] In a feasible implementation scheme, after executing the generation of the optimized dubbing audio corresponding to the text information to be dubbed, the processor is further used to: if the optimized dubbing audio does not meet the dubbing requirement, generate the voice information corresponding to the text information to be dubbed according to the target pitch requirement;

[0168] According to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed, the voice information is processed to generate the target dubbing audio corresponding to the text information to be dubbed.

[0169] In a feasible implementation scheme, when the processor processes the speech information according to the target dubbing person and the emotion type corresponding to the text information to be dubbed, and generates the target dubbing audio corresponding to the text information to be dubbed, the processor is specifically used to: determine the target dubbing timbre according to the target dubbing person, the emotion type corresponding to the text information to be dubbed, the mapping relationship between the pre-constructed identification of each candidate dubbing person and at least one timbre corresponding to each candidate dubbing person, and the corresponding relationship between each timbre corresponding to each candidate dubbing person and the emotion type;

[0170] The voice information is processed according to the target dubbing timbre, the original timbre of the voice information is adjusted to the target dubbing timbre, and the target dubbing audio is generated.

[0171] In a feasible implementation scheme, when the processor determines the target role and the target dubbing person, it is specifically used to: determine the target role from the roles to be dubbed; determine the target dubbing person who matches the target role based on the label information of each candidate dubbing person; the label information includes: personality, age, gender, and timbre.

[0172] Through the above method, when the graphical user interface in the terminal device is a character dubbing editing interface, the server can determine the target character and the target dubbing person corresponding to the target character based on the user's operation, so that the target dubbing timbre that matches the text information to be dubbed can be determined based on the target dubbing person and the emotional type corresponding to the text information to be dubbed of the character, so as to perform speech synthesis processing on the text information to be dubbed according to the target dubbing timbre, automatically realize the text-to-speech conversion processing, and obtain the dubbing audio corresponding to the text information to be dubbed, effectively improving the dubbing efficiency and at a low cost. Moreover, through the target dubbing person, the candidate timbre of the target dubbing person can be obtained, and according to the emotional type corresponding to the text information to be dubbed, the target dubbing timbre that matches the text information to be dubbed better can be determined from each candidate timbre, so that the final dubbing audio effect is better, more matched with the character itself, and the character is more real.

[0173] In the embodiment of the present application, when the computer program is run by the processor, other machine-readable instructions can also be executed to execute other methods such as those in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment, which will not be repeated here.

[0174] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0175] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0177] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

Claims

1. A method for dubbing a role, characterized in that: include: Determine the target roles and target voice actors; Determine the text information to be dubbed corresponding to the target character; Determining a target dubbing timbre according to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed; The text information to be dubbed is converted according to the target dubbing timbre to generate initial dubbing audio.

2. The method according to claim 1, characterized in that The step of determining a target dubbing timbre according to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed includes: Determine at least one timbre corresponding to the target dubbing person according to the mapping relationship between the target dubbing person and each candidate dubbing person and at least one timbre corresponding to each candidate dubbing person; each timbre corresponds to one emotion type; Determine the dubbing timbre of the target dubbing person under the emotion type according to the emotion type corresponding to the text information to be dubbed and the pre-established correspondence between the timbres corresponding to the emotion types of the candidate dubbing persons; The dubbing timbre of the target dubbing person under the emotion type is determined as the target dubbing timbre.

3. The method according to claim 1, characterized in that The step of converting the text information to be dubbed according to the target dubbing timbre to generate initial dubbing audio includes: Converting the text information to be dubbed into a phoneme sequence; According to the target dubbing timbre, the phoneme sequence is subjected to text-to-speech conversion processing to generate an initial dubbing audio corresponding to the text information to be dubbed, and the initial dubbing audio has the target dubbing timbre.

4. The method according to claim 1, characterized in that: After converting the text information to be dubbed according to the target dubbing timbre to generate the initial dubbing audio, the method includes: In response to the input dubbing modification instruction of the target character in the text information to be dubbed, the dubbing of the target character in the initial dubbing audio is modified to generate the optimized dubbing audio corresponding to the text information to be dubbed; the dubbing modification instruction includes at least one of the following: pronunciation modification instruction, tone insertion instruction, pitch modification instruction, and sound length modification instruction.

5. The method according to claim 1, characterized in that After converting the text information to be dubbed according to the target dubbing timbre to generate the initial dubbing audio, the method includes: In response to the input adjustment operation of the emotional intensity and the audio speech rate of the initial dubbing audio, an optimized dubbing audio corresponding to the text information to be dubbed is generated.

6. The method according to claim 4 or 5, characterized in that: After generating the optimized dubbing audio corresponding to the text information to be dubbed, the method further includes: If the optimized dubbing audio does not meet the dubbing requirement, generating voice information corresponding to the text information to be dubbed according to the target pitch requirement; The voice information is processed according to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed, and a target dubbing audio corresponding to the text information to be dubbed is generated.

7. The method according to claim 6, characterized in that The processing of the voice information according to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed to generate the target dubbing audio corresponding to the text information to be dubbed includes: Determine the target dubbing timbre according to the target dubbing artist, the emotion type corresponding to the text information to be dubbed, the pre-established mapping relationship between each candidate dubbing artist and at least one timbre corresponding to each candidate dubbing artist, and the correspondence relationship between each timbre corresponding to each candidate dubbing artist and the emotion type; The voice information is processed according to the target dubbing timbre, the original timbre of the voice information is adjusted to the target dubbing timbre, and the target dubbing audio is generated.

8. The method according to claim 1, characterized in that Determining the target role and the target voice actor includes: Determine the target role from among the roles to be dubbed; According to the label information of each candidate voice actor, a target voice actor matching the target role is determined; the label information includes: personality, age, gender, and timbre.

9. A character dubbing device, characterized in that: include: Determine the module and process the module; The determination module is used to determine the target role and the target dubbing person; The determination module is used to determine the target dubbing timbre according to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed; The determination module is used to determine the target dubbing timbre according to the target dubbing personnel and the emotion type corresponding to the text information to be dubbed; The processing module is used to convert the text information to be dubbed according to the target dubbing timbre to generate initial dubbing audio.

10. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the program instructions to perform the character dubbing method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the character dubbing method according to any one of claims 1 to 8 is executed.

Images