A processing method and device for video synthesis based on personalized voice

Abstract

The present application discloses a processing method and device for video synthesis based on personalized voice, which obtains data to be processed and a video object in an authorized state, and performs intelligent audit on the data to be processed through intelligent audit technology to obtain an audit result. If the audit result is legal data to be processed, the sound line of the video object is simulated through preset simulation rules to obtain the target voice, and the target voice is synthesized with the video corresponding to the video object to obtain a personalized video. Through the above scheme, there is no need to manually intercept and splice personalized voice word by word. It is only necessary to use a deep learning method to simulate the voice line of the video object to synthesize the personalized voice, and synthesize it with the video corresponding to the subsequent video object, thereby improving the effect, diversity and fun of personalized voice and video synthesis. In addition, the compliance of the content input by the user is reviewed through intelligent audit technology, and there is no need for manual participation in the review, thereby improving the efficiency of the review.

Description

Technical Field

[0001] The present application relates to the technical field of audio and video processing, and more specifically, to a processing method and device for video synthesis based on personalized voice. Background Art

[0002] With the improvement of computing power and the collection of Internet data, artificial intelligence has entered a new stage of development. Artificial intelligence is gradually changing the way people interact with computers. An important part of the human-computer interaction process is to simulate real people and interact with users. Combined with speech synthesis and speech conversion technology, personalized speech and video synthesis can be achieved.

[0003] In the prior art, if a character in a video is to speak text content specified by a user, an editor is required to search for words in all the words spoken by the character in the video that have the same pronunciation as the words in the text, and splice them into a paragraph.

[0004] However, the speech that is manually intercepted and spliced ​​word by word may have speech incoherence and stiff speech, which makes the personalized speech and video synthesis effect poor and less interesting. Summary of the invention

[0005] In view of this, the present application discloses a processing method and device for video synthesis based on personalized voice, aiming to improve the effect, diversity and fun of personalized voice and video synthesis. In addition, the content input by the user is reviewed for compliance through intelligent review technology, without the need for manual review, thereby improving the efficiency of the review.

[0006] In order to achieve the above purpose, the disclosed technical solution is as follows:

[0007] The first aspect of the present application discloses a processing method for video synthesis based on personalized voice, the method comprising:

[0008] Acquire data to be processed and a video object in an authorized state; the data to be processed includes personalized text data and / or personalized audio data;

[0009] Through intelligent audit technology, the data to be processed is intelligently audited to obtain audit results; the intelligent audit is used to detect the legitimacy of the data to be processed;

[0010] If the audit result is a legitimate data to be processed, the voice line of the video object is simulated by a preset simulation rule to obtain a target voice; the target voice is a personalized voice that is consistent with the voice line of the video object and is generated according to the legitimate data to be processed;

[0011] The target voice is synthesized with the video corresponding to the video object to obtain a personalized video.

[0012] Preferably, the obtaining of the data to be processed and the video object in the authorized state includes:

[0013] Obtain the data to be processed and the video object in the authorized state from the preset information source.

[0014] Preferably, the intelligent auditing of the data to be processed is performed by intelligent auditing technology to obtain the auditing results, including:

[0015] Reviewing the data content in the data to be processed through intelligent review technology;

[0016] When it is monitored that the data content in the data to be processed is consistent with the preset illegal data content, determining that the data to be processed is illegal data to be processed;

[0017] When it is monitored that the data content in the data to be processed is inconsistent with the preset illegal data content, the data to be processed is determined to be legal data to be processed.

[0018] Preferably, when the data to be processed includes personalized text data, if the audit result is that the data to be processed is legal, the voice of the video object is simulated by a preset simulation rule to obtain a target voice, including:

[0019] If the audit result is legal data to be processed, the sound feature information corresponding to the video object is extracted through the voiceprint recognition model, and the sound feature information is embedded and encoded into a vector of fixed dimension; the vector of fixed dimension is used to represent the sound feature of the video object;

[0020] The personalized text data and the fixed-dimensional vector are predicted by a preset cyclic sequence to sequence feature prediction network to obtain a first Mel-spectrogram corresponding to the video object; the first Mel-spectrogram is a Mel-spectrogram obtained by the preset cyclic sequence to sequence feature prediction network;

[0021] The first mel-spectrogram is converted into a first time series sound waveform graph by a speech synthesizer, and a target speech is obtained based on the first time series sound waveform graph.

[0022] Preferably, if the audit result is legal data to be processed, simulating the sound line of the video object by a preset simulation rule to obtain the target voice includes:

[0023] If the audit result is legal data to be processed, obtaining multiple sound feature information corresponding to the video object;

[0024] Synthesizing multiple sound feature information corresponding to the video object through a preset non-autoregressive speech synthesis model architecture to obtain a second Mel-spectrogram; the second Mel-spectrogram is a Mel-spectrogram obtained through the preset non-autoregressive speech synthesis model architecture;

[0025] The second mel-spectrogram is converted into a second time series sound waveform graph by a speech synthesizer, and the target speech is obtained based on the second time series sound waveform graph.

[0026] Preferably, it also includes:

[0027] If the data to be processed is illegal data to be processed, illegal prompt information is generated.

[0028] Preferably, synthesizing the target voice with the video corresponding to the video object to obtain a personalized video includes:

[0029] According to preset special effects, the target voice is synthesized with the video corresponding to the video object to obtain a personalized video.

[0030] The second aspect of the present application discloses a processing device for performing video synthesis based on personalized voice, the device comprising:

[0031] An acquisition unit, used for acquiring data to be processed and a video object in an authorized state; the data to be processed includes personalized text data and / or personalized audio data;

[0032] An audit unit, used to perform an intelligent audit on the data to be processed through intelligent audit technology to obtain an audit result; the intelligent audit is used to detect the legitimacy of the data to be processed;

[0033] A simulation unit, for simulating the voice of the video object according to a preset simulation rule to obtain a target voice if the audit result is a legitimate data to be processed; the target voice is a personalized voice that is consistent with the voice of the video object and is generated according to the legitimate data to be processed;

[0034] The synthesis unit is used to synthesize the target voice with the video corresponding to the video object to obtain a personalized video.

[0035] Preferably, the acquisition unit is specifically used for:

[0036] Obtain the data to be processed and the video object in the authorized state from the preset information source.

[0037] Preferably, the review unit includes:

[0038] An audit module, used to audit the data content in the data to be processed through intelligent audit technology;

[0039] A first determination module is used to determine that the data to be processed is illegal data to be processed when it is monitored that the data content in the data to be processed is consistent with the preset illegal data content;

[0040] The second determination module is used to determine that the data to be processed is legal data to be processed when it is monitored that the data content in the data to be processed is inconsistent with the preset illegal data content.

[0041] It can be known from the above technical scheme that the present application discloses a processing method and device for video synthesis based on personalized voice, obtains data to be processed and a video object in an authorized state, the data to be processed includes personalized text data and / or personalized audio data, and through intelligent audit technology, the data to be processed is intelligently audited to obtain the audit result, and the intelligent audit is used to detect the legality of the data to be processed. If the audit result is legal data to be processed, the sound line of the video object is simulated by a preset simulation rule to obtain a target voice, and the target voice is a personalized voice that is consistent with the sound line of the video object generated according to the legal data to be processed, and the target voice is synthesized with the video corresponding to the video object to obtain a personalized video. Through the above scheme, there is no need to manually intercept and splice personalized voice word by word, only the sound line of the video object is simulated by a deep learning method to synthesize the personalized voice, and it is synthesized with the video corresponding to the subsequent video object, thereby improving the effect, diversity and fun of personalized voice and video synthesis. In addition, the content input by the user is audited by intelligent audit technology to see whether it is compliant, and no manual participation in the audit is required, thereby improving the efficiency of the audit. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0043] Figure 1 A flowchart of a processing method for performing video synthesis based on personalized voice disclosed in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of the structure of a processing device for performing video synthesis based on personalized voice disclosed in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0048] As can be seen from the background technology, in the prior art, if a character in a video is to speak a text content specified by a user, the editor needs to find the content with the same pronunciation as the words in the text in all the words spoken by the character in the video, and splice them into a paragraph. However, the speech manually intercepted and spliced ​​word by word will have speech incoherence and stiff speech, which makes the personalized speech and video synthesis effect poor and low in interest.

[0049] In order to solve the above problems, the embodiment of the present application discloses a processing method and device for video synthesis based on personalized voice, which does not require manual interception and word-by-word splicing of personalized voice. It only needs to use deep learning methods to simulate the sound line of the video object to synthesize personalized voice, and synthesize it with the video corresponding to the subsequent video object, thereby improving the effect, diversity and fun of personalized voice and video synthesis. In addition, the compliance of the content input by the user is reviewed by intelligent review technology, without the need for manual review, thereby improving the efficiency of the review. The specific implementation method is illustrated by the following embodiment.

[0050] refer to Figure 1 As shown, a processing method and device for performing video synthesis based on personalized voice disclosed in an embodiment of the present application are shown, and the processing method and device for performing video synthesis based on personalized voice mainly include the following steps:

[0051] S101: Acquire data to be processed and a video object in an authorized state; the data to be processed includes personalized text data and / or personalized audio data.

[0052] In S101, the video objects include celebrities, animation and cartoon characters, and other virtual characters, cartoon images, and the like.

[0053] The words that the user decides to let the video object say may be personalized text data (such as text content input by the user) and / or personalized audio data (such as audio data recorded by the user).

[0054] The user selects a video character or a virtual character such as an animation that is in an authorized state. The video character has obtained the authorization of the person himself, and the virtual characters such as animation also have copyright.

[0055] Obtaining the authorization of the video character and the copyright of virtual characters such as animation and cartoons means obtaining the video object in an authorized state.

[0056] The determination of specific video objects is set by technical personnel based on actual conditions, and this application does not make any specific limitations.

[0057] For example, if a celebrity agrees to let us use his voice to synthesize part of the audio, generally, the scope of use of this part of the audio will be limited, such as only being used to generate a blessing video, etc.

[0058] The user can decide what the video object wants to say. The video object can obtain what the user wants to say by inputting personalized text data and / or personalized audio data.

[0059] Obtain the data to be processed and the video object in the authorized state from the preset information source.

[0060] The preset information source may be an activity page through an application (APP) or an analysis link, etc. For example, the preset information source, i.e., the activity entry of the activity page may be a Chinese Valentine's Day party, a New Year's Eve live party, a variety show related to a star, an e-commerce platform, etc.

[0061] S102: Through intelligent audit technology, the data to be processed is intelligently audited to obtain the audit result; the intelligent audit is used to detect the legality of the data to be processed. If the audit result is legal data to be processed, S103 is executed; if the audit result is illegal data to be processed, S105 is executed.

[0062] In S102, intelligent audit technology is used to perform intelligent audit on the data to be processed (personalized text data or personalized audio data input by the user), and legal data to be processed and illegal data to be processed (such as prohibited data, spam data, etc.) are accurately identified.

[0063] The intelligent audit technology can be a text content analysis algorithm or other analysis algorithms. The specific intelligent audit technology is determined by the technicians according to the actual situation, and this application does not make specific restrictions. The intelligent audit technology of this application preferably uses a text content analysis algorithm.

[0064] Intelligent review technology is used to automatically review whether the audio or text input by the user is compliant. The content input by the user is detected to effectively avoid the risks and brand damage caused by violations. In addition, intelligent review technology is used to review whether the content input by the user is compliant, without the need for manual review, which greatly increases efficiency.

[0065] Intelligent review technology supports the recognition of variants such as pinyin, homophones, decomposed characters, similar characters, and allusions.

[0066] Specifically, the intelligent audit technology is used to conduct intelligent audit on the data to be processed, and the process of obtaining the audit results is shown in A1-A3.

[0067] A1: Use intelligent audit technology to audit the data content in the data to be processed.

[0068] A2: When it is monitored that the data content in the data to be processed is consistent with the preset illegal data content, the data to be processed is determined to be illegal data to be processed.

[0069] The preset illegal data content includes pre-stored prohibited and spam content.

[0070] The determination of the preset illegal data content is set by the technical personnel according to the actual situation, and this application does not make any specific limitation.

[0071] In order to facilitate understanding of the process of determining that the data to be processed is illegal data to be processed when the data content in the data to be processed is detected to be consistent with the preset illegal data content, an example is given here to illustrate:

[0072] For example, prohibited content is pre-stored in the preset illegal data content. When it is monitored that the data content in the data to be processed involves prohibited content that is consistent with the prohibited content in the preset illegal data content, the data to be processed is determined to be illegal data to be processed.

[0073] A3: When it is detected that the data content in the data to be processed is inconsistent with the preset illegal data content, the data to be processed is determined to be legal data to be processed.

[0074] S103: The voice line of the video object is simulated by a preset simulation rule to obtain a target voice; the target voice is a personalized voice that is generated according to the legal data to be processed and is consistent with the voice line of the video object.

[0075] In S103, when the data to be processed includes personalized text data and the audit result is that the data to be processed is legal, a voice identical to the voice in the video object is generated by simulating the voice of the video object and based on the content of the personalized text data and / or personalized audio data input by the user.

[0076] For example, the user selects a star as the video object, and the personalized text data entered by the user is "Happy New Year, good health". The voice of the star is simulated, and a voice similar to the star's middle voice is generated, and a blessing video of "Happy New Year, good health" is said in the same voice as the star's voice.

[0077] Specifically, the process of simulating the sound line of the video object by preset simulation rules to obtain the target voice includes the first method and the second method:

[0078] The specific process of the first method is shown in B1-B3.

[0079] B1: If the audit result is legal data to be processed, the voice feature information corresponding to the video object is extracted through the voiceprint recognition model, and the voice feature information is embedded and encoded into a vector of fixed dimension; the vector of fixed dimension is used to represent the sound features of the video object;

[0080] Among them, a voiceprint recognition model is established through a sound feature encoder to obtain the sound potential characteristics of the video object.

[0081] The voiceprint recognition model is a model that achieves the purpose of identifying unknown voices by analyzing the characteristics of one or more voice signals. The theoretical basis of voiceprint recognition is that each voice has unique characteristics, and the voices of different people can be effectively distinguished through these characteristics.

[0082] B2: Predict personalized text data and fixed-dimensional vectors through a preset cyclic sequence-to-sequence feature prediction network to obtain a first mel-spectrogram corresponding to the video object; the first mel-spectrogram is a mel-spectrogram obtained through a preset cyclic sequence-to-sequence feature prediction network.

[0083] Among them, the preset cyclic sequence to sequence feature prediction network is the cyclic sequence to sequence feature prediction network Tacotron2 with attention.

[0084] Based on Tacotron2, a recurrent sequence-to-sequence feature prediction network with attention, it is capable of end-to-end speech synthesis.

[0085] The character sequence input by the user (personalized text data of the data to be processed) and the vector of fixed dimension (sound feature information of the video object) are converted into the first Mel-spectrogram corresponding to the video object through the preset cyclic sequence to sequence feature prediction network.

[0086] B3: Convert the first Mel-spectrogram into a first time series sound waveform through a speech synthesizer, and obtain the target speech based on the first time series sound waveform.

[0087] The first Mel-spectrogram (spectral domain) is converted into a first time series sound waveform (time domain) by a speech synthesizer to complete the synthesis of the target speech.

[0088] The specific process of the second method is shown in C1-C3.

[0089] C1: If the audit result is legal data to be processed, obtain multiple sound feature information corresponding to the video object.

[0090] Among them, multiple sound feature information corresponding to the video object is trained. For example, two hundred sound feature information corresponding to the video object can be trained in advance to provide higher accuracy for the target speech obtained subsequently.

[0091] C2: synthesize multiple sound feature information corresponding to the video object through a preset non-autoregressive speech synthesis model architecture to obtain a second Mel-spectrogram; the second Mel-spectrogram is a Mel-spectrogram obtained through a preset non-autoregressive speech synthesis model architecture.

[0092] Among them, the preset non-autoregressive speech synthesis model architecture is the non-autoregressive speech synthesis model architecture FastSpeech2.

[0093] C3: Convert the second mel-spectrogram into a second time series sound waveform through a speech synthesizer, and obtain the target speech based on the second time series sound waveform.

[0094] The second mel-spectrogram (spectral domain) is converted into a second time series sound waveform (time domain) by a speech synthesizer to complete the synthesis of the target speech.

[0095] S104: Synthesize the target voice and the video corresponding to the video object to obtain a personalized video.

[0096] In S104, the target voice is synthesized with the video corresponding to the video object according to the preset special effects to obtain a personalized video.

[0097] The preset special effects include flower characters, sound effects, background music and other special effects. The specific preset characteristics are determined by technical personnel according to actual conditions, and this application does not make specific limitations.

[0098] Combine the target voice with the video recorded by the video object, and add special effects such as flower words, sound effects, background music, etc. to synthesize a complete, customized, smooth, and real personalized video, such as a blessing video.

[0099] The application can be used in live broadcasts of evening parties, celebrity-related variety shows, sending blessings on special holidays, cheering videos for special events, etc.

[0100] Using Artificial Intelligence (AI) technology, we can support mass production of blessing videos in a short period of time and send personalized customized blessing videos to people across the country.

[0101] Personalized videos can be downloaded, shared, and configured as red envelope covers.

[0102] S105: Generate illegal prompt information.

[0103] In S105, illegal prompt information is generated to remind the user that the audit result is illegal data to be processed. If the audit result is illegal data to be processed, the personalized video cannot be synthesized.

[0104] In the embodiment of the present application, there is no need to manually intercept and splice personalized voice word by word. It is only necessary to use deep learning methods to simulate the voice of the video object to synthesize personalized voice, and synthesize it with the video corresponding to the subsequent video object, thereby improving the effect, diversity and fun of personalized voice and video synthesis. In addition, the intelligent review technology is used to review whether the content input by the user is compliant, without the need for manual review, thereby improving the efficiency of the review.

[0105] Based on the above embodiments Figure 1 A processing method for video synthesis based on personalized voice is disclosed, and an embodiment of the present application also discloses a processing device for video synthesis based on personalized voice, such as Figure 2 As shown, the processing device for performing video synthesis based on personalized voice includes an acquisition unit 201, a review unit 202, a simulation unit 203 and a synthesis unit 204.

[0106] The acquisition unit 201 is used to acquire the data to be processed and the video object in the authorized state; the data to be processed includes personalized text data and / or personalized audio data.

[0107] The audit unit 202 is used to perform an intelligent audit on the data to be processed through intelligent audit technology to obtain an audit result; the intelligent audit is used to detect the legitimacy of the data to be processed.

[0108] The simulation unit 203 is used to simulate the voice of the video object according to preset simulation rules to obtain the target voice if the audit result is legal data to be processed; the target voice is a personalized voice consistent with the voice of the video object generated according to the legal data to be processed.

[0109] The synthesis unit 204 is used to synthesize the target voice with the video corresponding to the video object to obtain a personalized video.

[0110] Furthermore, the acquisition unit 201 is specifically configured to acquire the data to be processed and the video objects in the authorized state from a preset information source.

[0111] Furthermore, the review unit 202 includes a review module, a first determination module and a second determination module.

[0112] The audit module is used to audit the data content in the data to be processed through intelligent audit technology.

[0113] The first determination module is used to determine that the data to be processed is illegal data to be processed when it is monitored that the data content in the data to be processed is consistent with the preset illegal data content.

[0114] The second determination module is used to determine that the data to be processed is legal data to be processed when it is monitored that the data content in the data to be processed is inconsistent with the preset illegal data content.

[0115] Furthermore, the simulation unit 203 includes an extraction module, a prediction module and a first conversion module.

[0116] The extraction module is used to extract the sound feature information corresponding to the video object through the voiceprint recognition model if the audit result is legal data to be processed, and embed the sound feature information into a vector of fixed dimension; the vector of fixed dimension is used to characterize the sound features of the video object.

[0117] The prediction module is used to predict personalized text data and fixed-dimensional vectors through a preset cyclic sequence to sequence feature prediction network to obtain a first Mel-spectrogram corresponding to the video object; the first Mel-spectrogram is a Mel-spectrogram obtained through a preset cyclic sequence to sequence feature prediction network.

[0118] The first conversion module is used to convert the first Mel-spectrogram into a first time series sound waveform through a speech synthesizer, and obtain the target speech based on the first time series sound waveform.

[0119] Furthermore, the simulation unit 203 includes an acquisition module, a synthesis module and a second conversion module.

[0120] The acquisition module is used to obtain multiple sound feature information corresponding to the video object if the audit result is legal data to be processed.

[0121] A synthesis module is used to synthesize multiple sound feature information corresponding to the video object through a preset non-autoregressive speech synthesis model architecture to obtain a second Mel-spectrogram; the second Mel-spectrogram is a Mel-spectrogram obtained through the preset non-autoregressive speech synthesis model architecture.

[0122] The second conversion module is used to convert the second Mel-spectrogram into a second time series sound waveform through a speech synthesizer, and obtain the target speech based on the second time series sound waveform.

[0123] Furthermore, the processing device for performing video synthesis based on personalized voice also includes a generating unit.

[0124] The generating unit is used to generate illegal prompt information if the data to be processed is illegal data to be processed.

[0125] Furthermore, the synthesis unit 204 is specifically configured to synthesize the target voice and the video corresponding to the video object according to a preset special effect to obtain a personalized video.

[0126] In the embodiment of the present application, there is no need to manually intercept and splice personalized voice word by word. It is only necessary to use deep learning methods to simulate the voice of the video object to synthesize personalized voice, and synthesize it with the video corresponding to the subsequent video object, thereby improving the effect, diversity and fun of personalized voice and video synthesis. In addition, the intelligent review technology is used to review whether the content input by the user is compliant, without the need for manual review, thereby improving the efficiency of the review.

[0127] An embodiment of the present application further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned processing method for video synthesis based on personalized voice.

[0128] The present application also provides an electronic device, whose structural diagram is as follows: Figure 3 As shown, it specifically includes a memory 301 and one or more instructions 302, wherein one or more instructions 302 are stored in the memory 301 and are configured to be executed by one or more processors 303 to execute the one or more instructions 302 to execute the above-mentioned processing method for video synthesis based on personalized voice.

[0129] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0130] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0131] The steps in the methods of the embodiments of the present application can be adjusted in order, combined, and deleted according to actual needs.

[0132] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0133] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0134] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A processing method for video synthesis based on personalized voice, characterized in that: The method comprises: Acquire data to be processed and a video object in an authorized state; the data to be processed includes personalized text data and / or personalized audio data; Through intelligent audit technology, the data to be processed is intelligently audited to obtain audit results; the intelligent audit is used to detect the legitimacy of the data to be processed; If the audit result is a legitimate data to be processed, the voice line of the video object is simulated by a preset simulation rule to obtain a target voice; the target voice is a personalized voice that is consistent with the voice line of the video object and is generated according to the legitimate data to be processed; Synthesizing the target voice with the video corresponding to the video object to obtain a personalized video; Wherein, when the data to be processed includes personalized text data, if the audit result is that the data to be processed is legal, the sound line of the video object is simulated by a preset simulation rule to obtain the target voice, including: If the audit result is legal data to be processed, the sound feature information corresponding to the video object is extracted through the voiceprint recognition model, and the sound feature information is embedded and encoded into a vector of fixed dimension; the vector of fixed dimension is used to characterize the sound feature of the video object; the voiceprint recognition model is a model established based on the sound feature encoder; The personalized text data and the fixed-dimensional vector are predicted by a cyclic sequence-to-sequence feature prediction network with attention, so as to obtain a first Mel-spectrogram corresponding to the video object; the first Mel-spectrogram is a Mel-spectrogram obtained by a preset cyclic sequence-to-sequence feature prediction network; The first mel-spectrogram is converted into a first time series sound waveform graph by a speech synthesizer, and a target speech is obtained based on the first time series sound waveform graph.

2. The method according to claim 1, characterized in that The step of obtaining the data to be processed and the video object in the authorized state includes: Obtain the data to be processed and the video object in the authorized state from the preset information source.

3. The method according to claim 1, characterized in that The intelligent audit technology is used to intelligently audit the data to be processed to obtain audit results, including: Reviewing the data content in the data to be processed through intelligent review technology; When it is monitored that the data content in the data to be processed is consistent with the preset illegal data content, determining that the data to be processed is illegal data to be processed; When it is monitored that the data content in the data to be processed is inconsistent with the preset illegal data content, the data to be processed is determined to be legal data to be processed.

4. The method according to claim 3, characterized in that If the audit result is legal data to be processed, the sound line of the video object is simulated by a preset simulation rule to obtain a target voice, including: If the audit result is legal data to be processed, obtaining multiple sound feature information corresponding to the video object; Synthesizing multiple sound feature information corresponding to the video object through a preset non-autoregressive speech synthesis model architecture to obtain a second Mel-spectrogram; the second Mel-spectrogram is a Mel-spectrogram obtained through the preset non-autoregressive speech synthesis model architecture; The second mel-spectrogram is converted into a second time series sound waveform graph by a speech synthesizer, and the target speech is obtained based on the second time series sound waveform graph.

5. The method according to claim 3, characterized in that: Also includes: If the data to be processed is illegal data to be processed, illegal prompt information is generated.

6. The method according to claim 1, characterized in that The step of synthesizing the target voice with the video corresponding to the video object to obtain a personalized video includes: According to preset special effects, the target voice is synthesized with the video corresponding to the video object to obtain a personalized video.

7. A processing device for video synthesis based on personalized voice, characterized in that: The device comprises: An acquisition unit, used for acquiring data to be processed and a video object in an authorized state; the data to be processed includes personalized text data and / or personalized audio data; An audit unit, used to perform an intelligent audit on the data to be processed through intelligent audit technology to obtain an audit result; the intelligent audit is used to detect the legitimacy of the data to be processed; A simulation unit, for simulating the voice of the video object according to a preset simulation rule to obtain a target voice if the audit result is a legitimate data to be processed; the target voice is a personalized voice that is consistent with the voice of the video object and is generated according to the legitimate data to be processed; A synthesis unit, used for synthesizing the target voice with the video corresponding to the video object to obtain a personalized video; Wherein, when the data to be processed includes personalized text data, if the audit result is that the data to be processed is legal, the sound line of the video object is simulated by a preset simulation rule to obtain the target voice, including: If the audit result is legal data to be processed, the sound feature information corresponding to the video object is extracted through the voiceprint recognition model, and the sound feature information is embedded and encoded into a vector of fixed dimension; the vector of fixed dimension is used to characterize the sound feature of the video object; the voiceprint recognition model is a model established based on the sound feature encoder; The personalized text data and the fixed-dimensional vector are predicted by a cyclic sequence-to-sequence feature prediction network with attention, so as to obtain a first Mel-spectrogram corresponding to the video object; the first Mel-spectrogram is a Mel-spectrogram obtained by a preset cyclic sequence-to-sequence feature prediction network; The first mel-spectrogram is converted into a first time series sound waveform graph by a speech synthesizer, and a target speech is obtained based on the first time series sound waveform graph.

8. The device according to claim 7, characterized in that The acquisition unit is specifically used for: Obtain the data to be processed and the video object in the authorized state from the preset information source.

9. The device according to claim 7, characterized in that The review unit includes: An audit module, used to audit the data content in the data to be processed through intelligent audit technology; A first determination module is used to determine that the data to be processed is illegal data to be processed when it is monitored that the data content in the data to be processed is consistent with the preset illegal data content; The second determination module is used to determine that the data to be processed is legal data to be processed when it is monitored that the data content in the data to be processed is inconsistent with the preset illegal data content.

Images