Multimedia data generation method, device, electronic device and computer storage medium

Abstract

The embodiments of the present invention provide a method, device, electronic device and computer storage medium for generating multimedia data. The method for generating multimedia data includes: performing energy spectrum and differential spectrum analysis on the acquired human voice audio to be processed, and determining the human voice syllable information in the human voice audio to be processed according to the analysis results; processing the human voice syllable indicated by the human voice syllable information according to the beat information of the accompaniment audio to generate a target human voice audio that matches the accompaniment audio; and synthesizing the target human voice audio and the accompaniment audio to generate multimedia data. The embodiments of the present invention can improve the applicability.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a method, device, electronic device and computer storage medium for generating multimedia data. Background Art

[0002] With the development and maturity of science and Internet technology, the types and functions of applications are becoming more and more abundant. For example, there are instant messaging applications for users to conduct daily communications and exchanges, and playback applications that provide multimedia resources such as videos. Karaoke applications include but are not limited to applications based on desktop PCs, applets based on platforms such as WeChat, applications based on smart mobile devices, and web applications based on web technology. Because they have the functions of recording and publishing songs, and have social functions, they are deeply loved by the majority of users. However, in existing Karaoke applications, when users generate multimedia data, such as audio data, they usually record the user's voice while playing the accompaniment, and then synthesize the recorded voice and accompaniment according to the timestamp to generate a song.

[0003] This makes the way the application generates songs relatively simple, and users need to sing in sync with the accompaniment. It cannot be used in some scenes where it is inconvenient to make sound or in scenes with noisy ambient sound, resulting in poor applicability. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a multimedia data generation solution to solve part or all of the above problems.

[0005] According to a first aspect of an embodiment of the present invention, a method for generating multimedia data is provided, which includes: performing energy spectrum and differential spectrum analysis on acquired vocal audio to be processed, and determining vocal syllable information in the vocal audio to be processed according to the analysis results; processing the vocal syllables indicated by the vocal syllable information according to the beat information of an accompaniment audio to generate a target vocal audio that matches the accompaniment audio; and synthesizing the target vocal audio and the accompaniment audio to generate multimedia data.

[0006] According to a second aspect of an embodiment of the present invention, a multimedia data generation method is provided, comprising: performing energy spectrum and differential spectrum analysis on acquired vocal audio to be processed, and determining vocal syllable information in the vocal audio to be processed according to the analysis results; obtaining node information for vocal matching in accompaniment audio and / or video data without audio, and processing the vocal syllables indicated by the vocal syllable information according to the node information to generate target vocal audio; and synthesizing at least one of the accompaniment audio and the video data with the target vocal audio to generate multimedia data.

[0007] According to a third aspect of an embodiment of the present invention, a multimedia data processing method is provided, comprising: collecting audio data containing vocal audio according to a trigger operation; acquiring multimedia data generated according to the audio data, wherein the multimedia data obtains target vocal audio by identifying vocal syllable information of the audio data and processing the vocal syllables indicated by the vocal syllable information according to node information for vocal matching in accompaniment audio and / or video data without audio, and synthesizing at least one of the accompaniment audio and the video data without audio with the target vocal audio; and providing the multimedia data on a display interface.

[0008] According to a fourth aspect of an embodiment of the present invention, a multimedia data generating device is provided, comprising: a first analysis module, used to perform energy spectrum and differential spectrum analysis on the acquired human voice audio to be processed, and determine the human voice syllable information in the human voice audio to be processed according to the analysis results; a first target human voice generation module, used to process the human voice syllables indicated by the human voice syllable information according to the beat information of the accompaniment audio, so as to generate a target human voice audio matching the accompaniment audio; a first synthesis module, used to synthesize the target human voice audio and the accompaniment audio to generate multimedia data.

[0009] According to a fifth aspect of an embodiment of the present invention, a multimedia data generating device is provided, comprising: a second analysis module, used to perform energy spectrum and differential spectrum analysis on the acquired human voice audio to be processed, and determine the human voice syllable information in the human voice audio to be processed according to the analysis results; a second target human voice generation module, used to obtain node information for human voice matching in accompaniment audio and / or video data without audio, and process the human voice syllables indicated by the human voice syllable information according to the node information to generate target human voice audio; a second synthesis module, used to synthesize at least one of the accompaniment audio and the video data with the target human voice audio to generate multimedia data.

[0010] According to a sixth aspect of an embodiment of the present invention, a multimedia data processing device is provided, comprising: an acquisition module, for acquiring audio data and data to be synthesized containing human voice audio according to a trigger operation; a multimedia acquisition module, for acquiring multimedia data generated according to the audio data, wherein the multimedia data is obtained by identifying the human voice syllable information of the audio data and processing the human voice syllables indicated by the human voice syllable information according to the node information for human voice matching in the accompaniment audio and / or the video data without audio, obtaining the target human voice audio, and synthesizing at least one of the accompaniment audio and the video data without audio with the target human voice audio; and a display module, for providing the multimedia data on a display interface.

[0011] According to a seventh aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction enables the processor to perform operations corresponding to the multimedia data generation method described in the first aspect to the second aspect or operations corresponding to the multimedia data processing method described in the third aspect.

[0012] According to an eighth aspect of an embodiment of the present invention, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the multimedia data generation method as described in the first aspect to the second aspect or the multimedia data processing method as described in the third aspect is implemented.

[0013] According to the multimedia data generation scheme provided by the embodiment of the present invention, when analyzing the human voice audio to obtain the human voice syllable information, the energy spectrum and the differential spectrum of the spectrum of the human voice audio are used in combination, so that the analysis is more comprehensive and the obtained human voice syllable information is more accurate, so that the human voice syllable indicated by the human voice syllable information can be processed according to the beat information of the accompaniment audio to generate the target human voice audio, and the target human voice audio is synthesized with the accompaniment audio to generate multimedia data. In this way, the beat matching of the human voice syllable of the target human voice audio and the accompaniment audio can be achieved, thereby solving the problem in the prior art that the user has to match the accompaniment beat by himself and the user must record the sound during the playing of the accompaniment, thereby avoiding the limitation of the use environment and improving the applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0015] Figure 1a is a flowchart of a method for generating multimedia data according to Embodiment 1 of the present invention;

[0016] Figure 1b This is a usage scenario diagram of a multimedia data generation method according to Embodiment 1 of the present invention;

[0017] Figure 2 is a flowchart of a method for generating multimedia data according to Embodiment 2 of the present invention;

[0018] Figure 3 is a flowchart of a method for generating multimedia data according to Embodiment 3 of the present invention;

[0019] Figure 4 is a flowchart of a method for generating multimedia data according to a fourth embodiment of the present invention;

[0020] Figure 5 A flowchart of the steps of a method for generating multimedia data according to a usage scenario of the present invention;

[0021] Figure 6a is a flowchart of a method for generating multimedia data according to Embodiment 5 of the present invention;

[0022] Figure 6b This is a usage scenario diagram of a multimedia data generation method according to Embodiment 5 of the present invention;

[0023] Figure 7 is a flowchart of a method for generating multimedia data according to Embodiment 6 of the present invention;

[0024] Figure 8 is a flowchart of a method for generating multimedia data according to Embodiment 7 of the present invention;

[0025] Figure 9a is a flowchart of a multimedia data processing method according to Embodiment 8 of the present invention;

[0026] Figure 9b This is an interface change diagram in a usage scenario diagram of a multimedia data processing method according to Embodiment 8 of the present invention;

[0027] Fig.9c This is an interface change diagram in a usage scenario diagram of another multimedia data processing method according to Embodiment 8 of the present invention;

[0028] Fig.10 is a structural block diagram of a multimedia data generating device according to Embodiment 9 of the present invention;

[0029] Fig.11 is a structural block diagram of a multimedia data generating device according to Embodiment 10 of the present invention;

[0030] Fig.12 is a structural block diagram of a multimedia data processing device according to Embodiment 11 of the present invention;

[0031] Fig.13 FIG. 1 is a schematic diagram of the structure of an electronic device according to Embodiment 12 of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in the field based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0033] The specific implementation of the embodiment of the present invention is further described below in conjunction with the accompanying drawings of the embodiment of the present invention.

[0034] Embodiment 1

[0035] Reference Figure 1a , shows a flowchart of the steps of a multimedia data generation method according to embodiment 1 of the present invention.

[0036] The multimedia data generation method of this embodiment includes the following steps:

[0037] Step S102: performing energy spectrum and differential spectrum analysis on the acquired human voice audio to be processed, and determining human voice syllable information in the human voice audio to be processed according to the analysis results.

[0038] The human voice audio to be processed may include human voice audio recorded in real time by the user or pre-recorded human voice audio uploaded by the user, and the recording environment may be any environment. The human voice contained in the human voice audio may be in any language (such as a dialect) or may only contain some sounds of phoneme combinations (for example, a person imitating the sounds of animals). Phonemes can be understood as sounds with certain meanings emitted by human vocal organs.

[0039] The spectrum of the human voice audio may be a Mel spectrum, and the Mel spectrum may be a spectrum obtained by inputting the human voice audio into a Mel-scale filter bank and transforming the Mel-scale filter bank. The energy spectrum of the spectrum of the human voice audio may be used to indicate the signal energy per unit frequency. The difference spectrum of the spectrum of the human voice audio may be a spectrum obtained by performing a difference operation on the Mel spectrum.

[0040] By analyzing the energy spectrum and difference spectrum in the Mel spectrum, the human voice syllable information in the human voice audio can be obtained. The human voice syllable information is used to indicate each syllable in the human voice. Since the energy spectrum and difference spectrum in the Mel spectrum of the human voice audio are used in combination when analyzing the human voice audio, the syllable indicated in the human voice syllable information obtained by analysis is more accurate, which is more conducive to the subsequent processing of the syllable.

[0041] Depending on the needs, the human voice syllable information may include different information. For example, in this embodiment, the human voice syllable information includes time information of the human voice syllable and consonant ratio information of the human voice syllable.

[0042] Among them, the time information of each vocal syllable can be expressed in the form of [st, et], where st indicates the start time, that is, the start time of the vocal syllable in the vocal audio, and et indicates the end time, that is, the end time of the vocal syllable in the vocal audio.

[0043] The consonant ratio information is used to indicate the consonant part and the vowel part in a vocal syllable. Usually, the consonant part of a vocal syllable comes first and the vowel part comes second. For example, if the time information of a vocal syllable is [1:00, 1:10] and the consonant ratio information is 60%, then the consonant part is [1:00, 1:06] and the vowel part is [1:06, 1:10].

[0044] Of course, in other embodiments, the human voice syllable information may also include other information, which is not limited in this embodiment.

[0045] Step S104: processing the vocal syllables indicated by the vocal syllable information according to the beat information of the accompaniment audio to generate target vocal audio matching the accompaniment audio.

[0046] The accompaniment audio may be an audio uploaded by a user or an audio selected from an accompaniment library. The beat information of the accompaniment audio includes but is not limited to the beats per minute (BPM, beat per minute).

[0047] Those skilled in the art can perform appropriate processing on the vocal syllables as needed, as long as the target vocal audio that matches the accompaniment audio can be generated.

[0048] For example, the human voice syllables are subjected to syllable extraction processing, note quantization processing, beat matching processing, etc. Syllable extraction processing is used to obtain syllable segments of each human voice syllable from the human voice audio, and any appropriate method can be used for processing. Note quantization processing is used to quantize the human voice syllables into corresponding notes, and the quantization method can be determined as needed. Beat matching processing is used to match the human voice syllables with the beats of the accompaniment audio, and the matching method can be any appropriate method.

[0049] In this embodiment, the target human voice audio refers to the audio produced by a person in a musical work (such as a rap work) in addition to the accompaniment audio. It can be pure human voice audio containing only human voice, or it can be audio containing human voice and other background sounds, as long as it matches the beat of the accompaniment audio.

[0050] Step S106: synthesize the target vocal audio and the accompaniment audio to generate multimedia data.

[0051] In this embodiment, the target vocal audio and the accompaniment audio may be synthesized in any appropriate manner, for example, directly superimposing the target vocal audio and the accompaniment audio, or superimposing the target vocal audio and / or the accompaniment audio after performing sound effect processing.

[0052] If the multimedia data to be generated is audio data, the synthesized audio obtained by directly synthesizing the target vocal audio and the accompaniment audio can be used as the generated multimedia data.

[0053] The implementation process of the multimedia data generation method is described below in conjunction with a specific usage scenario.

[0054] like Figure 1b As shown, the multimedia data generation method can be executed by a synthesis module, and the synthesis module can be configured on the server side (the server side includes a server and / or a cloud), or can be configured locally on the terminal device.

[0055] In this usage scenario, the synthesis module obtains the to-be-processed human voice audio (hereinafter referred to as human voice audio) containing the human voice of the user, and performs energy spectrum analysis and differential spectrum analysis on the human voice audio to obtain human voice syllable information, which includes the time information of the human voice syllables in the to-be-processed human voice audio. Due to the use of this comprehensive analysis method, the time information of the analyzed human voice syllables is more accurate.

[0056] In addition, the synthesis module processes the vocal syllables in the vocal audio according to the beat information of the accompaniment audio, and obtains the target vocal audio matching the beat information. The accompaniment audio may be a preset default accompaniment audio, or may be determined according to the user's operation. The beat information of the accompaniment audio may be obtained in advance, or obtained by analyzing the accompaniment audio.

[0057] Afterwards, the accompaniment audio and the target vocal audio are synthesized to generate the required multimedia data.

[0058] Through this embodiment, when analyzing the human voice audio to obtain the human voice syllable information, the energy spectrum and the differential spectrum of the spectrum of the human voice audio are used comprehensively, so that the analysis is more comprehensive and the obtained human voice syllable information is more accurate, so that the human voice syllable indicated by the human voice syllable information can be processed according to the beat information of the accompaniment audio, and the target human voice audio is generated. The target human voice audio and the accompaniment audio are synthesized to generate multimedia data. In this way, the beat matching of the human voice syllable of the target human voice audio and the accompaniment audio can be achieved, thereby solving the problem in the prior art that the user has to match the accompaniment beat by himself, and the user must record the sound during the playing of the accompaniment, thereby avoiding the limitation of the use environment and improving the applicability.

[0059] The multimedia data generating method of this embodiment can be executed by any appropriate electronic device with data processing capability, including but not limited to: a server, a mobile terminal (such as a tablet computer, a mobile phone, etc.) and a PC, etc.

[0060] Embodiment 2

[0061] Reference Figure 2 , shows a flowchart of the steps of a multimedia data generating method according to Embodiment 2 of the present invention.

[0062] The multimedia data generation method of this embodiment includes the aforementioned steps S102 to S106.

[0063] Wherein, the step S106 includes the following sub-steps:

[0064] Sub-step S1061: adding sound effects to the target vocal audio and the accompaniment audio respectively.

[0065] After the target vocal audio is generated, in order to make the audio effect in the generated multimedia data better, make the vocals fuller, and have the required sound effects, sound effects are added to the target vocal audio and the accompaniment audio.

[0066] Different sound effects can be added according to different needs. For example, if the audio in the multimedia data to be generated is rap audio, you can add sound effects such as scratching sound effects and tape sound effects to the accompaniment audio, and add sound effects such as delay sound effects and loop sound effects to the target vocal audio.

[0067] For another example, if the audio in the multimedia data to be generated is other styles of audio, stereo sound effects can be added to the accompaniment audio and the target vocal audio, and this embodiment does not limit this. The following takes the generation of rap multimedia data as an example to illustrate the process of adding sound effects:

[0068] In a specific implementation, according to the pause in the target vocal audio, sound effects are added to the audio segments corresponding to the pause in the target vocal audio and the accompaniment audio. In order to obtain a better auditory effect, different sound effects are added to the corresponding audio segments according to the different pause durations.

[0069] For example, sub-step S1061 includes sub-steps S1061a to S1061f. For a longer pause, sub-steps S1061a to S1061c in sub-step S1061 are executed; for a shorter pause, sub-steps S1061d to S1061f in sub-step S1061 are executed.

[0070] Of course, in other embodiments, other bases may be used to add sound effects, and this embodiment does not limit this.

[0071] Specifically, the process of implementing sub-step S1061 through sub-step S1061a to sub-step S1061f is as follows:

[0072] Sub-step S1061a: Obtaining time information of a first pause in the target vocal audio whose pause duration is greater than a first preset duration.

[0073] The specific value of the first preset duration can be determined as needed, and this embodiment does not limit this.

[0074] The first pause may be a pause in the target human voice audio. When acquiring the first pause, it may be determined according to the human voice syllable indicated in the human voice syllable information obtained in step S102.

[0075] For example, if the time interval between two adjacent vocal syllables is greater than a first preset duration, it is determined as the first pause. Alternatively, it is determined based on the spectrum in the target vocal audio. For example, if the blank time of a certain frequency in the target vocal audio is greater than a first preset duration, it is determined as the first pause.

[0076] The time information of the first pause may include the start time and the end time of the first pause. For example, if the time information of a first pause is [1:01, 1:04], it means that the first pause is from 1 minute 01 seconds to 1 minute 04 seconds of the target human voice audio.

[0077] Of course, those skilled in the art may use other methods to indicate the first pause as needed, and this embodiment does not limit this.

[0078] Sub-step S1061b: determining at least one corresponding first audio segment from the target vocal audio according to the time information of the first pause, and determining at least one corresponding second audio segment from the accompaniment audio.

[0079] There may be one or more first pauses in a target vocal audio, and the first audio segment in the target vocal audio corresponding to each first pause is determined according to the time information of each first pause. For example, if the time information of the first pause A is [1:01, 1:04], then the first audio segment in this time period is obtained from the target vocal audio according to the time information, and the time information of the first pause B is [1:20, 1:25], then another first audio segment in this time period is obtained from the target vocal audio according to the time information.

[0080] Since the target vocal audio is generated according to the beat information of the accompaniment audio, the second audio segment of the corresponding time period can be obtained from the accompaniment audio according to the time information of the first pause in the target vocal audio. The acquisition method can be the same as the method of acquiring the first audio segment, so it is not repeated here. Of course, in other embodiments, the second audio segment can be acquired in a different manner from the method of acquiring the first audio segment, and this embodiment does not limit this.

[0081] Sub-step S1061c: adding sound effects to the first audio segment and the second audio segment respectively, to generate target vocal audio and accompaniment audio after adding sound effects.

[0082] In a specific implementation, sub-step S1061c includes the following process:

[0083] Process A1: According to a preset first sound effect algorithm, delay processing is performed on each of the first audio segments, a first effect segment corresponding to each of the first audio segments is obtained, and each of the first audio segments in the target vocal audio is replaced with the corresponding first effect segment.

[0084] In this embodiment, adding a delay effect to the first audio clip is taken as an example for description.

[0085] In this case, the preset first sound effect algorithm may be any appropriate algorithm for achieving a delay effect (ie, achieving the effect produced by delaying the playback of a segment of audio according to a certain rule).

[0086] For each first audio segment, a first sound effect algorithm is used to calculate it to obtain a first effect segment corresponding to the first audio segment and having an auditory delayed playback effect, and the first audio segment in the target vocal audio is replaced with the first effect segment, so that during the process of playing the generated multimedia data, the played first effect segment can have a delayed effect.

[0087] The first sound effect algorithm may be any algorithm that can achieve a delay effect.

[0088] Process B1: mute each second audio segment according to a preset second sound effect algorithm, obtain a second effect segment corresponding to each second audio segment, and replace each second audio segment in the accompaniment audio with the corresponding second effect segment.

[0089] In this embodiment, adding a mute effect to the second audio segment is taken as an example for description.

[0090] In this case, the preset second sound effect algorithm may be an algorithm for realizing a mute effect. For each second audio segment, the second sound effect algorithm is used to calculate it, and a second effect segment corresponding to the second audio segment that is mute in hearing is obtained, and each second audio segment in the accompaniment audio is replaced with the corresponding second effect segment. In this way, during the process of playing the generated multimedia data, the second effect segment can be played with a mute effect.

[0091] The second sound effect algorithm may be any algorithm that can achieve a mute effect.

[0092] Sub-step S106d: obtaining time information of a second pause in the target vocal audio, the pause duration of which is less than or equal to the first preset duration and greater than the second preset duration.

[0093] The value of the second preset duration can be determined as needed, and this embodiment does not limit this. For example, if the second pause corresponds to a shorter pause in the target vocal audio (such as a quarter note rest), the value of the second preset duration can be determined accordingly.

[0094] In this embodiment, the method of determining the second pause from the target human voice audio is the same as the method of determining the first pause, so it will not be repeated. Of course, in other embodiments, different methods can also be used to determine the second pause.

[0095] The time information of the second pause may include the start time and the end time of the second pause. For example, if the time information of a second pause is [1:01, 1:02], it means that the second pause is from 1 minute 01 seconds to 1 minute 02 seconds of the target vocal audio.

[0096] Of course, those skilled in the art may use other methods to indicate the second pause as needed, and this embodiment does not limit this.

[0097] Sub-step S1061e: determining at least one corresponding third audio segment from the accompaniment audio according to the time information of the second pause.

[0098] As mentioned above, since the target vocal audio is generated based on the beat information of the accompaniment audio, the audio segment of the corresponding time period in the accompaniment audio, that is, the third audio segment, can be obtained based on the time information of the second pause in the target vocal audio.

[0099] Sub-step S1061f: adding sound effects to each of the third audio clips to obtain corresponding accompaniment audio.

[0100] Different sound effects can be processed on the third audio clip according to different requirements. For example, if the multimedia data to be generated is in rap style, a scratching sound effect (i.e., a sound effect that makes the audio playback repeat and speed up) can be added to part or all of the third audio clips as needed, or a tape sound effect (i.e., a sound effect that makes the audio playback speed faster or slower) can be added to part or all of the third audio clips as needed. There are two types of tape sound effects: one is a speeded-up tape sound effect that speeds up the playback, and the other is a slowed-down tape sound effect that slows down the playback.

[0101] For example, for the third audio segment A, the third audio segment is subjected to audio reciprocation processing to obtain the accompaniment audio after the sound effect processing. Specifically, a preset scratching algorithm is used, and the scratching algorithm can achieve the effect of performing audio reciprocation processing on the third audio segment A, the third audio segment A is calculated to obtain the third effect segment A that makes it auditorily have a scratching effect, and the third effect segment A is used to replace the third audio segment A in the accompaniment audio to obtain the accompaniment audio after the sound effect processing.

[0102] Among them, the scratching algorithm can calculate an effect segment for speeding up the first half of the third audio segment A, and calculate an effect segment for slowing down the second half of the third audio segment A. After splicing the two effect segments, the third effect segment A can be obtained.

[0103] The scratching algorithm may be any algorithm capable of achieving a scratching effect.

[0104] And / or, for the third audio segment B, the third audio segment is played at a higher speed to obtain the accompaniment audio after the sound effect processing. Specifically, a preset speed-up tape algorithm is used, and the speed-up tape algorithm can achieve the effect of playing the third audio segment B at a higher speed, the third audio segment B is calculated to obtain the third effect segment B that has the auditory speed-up tape effect, and the third effect segment B is used to replace the third audio segment B in the accompaniment audio to obtain the accompaniment audio after the sound effect processing.

[0105] And / or, for the third audio segment C, the third audio segment is played back at a reduced speed to obtain the accompaniment audio after the sound effect processing. Specifically, a preset slow-down tape algorithm is used, which can achieve the effect of playing back at a reduced speed for the third audio segment B, the third audio segment C is calculated to obtain a third effect segment C that has an auditory slow-down tape effect, and the third effect segment C is used to replace the third audio segment C in the accompaniment audio to obtain the accompaniment audio after the sound effect processing.

[0106] The speeding up the tape algorithm and the slowing down the tape algorithm may be any appropriate algorithm capable of achieving the corresponding effects.

[0107] Of course, in other embodiments, other sound effects may be added to the first audio clip, the second audio clip and the third audio clip, for example, a looping sound effect, that is, an auditory effect of looping an audio clip, and when the audio clip is looped, the audio clip will cover the subsequent audio clips, etc.

[0108] Sub-step S1062: synthesizing the target vocal audio and the accompaniment audio after the sound effect is added to generate the multimedia data.

[0109] After adding sound effects to the target vocal audio and accompaniment audio, the target vocal audio and / or accompaniment audio can be further processed according to the quality of the target vocal audio (such as loudness, clarity, etc.) and / or the quality of the accompaniment audio, and the processed target vocal audio and accompaniment audio are synthesized to generate multimedia data.

[0110] For example, in this embodiment, in order to make the audio sound clearer and have sufficient loudness, sub-step S1062 includes sub-steps S1062a to S1062c.

[0111] Specifically, the process of executing sub-steps S1062a to S1062c to implement sub-step S1062 is as follows:

[0112] Sub-step S1062a: Calculate the loudness of the target vocal audio after the sound effect is added and the loudness of the accompaniment audio after the sound effect is processed, and perform loudness gain processing on the target vocal audio after the sound effect processing and the accompaniment audio after the sound effect processing according to the loudness of the target vocal audio after the sound effect processing and the loudness of the accompaniment audio after the sound effect processing.

[0113] Loudness is the strength of sound perceived by human ears. Those skilled in the art can calculate the loudness of the target human voice audio after sound effect processing and the loudness of the accompaniment audio after sound effect processing in any appropriate manner as needed, and this embodiment does not limit this.

[0114] Those skilled in the art may use any appropriate method to perform loudness gain processing on the target human voice audio after sound effect processing and the accompaniment audio after sound effect processing. The loudness gain may be a positive gain (i.e., increase the loudness) or a negative gain (i.e., reduce the loudness), which is not limited in this embodiment.

[0115] Sub-step S1062b: superimposing the target vocal audio after gain processing with the accompaniment audio to obtain a synthesized audio.

[0116] After performing gain processing on the target vocal audio and the accompaniment audio respectively, the target vocal audio and the accompaniment audio are superimposed (ie, mixed) to obtain a synthesized audio, so that the target vocal audio and the accompaniment audio are synthesized into one audio.

[0117] Sub-step S1062c: Generate the multimedia data according to the synthesized audio.

[0118] In a specific implementation, sub-step S1062c is implemented as: performing total gain processing on the synthesized audio, and generating the multimedia data according to the synthesized audio after the total gain.

[0119] For example, using a compressor to process the total gain of the synthesized audio can also be understood as adding a compression effect to the synthesized audio, that is, processing the dynamic range of the synthesized audio to make the processed synthesized audio fuller and better audible. The compression effect is an extreme case of the compression effect. The compression effect is a common dynamic range processing method for audio, which reduces the dynamic range of the audio.

[0120] If the multimedia data is audio data, the synthesized audio after the total gain can be output as audio with accompaniment. If the multimedia data is video data, the synthesized audio after the total gain can be superimposed with the image data to obtain the multimedia data.

[0121] Through this embodiment, the first audio segment in the target vocal audio, the second audio segment and / or the third audio segment in the accompaniment audio are obtained according to the time information of the first pause and / or the time information of the second pause, and sound effects are added to them, so that the sound effects of the target vocal audio and the accompaniment audio are richer, and different styles of audio can be synthesized, so that the obtained multimedia data is auditorily fuller and more interesting.

[0122] By processing the loudness gain of the target vocal audio and accompaniment audio, the loudness can be more balanced, giving the user a better listening experience without damaging the user's hearing due to excessive loudness.

[0123] In addition, when analyzing the human voice audio to obtain the human voice syllable information, the energy spectrum and the differential spectrum of the spectrum of the human voice audio are used in combination, so that the analysis is more comprehensive and the obtained human voice syllable information is more accurate, so that the human voice syllable indicated by the human voice syllable information can be processed according to the beat information of the accompaniment audio, and the target human voice audio is generated. The target human voice audio and the accompaniment audio are synthesized to generate multimedia data. In this way, the beat of the human voice syllable of the target human voice audio and the accompaniment audio can be matched, thereby solving the problem in the prior art that the user has to match the accompaniment beat by himself and the user must record the sound during the playing of the accompaniment, thereby avoiding the limitation of the use environment and improving the applicability.

[0124] The multimedia data generating method of this embodiment can be executed by any appropriate electronic device with data processing capability, including but not limited to: a server, a mobile terminal (such as a tablet computer, a mobile phone, etc.) and a PC, etc.

[0125] Embodiment 3

[0126] Reference Figure 3 , shows a flowchart of the steps of a multimedia data generation method according to embodiment 3 of the present invention.

[0127] The multimedia data generation method of this embodiment includes the aforementioned steps S102 to S106, wherein sub-step S106 can be implemented in the manner described in the second embodiment, or in other manners.

[0128] Wherein, step S102 includes the following sub-steps:

[0129] Sub-step S1021: performing noise reduction processing on the acquired original audio.

[0130] This step is an optional step, and those skilled in the art can determine whether to execute sub-step S1021 as needed.

[0131] The original audio may be any audio containing human voice, which may be recorded by the user in real time, or may be pre-recorded audio loaded from a storage space, which is not limited in this embodiment.

[0132] When the user inputs the original audio, the original audio is subjected to noise reduction processing to remove the environmental noise in the original audio. Those skilled in the art can use any appropriate method to perform noise reduction as needed, for example, using the noise reduction method of webrtc for noise reduction, or its improved method for noise reduction.

[0133] Sub-step S1022: performing human voice activity detection on the noise-reduced audio.

[0134] This step is an optional step, and those skilled in the art can determine whether to execute sub-step S1022 as needed.

[0135] The part of the audio with human voice in the de-noised audio can be obtained through voice activity detection (VAD), so that subsequent processing can be performed based on the part with human voice. Those skilled in the art can use any appropriate method to perform voice activity detection.

[0136] Sub-step S1023: Analyze the energy spectrum of the frequency spectrum of the human vocal audio to be processed to obtain a first sequence for indicating each human vocal syllable in the human vocal audio to be processed; and analyze the differential spectrum of the frequency spectrum of the human vocal audio to be processed to obtain a second sequence for indicating each human vocal syllable in the human vocal audio to be processed.

[0137] In a specific implementation, the following process is included:

[0138] Process A2: Calculate the Mel spectrum of the human voice audio, and then calculate the corresponding energy spectrum and difference spectrum based on the Mel spectrum, where each column in the difference spectrum is differentiated, and when the difference is negative, it is directly set to 0.

[0139] Process B2: Sum the difference spectrum by column to obtain a one-dimensional vector.

[0140] Process C2: Perform Gaussian smoothing on the one-dimensional vector obtained in process B2.

[0141] Process D2: Normalize the result obtained in process C2.

[0142] Process E2: Find peaks on the result obtained in process D2, that is, find the reasonable local maximum (peaks) in this one-dimensional vector, and the output result is the second sequence.

[0143] Process F2: Execute the slice process, that is, perform a similar peak selection process on the energy spectrum obtained in process A2 (which is one-dimensional in itself), but this process outputs not the local maximum value, but a one-dimensional vector isVowels of the same size as the energy spectrum, which contains only 0 and 1. Among them, 0 means "no vowel" and 1 means "yes vowel". After that, the one-dimensional vector isVowels is corrected using the local maximum value calculated by the differential spectrum. The more formal can be determined as needed. For example, in the part where isVowels is 0, if there is a local maximum value, isVowels will be set to 1 in the range of 8 frames starting from the local maximum value and ending in the frame where isVowels is 1 after the local maximum value.

[0144] Process G2: Based on the output result of the human voice activity detection process, that is, the output part with human voice (denoted as regions), the part with human voice and isVowels are corrected with each other, that is, the start time and end time of the two are compared, and the details are adjusted. The first sequence is generated according to the adjusted isVowels.

[0145] Sub-step S1024: determining the human voice syllable information in the human voice audio according to the first sequence and the second sequence.

[0146] In a specific implementation, sub-step S1024 includes the following process:

[0147] Process H2: According to the adjusted isVowels, the target human voice audio is segmented to obtain the notes that meet the requirements. After obtaining the notes, a new audio segmentation is calculated according to the speech recognition engine. Within the range of each note, if there are more than two new audio segmentation points, the note is segmented again according to the new audio segmentation, otherwise no processing is performed.

[0148] Process I2: Output each vocal syllable information based on the obtained note information and the adjusted isVowels information.

[0149] The human voice syllable information may include time information of the human voice syllable and consonant ratio information. The time information of the human voice syllable may be a [st, et] sequence, where st indicates the start time and et indicates the end time.

[0150] The proportion of consonants in each vocal syllable to the entire vocal syllable can be determined by analyzing the proportion of isVowel 0 in each note.

[0151] Sub-step S1025: pre-processing the human voice audio.

[0152] This step is an optional step, and those skilled in the art can determine whether to execute sub-step S1025 as needed.

[0153] Depending on the needs, the pre-processing process may be different. For example, in this embodiment, the pre-processing includes compression processing, equalization processing and dynamic equalization processing. Of course, in other embodiments, the pre-processing may only include part of the aforementioned processing, or may include processing other than the aforementioned processing.

[0154] Among them, compression processing is to process the dynamic range of audio, making the dynamic range of audio smaller so that the human voice sounds fuller and relatively louder.

[0155] Equalization processing is to perform different gain or attenuation operations on components in different frequency bands in the audio. The amount of gain and attenuation can be determined as needed.

[0156] Dynamic equalization is to make different gains or attenuations to the components of different frequency bands in the audio. The amount of gain and attenuation will change dynamically with the audio condition. Equalization and dynamic equalization can make the human voice more natural.

[0157] Through this embodiment, by performing voice activity detection on the noise-reduced audio, the vocal syllable information of the vocal audio is determined according to the results of the voice activity detection, the energy spectrum analysis results and the differential spectrum analysis results, so that the obtained vocal syllable information is more accurate.

[0158] In addition, when analyzing the human voice audio to obtain the human voice syllable information, the energy spectrum and the differential spectrum of the spectrum of the human voice audio are used in combination, so that the analysis is more comprehensive and the obtained human voice syllable information is more accurate, so that the human voice syllable indicated by the human voice syllable information can be processed according to the beat information of the accompaniment audio, and the target human voice audio is generated. The target human voice audio and the accompaniment audio are synthesized to generate multimedia data. In this way, the beat of the human voice syllable of the target human voice audio and the accompaniment audio can be matched, thereby solving the problem in the prior art that the user has to match the accompaniment beat by himself and the user must record the sound during the playing of the accompaniment, thereby avoiding the limitation of the use environment and improving the applicability.

[0159] The multimedia data generating method of this embodiment can be executed by any appropriate electronic device with data processing capability, including but not limited to: a server, a mobile terminal (such as a tablet computer, a mobile phone, etc.) and a PC, etc.

[0160] Embodiment 4

[0161] Reference Figure 4 , shows a flowchart of the steps of a multimedia data generation method according to embodiment 4 of the present invention.

[0162] The multimedia data generation method of this embodiment includes the aforementioned steps S102 to S106. Step S106 can be implemented in the manner described in the second embodiment or in other manners. Step S102 can be implemented in the manner described in the third embodiment or in other manners.

[0163] Wherein, the step S104 includes the following sub-steps:

[0164] Sub-step S1041: performing audio segmentation processing on the human voice audio according to the time information of each human voice syllable in the human voice syllable information, and obtaining syllable segments corresponding to each human voice syllable.

[0165] For example, the human voice audio is segmented according to the time information, and each syllable therein is extracted, that is, by cutting the human voice audio, the syllable segments corresponding to the time information of the human voice syllables are obtained.

[0166] Sub-step S1042: performing quantization processing on the syllable segments of each of the human voice syllables, and quantizing the syllable segments of each of the human voice syllables into corresponding musical notes.

[0167] In a specific implementation, the extracted syllable segments are quantized to notes according to a preset quantization rule, for example, quarter notes, eighth notes, sixteenth notes, etc.

[0168] Since each syllable segment has a corresponding length, we can calculate the median length of all syllable segments and use this median as a "unit length" (unitdur). Then, we divide the length of each syllable segment by the unit length and take the logarithm of the result, i.e., log(x / unidur) / log(2) as a quantitative statistical value. Here, x indicates the length of a syllable segment.

[0169] Compare the quantization statistics with the four values ​​{-1, 0, 1, 2} to find the closest value, and then determine the quantized note based on the closest value.

[0170] For example, if the closest value is -1, the quantized notes are 16th notes. If the closest value is 0, the quantized notes are 16th notes. If the closest value is 1, the quantized notes are 8th notes. If the closest value is 2, the quantized notes are 4th notes.

[0171] Of course, in other embodiments, other methods may be used for quantization, and this embodiment does not limit this.

[0172] Sub-step S1043: Generate target vocal audio that matches the accompaniment audio according to the notes corresponding to each of the syllable segments, the consonant proportion information corresponding to each of the syllable segments, and the beat information of the accompaniment audio.

[0173] In a specific implementation, sub-step S1043 can be implemented as follows: determining the consonant part and the vowel part in the syllable segment according to the consonant proportion information; stretching or compressing the vowel part in the syllable segment according to the beat information of the accompaniment audio, so that the time length of the vowel part matches the time length indicated by the note corresponding to the audio segment; and generating a cappella audio matching the accompaniment audio according to each of the processed syllable segments.

[0174] After the syllable fragments are quantized to notes, the quantized notes are matched with the beats in the accompaniment audio according to the beat information of the accompaniment audio (the beat information includes BPM, i.e. beats per minute), and then the corresponding syllable fragments are compressed or stretched according to the time length indicated by each note.

[0175] During compression or stretching, only the vowel part of the syllable segment is processed so that the overall time length of the syllable segment matches the time length indicated by the note. This can change the time length of the syllable segment without changing the timbre and pitch of the syllable segment, thus avoiding distortion.

[0176] After processing each syllable segment, the processed syllable segments are linked together in chronological order to form an a cappella audio.

[0177] Through this embodiment, the syllable fragments are quantized to corresponding notes, and the time length of each syllable fragment is adjusted according to the quantized notes and the beat information of the accompaniment audio, so that the generated a cappella audio can correspond well to the rhythm of the accompaniment audio, thereby ensuring the correspondence of the subsequent synthesized audio and improving the synthesis effect.

[0178] In addition, when analyzing the human voice audio to obtain the human voice syllable information, the energy spectrum and the differential spectrum of the spectrum of the human voice audio are used in combination, so that the analysis is more comprehensive and the obtained human voice syllable information is more accurate, so that the human voice syllable indicated by the human voice syllable information can be processed according to the beat information of the accompaniment audio, and the target human voice audio is generated. The target human voice audio and the accompaniment audio are synthesized to generate multimedia data. In this way, the beat of the human voice syllable of the target human voice audio and the accompaniment audio can be matched, thereby solving the problem in the prior art that the user has to match the accompaniment beat by himself and the user must record the sound during the playing of the accompaniment, thereby avoiding the limitation of the use environment and improving the applicability.

[0179] The multimedia data generating method of this embodiment can be executed by any appropriate electronic device with data processing capability, including but not limited to: a server, a mobile terminal (such as a tablet computer, a mobile phone, etc.) and a PC, etc.

[0180] Usage scenarios

[0181] like Figure 5 As shown, it shows a schematic diagram of the process of generating rap audio according to the multimedia data generation method in this usage scenario. In this usage scenario, taking the execution of the multimedia data generation method of the above-mentioned embodiments 1 to 4 by a smart terminal as an example, the process of generating rap audio is described.

[0182] Step A: Get the initial audio input.

[0183] For example, the user can input the initial audio by operating the intelligent terminal. The initial audio can be the audio recorded in advance or the audio recorded in real time.

[0184] Step B: Perform general noise reduction processing on the initial input audio to eliminate environmental noise as much as possible.

[0185] Step C: Perform human voice activity detection on the noise-reduced audio.

[0186] The audio after noise reduction is human voice audio. Through human voice activity detection, it is possible to determine which time periods in the audio after noise reduction have human voices and which time periods have no human voices. The time periods with human voices are human voice activity areas, and the time periods without human voices are no human voice activity areas.

[0187] Step D: Perform sound head analysis on the vocal audio.

[0188] The onset can refer to the start time of a note in audio, or the start time of a phoneme in speech. In this usage scenario, onset analysis is used to determine the start and end time of each vocal syllable in the human voice audio, as well as the consonant and vowel parts of each vocal syllable (this information can be represented by the consonant ratio).

[0189] The results of the sound head analysis can be cached in the cache library for subsequent use. Some accompaniment audio can also be cached in the cache library, so that the accompaniment audio can be loaded faster.

[0190] Step E: Pre-process the vocal audio.

[0191] For example, for the human voice audio after noise reduction, compression processing, equalization processing and dynamic equalization processing are performed to make the human voice louder and clearer. The pre-processed human voice audio can be cached in the cache library for subsequent use.

[0192] Step F: Extract syllables

[0193] According to the sound head analysis results, the human voice audio is segmented and processed, and the syllable segments corresponding to the human voice syllables in the human voice audio are extracted. For Chinese, a syllable is equivalent to a word.

[0194] Step G: Perform note quantization processing.

[0195] According to certain quantization rules, all extracted syllable segments are quantized to corresponding notes, such as quarter notes, eighth notes, and sixteenth notes.

[0196] Step H: Generate a cappella audio (ie, target vocal audio).

[0197] According to the quantized notes and the beat information (ie BPM) of the selected accompaniment audio, the vowel part in each syllable segment is compressed or stretched to reach the time length indicated by the corresponding note, and the processed syllable segments are linked together to form a cappella audio.

[0198] The accompaniment audio may be an accompaniment audio selected by a user from an accompaniment library. The accompaniment library stores all accompaniment audios for rap. The used accompaniment audio cached in the cache library may be an accompaniment audio determined according to the usage popularity of all users, which may be part of the accompaniment audio stored in the accompaniment library.

[0199] Step I: Post-process the a cappella audio and accompaniment audio.

[0200] According to the a cappella audio and the accompaniment audio, a variety of audio effects are respectively processed on the a cappella and the accompaniment to increase the richness of the audio. These audio effects include but are not limited to loop effect, delay effect, scratch effect, and tape effect.

[0201] Step J: Perform automatic mixing processing.

[0202] The loudness of the a cappella audio and the accompaniment audio are calculated respectively, and the corresponding loudness gain processing is performed on the a cappella audio and the accompaniment audio according to their respective loudness, and then the a cappella audio and the accompaniment audio are superimposed together to generate a synthetic audio. Finally, a compressor is used to process the synthetic audio with a total gain, and a rap audio with accompaniment is output.

[0203] Through the process of this usage scenario, users do not need to follow the accompaniment and read the text to generate vocal audio. The energy spectrum and differential spectrum are used to analyze the vocal audio, making the vocal syllable information obtained by the analysis more accurate, thereby making the subsequent generation of target vocal audio more effective, thereby ensuring that multimedia data that meets the requirements can be generated and improving applicability.

[0204] In the process of generating Arabella audio, by quantizing the human voice syllables to notes and generating Arabella audio according to the beat information of the accompaniment audio, it is ensured that the rhythm of the generated Arabella audio matches the accompaniment audio, thereby ensuring the effect of the generated multimedia data.

[0205] Embodiment 5

[0206] Reference Figure 6a , shows a schematic flow chart of the steps of a multimedia data generation method according to Embodiment 5 of the present invention.

[0207] In this embodiment, the multimedia data generating method includes:

[0208] Step S602: performing energy spectrum and differential spectrum analysis on the acquired human voice audio to be processed, and determining human voice syllable information in the human voice audio to be processed according to the analysis results.

[0209] The human voice audio to be processed may be audio containing human voice recorded by the user in real time, or audio containing human voice recorded in advance, or audio simulating human voice converted from text input by the user, and so on.

[0210] The implementation method of performing energy spectrum and differential spectrum analysis on the processed human voice audio can be implemented in the same manner as in Example 1, so it will not be repeated here. Since the energy spectrum and differential spectrum in the Mel spectrum of the human voice audio are used in combination when analyzing the human voice audio, the syllable segment obtained by the analysis is more accurate, and the multimedia data synthesized after the subsequent processing of the syllable segment is more authentic and natural, and more realistic.

[0211] In this embodiment, the obtained vocal syllable information includes time information of the vocal syllable. As required, the vocal syllable information may further include consonant ratio information of the vocal syllable.

[0212] The time information of a vocal syllable may indicate the time of a vocal syllable in the form of a start time and an end time.

[0213] The consonant proportion information can be used to indicate the consonant part and the vowel part in the human voice syllable in the form of a consonant ratio value (or a vowel ratio value). Because the consonant usually comes first and the vowel comes second in a human voice syllable, the consonant part time and the vowel part time can be determined based on the time information of the human voice syllable and the consonant ratio value (or the vowel ratio value).

[0214] Of course, the consonant end time or vowel start time may also be directly used to indicate the vowel part and the consonant part in a vocal syllable, and this embodiment does not limit this.

[0215] Step S604: Obtain node information for vocal matching in the accompaniment audio and / or the video data without audio, and process the vocal syllables indicated by the vocal syllable information according to the node information to generate target vocal audio.

[0216] The accompaniment audio and the video data without audio (hereinafter referred to as video data) may be collectively referred to as data to be synthesized.

[0217] If the multimedia data to be generated is audio and video data, the data to be synthesized at least includes video data. When the multimedia data includes accompaniment, the data to be synthesized may also include accompaniment audio.

[0218] If the multimedia data to be generated is audio data, the data to be synthesized only includes the accompaniment audio.

[0219] The node information used for voice matching in the video data can be understood as the time information of the video clips that need dubbing. For example, the video clips that need dubbing include but are not limited to the video clips of the characters speaking.

[0220] The node information of the accompaniment audio can be understood as the beat information of the accompaniment audio. The beat information has been described in detail in the aforementioned embodiment and will not be repeated here.

[0221] It should be noted that the node information may be included in the accompaniment audio and / or video data in advance, for example, the time information of the video clips of each character speaking in the video data may be predetermined and marked in the video data, or the beat of the accompaniment audio may be marked in the accompaniment audio, etc. Of course, the node information may also be obtained by analyzing and processing the accompaniment audio and / or video data in real time.

[0222] A person skilled in the art may process the vocal audio according to the node information in any appropriate manner, such as performing syllable segment extraction processing, syllable segment stretching processing or compression processing, sound effect adding processing, etc., as long as the target vocal audio matching the node information can be obtained.

[0223] Step S606: synthesize at least one of the accompaniment audio and the video data with the target vocal audio to generate multimedia data.

[0224] In the scenario of dubbing video data, the video data is synthesized with target vocal audio, or the video data, accompaniment audio and target vocal audio are synthesized.

[0225] In the audio synthesis scenario, the accompaniment audio and the target vocal audio are synthesized.

[0226] Through this embodiment, when analyzing the human voice audio to obtain the human voice syllable information, the energy spectrum and the differential spectrum of the spectrum of the human voice audio are used comprehensively, so that the analysis is more comprehensive and the obtained human voice syllable information is more accurate, so that the human voice syllable indicated by the human voice syllable information can be processed according to the node information in the future to generate the target human voice audio, and the target human voice audio is synthesized with at least one of the accompaniment audio and the video data to generate multimedia data. In this way, the beat matching of the human voice syllable of the target human voice audio and the accompaniment audio can be achieved, thereby solving the problem of relying on the user to match the node information by himself to record the human voice audio in the prior art, thereby avoiding the limitation of the use environment and improving the applicability.

[0227] The following is an explanation of the multimedia data generation process in combination with specific usage scenarios:

[0228] like Figure 6bAs shown, in this usage scenario, the multimedia data generation method is executed by a synthesis module. Moreover, the synthesis module is configured on the server side (the server side includes a server and / or a cloud), and the server side generates multimedia data and returns it to the terminal device. Configuring the synthesis module on the server side can make full use of the strong data processing capability of the server side, and improve the processing speed while ensuring the processing effect of human voice audio and video data. Of course, in other usage scenarios, the synthesis module can also be configured locally on the user's terminal device to realize the function of generating multimedia data locally on the terminal device.

[0229] In this usage scenario, the user selects human voice audio and data to be synthesized through the terminal device. The multimedia data generation process is as follows:

[0230] like Figure 6b As shown, the user inputs the human voice audio to be processed (hereinafter referred to as human voice audio) and the data to be synthesized (the data to be synthesized in this usage scenario is video data) through the terminal device, and sends them to the synthesis module.

[0231] Specifically, in the synthesis module, the spectrum of the human voice audio is analyzed by energy spectrum and differential spectrum to obtain human voice syllable information, which is used to indicate the time information of each human voice syllable in the human voice audio. In this way, the purpose of comprehensively utilizing the energy spectrum and differential spectrum to accurately determine each human voice syllable in the human voice audio is achieved. Compared with the method of extracting human voice syllables in the prior art, this method is more accurate, thereby ensuring the authenticity and naturalness of the subsequently synthesized multimedia data.

[0232] In addition, in the synthesis module, node information is also obtained from the video data and / or the accompaniment audio, that is, the time information of the video clip that needs to be matched with the human voice is obtained from the video data, and / or the beat information is obtained from the accompaniment audio.

[0233] According to the node information, the human voice syllables indicated by the human voice syllable information are appropriately processed to obtain the target human voice audio matching the node information.

[0234] Then the target human voice audio and video data are synthesized to generate multimedia data after dubbing the video data, and the synthesized multimedia data is output to the terminal device. After receiving the multimedia data, the terminal device can display the multimedia data to the user.

[0235] Embodiment 6

[0236] Reference Figure 7 , shows a schematic flow chart of the steps of a multimedia data generating method according to Embodiment 6 of the present invention.

[0237] In this embodiment, the implementation process of the multimedia data generation method is specifically described for the case where the data to be synthesized only includes video data. The multimedia data generation method includes steps S602 to S606 described in the sixth embodiment.

[0238] The implementation process of step S602 may adopt the implementation process of step S602 in embodiment 6, so it will not be described in detail here.

[0239] In this embodiment, for the case where the data to be synthesized only includes video data, step S604 includes the following sub-steps:

[0240] Sub-step S6041a: Obtaining time information of the video segment to be matched with human voice in the video data.

[0241] The video clip to be matched with human voice can be understood as a video clip that needs to be dubbed, for example, a video clip in which a person speaks a line in the video data.

[0242] In a specific implementation, sub-step S6041a can be implemented as: performing image recognition processing on at least part of the image frames in the video data, and based on the image recognition results, determining a video segment containing continuous lip opening and closing as a video segment to be matched with human voice; and obtaining time information of the video segment to be matched with human voice.

[0243] For example, the image frames in the video data are input into a trained neural network model capable of lip recognition, and the neural network model is used to identify whether the mouth shape of the person in the image frame is open or closed, and then determine a video segment consisting of multiple image frames with continuous changes in the opening and closing of the mouth shape. The neural network model can be any model capable of image recognition, such as a convolutional neural network model.

[0244] The start time and end time of the multiple continuous image frames can be used as the time information of the corresponding video segment.

[0245] Sub-step S6042a: performing audio segmentation processing on the human voice audio according to the time information of each human voice syllable in the human voice syllable information, and obtaining syllable segments corresponding to each human voice syllable.

[0246] For example, the human voice audio is segmented according to the time information, and each syllable therein is extracted, that is, by cutting the human voice audio, the syllable segments corresponding to the time information of the human voice syllables are obtained.

[0247] It should be noted that there is no strict time sequence relationship between step S6041a and step S6042a, and the two steps can be executed in any order or in parallel.

[0248] Sub-step S6043a: According to the time information of the video clip and the time information of each of the human voice syllables, stretch or compress the syllable segments corresponding to each of the video clips to generate the target human voice audio.

[0249] Take a video clip of a character speaking lines in the video data as an example, the time information of the video clip is [1:00,1:03]. Based on this, it is determined that the syllable segments of the human voice corresponding to the video clip are 9, and the total time length is 2 seconds. In this case, in order to avoid the inconsistency between the sound and the picture of the video clip after dubbing, at least part of the syllable segments are stretched to increase the duration of the vowel part, so as to generate the target human voice audio that matches the time information of the video clip, that is, to adapt the total time length of the syllable segments corresponding to each video clip in the target human voice audio to it.

[0250] Similarly, if the total duration of the syllable segments corresponding to a certain video segment is greater than the duration of the video segment, the vowel parts of at least some of the syllable segments may be compressed.

[0251] In step S606, the acquired target human voice audio and video data may be synthesized to generate multimedia data, that is, video data with dubbing.

[0252] Through this embodiment, it is possible to process syllable segments of vocal syllables according to the time information of the video segment in the video data to obtain target vocal audio that matches the time information of the video segment, and then synthesize the target vocal audio and video data to generate multimedia data, thereby achieving automatic dubbing of the video data and ensuring the authenticity and naturalness of the multimedia data after dubbing.

[0253] Embodiment 7

[0254] Reference Figure 8 , shows a schematic flow chart of the steps of a multimedia data generating method according to Embodiment 7 of the present invention.

[0255] In this embodiment, the implementation process of the multimedia data generation method is specifically described for the case where the data to be synthesized includes video data and accompaniment audio. The multimedia data generation method includes steps S602 to S606 described in the sixth embodiment.

[0256] The implementation process of step S602 may adopt the implementation process of step S602 in embodiment 6, so it will not be described in detail here.

[0257] In this embodiment, for the case where the data to be synthesized includes video data and accompaniment audio, step S604 includes the following sub-steps:

[0258] Sub-step S6041b: obtaining the beat information in the accompaniment audio, and obtaining the syllable segments corresponding to each of the vocal syllables according to the vocal syllable information.

[0259] The beat information in the accompaniment audio includes but is not limited to the number of beats per minute.

[0260] In this embodiment, since the vocal syllable information includes the time information of the vocal syllables, obtaining the syllable segments corresponding to each of the vocal syllables according to the vocal syllable information can be implemented as follows: performing audio segmentation processing on the vocal audio according to the time information of each of the vocal syllables in the vocal syllable information to obtain the syllable segments corresponding to each of the vocal syllables.

[0261] The human voice audio may be segmented according to the time information of the human voice syllables in the human voice syllable information to obtain the syllable segments of each human voice syllable.

[0262] Sub-step S6042b: stretching or compressing the syllable segments corresponding to each beat according to the beat information to generate the target vocal audio.

[0263] In a specific implementation, the syllable segment is quantized to be quantized into corresponding notes. For example, the quantization is implemented in the manner of sub-step S1042 in the fourth embodiment.

[0264] The quantized notes are matched to the beats in the accompaniment audio, and then the corresponding syllable fragments are compressed or stretched according to the time length indicated by each note.

[0265] During compression or stretching, only the vowel part of the syllable segment is processed so that the overall time length of the syllable segment matches the time length indicated by the note. This can change the time length of the syllable segment without changing the timbre and pitch of the syllable segment, thus avoiding distortion.

[0266] The transformed syllable segments are spliced ​​together to form the target vocal audio.

[0267] In this embodiment, step S606 includes the following sub-steps:

[0268] Sub-step S6061: Obtaining time information of the video segment to be matched with human voice in the video data.

[0269] In a specific implementation, each image frame in the video data can be subjected to image recognition processing through a trained neural network model, and the video data can be segmented into multiple video segments according to the image recognition results, and the time information of each video segment can be obtained.

[0270] Alternatively, the video data may be manually divided into a plurality of video segments to be matched with human voices, and the time information of each video segment may be obtained.

[0271] Sub-step S6062: According to the beat information in the accompaniment audio and the time information of the video clip, the video clip corresponding to each beat in the accompaniment audio is stretched or compressed to generate video data containing the processed video clip.

[0272] Taking the generation of an MV of a rap song as an example, the video clip corresponding to each beat in the accompaniment audio can be determined based on the beat information in the accompaniment audio and the time information of the video clip. If the time length of the video clip is greater than the time length of the corresponding beat, the video clip is compressed, that is, some image frames in the video clip are deleted or the display time of at least some of the image frames is reduced; conversely, if the time length of the video clip is less than the time length of the corresponding beat, the video clip is stretched, that is, the image frames in the video clip are increased or the display time of at least some of the image frames is increased.

[0273] Afterwards, the processed video segments are spliced ​​in chronological order to generate video data including the processed video segments.

[0274] Sub-step S6063: synthesizing the target human voice audio, the accompaniment audio and the video data containing the processed video clip to generate the multimedia data. Through this embodiment, the human voice audio and video data to be processed are processed based on the beat information of the accompaniment audio, and the target human voice audio and the video data containing the processed video clip that match the beat of the accompaniment audio are obtained, and then the target human voice audio, the accompaniment audio and the video data containing the processed video clip are synthesized to generate the multimedia data, so that the synthesized multimedia data is more natural and real.

[0275] Embodiment 8

[0276] See also Figure 9a , showing a schematic flow chart of the steps of a multimedia data processing method according to Embodiment 8 of the present invention.

[0277] In this embodiment, the multimedia data processing method includes the following steps:

[0278] Step S902: collecting audio data including human voice audio and data to be synthesized according to the trigger operation.

[0279] In this embodiment, the multimedia data processing method is executed by the terminal device, and the multimedia data is generated by the terminal device according to the audio data. Of course, in other embodiments, the multimedia data can also be generated by the server (the server includes a server and / or a cloud) according to the audio data.

[0280] like Figure 9b As shown, the multimedia data processing method is executed by the embedded module of the multimedia playback application on the terminal device as an example to illustrate it. In this case, the terminal device can display an interface as shown in interface 1. When the trigger operation of adding the human voice audio option in the user trigger interface 1 is received, the recording device on the terminal device is called to collect the user's human voice and generate audio data containing human voice audio. Alternatively, when the trigger operation of adding the human voice audio option in the user trigger interface 1 is received, the storage space of the terminal device is read to obtain the audio data containing human voice audio selected by the user.

[0281] The data to be synthesized includes accompaniment audio and / or video data without audio.

[0282] When receiving user Figure 9b When the option of adding data to be synthesized in the middle interface 2 is triggered, candidate accompaniment audio and / or candidate video data are displayed, and based on the user's selection operation of the candidate accompaniment audio and / or the selection operation of the candidate video data, the selected accompaniment audio and / or video data is determined as the data to be synthesized.

[0283] After obtaining the audio data and the data to be synthesized, it is displayed as follows Figure 9b The interface shown in the middle interface 3.

[0284] Step S904: Acquire multimedia data generated according to the audio data.

[0285] The multimedia data may be generated by a synthesis module. The synthesis module may be configured locally on the terminal device, and the audio data may be processed locally on the terminal device to generate multimedia data. Alternatively, the synthesis module may be configured on the server side, and the terminal device may send the audio data to the synthesis module on the server side, and the synthesis module may process the audio data and generate multimedia data.

[0286] The multimedia data is generated in the following manner:

[0287] Identify the vocal syllable information of the audio data, and process the vocal syllables indicated by the vocal syllable information according to the node information for vocal matching in the accompaniment audio and / or the video data without audio to obtain the target vocal audio, and synthesize at least one of the accompaniment audio and the video data without audio with the target vocal audio to obtain multimedia data.

[0288] The human voice syllable information of the audio data can be identified by the implementation method of step S102 or step 602 in the above embodiment, so it will not be repeated. The human voice syllable information can include the time information of the human voice syllable. If necessary, it can also include the proportion of consonants in the human voice syllable.

[0289] The information of the accompaniment audio and / or the video data without audio (hereinafter referred to as video data) can be predetermined, or determined by the terminal device according to the user's selection operation.

[0290] The node information in the accompaniment audio used for vocal matching may be beat information, but is not limited thereto.

[0291] The node information in the video data used for voice matching may be time information of the video segment to be matched with the voice, but is not limited thereto.

[0292] Those skilled in the art can, as needed, use any appropriate method to process the syllable segments corresponding to the vocal syllables indicated by the vocal syllable information according to the node information (for example, quantization processing, stretching processing, compression processing, etc.), as long as the target vocal audio matching the node information can be obtained.

[0293] After the target vocal audio is obtained, at least one of the accompaniment audio and the video data is synthesized with the target vocal audio to generate multimedia data.

[0294] Of course, in other embodiments, the multimedia data may be generated by using the multimedia data generation method described in any one of the aforementioned embodiments 1 to 7.

[0295] Step S906: providing the multimedia data on a display interface.

[0296] After acquiring the multimedia video, the terminal device can display Figure 9b The interface shown in the middle interface 4 provides multimedia data to the user, so that the user can watch or perform other operations. Optionally, in a specific usage scenario, the multimedia data processing method is described below by taking the execution of the multimedia data processing method by an embedded module in a social application on a terminal device as an example. Of course, in other usage scenarios, the application can be any other appropriate application.

[0297] Taking a social application as an example, multimedia data received from other users can be displayed in a chat window of the social application. Multimedia data may include at least one of audio data, video data, and image data. Among them, video data may be audio and video data or non-audio and video data. The interface diagram of displaying received multimedia data in a chat window is shown in FIG. Fig.9c As shown in the middle interface 1.

[0298] The user can trigger an editing interface (such as a menu) for processing the received multimedia data by, for example, long pressing the received multimedia data. Fig.9c(As shown in the middle interface 2). Through the editing interface, the user can use the embedded module to edit the received multimedia data, thereby generating the edited multimedia data and displaying the edited multimedia data.

[0299] For example, if the received multimedia data is a non-audio animated picture, the user can use the embedded module to execute the aforementioned multimedia data processing method to generate a dubbing animated picture. If the received multimedia data is a singing video data, the user can use the embedded module to execute the aforementioned multimedia data processing method to generate a duet video, and so on.

[0300] In this usage scenario, if Fig.9c As shown in the middle interface 2, the editing interface includes a first area for displaying editing function options, a second area for displaying received multimedia data, and a third area for previewing multimedia data edited by the user.

[0301] One or more editing function options are configured in the first area, and the editing function options include but are not limited to a first option for audio and a second option for image frames in the video.

[0302] Among them, the first option for audio includes but is not limited to "dubbing options", "sound effect options" and "style options".

[0303] The dubbing option is used to replace the audio in video data with audio with new audio, or to add new audio to video data without audio, audio data, and image data.

[0304] The sound effect option is used to add sound effects to the audio, such as DJ sound effects, echo effects, voice changing effects, etc.

[0305] The style option is used to process the audio into the corresponding style, such as heavy metal style, rock style, ethnic style, etc.

[0306] The second options for the image frames in the video include but are not limited to “video options”, “filter options”, and “texture options”.

[0307] The Video option is used to add new video or image frames.

[0308] The filter option is used to add filters to one or more image frames in a video, where the filters include blur filters, color filters, etc.

[0309] The Texture option is used to add a texture to one or more image frames in the video.

[0310] In this way, when editing received multimedia data, the user can select the desired style according to the demand, which can improve the interactivity on the one hand, and enable the user to edit the desired multimedia data more easily on the other hand.

[0311] It should be noted that the editing function options are not limited to the options listed above, and may also include other options.

[0312] The following uses the generation of a duet video as an example to illustrate the process of the embedded module executing the multimedia data processing method.

[0313] When the user long presses Fig.9c When receiving multimedia data as shown in the middle interface 1, candidate options such as "duet video option" and "dubbing animated picture option" can be displayed in the interface for the user to choose. The corresponding editing interface (such as Fig.9c (as shown in the middle interface 2).

[0314] Taking the generation of a duet video as an example, the received multimedia data is displayed in the second area of ​​the editing interface (the singing video is used as an example in this usage scenario). If the user triggers the dubbing option, the user can choose to add audio by recording the sound in real time, uploading the recorded audio, or inputting text and converting it into audio in real time. Of course, the user can also add a new video by triggering the video option. If the user does not add a new video, it means that the video in the received multimedia data is edited.

[0315] The embedded module executes the multimedia data processing method according to the acquired audio and / or video added by the user, obtains synthesized multimedia data (ie, edited multimedia data), and displays the synthesized multimedia data in real time in the third area.

[0316] During this process, the user can also trigger the sound effect option and / or style option, and the embedded module adds the required sound effect and / or style to the audio according to the user's triggering operation. And / or, the user can also trigger the filter option and / or sticker option, and the embedded module adds the filter effect and / or sticker to the video according to the user's triggering operation.

[0317] After the edited multimedia data required by the user is generated, the received multimedia data and the edited multimedia data can be merged to generate a duet video. The generated duet video can be displayed in the interface of the chat window, and the display interface is as follows: Fig.9c As shown in the middle interface 3. It should be noted that the interface for displaying the duet video is not limited to the chat window, and it can also be displayed in any other appropriate manner, which is not limited in this embodiment.

[0318] The process of generating a dubbing animated image is similar to the process of generating a duet video, except that the process of merging the received multimedia data and the edited multimedia data is omitted, so it will not be repeated here.

[0319] Through this embodiment, audio data containing human voice audio can be collected, and by processing the audio data, real and natural multimedia data that meets the needs can be obtained, and the multimedia data can be provided to the user. In this way, automatic generation of multimedia data is achieved, and the authenticity and naturalness of the generated multimedia data can be guaranteed.

[0320] Embodiment 9

[0321] Reference Fig.10 , shows a structural block diagram of a multimedia data generating device according to Embodiment 9 of the present invention.

[0322] The multimedia data generating device of this embodiment includes: a first analysis module 1002, which is used to perform energy spectrum and differential spectrum analysis on the acquired human voice audio to be processed, and determine the human voice syllable information in the human voice audio to be processed according to the analysis results; a first target human voice generation module 1004, which is used to process the human voice syllables indicated by the human voice syllable information according to the beat information of the accompaniment audio to generate a target human voice audio that matches the accompaniment audio; a first synthesis module 1006, which is used to synthesize the target human voice audio and the accompaniment audio to generate multimedia data.

[0323] Optionally, the first synthesis module 1006 includes: a sound effect processing module 10061, used to add sound effects to the target vocal audio and the accompaniment audio respectively; a multimedia data generation module 10062, used to synthesize the target vocal audio and the accompaniment audio after the sound effects are added to generate the multimedia data.

[0324] Optionally, the sound effect processing module 10061 is used to obtain time information of a first pause in the target vocal audio whose pause duration is greater than a first preset duration; based on the time information of the first pause, determine at least one corresponding first audio segment from the target vocal audio, and determine at least one corresponding second audio segment from the accompaniment audio; and add sound effect processing to the first audio segment and the second audio segment respectively to generate the corresponding target vocal audio and accompaniment audio after the sound effects are added.

[0325] Optionally, when the sound effect processing module 10061 adds sound effects to the first audio clip and the second audio clip respectively to generate the corresponding target vocal audio and accompaniment audio after the sound effects are added, the sound effect processing module 10061 performs delay processing on each of the first audio clips according to a preset first sound effect algorithm, obtains the first effect clip corresponding to each of the first audio clips, and replaces each of the first audio clips in the target vocal audio with the corresponding first effect clip; according to a preset second sound effect algorithm, performs mute processing on each of the second audio clips, obtains the second effect clip corresponding to each of the second audio clips, and replaces each of the second audio clips in the accompaniment audio with the corresponding second effect clip.

[0326] Optionally, the sound effect processing module 10061 is also used to obtain time information of a second pause in the target vocal audio, the pause duration of which is less than or equal to a first preset duration and greater than a second preset duration; based on the time information of the second pause, determine at least one corresponding third audio segment from the accompaniment audio; add sound effect processing to each of the third audio segments to obtain the corresponding accompaniment audio.

[0327] Optionally, when adding sound effects to each of the third audio clips to obtain the corresponding accompaniment audio, the sound effect processing module 10061 performs audio reciprocating processing on each of the third audio clips to obtain the corresponding accompaniment audio; and / or, performs playback speed-up processing on each of the third audio clips to obtain the corresponding accompaniment audio; and / or, performs playback speed-down processing on each of the third audio clips to obtain the corresponding accompaniment audio.

[0328] Optionally, the multimedia data generating module 10062 is used to calculate the loudness of the target vocal audio and the loudness of the accompaniment audio after the sound effect is added, and perform loudness gain processing on the target vocal audio and the accompaniment audio respectively according to the loudness of the target vocal audio and the loudness of the accompaniment audio; superimpose the target vocal audio after gain processing with the accompaniment audio to obtain a synthesized audio; and generate the multimedia data according to the synthesized audio.

[0329] Optionally, the multimedia data generating module 10062 is used to perform total gain processing on the synthesized audio when generating the multimedia data according to the synthesized audio, and generate the multimedia data according to the synthesized audio after the total gain.

[0330] Optionally, the first analysis module 1002 includes: a sequence generation module 10021, which is used to analyze the energy spectrum of the frequency spectrum of the human voice audio to be processed to obtain a first sequence for indicating each human voice syllable in the human voice audio to be processed; and, to analyze the differential spectrum of the frequency spectrum of the human voice audio to obtain a second sequence for indicating each human voice syllable to be processed in the human voice audio; a determination module 10022, which is used to determine the human voice syllable information in the human voice audio to be processed based on the first sequence and the second sequence.

[0331] Optionally, the frequency spectrum of the human voice audio to be processed is a Mel frequency spectrum.

[0332] Optionally, the vocal syllable information includes time information of the vocal syllable and consonant ratio information of the vocal syllable.

[0333] Optionally, the first target vocal generation module 1004 includes: a segmentation module 10041, used to perform audio segmentation processing on the vocal audio according to the time information of each vocal syllable in the vocal syllable information, and obtain syllable segments corresponding to each vocal syllable; a quantization module 10042, used to perform quantization processing on the syllable segments of each vocal syllable, and quantize the syllable segments of each vocal syllable into corresponding notes; an a cappella generation module 10043, used to generate a target vocal audio that matches the accompaniment audio according to the notes corresponding to each syllable segment, the consonant proportion information corresponding to each syllable segment, and the beat information of the accompaniment audio.

[0334] Optionally, the a cappella generation module 10043 is used to determine the consonant part and the vowel part in the syllable segment according to the consonant proportion information; according to the beat information of the accompaniment audio, stretch or compress the vowel part in the syllable segment so that the time length of the vowel part matches the time length indicated by the note corresponding to the audio segment; and generate a cappella audio matching the accompaniment audio according to each of the processed syllable segments.

[0335] The multimedia data generating device of this embodiment is used to implement the corresponding multimedia data generating methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the multimedia data generating device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0336] Embodiment 10

[0337] Reference Fig.11 , a structural block diagram of a multimedia data generating device according to embodiment 10 of the present invention.

[0338] In this embodiment, the multimedia data generating device includes: a second analysis module 1102, which is used to perform energy spectrum and differential spectrum analysis on the acquired human voice audio to be processed, and determine the human voice syllable information in the human voice audio to be processed according to the analysis results; a second target human voice generation module 1104, which is used to obtain node information for human voice matching in the accompaniment audio and / or the video data without audio, and process the human voice syllables indicated by the human voice syllable information according to the node information to generate the target human voice audio; a second synthesis module 1106, which is used to synthesize at least one of the accompaniment audio and the video data with the target human voice audio to generate multimedia data.

[0339] Optionally, the vocal syllable information includes time information of the vocal syllables; the second target vocal generation module 1104 includes: a video processing module 11041, used to obtain the time information of the video segment to be matched with the vocal in the video data; a syllable segment extraction module 11042, used to perform audio segmentation processing on the vocal audio according to the time information of each vocal syllable in the vocal syllable information, and obtain syllable segments corresponding to each vocal syllable; a vocal syllable processing module 11043, used to stretch or compress the syllable segments corresponding to each video segment according to the time information of the video segment and the time information of each vocal syllable, so as to generate the target vocal audio.

[0340] Optionally, the video processing module 11041 is used to perform image recognition processing on at least part of the image frames in the video data, and based on the image recognition result, determine a video segment containing continuous lip opening and closing as a video segment to be matched with human voice; and obtain time information of the video segment to be matched with human voice.

[0341] Optionally, the second target vocal generation module 1104 is used to obtain the beat information in the accompaniment audio, and to obtain the syllable segments corresponding to each of the vocal syllables according to the vocal syllable information; and to stretch or compress the syllable segments corresponding to each beat according to the beat information to generate the target vocal audio.

[0342] Optionally, the vocal syllable information includes time information of the vocal syllables; the second target vocal generation module 1104 is used to perform audio segmentation processing on the vocal audio according to the time information of each vocal syllable in the vocal syllable information when obtaining the syllable segments corresponding to each vocal syllable according to the vocal syllable information, so as to obtain the syllable segments corresponding to each vocal syllable.

[0343] Optionally, the second synthesis module 1106 is used to obtain the time information of the video segment to be matched with the human voice in the video data; according to the beat information in the accompaniment audio and the time information of the video segment, the video segment corresponding to each beat in the accompaniment audio is stretched or compressed to generate video data containing the processed video segment; the target human voice audio, the accompaniment audio and the video data containing the processed video segment are synthesized to generate the multimedia data.

[0344] The multimedia data generating device of this embodiment is used to implement the corresponding multimedia data generating methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the multimedia data generating device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0345] Embodiment 11

[0346] Reference Fig.12 , shows a structural block diagram of a multimedia data processing device according to embodiment 11 of the present invention.

[0347] In this embodiment, the multimedia data processing device includes: an acquisition module 1202, which is used to acquire audio data containing human voice audio according to a trigger operation; a multimedia acquisition module 1204, which is used to acquire multimedia data generated according to the audio data, wherein the multimedia data is obtained by identifying the human voice syllable information of the audio data, and processing the human voice syllables indicated by the human voice syllable information according to the node information for human voice matching in the accompaniment audio and / or the video data without audio, to obtain the target human voice audio, and synthesizing at least one of the accompaniment audio and the video data without audio with the target human voice audio; and a display module 1206, which is used to provide the multimedia data on a display interface.

[0348] The multimedia data processing device of this embodiment is used to implement the corresponding multimedia data processing methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the multimedia data processing device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0349] Embodiment 12

[0350] Reference Fig.13 , shows a schematic diagram of the structure of an electronic device according to Embodiment 12 of the present invention. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.

[0351] As shown in FIG. 1313 , the electronic device may include: a processor (processor) 1302 , a communication interface (Communications Interface) 1304 , a memory (memory) 1306 , and a communication bus 1308 .

[0352] in:

[0353] The processor 1302 , the communication interface 1304 , and the memory 1306 communicate with each other via a communication bus 1308 .

[0354] The communication interface 1304 is used to communicate with other electronic devices such as terminal devices or servers.

[0355] The processor 1302 is used to execute the program 1310, and specifically can execute the relevant steps in the above-mentioned multimedia data generation method embodiment.

[0356] Specifically, the program 1310 may include program codes, which include computer operation instructions.

[0357] The processor 1302 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention. The one or more processors included in the electronic device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0358] The memory 1306 is used to store the program 1310. The memory 1306 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0359] Program 1310 can be specifically used to enable the processor 1302 to perform the following operations: perform energy spectrum and differential spectrum analysis on the acquired vocal audio to be processed, and determine the vocal syllable information in the vocal audio to be processed based on the analysis results; process the vocal syllables indicated by the vocal syllable information based on the beat information of the accompaniment audio to generate a target vocal audio that matches the accompaniment audio; and synthesize the target vocal audio and the accompaniment audio to generate multimedia data.

[0360] In an optional embodiment, the program 1310 is also used to enable the processor 1302 to add sound effects to the target vocal audio and the accompaniment audio respectively when synthesizing the target vocal audio and the accompaniment audio to generate multimedia data; and synthesize the target vocal audio and the accompaniment audio after the sound effects are added to generate the multimedia data.

[0361] In an optional embodiment, program 1310 is also used to enable processor 1302 to obtain time information of a first pause in the target vocal audio whose pause duration is greater than a first preset duration when adding sound effects to the target vocal audio and the accompaniment audio respectively; determine at least one corresponding first audio segment from the target vocal audio according to the time information of the first pause, and determine at least one corresponding second audio segment from the accompaniment audio; add sound effects to the first audio segment and the second audio segment respectively, and generate the target vocal audio and accompaniment audio after the corresponding sound effects are added.

[0362] In an optional embodiment, the program 1310 is also used to enable the processor 1302 to add sound effects to the first audio clip and the second audio clip respectively to generate the corresponding target vocal audio and accompaniment audio after sound effect processing. According to a preset first sound effect algorithm, each of the first audio clips is delayed, and the first effect clip corresponding to each of the first audio clips is obtained, and each of the first audio clips in the target vocal audio is replaced with the corresponding first effect clip; according to a preset second sound effect algorithm, each of the second audio clips is muted, and the second effect clip corresponding to each of the second audio clips is obtained, and each of the second audio clips in the accompaniment audio is replaced with the corresponding second effect clip.

[0363] In an optional embodiment, program 1310 is also used to enable processor 1302 to obtain time information of a second pause in the target vocal audio whose pause duration is less than or equal to a first preset duration and greater than a second preset duration when adding sound effects to the target vocal audio and the accompaniment audio respectively; determine at least one corresponding third audio segment from the accompaniment audio based on the time information of the second pause; add sound effects to each of the third audio segments to obtain the corresponding accompaniment audio.

[0364] In an optional embodiment, program 1310 is also used to enable processor 1302 to perform audio reciprocating processing on each of the third audio clips to obtain the corresponding accompaniment audio when adding sound effects to each of the third audio clips to obtain the corresponding accompaniment audio; and / or, to speed up the playback of each of the third audio clips to obtain the corresponding accompaniment audio; and / or, to slow down the playback of each of the third audio clips to obtain the corresponding accompaniment audio.

[0365] In an optional embodiment, the program 1310 is also used to enable the processor 1302 to calculate the loudness of the target vocal audio and the loudness of the accompaniment audio after the sound effect is added when synthesizing the processed target vocal audio and the accompaniment audio to generate the multimedia data, and perform loudness gain processing on the target vocal audio and the accompaniment audio respectively according to the loudness of the target vocal audio and the loudness of the accompaniment audio; superimpose the target vocal audio after gain processing with the accompaniment audio to obtain synthesized audio; and generate the multimedia data according to the synthesized audio.

[0366] In an optional implementation, the program 1310 is also used to enable the processor 1302 to perform total gain processing on the synthesized audio when generating the multimedia data based on the synthesized audio, and generate the multimedia data based on the synthesized audio after the total gain.

[0367] In an optional embodiment, the program 1310 is also used to enable the processor 1302 to analyze the energy spectrum of the spectrum of the human vocal audio to be processed to obtain a first sequence indicating each human vocal syllable in the human vocal audio to be processed; and to analyze the differential spectrum of the spectrum of the human vocal audio to be processed to obtain a second sequence indicating each human vocal syllable in the human vocal audio to be processed; and to determine the human vocal syllable information in the human vocal audio based on the first sequence and the second sequence.

[0368] In an optional implementation, the frequency spectrum of the human voice audio to be processed is a Mel frequency spectrum.

[0369] In an optional implementation, the vocal syllable information includes time information of the vocal syllable and consonant ratio information of the vocal syllable.

[0370] In an optional embodiment, the program 1310 is also used to enable the processor 1302 to process the syllables indicated by the vocal syllable information according to the beat information of the accompaniment audio to generate a target vocal audio that matches the accompaniment audio, and then perform audio segmentation processing on the vocal audio according to the time information of each vocal syllable in the vocal syllable information to obtain syllable segments corresponding to the syllable segments of each vocal syllable; perform quantization processing on the syllable segments of each vocal syllable to quantize each vocal syllable into corresponding notes; and generate a target vocal audio that matches the accompaniment audio according to the notes corresponding to each syllable segment, the consonant proportion information corresponding to each syllable segment, and the beat information of the accompaniment audio.

[0371] In an optional embodiment, the program 1310 is also used to enable the processor 1302 to determine the consonant part and the vowel part in the syllable segment according to the consonant proportion information when generating the target vocal audio matching the accompaniment audio based on the notes corresponding to each syllable segment, the consonant proportion information corresponding to each syllable segment, and the beat information of the accompaniment audio; according to the beat information of the accompaniment audio, stretch or compress the vowel part in the syllable segment so that the time length of the vowel part matches the time length indicated by the notes corresponding to the audio segment; and generate a cappella audio matching the accompaniment audio based on each of the processed syllable segments.

[0372] Alternatively, program 1310 can be specifically used to enable processor 1302 to perform the following operations: perform energy spectrum and differential spectrum analysis on the acquired vocal audio to be processed, and determine the vocal syllable information in the vocal audio to be processed based on the analysis results; obtain node information for vocal matching in the accompaniment audio and / or the video data without audio, and process the vocal syllables indicated by the vocal syllable information based on the node information to generate target vocal audio; synthesize at least one of the accompaniment audio and the video data with the target vocal audio to generate multimedia data.

[0373] In an optional embodiment, the vocal syllable information includes time information of the vocal syllables; the program 1310 is also used to enable the processor 1302 to obtain node information for vocal matching in the accompaniment audio and / or the video data without audio, and process the vocal syllables indicated by the vocal syllable information according to the node information to generate the target vocal audio, and obtain the time information of the video segment to be vocal matched in the video data; perform audio segmentation processing on the vocal audio according to the time information of each vocal syllable in the vocal syllable information to obtain the syllable segment corresponding to each vocal syllable; perform stretching or compression processing on the syllable segment corresponding to each video segment according to the time information of the video segment and the time information of each vocal syllable to generate the target vocal audio.

[0374] In an optional embodiment, the program 1310 is also used to enable the processor 1302 to perform image recognition processing on at least part of the image frames in the video data when obtaining the time information of the video segment to be matched with the human voice in the video data, and determine, based on the image recognition result, the video segment containing continuous lip opening and closing as the video segment to be matched with the human voice; and obtain the time information of the video segment to be matched with the human voice.

[0375] In an optional embodiment, the program 1310 is also used to enable the processor 1302 to obtain node information for vocal matching in the accompaniment audio and / or the video data without audio, process the vocal syllables indicated by the vocal syllable information according to the node information to generate the target vocal audio, obtain the beat information in the accompaniment audio, and obtain the syllable segments corresponding to each of the vocal syllables according to the vocal syllable information; and stretch or compress the syllable segments corresponding to each beat according to the beat information to generate the target vocal audio.

[0376] In an optional embodiment, the vocal syllable information includes time information of the vocal syllables; the program 1310 is also used to enable the processor 1302 to perform audio segmentation processing on the vocal audio according to the time information of each vocal syllable in the vocal syllable information when obtaining the syllable segments corresponding to each vocal syllable according to the vocal syllable information, so as to obtain the syllable segments corresponding to each vocal syllable.

[0377] In an optional embodiment, the program 1310 is also used to enable the processor 1302 to obtain time information of a video segment to be matched with human voice in the video data when synthesizing at least one of the accompaniment audio and the video data with the target vocal audio to generate multimedia data; to stretch or compress the video segment corresponding to each beat in the accompaniment audio according to the beat information in the accompaniment audio and the time information of the video segment to generate video data containing the processed video segment; and to synthesize the target vocal audio, the accompaniment audio and the video data containing the processed video segment to generate the multimedia data.

[0378] Alternatively, program 1310 can be specifically used to enable processor 1302 to perform the following operations: collect audio data containing vocal audio according to a trigger operation; obtain multimedia data generated based on the audio data, wherein the multimedia data obtains target vocal audio by identifying vocal syllable information of the audio data and processing the vocal syllables indicated by the vocal syllable information according to node information for vocal matching in the accompaniment audio and / or the video data without audio, and synthesizing at least one of the accompaniment audio and the video data without audio with the target vocal audio; and providing the multimedia data on a display interface.

[0379] The specific implementation of each step in the program 1310 can refer to the corresponding description of the corresponding steps and units in the above-mentioned multimedia data generation method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, which will not be repeated here.

[0380] By using the electronic device of this embodiment, when analyzing the human voice audio to obtain the human voice syllable information, the energy spectrum and the differential spectrum of the spectrum of the human voice audio are used comprehensively, so that the analysis is more comprehensive and the obtained human voice syllable information is more accurate, so that the human voice syllable indicated by the human voice syllable information can be processed according to the beat information of the accompaniment audio, and the target human voice audio is generated. The target human voice audio and the accompaniment audio are synthesized to generate multimedia data. In this way, the beat matching of the human voice syllable of the target human voice audio and the accompaniment audio can be achieved, thereby solving the problem in the prior art that the user has to match the accompaniment beat by himself and the user must record the sound during the playing of the accompaniment, thereby avoiding the limitation of the use environment and improving the applicability.

[0381] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present invention can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present invention.

[0382] The method according to the embodiment of the present invention described above may be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the multimedia data generation method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the multimedia data generation method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the multimedia data generation method shown herein.

[0383] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present invention.

[0384] The above implementation methods are only used to illustrate the embodiments of the present invention, and are not limitations of the embodiments of the present invention. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present invention. The patent protection scope of the embodiments of the present invention should be defined by the claims.

Claims

1. A method for generating multimedia data, characterized in that: include: The difference spectrum of the Mel spectrum of the human voice audio to be processed is summed by column to obtain a one-dimensional vector, the one-dimensional vector is Gaussian smoothed and normalized, and then a peak selection process is performed to obtain a local maximum value, and the energy spectrum of the Mel spectrum is summed by column to obtain a one-dimensional vector isVowels, the one-dimensional vector isVowels is Gaussian smoothed and normalized, and then a peak selection process is performed, the one-dimensional vector isVowels is corrected by the local maximum value, and according to the output result of the human voice activity detection process, the start time and the end time of the human voice part and the corrected one-dimensional vector isVowels are compared by mutual correction between the two to make adjustments, and a first sequence is generated according to the adjusted one-dimensional vector isVowels; According to the adjusted one-dimensional vector isVowels, the human voice audio to be processed is segmented to obtain each syllable information, and according to each syllable information and the corrected one-dimensional vector isVowels, the human voice syllable information is output, wherein the new audio segmentation points calculated by the speech recognition engine and if there are more than two new audio segmentation points within the range of each syllable information, the syllable information is segmented again according to the new audio segmentation points; Processing the vocal syllables indicated by the vocal syllable information according to the beat information of the accompaniment audio to generate target vocal audio matching the accompaniment audio; The target vocal audio and the accompaniment audio are synthesized to generate multimedia data.

2. The method according to claim 1, characterized in that The step of synthesizing the target vocal audio and the accompaniment audio to generate multimedia data includes: Adding sound effects to the target vocal audio and the accompaniment audio respectively; The target vocal audio and the accompaniment audio after the sound effect is added are synthesized to generate the multimedia data.

3. The method according to claim 2, characterized in that The adding sound effects processing to the target vocal audio and the accompaniment audio respectively comprises: Acquire time information of a first pause in the target human voice audio whose pause duration is greater than a first preset duration; Determine at least one corresponding first audio segment from the target vocal audio according to the time information of the first pause, and determine at least one corresponding second audio segment from the accompaniment audio; Sound effects are added to the first audio segment and the second audio segment respectively to generate corresponding target vocal audio and accompaniment audio after the sound effects are added.

4. The method according to claim 3, characterized in that The adding sound effects to the first audio segment and the second audio segment respectively to generate target vocal audio and accompaniment audio after the sound effects are added, comprises: According to a preset first sound effect algorithm, delay processing is performed on each of the first audio segments, a first effect segment corresponding to each of the first audio segments is obtained, and each of the first audio segments in the target human voice audio is replaced with the corresponding first effect segment; According to a preset second sound effect algorithm, each of the second audio segments is muted, a second effect segment corresponding to each of the second audio segments is obtained, and each of the second audio segments in the accompaniment audio is replaced with the corresponding second effect segment.

5. The method according to claim 3, characterized in that: The adding sound effects to the target vocal audio and the accompaniment audio respectively also includes: Acquire time information of a second pause in the target human voice audio, the pause duration of which is less than or equal to the first preset duration and greater than a second preset duration; Determining at least one corresponding third audio segment from the accompaniment audio according to the time information of the second pause; Sound effects are added to each of the third audio clips to obtain corresponding accompaniment audio.

6. The method according to claim 5, characterized in that The adding sound effects to each of the third audio clips to obtain corresponding accompaniment audio includes: Performing audio reciprocating processing on each of the third audio clips to obtain corresponding accompaniment audio; and / or, Performing playback speed-up processing on each of the third audio clips to obtain corresponding accompaniment audio; and / or, The third audio segments are played at a reduced speed to obtain corresponding accompaniment audio.

7. The method according to claim 2, characterized in that: The target human voice audio and the accompaniment audio after the sound effect is added are synthesized to generate the multimedia data, including: Calculating the loudness of the target vocal audio and the loudness of the accompaniment audio after the sound effect is added, and performing loudness gain processing on the target vocal audio and the accompaniment audio respectively according to the loudness of the target vocal audio and the loudness of the accompaniment audio; Superimposing the gain-processed target vocal audio with the accompaniment audio to obtain a synthesized audio; The multimedia data is generated according to the synthesized audio.

8. The method according to claim 7, characterized in that The step of generating the multimedia data according to the synthesized audio comprises: Perform total gain processing on the synthesized audio, and generate the multimedia data according to the synthesized audio after the total gain.

9. The method according to claim 1, characterized in that: The vocal syllable information includes time information of the vocal syllable and consonant ratio information of the vocal syllable.

10. The method according to claim 9, characterized in that The step of processing the syllables indicated by the vocal syllable information according to the beat information of the accompaniment audio to generate a target vocal audio that matches the accompaniment audio includes: According to the time information of each of the human voice syllables in the human voice syllable information, the human voice audio is subjected to audio segmentation processing to obtain syllable segments corresponding to each of the human voice syllables; Quantizing the syllable segments of each of the human voice syllables to quantize the syllable segments of each of the human voice syllables into corresponding musical notes; According to the notes corresponding to each of the syllable segments, the consonant proportion information corresponding to each of the syllable segments, and the beat information of the accompaniment audio, a target vocal audio matching the accompaniment audio is generated.

11. The method according to claim 10, characterized in that The step of generating a target vocal audio that matches the accompaniment audio according to the notes corresponding to each of the syllable segments, the consonant proportion information corresponding to each of the syllable segments, and the beat information of the accompaniment audio comprises: Determine the consonant part and the vowel part in the syllable segment according to the consonant proportion information; According to the beat information of the accompaniment audio, stretching or compressing the vowel part in the syllable segment so that the time length of the vowel part matches the time length of the note indication corresponding to the syllable segment; Based on the processed syllable segments, a cappella audio matching the accompaniment audio is generated.

12. A method for generating multimedia data, characterized in that: include: The difference spectrum of the Mel spectrum of the human voice audio to be processed is summed by column to obtain a one-dimensional vector, the one-dimensional vector is Gaussian smoothed and normalized, and then a peak selection process is performed to obtain a local maximum value, and the energy spectrum of the Mel spectrum is summed by column to obtain a one-dimensional vector isVowels, the one-dimensional vector is Gaussian smoothed and normalized, and then a peak selection process is performed, the one-dimensional vector isVowels is corrected by the local maximum value, and according to the output result of the human voice activity detection process, the start time and the end time of the human voice part and the corrected one-dimensional vector isVowels are compared by mutual correction between the two to make adjustments, and a first sequence is generated according to the adjusted one-dimensional vector isVowels; According to the adjusted one-dimensional vector isVowels, the human voice audio to be processed is segmented to obtain each syllable information, and according to each syllable information and the corrected one-dimensional vector isVowels, the human voice syllable information is output, wherein the new audio segmentation points calculated by the speech recognition engine and if there are more than two new audio segmentation points within the range of each syllable information, the syllable information is segmented again according to the new audio segmentation points; Acquire node information for vocal matching in the accompaniment audio and / or the video data without audio, and process the vocal syllables indicated by the vocal syllable information according to the node information to generate target vocal audio; At least one of the accompaniment audio and the video data is synthesized with the target vocal audio to generate multimedia data.

13. The method according to claim 12, characterized in that The human voice syllable information includes time information of the human voice syllable; The step of obtaining node information for vocal matching in the accompaniment audio and / or the video data without audio, and processing the vocal syllables indicated by the vocal syllable information according to the node information to generate target vocal audio includes: Obtaining time information of a video segment to be matched with a human voice in the video data; According to the time information of each of the human voice syllables in the human voice syllable information, the human voice audio is subjected to audio segmentation processing to obtain syllable segments corresponding to each of the human voice syllables; According to the time information of the video clip and the time information of each of the human voice syllables, stretching or compressing the syllable segments corresponding to each of the video clips is performed to generate the target human voice audio.

14. The method according to claim 13, characterized in that The step of obtaining time information of a video segment to be matched with human voice in the video data includes: Performing image recognition processing on at least part of the image frames in the video data, and determining, based on the image recognition result, a video segment containing continuous lip opening and closing as a video segment to be matched with human voice; Get the time information of the video clip to be matched with the human voice.

15. The method according to claim 12, characterized in that The step of obtaining node information for vocal matching in the accompaniment audio and / or the video data without audio, and processing the vocal syllables indicated by the vocal syllable information according to the node information to generate target vocal audio includes: Acquire the beat information in the accompaniment audio, and acquire the syllable segments corresponding to each of the vocal syllables according to the vocal syllable information; The syllable segments corresponding to each beat are stretched or compressed according to the beat information to generate the target vocal audio.

16. The method according to claim 15, characterized in that The human voice syllable information includes time information of the human voice syllable; The step of acquiring the syllable segments corresponding to the human voice syllables according to the human voice syllable information includes: According to the time information of each of the human voice syllables in the human voice syllable information, the human voice audio is subjected to audio segmentation processing to obtain syllable segments corresponding to each of the human voice syllables.

17. The method according to claim 15, characterized in that The method comprises synthesizing at least one of the accompaniment audio and the video data with the target vocal audio to generate multimedia data, including: Obtaining time information of a video segment to be matched with a human voice in the video data; According to the beat information in the accompaniment audio and the time information of the video clip, the video clip corresponding to each beat in the accompaniment audio is stretched or compressed to generate video data containing the processed video clip; The target vocal audio, the accompaniment audio and the video data including the processed video clip are synthesized to generate the multimedia data.

18. A multimedia data processing method, characterized in that: include: Collecting audio data including human voice audio according to a trigger operation; Acquire multimedia data generated according to the audio data, wherein the multimedia data is obtained by identifying the vocal syllable information of the audio data and processing the vocal syllables indicated by the vocal syllable information according to the node information for vocal matching in the accompaniment audio and / or the video data without audio, obtaining the target vocal audio, and synthesizing at least one of the accompaniment audio and the video data without audio with the target vocal audio, wherein the vocal syllable information is obtained by summing the difference spectrum of the Mel spectrum of the vocal audio to be processed by column to obtain a one-dimensional vector, performing Gaussian smoothing and normalization on the one-dimensional vector, and performing peak selection processing to obtain a local maximum value, and summing the energy spectrum of the Mel spectrum by column to obtain a one-dimensional vector isVowels, performing Gaussian smoothing and normalization on the one-dimensional vector isVowels, and performing peak selection processing on the one-dimensional vector isVowels , correcting the one-dimensional vector isVowels through the local maximum value, according to the output result of the human voice activity detection process, by correcting the human voice part and the corrected one-dimensional vector isVowels to compare the start time and the end time of the two, so as to make adjustments, generating a first sequence according to the adjusted one-dimensional vector isVowels, segmenting the human voice audio to be processed according to the adjusted one-dimensional vector isVowels to obtain each syllable information, and outputting the human voice syllable information according to each syllable information and the corrected one-dimensional vector isVowels, wherein, according to the new audio segmentation point calculated by the speech recognition engine, and if there are more than two new audio segmentation points within the range of each syllable information, the syllable information is segmented again according to the new audio segmentation point; providing the multimedia data on the display interface.

19. A multimedia data generating device, characterized in that: include: A first analysis module is used to sum the difference spectrum of the Mel spectrum of the human voice audio to be processed by columns to obtain a one-dimensional vector, perform Gaussian smoothing and normalization on the one-dimensional vector, and then perform peak selection processing to obtain a local maximum value, and sum the energy spectrum of the Mel spectrum by columns to obtain a one-dimensional vector isVowels, perform Gaussian smoothing and normalization on the one-dimensional vector isVowels, and then perform peak selection processing, correct the one-dimensional vector isVowels according to the local maximum value, and according to the output result of the human voice activity detection process, compare the start time and end time of the human voice part and the corrected one-dimensional vector isVowels by mutual correction between the two to make adjustments, and generate a first sequence according to the adjusted one-dimensional vector isVowels; The first determination module divides the human voice audio to be processed to obtain each syllable information according to the adjusted one-dimensional vector isVowels, and outputs the human voice syllable information according to each syllable information and the corrected one-dimensional vector isVowels, wherein the new audio segmentation points calculated by the speech recognition engine and if there are more than two new audio segmentation points within the range of each syllable information, the syllable information is divided again according to the new audio segmentation points; A first target vocal generation module, configured to process the vocal syllables indicated by the vocal syllable information according to the beat information of the accompaniment audio, so as to generate a target vocal audio that matches the accompaniment audio; The first synthesis module is used to synthesize the target vocal audio and the accompaniment audio to generate multimedia data.

20. A multimedia data generating device, characterized in that: include: A second analysis module is used to sum the difference spectra of the Mel spectrum of the human voice audio to be processed by columns to obtain a one-dimensional vector, perform Gaussian smoothing and normalization on the one-dimensional vector, and then perform peak selection processing to obtain a local maximum value, and sum the energy spectrum of the Mel spectrum by columns to obtain a one-dimensional vector isVowels, perform Gaussian smoothing and normalization on the one-dimensional vector isVowels, and then perform peak selection processing, correct the one-dimensional vector isVowels according to the local maximum value, and according to the output result of the human voice activity detection process, compare the start time and end time of the human voice part and the corrected one-dimensional vector isVowels by mutual correction between the two to make adjustments, and generate a first sequence according to the adjusted one-dimensional vector isVowels; The second determination module divides the human voice audio to be processed to obtain each syllable information according to the adjusted one-dimensional vector isVowels, and outputs the human voice syllable information according to each syllable information and the corrected one-dimensional vector isVowels, wherein the new audio segmentation points calculated by the speech recognition engine and if there are more than two new audio segmentation points within the range of each syllable information, the syllable information is divided again according to the new audio segmentation points; A second target vocal generation module is used to obtain node information for vocal matching in the accompaniment audio and / or the video data without audio, and process the vocal syllables indicated by the vocal syllable information according to the node information to generate target vocal audio; The second synthesis module is used to synthesize at least one of the accompaniment audio and the video data with the target vocal audio to generate multimedia data.

21. A multimedia data processing device, characterized in that: include: A collection module, used for collecting audio data including human voice audio and data to be synthesized according to a trigger operation; A multimedia acquisition module is used to acquire multimedia data generated according to the audio data, wherein the multimedia data is obtained by identifying the vocal syllable information of the audio data and processing the vocal syllables indicated by the vocal syllable information according to the node information for vocal matching in the accompaniment audio and / or the video data without audio to obtain the target vocal audio, and synthesizing at least one of the accompaniment audio and the video data without audio with the target vocal audio, wherein the vocal syllable information is obtained by summing the difference spectrum of the Mel spectrum of the human voice audio to be processed by column to obtain a one-dimensional vector, performing Gaussian smoothing and normalization on the one-dimensional vector, and then performing peak selection processing to obtain a local maximum value, and summing the energy spectrum of the Mel spectrum by column to obtain a one-dimensional vector isVowels, and performing Gaussian After smoothing and normalization, peak selection processing is performed, the one-dimensional vector isVowels is corrected by the local maximum value, and according to the output result of the human voice activity detection process, the start time and the end time of the human voice part and the corrected one-dimensional vector isVowels are compared by mutual correction between the two to make adjustments, and a first sequence is generated according to the adjusted one-dimensional vector isVowels, and the human voice audio to be processed is segmented according to the adjusted one-dimensional vector isVowels to obtain each syllable information, and the human voice syllable information is output according to each syllable information and the corrected one-dimensional vector isVowels, wherein the new audio segmentation point is calculated according to the speech recognition engine, and if there are more than two new audio segmentation points within the range of each syllable information, the syllable information is segmented again according to the new audio segmentation point; The display module is used to provide the multimedia data on a display interface.

22. An electronic device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the multimedia data generation method described in any one of claims 1-11, or to perform operations corresponding to the multimedia data generation method described in any one of claims 12-17, or to perform operations corresponding to the multimedia data processing method described in claim 18.

23. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimedia data generating method as described in any one of claims 1 to 11, or implements the multimedia data generating method as described in any one of claims 12 to 17, or implements the multimedia data processing method as described in claim 18.

Images