A synthesis method, electronic device and computer storage medium

Abstract

The embodiment of the present application discloses a synthesis method, including: collecting a user's video and the user's singing voice for a reference song and dance, obtaining the singer's singing voice and the accompaniment sound of a musical instrument in the reference song, correcting the user's singing voice according to the user's video and the singer's singing voice to obtain the corrected user's singing voice, mixing the corrected user's singing voice and the accompaniment sound of the musical instrument to obtain the user's song, and synthesizing the user's song and the user's video to obtain a synthesized video of the user. The embodiment of the present application also provides an electronic device and a computer storage medium.

Description

Technical Field

[0001] The present application relates to a synthesis technology of video and music in an electronic device, and in particular to a synthesis method, an electronic device and a computer storage medium. Background Art

[0002] At present, short video applications are very popular. Users can make short videos with their mobile phones. Due to the limitations of singing tuning technology, the producers of singing short videos must have strong singing skills. However, for dance short videos or singing, since the played music is different from the final collected music, the rhythm of the video content and the collected music often does not match well. It can be seen that the existing short videos have a technical problem of poor synthesis effect when synthesized with the music. Summary of the invention

[0003] The embodiments of the present application provide a synthesis method, an electronic device and a computer storage medium, which can improve the synthesis effect of a short video when synthesized with music.

[0004] The technical solution of this application is implemented as follows:

[0005] In a first aspect, the present invention provides a synthesis method, comprising:

[0006] Collecting a video of the user and the user's singing voice for a reference song;

[0007] Acquire the singing voice of the singer and the accompaniment sound of the musical instrument in the reference song;

[0008] According to the video of the user and the singing voice of the singer, the singing voice of the user is modified to obtain the modified singing voice of the user;

[0009] Mixing the corrected singing voice of the user and the accompaniment sound of the instrument to obtain the song of the user;

[0010] The song of the user and the video of the user are synthesized to obtain a synthesized video of the user.

[0011] In a second aspect, an embodiment of the present application provides an electronic device, including:

[0012] A collection module, used for collecting a user's video and the user's singing voice for a reference song;

[0013] An acquisition module, used for acquiring the singing voice of the singer and the accompaniment sound of the musical instrument in the reference song;

[0014] A correction module, used for correcting the singing voice of the user according to the video of the user and the singing voice of the singer, to obtain the corrected singing voice of the user;

[0015] A mixing module, used for mixing the corrected singing voice of the user and the accompaniment sound of the instrument to obtain the song of the user;

[0016] A synthesis module is used to synthesize the user's song and the user's video to obtain a synthesized video of the user.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a storage medium storing instructions executable by the processor; the storage medium relies on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, the synthesis method described in one or more of the above embodiments is executed.

[0018] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing executable instructions. When the executable instructions are executed by one or more processors, the processor executes the synthesis method described in one or more of the above embodiments.

[0019] The embodiment of the present application provides a synthesis method, an electronic device and a computer storage medium, including: collecting a user's video and the user's singing voice for a reference song, obtaining the singer's singing voice and the accompaniment sound of the musical instrument in the reference song, correcting the user's singing voice according to the user's video and the singer's singing voice to obtain the corrected user's singing voice, mixing the corrected user's singing voice and the accompaniment sound of the musical instrument to obtain the user's song, and synthesizing the user's song and the user's video to obtain a synthesized video of the user; that is, in the embodiment of the present application, the collected user's singing voice is corrected using the collected user's video and the obtained singer's singing voice, so that the corrected user's singing voice is closer to the singer's singing voice and more matched with the user's video, then, the sound effect of the user's song obtained by mixing the corrected user's singing voice and the accompaniment sound of the musical instrument is not only improved, but also fully combined with the content of the user's video, so that the content of the synthesized video synthesized using the user's song and the user's video complements the soundtrack of the synthesized video, thereby improving the synthesis effect of the synthesized video. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of an optional synthesis method provided in an embodiment of the present application;

[0021] Figure 2A schematic diagram of a process flow of an example 1 of an optional synthesis method provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of a second example of an optional synthesis method provided in an embodiment of the present application;

[0023] Figure 4 A schematic flow chart of Example 3 of an optional synthesis method provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a process flow of Example 4 of an optional synthesis method provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of an optional electronic device provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the structure of another optional electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0028] Embodiment 1

[0029] The present application embodiment provides a synthesis method, Figure 1 A schematic diagram of an optional synthesis method provided in the embodiments of the present application is shown in FIG. Figure 1 As shown, the synthesis method may include:

[0030] S101: Collecting a user's video and the user's singing voice for a reference song;

[0031] In the related art, short video applications often have the situation that the content of the video does not match the soundtrack of the video, or the sound effect of the soundtrack of the video is poor. In order to improve the effect of the short video, a synthesis method is proposed in an embodiment of the present application. First, the user's video and the user's singing voice for a reference song are collected. Specifically, during the collection process, the user's singing voice is collected using the microphone of the electronic device, and the user's video is collected using the camera of the electronic device. In this way, the user's video and the user's singing voice can be collected.

[0032] In order to improve the quality of the collected singing voice of the user, in an optional embodiment, the method may further include:

[0033] When the user in the collected video faces away from the collection direction, noise suppression processing and / or signal amplification processing are performed on the collected sound signal of the user to obtain the singing sound of the user.

[0034] Specifically, since the microphone collects the user's singing voice and the camera collects the user's video at the same time, the user's orientation in the user's video directly affects the sound pickup effect of the microphone. In order to avoid the loss of the user's singing voice due to the user's turning, when it is detected that the user's orientation in the collected video is facing away from the collection direction, that is, facing away from the electronic device, the collected sound signal is subjected to noise suppression processing and / or signal amplification processing, and the processed sound signal is determined to be the user's singing voice.

[0035] It should be noted that the above-mentioned noise suppression processing can be performed on the collected sound signal to obtain a processed sound signal, and the collected sound signal can also be amplified to obtain a processed sound signal. The collected sound signal can also be first subjected to noise suppression processing and then subjected to signal amplification processing to obtain a processed sound signal. The collected sound signal can also be first subjected to signal amplification processing and then subjected to noise suppression processing to obtain a processed sound signal. Here, the embodiments of the present application do not make specific limitations on this.

[0036] In this way, the quality of the user's singing voice can be improved after processing, which is beneficial to improving the effect of the user's synthesized video.

[0037] S102: Acquire the singing voice of the singer and the accompaniment sound of the musical instrument in the reference song;

[0038] Specifically, for each reference song, the electronic device can store the singing voice of the singer of the reference song and the accompaniment sound of the musical instrument. For each reference song, the type of musical instrument included in the accompaniment sound of the musical instrument is related to the reference song. For example, the accompaniment sound of the musical instrument can include: the accompaniment sound of the piano, the accompaniment sound of the guitar, the accompaniment sound of the erhu, the accompaniment sound of the guzheng, etc.

[0039] Since the electronic device stores the singing voice of the singer and the accompaniment sound of the musical instrument of the reference song, after receiving the singing voice of each user for the reference song, the singing voice of the singer and the accompaniment sound of the musical instrument of the reference song can be directly obtained, and the singing voice of the singer and the accompaniment sound of the musical instrument can be separated from the reference song in real time. Here, the embodiment of the present application does not make any specific limitation on this.

[0040] S103: According to the user's video and the singer's singing voice, the user's singing voice is corrected to obtain a corrected user's singing voice;

[0041] Through S101 and S102, the user's video, the user's singing voice, the singer's singing voice and the accompaniment sounds of musical instruments can be obtained. In order to improve the synthesis effect of the user's synthesized video, the user's singing voice can be corrected according to the user's video and the singer's singing voice, so that the corrected user's singing voice is closer to the singer's singing voice and better matches the content in the user's video.

[0042] In order to correct the user's singing voice according to the user's video and the singer's singing voice, in an optional embodiment, S103 may include:

[0043] According to the singing voice of the singer, the singing voice of the user is corrected to obtain the singing voice of the user after preliminary correction;

[0044] According to the orientation of the user in the video of the user, the singing voice of the user after the preliminary correction is adjusted to obtain the singing voice of the user after the adjustment;

[0045] The adjusted user's singing voice is determined as the corrected user's singing voice.

[0046] Specifically, the singing voice of the user can be corrected according to the singing voice of the singer to obtain the preliminary corrected singing voice of the user, and then the singing voice of the user after the preliminary correction can be adjusted according to the direction of the user in the user's video to obtain the adjusted singing voice of the user, and finally the adjusted singing voice of the user can be determined as the corrected singing voice of the user; the singing voice of the user can also be adjusted according to the direction of the user in the user's video to obtain the adjusted singing voice of the user, and then the singing voice of the adjusted user can be corrected according to the singing voice of the singer to obtain the preliminary corrected singing voice of the user, and finally the singing voice of the user after the preliminary correction can be determined as the corrected singing voice of the user. Here, the embodiments of the present application do not make specific limitations on this.

[0047] Among them, the singing voice of the user is corrected according to the singing voice of the singer to obtain the preliminary corrected singing voice of the user, the singing voice of the user after the preliminary correction is adjusted according to the direction of the user in the user's video to obtain the adjusted singing voice of the user, and the adjusted singing voice of the user is determined as the corrected singing voice of the user. The singing voice of the user is corrected by using the singing voice of the singer so that the singing voice of the preliminary corrected user is close to the singing voice of the singer, and the singing voice of the user after the preliminary correction is adjusted by using the direction of the user in the user's video so that the adjusted singing voice of the user is more matched with the video content, so that the corrected singing voice of the user is conducive to improving the synthesis effect of the synthesized video.

[0048] Here, the overall loudness of the user's singing voice and the phase and loudness difference of the left and right channels can be corrected based on the user's position, whether the user is facing the microphone or facing away from the microphone, whether the user is singing, and other information.

[0049] Among them, in the correction of the singing voice of the user by the singing voice of the singer, in an optional embodiment, the singing voice of the user is corrected according to the singing voice of the singer to obtain the singing voice of the user after preliminary correction, including:

[0050] According to the singing voice of the singer, the user's singing voice is corrected by using a preset singing voice correction algorithm to obtain the user's singing voice after preliminary correction.

[0051] Specifically, a preset singing voice correction algorithm can be used to correct the user's singing voice based on the singer's singing voice.

[0052] Here, it should be pointed out that, in correcting the user's singing voice according to the singer's singing voice using a preset singing voice correction algorithm, the singer's singing voice and the user's singing voice can be segmented first, and for each corresponding segment, according to the segmentation of the singer's singing voice, the preset singing voice correction algorithm is used to correct the segmentation of the user's singing voice corresponding to the segment of the singer's singing voice, until each segment is corrected, thereby obtaining the user's singing voice after preliminary correction, wherein the duration of the above segments can be selected according to actual conditions.

[0053] Of course, the singing voice of the entire singer can also be corrected using a preset singing voice correction algorithm. Here, the embodiment of the present application does not make any specific limitations on this.

[0054] For the preset singing voice correction algorithm, the correction of different waveform features of the singing voice can be realized. In an optional embodiment, according to the singing voice of the singer, the singing voice of the user is corrected by using the preset singing voice correction algorithm to obtain the singing voice of the user after preliminary correction, including:

[0055] According to the singing voice of the singer, the audio amplitude in the singing voice of the user is corrected by using a preset singing voice correction algorithm to obtain the singing voice of the user after preliminary correction;

[0056] and / or, based on the singing voice of the singer, using a preset singing voice correction algorithm to correct the spectrum envelope of the singing voice of the user, to obtain the singing voice of the user after preliminary correction;

[0057] and / or, based on the singing voice of the singer, a preset singing voice correction algorithm is used to correct the spectrogram in the singing voice of the user to obtain the singing voice of the user after preliminary correction;

[0058] And / or, based on the singing voice of the singer, a preset singing voice correction algorithm is used to correct the harmonics in the compensated singing voice of the user to obtain the singing voice of the user after preliminary correction.

[0059] Specifically, each singing voice has a waveform of audio amplitude. Here, the waveform of the audio amplitude of the user's singing voice can be corrected according to the waveform of the audio amplitude of the singer's singing voice using a preset singing voice correction algorithm. The singing voice correction algorithm uses a smoothing method to correct the waveform of the audio amplitude, so that after preliminary correction, the waveform of the user's audio amplitude is close to the waveform of the singer's audio amplitude.

[0060] In this way, a smooth correction of the audio amplitude in the user's singing voice can be achieved, so that the breathing sound and microphone spraying sound in the user's singing voice can be removed after the preliminary correction, thereby improving the effect of the user's singing voice after the preliminary correction.

[0061] In addition, each singing voice also has a waveform of a spectrum envelope. Here, the waveform of the spectrum envelope of the user's singing voice can be corrected according to the waveform of the spectrum envelope of the singer's singing voice using a preset singing voice correction algorithm. The singing voice correction algorithm uses a smoothing method to correct the waveform of the spectrum envelope, so that after preliminary correction, the waveform of the user's spectrum envelope is close to the waveform of the singer's spectrum envelope.

[0062] In this way, it is possible to achieve a smooth correction of the spectral envelope of the user's singing voice after the preliminary correction. Similar to the above-mentioned smooth correction of the audio amplitude, the breathing sound and microphone spraying sound in the user's singing voice after the preliminary correction can be removed, thereby improving the effect of the user's singing voice after the preliminary correction.

[0063] In practical applications, after correcting the audio amplitude and spectrum envelope using the preset singing voice correction algorithm, many noises in singing such as breathing and microphone spraying can be basically removed, thereby greatly improving the sound effect of the user's singing voice after preliminary correction.

[0064] In practical applications, pleasant sounds often contain abundant harmonics, so harmonic correction is also very important for the correction of singing voices. In order to achieve the correction of harmonics in singing voices, specifically, each singing voice also has a harmonic waveform diagram, and here, the harmonic waveform diagram of the user's singing voice can be corrected according to the harmonic waveform diagram of the singer's singing voice using a preset singing voice correction algorithm, wherein the singing voice correction algorithm uses a smoothing method to correct the harmonic waveform diagram, so that after preliminary correction, the user's harmonic waveform diagram is close to the singer's harmonic waveform diagram.

[0065] When a singer has multiple harmonics, the user's preferred harmonics can be selected according to the user's preferences. Based on the user's preferred harmonics, a preset singing voice correction algorithm is used to correct the harmonics in the user's singing voice to obtain the user's singing voice after preliminary correction.

[0066] In this way, the harmonics in the singing voice of the user after the preliminary correction can be corrected, and the sound effect of the singing voice of the user after the preliminary correction can be improved.

[0067] Furthermore, each singing voice also has a waveform of a spectrogram, and here, the waveform of the spectrogram of the singing voice of the user can be corrected according to the waveform of the spectrogram of the singing voice of the singer using a preset singing voice correction algorithm, wherein the spectrogram contains the characteristics of the audio time domain and frequency domain, and in the correction of the waveform of the spectrogram, it can include correction of the time domain, correction of the frequency domain, or correction of both the time domain and the frequency domain, and in addition, the correction algorithm of the singing voice uses a smoothing method to correct the waveform of the spectrogram, so that the waveform of the spectrogram of the user after preliminary correction is close to the waveform of the spectrogram of the singer. In this way, the sound effect of the singing voice of the user after preliminary correction can be further improved.

[0068] Finally, some received singing voices of users may be out of tune. In order to eliminate the out of tune, the pitch of the user's singing voice may be corrected. In an optional embodiment, according to the singing voice of the singer, the singing voice of the user is corrected by using a preset singing voice correction algorithm to obtain the singing voice of the user after preliminary correction, including:

[0069] According to the singing voice of the singer, the pitch in the singing voice of the user is corrected by using a preset singing voice correction algorithm to obtain the singing voice of the user after the first pitch correction;

[0070] According to the accompaniment sound of the musical instrument, the pitch of the user's singing voice after the first pitch correction is corrected to obtain the user's singing voice after preliminary correction.

[0071] Specifically, the pitch of the user's singing voice can be corrected according to the singer's singing voice using a preset singing voice correction algorithm to obtain the user's singing voice after preliminary correction. The pitch of the user's singing voice can also be corrected according to the accompaniment sound of the musical instrument to obtain the singing voice of each user after preliminary correction. The pitch of the user's singing voice can also be corrected according to the singer's singing voice using a preset singing voice correction algorithm to obtain the user's singing voice after the first pitch correction. The pitch of the user's singing voice after the first pitch correction can then be corrected according to the accompaniment sound of the musical instrument to obtain the user's singing voice after preliminary correction. Here, the embodiments of the present application do not make specific limitations on this.

[0072] First, based on the singing voice of the singer, the pitch of the user's singing voice is corrected by using a preset singing voice correction algorithm to obtain the user's singing voice after the first pitch correction. Then, based on the accompaniment sound of the musical instrument, the pitch of the user's singing voice after the first pitch correction is corrected to obtain the user's singing song after the preliminary correction, in which each singing voice has a waveform of the pitch. Here, based on the waveform of the pitch of the singer's singing voice, the waveform of the pitch of the user's singing voice can be corrected by using a preset singing voice correction algorithm. Among them, the singing voice correction algorithm adopts a smoothing method to correct the waveform of the pitch, so that the waveform of the user's pitch after the first pitch correction is close to the waveform of the singer's pitch.

[0073] Since the accompaniment sound of the musical instrument can often determine the pitch of the reference song at certain nodes, here, the user's singing voice after the first pitch correction is corrected according to the accompaniment sound of the musical instrument, so as to further correct the out-of-tune part of the user's singing voice after the first pitch correction, and obtain the user's singing voice after preliminary correction.

[0074] Here, it should be noted that the above-mentioned preset singing voice correction algorithm mainly includes the correction of the waveform graphs of the following different characteristics of the singing voice: correction of the waveform graph of the audio amplitude, correction of the waveform graph of the spectrum envelope, correction of the waveform graph of the spectrogram, correction of the waveform graph of the pitch, etc. The singing voice correction algorithm may include the correction of the waveform graph of one or more of the above characteristics, and the order of correction of the waveform graph of each characteristic is not limited.

[0075] In addition, in order to realize the correction of the user's singing voice, a neural network model can also be used. In an optional embodiment, the user's singing voice is corrected according to the singer's singing voice to obtain the user's singing voice after preliminary correction, including:

[0076] The singer's singing voice and the user's singing voice are input into a pre-trained machine learning model to obtain the user's singing voice after preliminary correction.

[0077] Here, the machine learning model needs to be trained in advance. On the basis of the trained machine learning model, the singing voice of the singer and the singing voices of each user after compensation are input into the trained machine learning model together, so that the corrected singing voices of each user can be output. In order to obtain the trained machine learning model, in an optional embodiment, the above method further includes:

[0078] Acquire a sample data set; wherein the sample data set includes: the singing voice of the singer in the collected song, the singing voices of at least two users of the collected song, and the corrected singing voice of each user of the at least two users;

[0079] The sample data set is input into the machine learning model for training to obtain a trained machine learning model.

[0080] Specifically, a sample data set is first obtained, wherein the sample data set is: the singing voice of the singer in the collected song, the singing voices of at least two users of the collected song, and the corrected singing voice of each of the at least two users; that is, the song is first collected, and then the singing voice of the collected song is separated, and then the singing voices of at least two users of the song are collected, and a preset singing voice correction algorithm can be used to obtain the corrected singing voice of each of the at least two users, and the singing voice of the singer in the collected song, the singing voice of at least two users of the collected song, and the corrected singing voice of each of the at least two users constitute a sample data set. It should be noted that the more the singing voices of users in the sample data set, the better the correction effect of the trained machine learning model.

[0081] After obtaining the sample data set, the sample data set is input into the machine learning model for training to obtain a trained machine learning model. Here, after obtaining the trained machine learning model, the machine learning model can also be tested, and the machine training model that passes the test is determined as the trained machine learning model, and the machine learning model that fails the test continues to be trained until the test passes.

[0082] After obtaining the singing voice of the user after preliminary correction, in order to obtain the adjusted singing voice of the user, in an optional embodiment, the singing voice of the user after preliminary correction is adjusted according to the orientation of the user in the video of the user to obtain the adjusted singing voice of the user, including:

[0083] When the user in the video of the user is facing away from the collection direction, the audio amplitude of the singing voice of the user after preliminary correction is amplified to obtain the adjusted singing voice of the user.

[0084] Specifically, the singing voice of the user after the preliminary correction can be adjusted in combination with the user's orientation in the user's video. When the user in the user's video is facing away from the collection direction, the singing voice of the user collected by the microphone is often relatively small or even not collected at all. Therefore, the audio amplitude of the singing voice of the user after the preliminary correction can be amplified to obtain the adjusted singing voice of the user. Of course, other sound parameters of the singing voice of the user after the preliminary correction can also be adjusted in combination with actual conditions, and the embodiments of the present application do not make specific limitations on this.

[0085] In an optional embodiment, the above method may further include:

[0086] The singing songs of the corrected user are aligned based on the timestamp of the singer's singing voice, and the singing voice of the aligned user is re-determined as the singing voice of the corrected user.

[0087] Specifically, by separating the reference songs, we can not only obtain the singer's singing voice and the accompaniment sound of the musical instrument, but also the timestamp corresponding to the lyrics. Since the timestamp of the singer's singing voice is the same as the timestamp corresponding to the lyrics and the timestamp of the accompaniment sound of the musical instrument, the timestamp of the singer's singing voice is used to align the singing voice of the corrected user to obtain the aligned singing voice of the user, and the aligned singing voice of the user is re-determined as the singing voice of the corrected user.

[0088] In this way, the corrected user's singing voice is updated by aligning the timestamps to improve the sound effect of the corrected user's singing voice, which is beneficial to improving the effect of the synthesized video.

[0089] S104: Mixing the corrected singing voice of the user and the accompaniment sound of the musical instrument to obtain the user's song;

[0090] S105: Synthesize the user's song and the user's video to obtain a synthesized video of the user.

[0091] After obtaining the corrected singing voice of the user, the corrected singing voice of the user and the accompaniment sound of the musical instrument are mixed to obtain the user's song, which is close to the reference song and matches the user's video content. In this way, the synthesis effect of the user's synthetic video synthesized based on the user's song and the user's video is significantly improved.

[0092] In addition, in view of the situation that some users select the singing voice of a singer as the soundtrack for the user's video, in an optional embodiment, the above method may further include:

[0093] When receiving a selection instruction for a singer's singing voice, the audio amplitude of the singer's singing voice is corrected according to the direction of the user in the user's video to obtain a corrected singer's singing voice;

[0094] The corrected singing voice of the singer is synthesized with the user's video to obtain a synthesized video of the user.

[0095] Specifically, when a selection instruction for a singer's singing voice is received, that is, the user of the electronic device abandons the collected singing voice of the user and selects the singing voice of the singer. In order to match the singing voice of the singer with the video content of the user, some users require that the synthesized video contains environmental information of the current collection environment. Then it is necessary to correct the audio amplitude in the singing voice of the singer in combination with the video content in the user's video, such as the user's orientation, so as to obtain the corrected singing voice of the singer, and then synthesize the corrected singing voice of the singer and the user's video to obtain the user's synthesized video, which can improve the synthesis effect of the synthesized video.

[0096] In the correction of the audio amplitude of the singer's singing voice, in an optional embodiment, the audio amplitude of the singer's singing voice is corrected according to the direction of the user in the user's video to obtain the corrected singer's singing voice, including:

[0097] When the user in the user's video is facing away from the collection direction, the audio amplitude of the singer's singing voice is reduced to obtain a corrected singing voice of the singer.

[0098] Specifically, when the user in the user's video is facing away from the collection direction, the collected sound signals are generally weak. Therefore, here, in view of the situation where the user hopes to synthesize the video in combination with the collection environment information, it is necessary to reduce the audio amplitude of the singer's singing voice to obtain the corrected singer's singing voice. In this way, since the singer's singing voice is corrected in combination with the collection environment, the corrected singer's singing voice is more matched with the user's video content, achieving the user's desired spatial effect while improving the synthesis effect of the synthesized video.

[0099] After obtaining the user's synthesized video, in an optional embodiment, the method may further include:

[0100] According to the obtained lyrics of the reference song, the video content of the user's synthesized video and the strength of the accompaniment sound of the musical instrument, an optional special effect is determined for the user's synthesized video.

[0101] After the user's composite video is synthesized, optional special effects can be determined for the user's composite video according to the lyrics of the reference song, or according to the video content of the user's composite video, or according to the strength of the accompaniment sound of the musical instrument, or according to the lyrics of the reference song and the video content of the user's composite video, or according to the lyrics of the reference song and the strength of the accompaniment sound of the musical instrument, or according to the video content of the user's composite video and the strength of the accompaniment sound of the musical instrument, or according to the obtained lyrics of the reference song, the video content of the user's composite video and the strength of the accompaniment sound of the musical instrument, and here, the embodiments of the present application do not make specific limitations on this.

[0102] Among them, in determining the optional special effects for the user's synthesized video based on the lyrics of the obtained reference song, the video content of the user's synthesized video and the intensity of the accompaniment sound of the musical instrument, the optional special effects can be determined in combination with the keywords in the lyrics content of the reference song. For example, if the keyword is fireworks, the fireworks special effect can be determined as an optional special effect. It can also be combined with the video content. For example, if the video content is the user dancing Xinjiang dance, the dance clothes of Xinjiang dance can be used as an optional special effect. It can also be combined with the intensity of the musical instrument accompaniment sound. When the intensity reaches a certain threshold, the cheers are determined as an optional special effect. In this way, optional special effects are determined for the user's synthesized video.

[0103] In addition, it should be noted that the accompaniment sounds of musical instruments can also be corrected. For example, according to the preset target sound effects, the accompaniment sounds of different musical instruments can be modified accordingly, such as enhancing the corresponding instrument spectrum, changing the spatial position, changing the voice, making funny sounds, etc. Finally, all the modified accompaniment sounds of the musical instruments are mixed to obtain the final soundtrack result, which is saved in the same short video together with the video.

[0104] The following examples are given to illustrate the synthesis method described in one or more of the above embodiments.

[0105] Figure 2 A schematic diagram of an optional synthesis method according to an embodiment of the present application is shown in FIG. Figure 2As shown, the user terminal selects the original music (equivalent to the above-mentioned reference song) locally, and this original music passes through the vocal separation module to extract the original singer's vocals (equivalent to the singing voice of the above-mentioned singer) and generate the accompaniment. The original singer's vocals are recognized as lyrics through the speech-to-text module, and are recorded in the lyrics together with the lyrics timestamp information. The accompaniment is further separated into multiple independent instrument sounds such as drums, guitars, pianos, etc. (equivalent to the accompaniment sounds of the above-mentioned instruments) as needed through the instrument separation module. All results are stored locally for later use. In the above process, the specific separation algorithm can be any method, including traditional signal processing methods or deep learning methods.

[0106] Figure 3 The schematic diagram of the process of Example 2 of an optional synthesis method provided in the embodiment of the present application is as follows: Figure 3 As shown, the user terminal plays the original music, and the microphone collects the user's singing voice (equivalent to the singing voice of the above-mentioned user) and the external music and various environmental interference sounds. After passing through the audio signal processing module, the external music and environmental interference sounds are removed, leaving only the user's singing voice.

[0107] The camera obtains the user's real-time video (equivalent to the video of the above-mentioned user). In the video signal processing module, the image in the user's real-time video is enhanced and other corresponding video signal processing is performed. The result enters the video analysis module, where the user's orientation, whether singing, facing the microphone with his back to the microphone and other information are detected in real time. The result will be used as a reference for the singing tuning module.

[0108] The sound processing and mixing module can enhance or spatially render independent instruments according to the target scene set by the user to form a unique soundtrack. The user's singing voice after the singing tuning module will be added with voice-changing and funny effects according to the user's settings, and then mixed with the original accompaniment or the accompaniment after special sound effects in this sound processing and mixing module to form the final audio soundtrack output.

[0109] In the audio and video encoding and packaging module, the audio output results and the video output results are combined through corresponding encoders and packaging to form the final small video output.

[0110] Figure 4 The schematic diagram of the process of Example 3 of an optional synthesis method provided in the present application is as follows: Figure 4 As shown, by inputting the video (user's video) into the video analysis module, the constantly changing position information and orientation information of the person can be easily detected, and the surrounding environment information can also be detected.

[0111] Since the person is far away from the microphone and his position is constantly changing, it is necessary to enhance the directionality of the microphone input signal and suppress noise in the directional sound pickup module. The microphone array itself has the ability to detect the direction of the sound source. Combined with the person position of the video analysis module, it can obtain more accurate person position information and obtain better microphone directional sound pickup effect.

[0112] Different room sizes and layouts will result in different reverberation effects and audio attenuation effects, and different positions will also cause differences in left and right channel audio signals. Sometimes, users want the final accompaniment soundtrack to contain these spatial sound effect information; other times, users do not want the final accompaniment soundtrack to contain these spatial sound effect information. According to the user's settings, in the spatial sound effect compensation module, the input user singing voice is compensated for spatial sound effects. When these spatial sound effect information are needed in the final accompaniment soundtrack, the original singing voice (equivalent to the singing voice of the above-mentioned user) is modified according to the character position and spatial information, and the original singing voice has the same spatial sound effect information. The modified original singing voice will be used as the input of the audio tuning module; when these spatial sound effect information are not needed in the final accompaniment soundtrack, the user singing voice is modified according to the character position and spatial information to remove the left and right channel differences and reverberation attenuation effects caused by spatial sound effects.

[0113] The direction of a person will cause the directional sound pickup to fluctuate. It is impossible to directly modify the sound amplitude based on the unknown direction of the person. Combined with the person's direction information of the video analysis module, the audio attenuation caused by the person's direction can be calculated more specifically, so as to perform targeted audio amplification compensation for the user's singing voice, ensuring a stable audio signal amplitude when the user's direction changes.

[0114] After the user singing is corrected based on the spatial position information, the user singing voice is relatively stable and clean. There will be a delay with the actual accompaniment (the delay may be due to the hardware acquisition processing delay, or it may be the delay from the character to the microphone). Timestamp alignment needs to be performed in the timestamp alignment module. By calculating the correlation between the original singing voice and the user singing voice, the delay between the two can be calculated and the timestamps can be aligned. Since the delay of the original singing voice and the accompaniment is completely matched, the timestamp is aligned with the original singing voice, that is, aligned with the accompaniment timestamp.

[0115] The user's singing voice with timestamp alignment is then passed through an audio tuning module to undergo a series of audio tuning. The spectral characteristics of the original audio segment are analyzed, such as the fundamental frequency harmonic relationship, spectral distribution characteristics, etc., to perform spectral correction on the user's singing to ensure that the user's singing is close to the professional effect of the original singer.

[0116] It should be noted that the above-mentioned sound adjustment process is only an example reference. In actual situations, some modules can be enhanced and deleted to better utilize the spatial sound effects of the character's position for singing compensation or to balance the system's operating load capacity.

[0117] In the post-production process, you can also add atmosphere, animation, sound, etc. to the short video. The system's own preset mode effects are not very diverse and cannot meet the needs of uniqueness. You can give a post-production effect that matches the content of the short video and the soundtrack.

[0118] Figure 5 A schematic diagram of an optional synthesis method according to an embodiment of the present application is shown in Example 4. Figure 5 As shown, in the instrument analysis module, the position and strength of each independent instrument, such as drums, guitars, etc., can be analyzed to provide guidance on the intensity change, start and end time, position and other effects of the atmosphere animation sound effect; in the text analysis module, the lyrics are analyzed in full text, and guidance on the atmosphere animation sound effects that match the lyrics content or keywords is provided, such as fireworks, water flow, wind, cheers and other appropriate effects; in the video analysis module, the video is analyzed, and overall background modification guidance is provided according to the background characteristics, and guidance on the location of adding atmosphere animation is provided according to the user's clothing characteristics. In the integrated analysis module, these guidance clues are combined to provide several targeted post-processing effects for users to combine and choose, so that users can obtain unique and appropriate post-production effects.

[0119] Through the above examples, any music can be used as background music, and users can sing and dance by themselves, and the background music has good singing effect and more unique and appropriate post-production effect.

[0120] The embodiment of the present application provides a synthesis method, including: collecting a user's video and the user's singing voice for a reference song, obtaining the singer's singing voice and the accompaniment sound of the musical instrument in the reference song, correcting the user's singing voice according to the user's video and the singer's singing voice to obtain a corrected user's singing voice, mixing the corrected user's singing voice and the accompaniment sound of the musical instrument to obtain the user's song, and synthesizing the user's song and the user's video to obtain a synthesized video of the user; that is, in the embodiment of the present application, the collected user's video and the obtained singer's singing voice are used to correct the collected user's singing voice, so that the corrected user's singing voice is closer to the singer's singing voice and more consistent with the user's video. Then, the sound effect of the user's song obtained by mixing the corrected user's singing voice and the accompaniment sound of the musical instrument is not only improved, but also fully combined with the content of the user's video. In this way, the content of the synthesized video synthesized using the user's song and the user's video complements the soundtrack of the synthesized video, thereby improving the synthesis effect of the synthesized video.

[0121] Embodiment 2

[0122] Based on the same inventive concept, an embodiment of the present application provides an electronic device, Figure 6 A schematic diagram of the structure of an optional electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device includes: a collection module 61, an acquisition module 62, a correction module 63, a mixing module 64 and a synthesis module 65; wherein,

[0123] A collection module 61, used to collect a user's video and a user's singing voice for a reference song;

[0124] An acquisition module 62 is used to acquire the singing voice of the singer and the accompaniment sound of the musical instrument in the reference song;

[0125] A correction module 63, used for correcting the singing voice of the user according to the video of the user and the singing voice of the singer, to obtain the corrected singing voice of the user;

[0126] A mixing module 64 is used to mix the corrected singing voice of the user and the accompaniment sound of the musical instrument to obtain the user's song;

[0127] The synthesis module 65 is used to synthesize the user's song and the user's video to obtain the user's synthesized video.

[0128] In an optional embodiment, the correction module 63 is specifically configured to:

[0129] According to the singing voice of the singer, the singing voice of the user is corrected to obtain the singing voice of the user after preliminary correction;

[0130] According to the orientation of the user in the video of the user, the singing voice of the user after the preliminary correction is adjusted to obtain the singing voice of the user after the adjustment;

[0131] The adjusted user's singing voice is determined as the corrected user's singing voice.

[0132] In an optional embodiment, the correction module 63 corrects the user's singing voice according to the singer's singing voice, and obtains the user's singing voice after preliminary correction, including:

[0133] According to the singing voice of the singer, the user's singing voice is corrected by using a preset singing voice correction algorithm to obtain the user's singing voice after preliminary correction.

[0134] In an optional embodiment, the correction module 63 corrects the user's singing voice according to the singer's singing voice using a preset singing voice correction algorithm to obtain the user's singing voice after preliminary correction, including:

[0135] According to the singing voice of the singer, the audio amplitude in the singing voice of the user is corrected by using a preset singing voice correction algorithm to obtain the singing voice of the user after preliminary correction;

[0136] and / or, based on the singing voice of the singer, using a preset singing voice correction algorithm to correct the spectrum envelope of the singing voice of the user, to obtain the singing voice of the user after preliminary correction;

[0137] and / or, based on the singing voice of the singer, a preset singing voice correction algorithm is used to correct the spectrogram in the singing voice of the user to obtain the singing voice of the user after preliminary correction;

[0138] And / or, based on the singing voice of the singer, a preset singing voice correction algorithm is used to correct the harmonics in the user's singing voice to obtain the user's singing voice after preliminary correction.

[0139] In an optional embodiment, the correction module 63 corrects the user's singing voice according to the singer's singing voice using a preset singing voice correction algorithm to obtain the user's singing voice after preliminary correction, including:

[0140] According to the singing voice of the singer, the pitch in the singing voice of the user is corrected by using a preset singing voice correction algorithm to obtain the singing voice of the user after the first pitch correction;

[0141] According to the accompaniment sound of the musical instrument, the pitch of the user's singing voice after the first pitch correction is corrected to obtain the user's singing voice after preliminary correction.

[0142] In an optional embodiment, the correction module 63 corrects the user's singing voice according to the singer's singing voice, and obtains the user's singing voice after preliminary correction, including:

[0143] The singer's singing voice and the user's singing voice are input into a pre-trained machine learning model to obtain the user's singing voice after preliminary correction.

[0144] In an optional embodiment, the electronic device is further used for:

[0145] Acquire a sample data set; wherein the sample data set includes: the singing voice of the singer in the collected song, the singing voices of at least two users of the collected song, and the corrected singing voice of each user of the at least two users;

[0146] The sample data set is input into the machine learning model for training to obtain a trained machine learning model.

[0147] In an optional embodiment, the correction module 63 adjusts the singing voice of the user after preliminary correction according to the orientation of the user in the user's video, and obtains the adjusted singing voice of the user, including:

[0148] When the user in the video of the user is facing away from the collection direction, the audio amplitude of the singing voice of the user after preliminary correction is amplified to obtain the adjusted singing voice of the user.

[0149] In an optional embodiment, the electronic device is further used for:

[0150] When the user in the collected video faces away from the collection direction, noise suppression processing and / or signal amplification processing are performed on the collected sound signal of the user to obtain the singing sound of the user.

[0151] In an optional embodiment, the electronic device is further used for:

[0152] When receiving a selection instruction for a singer's singing voice, the audio amplitude of the singer's singing voice is corrected according to the direction of the user in the user's video to obtain a corrected singer's singing voice;

[0153] The corrected singing voice of the singer is synthesized with the user's video to obtain a synthesized video of the user.

[0154] In an optional embodiment, the electronic device corrects the audio amplitude in the singer's singing voice according to the user's orientation in the user's video, and obtains the corrected singer's singing voice, including:

[0155] When the user in the user's video is facing away from the collection direction, the audio amplitude of the singer's singing voice is reduced to obtain a corrected singing voice of the singer.

[0156] In an optional embodiment, the electronic device is further used for:

[0157] According to the obtained lyrics of the reference song, the video content of the user's synthesized video and the strength of the accompaniment sound of the musical instrument, an optional special effect is determined for the user's synthesized video.

[0158] In an optional embodiment, the electronic device is further used for:

[0159] The singing songs of the corrected user are aligned based on the timestamp of the singer's singing voice, and the singing voice of the aligned user is re-determined as the singing voice of the corrected user.

[0160] Figure 7 A schematic diagram of the structure of another optional electronic device provided in an embodiment of the present application, such as Figure 7As shown, an embodiment of the present application provides an electronic device 700, including: a processor 71 and a storage medium 72 storing instructions executable by the processor; the storage medium 72 relies on the processor 71 to perform operations through a communication bus 73, and when the instructions are executed by the processor, the synthesis method executed on the processor side in one or more of the above embodiments is executed.

[0161] It should be noted that in actual application, the various components in the terminal are coupled together through the communication bus 73. It is understandable that the communication bus 73 is used to realize the connection and communication between these components. In addition to the data bus, the communication bus 73 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as communication buses 73.

[0162] An embodiment of the present application provides a computer storage medium storing executable instructions. When the executable instructions are executed by one or more processors, the processors execute the synthesis method described in one or more of the above embodiments.

[0163] Among them, the computer-readable storage medium can be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and other memories.

[0164] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0165] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0166] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0168] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A synthesis method, characterized in that: include: Collecting a video of the user and the user's singing voice for a reference song; Acquire the singing voice of the singer and the accompaniment sound of the musical instrument in the reference song; According to the video of the user and the singing voice of the singer, the singing voice of the user is modified to obtain the modified singing voice of the user; Mixing the corrected singing voice of the user and the accompaniment sound of the instrument to obtain the song of the user; The user's song and the user's video are synthesized to obtain a synthesized video of the user; wherein the user's singing voice is corrected according to the user's video and the singer's singing voice to obtain the corrected user's singing voice, including: According to the singing voice of the singer, the singing voice of the user is corrected to obtain the singing voice of the user after preliminary correction; adjusting the singing voice of the user after preliminary correction according to the orientation of the user in the video of the user to obtain the adjusted singing voice of the user; The adjusted singing voice of the user is determined as the corrected singing voice of the user.

2. The method according to claim 1, characterized in that The method of correcting the singing voice of the user according to the singing voice of the singer to obtain the singing voice of the user after preliminary correction includes: According to the singing voice of the singer, the singing voice of the user is corrected by using a preset singing voice correction algorithm to obtain the singing voice of the user after preliminary correction.

3. The method according to claim 2, characterized in that The method of correcting the singing voice of the user by using a preset singing voice correction algorithm according to the singing voice of the singer to obtain the singing voice of the user after preliminary correction includes: According to the singing voice of the singer, the audio amplitude in the singing voice of the user is corrected by using a preset singing voice correction algorithm to obtain the singing voice of the user after preliminary correction; and / or, based on the singing voice of the singer, a preset singing voice correction algorithm is used to correct the spectrum envelope of the singing voice of the user to obtain the singing voice of the user after preliminary correction; and / or, based on the singing voice of the singer, a preset singing voice correction algorithm is used to correct the spectrogram in the singing voice of the user to obtain the singing voice of the user after preliminary correction; And / or, based on the singing voice of the singer, a preset singing voice correction algorithm is used to correct the harmonics in the compensated singing voice of the user to obtain the singing voice of the user after preliminary correction.

4. The method according to claim 2, characterized in that: The method of correcting the singing voice of the user by using a preset singing voice correction algorithm according to the singing voice of the singer to obtain the singing voice of the user after preliminary correction includes: According to the singing voice of the singer, the pitch in the singing voice of the user is corrected by using a preset singing voice correction algorithm to obtain the singing voice of the user after the first pitch correction; According to the accompaniment sound of the musical instrument, the pitch of the user's singing voice after the first pitch correction is corrected to obtain the user's singing voice after preliminary correction.

5. The method according to claim 2, characterized in that: The method of correcting the singing voice of the user according to the singing voice of the singer to obtain the singing voice of the user after preliminary correction includes: The singing voice of the singer and the singing voice of the user are input into a pre-trained machine learning model to obtain the singing voice of the user after preliminary correction.

6. The method according to claim 5, characterized in that The method further comprises: Acquire a sample data set; wherein the sample data set includes: the singing voice of the singer in the collected song, the singing voices of at least two users of the collected song, and the corrected singing voice of each user of the at least two users; The sample data set is input into the machine learning model for training to obtain the trained machine learning model.

7. The method according to claim 1, characterized in that The step of adjusting the singing voice of the user after preliminary correction according to the orientation of the user in the video of the user to obtain the adjusted singing voice of the user includes: When the user in the video of the user is facing away from the collection direction, the audio amplitude of the singing voice of the user after preliminary correction is amplified to obtain the adjusted singing voice of the user.

8. The method according to claim 1, characterized in that The method further comprises: When the direction of the user in the collected video of the user is facing away from the collection direction, the collected sound signal of the user is subjected to noise suppression processing and / or signal amplification processing to obtain the singing sound of the user.

9. The method according to claim 1, characterized in that: The method further comprises: When receiving a selection instruction for the singing voice of the singer, the audio amplitude of the singing voice of the singer is corrected according to the orientation of the user in the video of the user to obtain the corrected singing voice of the singer; The corrected singing voice of the singer and the video of the user are synthesized to obtain a synthesized video of the user.

10. The method according to claim 9, characterized in that The step of correcting the audio amplitude of the singer's singing voice according to the orientation of the user in the video of the user to obtain the corrected singing voice of the singer includes: When the user in the video of the user is facing away from the collection direction, the audio amplitude of the singer's singing voice is reduced to obtain a corrected singing voice of the singer.

11. The method according to claim 1, characterized in that: The method further comprises: According to the obtained lyrics of the reference song, the video content of the user's synthesized video and the strength of the accompaniment sound of the musical instrument, optional special effects are determined for the user's synthesized video.

12. The method according to claim 1, characterized in that The method further comprises: Based on the timestamp of the singer's singing voice, the corrected singing songs of the user are aligned, and the aligned singing voice of the user is re-determined as the corrected singing voice of the user.

13. An electronic device, characterized in that: include: A collection module, used for collecting a user's video and the user's singing voice for a reference song; An acquisition module, used for acquiring the singing voice of the singer and the accompaniment sound of the musical instrument in the reference song; A correction module, used for correcting the singing voice of the user according to the video of the user and the singing voice of the singer, to obtain the corrected singing voice of the user; A mixing module, used for mixing the corrected singing voice of the user and the accompaniment sound of the instrument to obtain the song of the user; A synthesis module, used for synthesizing the song of the user and the video of the user to obtain a synthesized video of the user; Wherein, the correction module is specifically used for: According to the singing voice of the singer, the singing voice of the user is corrected to obtain the singing voice of the user after preliminary correction; adjusting the singing voice of the user after preliminary correction according to the orientation of the user in the video of the user to obtain the adjusted singing voice of the user; The adjusted singing voice of the user is determined as the corrected singing voice of the user.

14. An electronic device, characterized in that: include: A processor and a storage medium storing instructions executable by the processor; The storage medium relies on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, the synthesis method described in any one of claims 1 to 12 is executed.

15. A computer storage medium, characterized in that Executable instructions are stored, and when the executable instructions are executed by one or more processors, the processors execute the synthesis method described in any one of claims 1 to 12.

Images