Data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method. The method comprises: in response to a first operation executed by a user, obtaining a first video, the first video comprising a first image set and a first voice, the timestamp of each frame of image in the first image set being the same as the timestamp of a corresponding first syllable in the first voice; in response to a second operation executed by the user, playing a second video; the second video comprises the first image set, a stretched first voice and background music, and the timestamp of a stretched first syllable in the stretched first voice is the same as the timestamp of a beat in the background music; the timestamp of each frame of image in the first image set in the second video is the same as the timestamp of the corresponding stretched first syllable in the stretched first voice.

Description

Technical Field

[0001] The present disclosure relates to the field of audio and video processing, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] There are currently a variety of audio and video processing applications on the market, and users can use these applications to process recorded audio and video, such as adding props to the video or adding background music to the video. Summary of the invention

[0003] The embodiments of the present disclosure provide a data processing method, device, electronic device and storage medium, which can convert speech into music with a certain musical style.

[0004] In a first aspect, an embodiment of the present disclosure provides a data processing method, including:

[0005] In response to a first operation performed by a user, a first video is acquired, wherein the first video includes a first image set and a first voice, and a timestamp of each frame image in the first image set is the same as a timestamp of a corresponding first syllable in the first voice;

[0006] In response to a second operation performed by the user, a second video is played; the second video includes the first image set, the stretched first voice and background music, the timestamp of the stretched first syllable in the stretched first voice is the same as the timestamp of the beat in the background music, and the timestamp of each frame image in the first image set in the second video is the same as the timestamp of the corresponding stretched first syllable in the stretched first voice.

[0007] In a second aspect, an embodiment of the present disclosure provides a data processing device, including:

[0008] an acquisition unit, configured to acquire a first video in response to a first operation performed by a user, wherein the first video includes a first image set and a first voice, wherein a timestamp of each frame image in the first image set is the same as a timestamp of a corresponding first syllable in the first voice;

[0009] A playback unit is used to play a second video in response to a second operation performed by the user; the second video includes the first image set, a stretched first voice and background music, the timestamp of a stretched first syllable in the stretched first voice is the same as the timestamp of the beat in the background music, and the timestamp of each frame image in the first image set in the second video is the same as the timestamp of the corresponding stretched first syllable in the stretched first voice.

[0010] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the method described in the first aspect above.

[0011] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method described in the first aspect above.

[0012] In one or more embodiments of the present disclosure, a second video can be played according to a user operation. Compared with the first video, the second video includes not only the first image set, but also the stretched first voice and background music. Furthermore, the timestamp of the stretched first syllable in the stretched first voice is the same as the timestamp of the beat in the background music, so that the stretched first voice and the background music are combined into music with a certain musical style, achieving the technical effect of converting voice into music. Furthermore, the timestamp of each frame image in the first image set in the second video is the same as the timestamp of the corresponding stretched first syllable in the stretched first voice, so that each frame image is displayed and played synchronously with the corresponding stretched first syllable during playback, thereby improving the user's experience of watching the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate one or more embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

[0014] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure;

[0015] Figure 2a A schematic diagram of obtaining a first video provided by an embodiment of the present disclosure;

[0016] Figure 2b A schematic diagram of obtaining a first video provided by an embodiment of the present disclosure;

[0017] Figure 3 A schematic diagram of stretching a first voice provided by an embodiment of the present disclosure;

[0018] Figure 4a A schematic diagram of a user selecting a music style provided by an embodiment of the present disclosure;

[0019] Figure 4b A schematic diagram of a user selecting a music style provided by another embodiment of the present disclosure;

[0020] Figure 5 A schematic diagram of stretching the first stressed syllable in each speech segment of a first speech provided by an embodiment of the present disclosure;

[0021] Figure 6 A first audio energy schematic diagram provided by an embodiment of the present disclosure;

[0022] Figure 7 A schematic diagram of stretching the timestamp of the second syllable provided by an embodiment of the present disclosure;

[0023] Figure 8 A schematic diagram of the volume peak of the first syllable provided by an embodiment of the present disclosure;

[0024] Fig. 9 A schematic diagram of the structure of a data processing device provided by an embodiment of the present disclosure;

[0025] Fig.10 A schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure, the technical solutions in one or more embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on one or more embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the protection scope of the present disclosure.

[0027] The disclosed embodiment provides a data processing method capable of converting speech into music with a certain musical style.

[0028] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the method includes:

[0029] Step S102, in response to a first operation performed by a user, obtaining a first video, the first video including a first image set and a first voice, the timestamp of each frame image in the first image set being the same as the timestamp of the corresponding first syllable in the first voice;

[0030] Step S104, in response to a second operation performed by the user, playing a second video; the second video includes a first image set, a stretched first voice and background music, the timestamp of a stretched first syllable in the stretched first voice is the same as the timestamp of the beat in the background music, and the timestamp of each frame image in the first image set in the second video is the same as the timestamp of the corresponding stretched first syllable in the stretched first voice.

[0031] Through this embodiment, the second video can be played according to the user operation. Compared with the first video, the second video includes not only the first image set, but also the stretched first voice and background music. In addition, the timestamp of the stretched first syllable in the stretched first voice is the same as the timestamp of the beat in the background music, so that the stretched first voice and the background music are combined into music with a certain musical style, achieving the technical effect of converting voice into music. In addition, the timestamp of each frame image in the first image set in the second video is the same as the timestamp of the corresponding stretched first syllable in the stretched first voice, so that each frame image is displayed and played synchronously with the corresponding stretched first syllable during playback, thereby improving the user's experience of watching the video.

[0032] The method in this embodiment can be executed by a terminal device running an audio and video processing application, and the terminal device includes but is not limited to electronic devices such as mobile phones, computers, tablet computers, car computers, wearable devices, etc. In this embodiment, the user can perform a first operation in the audio and video processing application, and the audio and video processing application responds to the first operation to obtain a first video. The first video includes a first image set and a first voice. The timestamp of each frame image in the first image set is the same as the timestamp of the corresponding first syllable in the first voice. For example. Each frame image in the first image set has a display timestamp, and each first syllable in the first voice has a start play timestamp. The display timestamp of each frame image in the first image set is the same as the start play timestamp of the corresponding first syllable in the first voice, that is, at the same time, a certain frame image in the first image set is synchronously displayed and played with the corresponding first syllable in the first voice, so that the first image set and the first voice cooperate with each other to improve the user's experience of watching the first video. Taking the first video as a narrative video as an example, a certain frame image in the first image set and the corresponding first syllable in the first voice are synchronously displayed and played at the same time, so as to achieve the effect of explaining each frame image in the first image set using the first voice.

[0033] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0034] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0035] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0036] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0037] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0038] In this embodiment, the first voice may be a sound made by a person, such as a speech voice, a story telling voice, a song singing voice, a nursery rhyme reading voice, etc. The first voice may also be a sound made by a machine imitating a human voice, such as a speech voice, a story telling voice, a song singing voice, a nursery rhyme reading voice, etc. It can be understood that this embodiment obtains the first voice of the user with the authorization and consent of the user, and in particular obtains the first voice made by a person with the authorization and consent of the user.

[0039] In one embodiment, in the above step S102, in response to a first operation performed by a user, obtaining a first video includes:

[0040] In response to a video upload operation performed by a user, obtaining a first video; or

[0041] In response to a recording operation performed by a user, a first video is acquired, where the first video includes a first voice recorded by the user.

[0042] In one case, the first operation includes a video upload operation. In response to the video upload operation performed by the user, the audio and video processing application obtains a first video uploaded by the user, where the first video includes a first image set and a first voice. Figure 2a A schematic diagram of obtaining a first video provided by an embodiment of the present disclosure, such as Figure 2aAs shown, the user can select the video to be uploaded in the terminal device and click the "Upload" button to upload the selected video as the first video to the audio and video processing application.

[0043] In another case, the first operation includes a recording operation. In response to the user performing the recording operation, the audio and video processing application obtains a first video, the first video including the first image set and a first voice recorded by the user. Figure 2b A schematic diagram of obtaining a first video provided by an embodiment of the present disclosure, such as Figure 2b As shown, after the user uploads the first image set in the audio and video processing application, the user can click the recording button in the audio and video processing application to record a matching first voice for the first image set, so that the audio and video processing application obtains the first voice, and uses the first voice and the pre-acquired first image set together as the first video. The process of the user uploading the first image set in the audio and video processing application can be that the user can select the first image set to be uploaded in the terminal device, and click the "Upload" button to upload the first image set to the audio and video processing application.

[0044] It can be seen that through this embodiment, the first video can be directly uploaded by the user, and the first voice in the first video can also be recorded by the user for the first image set, thereby meeting the user's first video uploading needs in different scenarios.

[0045] In this embodiment, in response to a second operation performed by the user, a second video may be played. The second video includes a first image set, a stretched first voice, and background music. The stretched first voice refers to a first syllable in the first voice whose timestamp is stretched, and the timestamp of the first syllable includes but is not limited to a start playback timestamp, so the stretched first voice includes the stretched first syllable.

[0046] When there is no silent segment in the first speech, the end playback timestamp of the previous first syllable is the start playback timestamp of the next first syllable. When there is a silent segment in the first speech, the end playback time of the previous first syllable is not the start playback time of the next first syllable, and the end playback time of the previous first syllable can be obtained by detecting whether there is sound. In this embodiment, stretching refers to moving the start playback timestamp of each first syllable, for example, moving the start playback timestamp forward or backward. The silent segment is a segment that does not contain any sound and is detected in the first speech by sound detection technology.

[0047] The timestamp of the stretched first syllable in the stretched first speech is the same as the timestamp of the beat in the background music, and the timestamp of the beat includes but is not limited to the start timestamp of the beat. Since the timestamp of the stretched first syllable in the stretched first speech is the same as the timestamp of the beat in the background music, when the second video is played, the stretched first speech can be played in conjunction with the beat in the background music, thereby improving the musicality of the second video.

[0048] According to the previous introduction, before the timestamp of the first syllable is stretched, at the same moment, a frame image in the first image set and the corresponding first syllable in the first voice are displayed and played synchronously, so that the first image set and the first voice cooperate with each other to improve the user's experience of watching the first video.

[0049] In order to ensure that the first image set and the stretched first syllable also cooperate with each other, in this embodiment, the timestamp of each frame image in the first image set in the second video is the same as the timestamp of the corresponding stretched first syllable in the stretched first voice. That is, if at the same time, a frame image in the first image set is synchronously displayed and played with the corresponding first syllable in the first voice, then after stretching, at the same time, a frame image in the first image set is also synchronously displayed and played with the corresponding stretched first syllable in the first voice, so that the first image set and the stretched first voice cooperate with each other, improving the user's experience of watching the second video. In a specific example, in the first video, image A in the first image set is synchronously displayed and played with the first syllable 1 in the first voice, so in the second video, image A in the first image set is also synchronously displayed and played with the stretched first syllable 1 in the first voice.

[0050] In one embodiment, the first correspondence between each frame image in the first image set and the first syllable in the first speech is used as the second correspondence between each frame image in the first image set and the stretched first syllable in the first speech, that is, if a frame image corresponds to a first syllable in the first speech, then the frame image also corresponds to the stretched first syllable. According to the second correspondence, the timestamp of each frame image in the first image set in the second video is set to be the same as the timestamp of the corresponding stretched first syllable in the stretched first speech, so that each frame image in the first image set in the second video is displayed and played synchronously with the corresponding stretched first syllable in the stretched first speech.

[0051] Figure 3 A schematic diagram of stretching a first voice provided by an embodiment of the present disclosure, Figure 3 In the example, dots represent the first syllable and the stretched first syllable, and the grid represents the beat in the background music. Figure 3As shown, the essence of stretching is to change the start playing timestamp of the first syllable in the first voice. For each first syllable, its playing timestamp is changed after stretching, and the playing timestamp of the first syllable after stretching is the same as the timestamp of the beat in the background music. The playing timestamp of the first syllable can be the start playing timestamp of the first syllable. The timestamp of the beat in the background music can be the start timestamp of the beat in the background music.

[0052] In one embodiment, in step S104, in response to a second operation performed by the user, playing the second video includes:

[0053] In response to the user selecting one of the multiple music style controls, the second video is played; and the background music has the music style corresponding to the one music style control.

[0054] In this embodiment, the user enters the video processing interface of the audio and video processing application, the audio and video processing application obtains the user's first video, obtains accompaniment music according to the default music style, stretches the first voice based on the timestamp of the beat in the accompaniment music, adds instrumental sounds to the accompaniment music based on the stretched first voice to obtain background music, generates and plays a second video based on the stretched first voice, background music and first image set, and if the user determines that the default music style is not satisfactory based on the played second video, the user can switch the selected music style control among multiple music style controls to switch the music style of the accompaniment music. The audio and video processing application re-acquires the accompaniment music according to the music style of the switched accompaniment music, stretches the first voice based on the timestamp of the beat in the accompaniment music, adds instrumental sounds to the accompaniment music based on the stretched first voice to obtain background music, generates and plays a second video based on the stretched first voice, background music and first image set.

[0055] It can be understood that since the background music is obtained by adding instrumental sounds to the accompaniment music, the background music has the music style of the accompaniment music. The music style of the accompaniment music is related to the timestamps of the beats in the accompaniment music. Therefore, switching the music style of the accompaniment music has the effect of replacing the accompaniment music and changing the timestamps of the beats in the accompaniment music. Therefore, the audio and video processing application needs to re-acquire the accompaniment music in response to the music style switched by the user, re-stretch the first voice and play the second video.

[0056] Therefore, in this embodiment, the audio and video processing application can not only obtain the accompaniment music according to the default music style, stretch the first voice based on the timestamp of the beat in the accompaniment music and play the second video, but also can redetermine the music style of the accompaniment music in response to the user's selection of one of the multiple music style controls, obtain new accompaniment music based on the redetermined music style of the accompaniment music, and re-stretch the first voice based on the timestamp of the beat in the new accompaniment music and play the second video, thereby meeting the user's need to select the music style of the accompaniment music. Since the background music has the music style of the accompaniment music, it also meets the user's need to select the music style of the background music.

[0057] Figure 4a A schematic diagram of a user selecting a music style provided by an embodiment of the present disclosure, such as Figure 4a As shown, in a specific embodiment, after the audio and video processing application obtains the first video, the user can first click on the voice-to-song control, and the audio and video processing application obtains the accompaniment music according to the music style of the default accompaniment music, stretches the first voice based on the timestamp of the beat in the accompaniment music, adds instrumental sounds to the accompaniment music based on the stretched first voice to obtain background music, and generates and plays the second video based on the stretched first voice, background music and first image set. If the user is not satisfied with the default music style, the user can switch the selected music style control among multiple music style controls by sliding the screen horizontally, thereby switching the music style of the accompaniment music, and the audio and video processing application obtains new accompaniment music according to the music style of the switched accompaniment music, and re-stretches the first voice based on the timestamp of the beat in the new accompaniment music and plays the second video.

[0058] Figure 4b A schematic diagram of a user selecting a music style provided by another embodiment of the present disclosure is shown in FIG. Figure 4b As shown, in another specific embodiment, after the audio and video processing application obtains the first video, the user can first click on the voice-to-song control, and the audio and video processing application obtains the accompaniment music according to the music style of the default accompaniment music, stretches the first voice based on the timestamp of the beat in the accompaniment music, adds instrumental sounds to the accompaniment music based on the stretched first voice to obtain background music, and generates and plays the second video based on the stretched first voice, background music and first image set. If the user is not satisfied with the default music style, the user can switch the selected music style control among multiple music style controls by clicking the switch button, thereby switching the music style of the accompaniment music, and the audio and video processing application obtains new accompaniment music according to the music style of the switched accompaniment music, and re-stretches the first voice based on the timestamp of the beat in the new accompaniment music and plays the second video.

[0059] It can be seen that through this embodiment, the user can select the required music style from a variety of music styles, and play the second video based on the selected music style, so that the background music has the selected music style, satisfying the user's need to select the music style of the background music.

[0060] In one embodiment, the background music includes a first audio track and a second audio track; the first audio track includes an instrument audio track, and the second audio track includes an audio track other than the instrument audio track in the background music;

[0061] The timestamp of the stretched first syllable in the stretched first speech is the same as the timestamp of the instrumental sound in the first track, and the timestamp of the stretched first syllable in the stretched first speech is the same as the timestamp of the beat in the second track.

[0062] In this embodiment, the background music includes a first audio track and a second audio track. The first audio track is the instrument audio track corresponding to the instrument sound added to the accompaniment music mentioned above, and the second audio track is the accompaniment audio track corresponding to the accompaniment music mentioned above. Since the background music is composed of the first audio track and the second audio track, the second audio track can also be called the audio track in the background music except the instrument audio track.

[0063] For a stretched first syllable, a first track, and a second track, wherein the first syllable is stretched according to the timestamp of the beat of the second track, so the timestamp of the stretched first syllable in the stretched first voice is the same as the timestamp of the beat in the second track. When adding the first track to the second track, it is necessary to add an instrumental sound at the timestamp of the stretched first syllable, for example, adding a drum sound at the start playback timestamp of the stretched first syllable, so the timestamp of the stretched first syllable in the stretched first voice is the same as the timestamp of the instrumental sound in the first track, and the timestamp of the instrumental sound can be the start timestamp of the instrumental sound.

[0064] In a specific embodiment, after the start playback timestamp of each first syllable in the first speech is stretched according to the timestamp of the beat in the second audio track, the representative pitch of each stretched first syllable in the stretched first speech is determined, and the representative pitch of each stretched first syllable can be the pitch with the largest number of occurrences among the various pitches possessed by each stretched first syllable, and an instrumental sound with the same pitch as the representative pitch of each stretched first syllable is generated, such as a drum sound with the same pitch as the representative pitch of each stretched first syllable, and an instrumental sound with the same pitch is added to the start playback timestamp of each stretched first syllable, so that the stretched first syllable in the stretched first speech is played synchronously with the instrumental sound, and the added instrumental sound forms the first audio track.

[0065] It can be seen that through this embodiment, the background music includes a first track and a second track, the first track includes an instrument track, the second track includes tracks in the background music other than the instrument track, the timestamp of the stretched first syllable in the stretched first speech is the same as the timestamp of the instrument sound in the first track, and the timestamp of the stretched first syllable in the stretched first speech is the same as the timestamp of the beat in the second track, and by adding the instrument track and setting the timestamp of the instrument sound, the stretched first syllable in the stretched first speech can be played synchronously with the instrument sound in the first track, so that the vocals in the stretched first speech sound more rhythmic and increase the musicality of the background music.

[0066] In a specific embodiment, the above method process can also select accompaniment music corresponding to the first voice in the database according to the music style and the beat information of the first voice, wherein the music style includes the default music style of the audio and video processing application and the music style selected by the user, and the music style can be exemplified by R&B, RAP, etc. The beat information of the first voice includes the BPM (Beat Per Minute) of the first voice, and the accompaniment music is used to form the second audio track.

[0067] In this embodiment, first, the start playback timestamp of each first syllable is detected, for example, the start playback timestamp of each first syllable is detected by a DSP (Digital Signal Process) algorithm, and the start playback timestamps of two adjacent first syllables are subtracted to obtain the duration of each first syllable. Then, the beat information of the first voice is determined according to the duration of each first syllable. The median of the duration can be selected from the duration of each first syllable, and the beat information of the first voice is calculated according to the median, for example, the BPM of the first voice is obtained by dividing 60 by the median as the beat information of the first voice.

[0068] The database stores a plurality of accompaniment music, each of which corresponds to a music style and a beat information. For example, the database stores 16 accompaniment music of four music styles, wherein each music style has four accompaniment music with different BPMs. Based on this, according to the music style and the beat information of the first voice, music whose music style is consistent with the above music style and whose beat information matches the beat information of the first voice is selected in the database as the accompaniment music corresponding to the first voice.

[0069] In this embodiment, various accompaniment music with music styles consistent with the above-mentioned music styles can be selected from the database first, and each selected accompaniment music corresponds to a BPM range, and the BPM ranges corresponding to the selected accompaniment music are different. It is determined whether the BPM of the first voice falls within the BPM range corresponding to any selected accompaniment music. If not, the BPM of the first voice is processed, such as multiplied by 2 or divided by 2, and it is determined again whether the BPM of the first voice falls within the BPM range corresponding to any selected accompaniment music, and the BPM of the first voice is processed until the BPM of the first voice falls within the BPM range corresponding to a selected accompaniment music, and the accompaniment music that falls within is used as the accompaniment music corresponding to the first voice.

[0070] In the database, each accompaniment music corresponds to a BPM range, and the BPM range can be obtained by fine-tuning the BPM of the accompaniment music. For example, if the BPM corresponding to a certain accompaniment music is 70, then its corresponding BPM range can be [70-2.5, 70+2.5]. By fine-tuning the BPM corresponding to the accompaniment music into a range, the efficiency of selecting the accompaniment music corresponding to the first voice in the database can be improved.

[0071] It can be seen that through this embodiment, the beat information of the first voice can be determined efficiently and quickly according to the duration of each first syllable in the first voice, and the appropriate accompaniment music can be accurately matched for the first voice in the database according to the music style and the beat information of the first voice.

[0072] In one embodiment, the timestamp of the first stressed syllable in each speech segment in the stretched first speech is the same as the timestamp of the first beat of the bar in the second audio track, or the same as the timestamp of the middle beat of the bar in the second audio track;

[0073] The speech segmentation is obtained by dividing the first speech according to the silent segments in the first speech; the beats include the first beat of each measure in the second audio track and the middle beat of each measure in the second audio track.

[0074] In this embodiment, there is a silent segment in the first speech, and the first speech can be divided according to the silent segment in the first speech to obtain multiple speech segments, and two adjacent speech segments are separated by a silent segment. The silent segment refers to a segment that is determined by the DSP algorithm to not contain any sound information, and the speech segment refers to a segment that is determined by the DSP algorithm to contain sound.

[0075] Each speech segment of the first speech includes at least one stressed syllable, and the stressed syllable is the first syllable whose audio energy is greater than the audio energy of the two adjacent first syllables. In this embodiment, the first stressed syllable is determined in each speech segment, and the first stressed syllable is the stressed syllable with the earliest timestamp. The beats in the above-mentioned second audio track include the first beat of each measure in the second audio track and the middle beat of each measure in the second audio track. For example, a measure in the second audio track includes 4 beats, and the beats in the above-mentioned second audio track include the first beat of each measure and the beat corresponding to the middle position of each measure.

[0076] In this embodiment, for the first stressed syllable in each speech segment of the first speech, the start playback timestamp of the first stressed syllable is stretched according to the start playback timestamp of the first beat of each bar in the second audio track and the start playback timestamp of the middle beat of each bar, and the start playback timestamp of the first stressed syllable after stretching is the same as the start playback timestamp of the first beat of the bar in the second audio track, or is the same as the playback timestamp of the middle beat of the bar in the second audio track.

[0077] In a specific example, first, for the first stressed syllable in the first speech segment of the first speech, the start playback timestamp of the syllable is stretched to be the same as the start playback timestamp of the first beat of the first bar in the second audio track. Then, for the first stressed syllable in the second speech segment of the first speech, while the timestamp interval between each first syllable in the same speech segment remains unchanged, the start playback timestamp of the first stressed syllable in the second speech segment is stretched to be the same as the start playback timestamp of the first beat of a bar in the second audio track, or to be the same as the start playback timestamp of the middle beat of a bar. And so on, the start playback timestamps of the first stressed syllable in each subsequent speech segment are stretched.

[0078] Figure 5 A schematic diagram of stretching the first stressed syllable in each speech segment of the first speech provided by an embodiment of the present disclosure, such as Figure 5 As shown, $ represents a stressed syllable, and * represents an unstressed syllable. For the first stressed syllable in each speech segment of the first speech, while the timestamp interval between the first syllables in the same speech segment remains unchanged, the start playback timestamp of the first stressed syllable in each speech segment is stretched to be the same as the start playback timestamp of the first beat of a measure in the second audio track, or to be the same as the start playback timestamp of the middle beat of a measure. Figure 5In the figure, when the time stamp interval between the first syllables in the same speech segment remains unchanged, the first stressed syllable of the second speech segment in the figure is moved to the middle beat of the fourth measure due to the longer length of the first speech segment in the figure. Considering that the first beat of the fifth measure is after the middle beat of the fourth measure, if it is moved to the first beat of the fifth measure, the playback of the first stressed syllable of the second speech segment in the figure will be delayed. Therefore, it is not chosen to move to the first beat of the fifth measure.

[0079] It can be seen that through this embodiment, the timestamp of the first stressed syllable in each speech segment of the first speech can be stretched to be the same as the timestamp of the first beat of the bar in the second audio track, or to be the same as the timestamp of the middle beat of the bar in the second audio track, so that the timestamp of the stretched first stressed syllable in each speech segment can match the rhythm of the second audio track, thereby improving the user's second video browsing experience.

[0080] In one embodiment, the method further includes:

[0081] Determining the audio energy of each first syllable in the first speech;

[0082] For each first syllable in the first speech, judging whether the first syllable is a stressed syllable according to the audio energy of the first syllable and the audio energy of the first syllable adjacent to the first syllable;

[0083] The first stressed syllable in each speech segment is determined according to whether each first syllable is a stressed syllable.

[0084] In this embodiment, for each first syllable in each first speech, since the first syllable has a certain pronunciation duration, the sampling values ​​of each audio energy size of the first syllable within the pronunciation duration are obtained, and the sampling values ​​of each audio energy size are subjected to root mean square processing to obtain the audio energy of the first syllable. For each first syllable in each first speech, according to the audio energy of the first syllable and the audio energy of the adjacent first syllables of the first syllable in the first speech (the adjacent first syllables may be located in different speech segments), the first syllable whose audio energy is greater than the audio energy of the two adjacent first syllables is determined as a stressed syllable. After determining each stressed syllable in the first speech, the first stressed syllable in each speech segment can be determined, and the first stressed syllable is the stressed syllable with the earliest timestamp in each speech segment.

[0085] Figure 6 A first audio energy diagram provided by an embodiment of the present disclosure is shown as follows: Figure 6 As shown, in the first speech, each first syllable has its own audio energy. Figure 6The horizontal axis represents the syllable number, and the vertical axis represents the audio energy. If the audio energy of a first syllable is greater than the audio energies of the two adjacent first syllables, the audio energy of the first syllable presents a local peak state, so the first syllable is a stressed syllable.

[0086] It can be seen that, through this embodiment, each stressed syllable in the first speech can be quickly determined by calculating the audio energy of the first syllable, thereby identifying the first stressed syllable in each speech segment.

[0087] In one embodiment, the method further includes:

[0088] Determine a start timestamp and an end timestamp of a second audio track according to a timestamp of a first stressed syllable stretched in the first speech segment and a timestamp of a first stressed syllable stretched in the last speech segment;

[0089] According to the start timestamp and end timestamp of the second audio track, the music in the second audio track is played in a loop.

[0090] In this embodiment, after the timestamp of the first stressed syllable in each speech segment, for example, the start playback timestamp, is stretched, based on the timestamp of the stretched first stressed syllable in the first speech segment in the first speech, while ensuring that the timestamp interval between the first stressed syllable in the first speech segment and the first syllable in the first speech segment remains unchanged, the timestamp of the first syllable in the first speech segment is stretched, that is, the timestamp of the first syllable in the first speech segment is shifted based on the timestamp of the stretched first stressed syllable in the first speech segment.

[0091] After the timestamp stretching of the first stressed syllable in each speech segment is completed, the timestamp of the last syllable in the last speech segment in the first speech is stretched according to the timestamp of the stretched first stressed syllable in the last speech segment, while ensuring that the timestamp interval between the first stressed syllable in the last speech segment and the last syllable in the last speech segment remains unchanged, that is, the timestamp of the last syllable in the last speech segment is shifted according to the timestamp of the stretched first stressed syllable in the last speech segment.

[0092] In this embodiment, the start timestamp of the second track is also set to be equal to the timestamp of the first stressed syllable stretched in the first voice segment, and the end timestamp of the second track is equal to the timestamp of the last stretched syllable in the last voice segment, so that the accompaniment music in the second track is played from the first stressed syllable in the first voice segment to the last syllable in the last voice segment. Since the accompaniment music stored in the database is music that can be played in a loop, the music in the second track is played in a loop according to the start timestamp and the end timestamp of the second track, and the music in the second track is the accompaniment music. For example, the length of the accompaniment music obtained from the database is 50 seconds, and the duration before the start timestamp and the end timestamp of the second track is 150 seconds, then the accompaniment music is played in a loop three times.

[0093] It can be seen that through this embodiment, the start timestamp and the end timestamp of the second audio track can be determined, and according to the start timestamp and the end timestamp of the second audio track, the music in the second audio track is played in a loop, that is, the accompaniment music is played in a loop, so as to achieve the goal of determining the playing duration of the accompaniment music based on the duration of the stretched first voice, thereby ensuring that the user's semantics in the stretched first voice are not lost, and generating the second video while retaining the user's complete semantics.

[0094] In one example, accompaniment music is first selected from a database based on the music style and the beat information of the first speech, and then the timestamp of the first stressed syllable in each speech segment in the first speech is stretched to be the same as the timestamp of the first beat of a measure of the accompaniment music, or to be the same as the timestamp of the middle beat of a measure in the accompaniment music. During the stretching process, if the duration of the accompaniment music is too short, the accompaniment music is played in a loop to complete the stretching of the timestamp of the first stressed syllable in the last speech segment.

[0095] Then, the start timestamp and the end timestamp of the accompaniment music are determined according to the timestamp of the first stretched stressed syllable in the first speech segment and the timestamp of the first stretched stressed syllable in the last speech segment, and the accompaniment music is played in a loop according to the start timestamp and the end timestamp of the accompaniment music to form a second audio track.

[0096] In one embodiment, the method further includes:

[0097] For a second syllable in the first speech, selecting a matching first beat for the second syllable from the two beats according to the audio energy of two beats whose timestamps in the second audio track are closest to the timestamp of the second syllable before stretching and the timestamps of the two beats;

[0098] Stretch the timestamp of the second syllable to be the same as the timestamp of the first beat;

[0099] Among them, the second syllables are syllables in each first syllable of the first speech except the first stressed syllable in each speech segment, the first syllable in the first speech segment and the last syllable in the last speech segment; the speech segments are obtained by dividing the first speech according to the silent segments in the first speech.

[0100] In this embodiment, each first syllable of the first speech includes a second syllable, and the second syllable is a syllable of each first syllable of the first speech except the first stressed syllable in each speech segment, the first syllable in the first speech segment, and the last syllable in the last speech segment. Through the previous process, the timestamps of the first stressed syllable in each speech segment, the first syllable in the first speech segment, and the last syllable in the last speech segment have been stretched, so the timestamp of the second syllable needs to be stretched here.

[0101] In this embodiment, the second audio track has multiple beats, and the beat can be the first beat of each measure in the second audio track, or the last beat of each measure, or the beat at a preset position such as the position of each beat, the position of each half beat, and the position of each quarter beat. The timestamp of the beat can be the start playing timestamp of the beat.

[0102] In this embodiment, each beat of the second audio track has corresponding audio energy. As an example, for each beat of the second audio track, the audio energy of the second audio track at the beat can be determined based on the sampled value of the audio energy of the second audio track at the beat. When there are multiple sampled values ​​of the audio energy of the second audio track at the beat, the multiple sampled values ​​can be subjected to root mean square processing to obtain the audio energy of the second audio track at the beat. As an example, the second audio track is divided into multiple beats in a quarter beat manner, the first beat is the starting position of the first measure in the second audio track, the second beat is the position of a quarter beat of the first measure in the second audio track, and the third beat is the position of a half beat of the first measure in the second audio track. For the first beat, the sampling values ​​of each sampling point in the second audio track within a time interval from the starting position of the first measure to the position of a quarter beat of the first measure are obtained, and each sampling value is subjected to root mean square processing to obtain the audio energy size of the second audio track at the first beat. For the second beat, the sampling values ​​of each sampling point in the second audio track within a time interval from the position of a quarter beat of the first measure to the position of a half beat of the first measure are obtained, and each sampling value is subjected to root mean square processing to obtain the audio energy size of the second audio track at the second beat.

[0103] Of course, the audio energy of the second audio track can also be represented by an audio energy graph. For example, the playback timestamp of each beat is used as the horizontal axis, the audio energy is used as the vertical axis, and the graph has a columnar pattern corresponding to each beat, and the color of the columnar pattern represents the audio energy of the beat.

[0104] In this embodiment, for each second syllable, two beats whose timestamps are closest to the timestamp of the second syllable before stretching are determined in each beat of the second audio track, that is, a beat before the second syllable before stretching and a beat after the second syllable before stretching in time sequence, for example, in playback time sequence, are determined, and according to the audio energy of these two beats and the timestamps of these two beats, for example, the starting playback timestamp, a matching first beat is selected for the second syllable in the two beats, and the timestamp of the second syllable is stretched to be the same as the timestamp of the first beat.

[0105] It can be seen that through this embodiment, for the second syllable in the first speech, based on the audio energy of the two beats closest to the timestamp of the second syllable before stretching and the timestamps of the two beats in the second audio track, a matching first beat can be selected for the second syllable in the two beats, and the timestamp of the second syllable can be stretched to be the same as the timestamp of the first beat, thereby achieving the effect of stretching the timestamp of the second syllable according to the rhythm of the second audio track.

[0106] In one embodiment, according to the audio energy of two beats whose timestamps in the second audio track are closest to the timestamp of the second syllable before stretching and the timestamps of the two beats, selecting a matching first beat for the second syllable in the two beats comprises:

[0107] A matching first beat is selected for the second syllable in the two beats based on the audio energy of the two beats, the first time length required for the timestamp of the second syllable before stretching to be stretched to the timestamps of the two beats, and the change in the second time length; the second time length includes the time length between the timestamp of the second syllable before stretching and the first timestamp.

[0108] First, obtain the audio energy of the above-mentioned two beats, obtain the timestamp of the second syllable before stretching and the timestamps of these two beats, and calculate the first time length required to stretch the timestamp of the second syllable before stretching to the timestamp of each beat in the two beats. One beat corresponds to one first time length, and the first time length is equal to the difference between the timestamp of the second syllable before stretching and the timestamp of each beat.

[0109] Next, obtain the change in the second time length, where the second time length includes the time length between the timestamp before the second syllable is stretched and the first timestamp. Since the timestamp of the second syllable is stretched, the second time length will be changed, and the change in the second time length is equal to the second time length before the second syllable is stretched minus the second time length after the second syllable is stretched. The second time length before the second syllable is stretched is the time length between the timestamp before the second syllable is stretched and the first timestamp. The second time length after the second syllable is stretched is the time length between the timestamp of the stretched second syllable and the first timestamp after the timestamp of the second syllable is stretched to each beat in the above two beats, that is, equal to the time length between the timestamp of each beat in the above two beats and the first timestamp. Each beat corresponds to a changed second time length, and also corresponds to a change in the second time length.

[0110] Finally, based on the audio energy of the above two beats, the first time length corresponding to each beat, and the change in the second time length corresponding to each beat, a matching first beat is selected for the second syllable in the above two beats.

[0111] It can be seen that through this embodiment, it is possible to select a matching first beat for the second syllable in the above two beats based on the audio energy of the above two beats, the first time length required for stretching the timestamp of the second syllable before stretching to the timestamp of the above two beats, and the change in the second time length, thereby achieving the technical effect of accurately determining the first beat.

[0112] In one embodiment, the first beat is a beat selected from the above two beats according to the order of audio energy size, so that the first time length is less than the length threshold and the change in the second time length meets the change requirement; the first timestamp includes the timestamp of the stretched first syllable before the second syllable and / or the timestamp of the stretched first syllable after the second syllable.

[0113] In this embodiment, it is first determined whether the beat with greater audio energy among the two most recent beats is the first beat. The specific determination process is to determine whether the first time length corresponding to the beat with greater audio energy is less than a length threshold, and to determine whether the change in the second time length corresponding to the beat with greater audio energy meets a change requirement. The change requirement may be that the change in the second time length is less than a change threshold, or that the ratio of the change in the second time length divided by the second time length before the second syllable is stretched is less than a ratio threshold.

[0114] If the first time length corresponding to the beat with larger audio energy is less than the length threshold, and the change in the corresponding second time length meets the change requirement, then the beat is taken as the first beat. Otherwise, it is determined whether the beat with smaller audio energy in the two most recent beats is the first beat. If the beat with smaller audio energy is not the first beat either, the second audio track can be re-divided to obtain finer-grained beats, and the first beat is selected for the second syllable in the finer-grained beats.

[0115] The process of selecting the first beat for the second syllable in a finer-grained beat is similar to the previous process, and also based on the audio energy of the two finer-grained beats closest to the timestamp of the second syllable before stretching in the second audio track and the timestamps of the two finer-grained beats, a matching first beat is selected for the second syllable in the two finer-grained beats.

[0116] In this embodiment, the first timestamp includes one or more of the timestamp of the stretched first syllable before the second syllable and the timestamp of the stretched first syllable after the second syllable. The timestamp of the stretched first syllable remains unchanged due to the stretching process. After the second syllable is stretched to any beat, the change in the time length between the second syllable and the first timestamp is measured, and whether the stretching distance of the second syllable is too long can be measured to avoid stretching the second syllable too long. Similarly, measuring the relationship between the first time length and the length threshold can also avoid stretching the second syllable too long.

[0117] Figure 7 A schematic diagram of stretching the timestamp of the second syllable provided by an embodiment of the present disclosure, such as Figure 7 As shown, the serial numbers of each second syllable are 1-7, wherein syllable 1 is the first syllable of a voice segment, and its timestamp is fixed; syllable 2 is the first stressed syllable of the voice segment, and its timestamp is also fixed, which is the same as the timestamp of the first beat of the first measure in the second audio track in the figure; syllable 7 is the first stressed syllable of the next voice segment, and its timestamp is also fixed, which is the same as the timestamp of the first beat of the third measure in the second audio track in the figure.

[0118] Figure 7 In the figure, the second syllables are 3, 4, 5, and 6. For syllable 4, the two nearest beats are the third beat of the first measure and the first beat of the second measure. Through the above process, the first beat of the second measure is determined as the first beat. Therefore, the timestamp of syllable 4 is stretched to be the same as the timestamp of the first beat of the second measure. Figure 7, the first timestamp includes the stretched timestamps of syllable 2 and syllable 7. The second time variation includes the variation of the time length between syllable 4 and syllable 2 before and after the timestamp of syllable 4 is stretched, and the variation of the time length between syllable 4 and syllable 7.

[0119] In addition, according to the previous introduction, the second audio track plays from the first stressed syllable in the first speech segment to the last syllable in the last speech segment. Therefore, in the case that the first speech segment in the figure is the first speech segment of the first speech, for the syllables between the first syllable and the first stressed syllable in the first speech segment in the figure, after the timestamps of the first syllable and the first stressed syllable in the first speech segment in the figure are fixed, the syllables between the two syllables do not have a corresponding second audio track, and the timestamps of the syllables between the two syllables do not need to be stretched.

[0120] It can be seen that through the embodiment, since the first beat is selected from the two most recent beats according to the order of audio energy size, so that the first time length is less than the length threshold and the change in the second time length meets the change requirement, the first beat can make the stretching amount of the timestamp of the second syllable be within an appropriate range, avoiding stretching the second syllable too long.

[0121] As mentioned above, the second audio track has multiple beats, and the beat can be the first beat of each measure in the second audio track, or the last beat of each measure, or the beat at a preset position such as the position of each beat, the position of each half beat, and the position of each quarter beat. The timestamp of the beat can be the start playing timestamp of the beat.

[0122] In one embodiment, the beats in the second audio track can be divided in the second audio track according to a division rule, and the division rule includes multiple sub-rules with a hierarchical relationship. When the sub-rules of the previous level do not meet the selection conditions for the first beat of the second syllable, the second audio track is divided according to the sub-rules of the next level and the first beat corresponding to the second syllable is reselected.

[0123] Specifically, Figure 7For example, first, according to the sub-rule that each beat in the second track is a beat, the second track is divided into multiple beats. For syllable 4, the two nearest beats corresponding to it are the third beat of the first measure in the figure and the first beat of the second measure. If, through the above process, the third beat of the first measure and the first beat of the second measure are not the first beat, then according to the sub-rule of the next level, the sub-rule that each half beat in the second track is a beat, the second track is divided into multiple beats, and the first beat is reselected for syllable 4. If syllable 4 corresponds to the 3.8th beat of the first measure of the second track before stretching, after dividing the second track into each half beat as a beat, the two nearest beats corresponding to syllable 4 are the 3.5th beat of the first measure in the figure and the first beat of the second measure. By dividing the beats in the second track in multiple levels, each second syllable can be matched with a suitable first beat.

[0124] In one embodiment, the method further includes:

[0125] For the second syllable in the first voice, according to the stretching order of each second syllable, stretch the timestamp of the second syllable with the earlier stretching order;

[0126] Based on the stretching amount and stretching order of the timestamp of the second syllable that comes first in the stretching order, stretching the timestamp of the second syllable that comes later in the stretching order;

[0127] Among them, the second syllables are syllables in each first syllable of the first speech except the first stressed syllable in each speech segment, the first syllable in the first speech segment and the last syllable in the last speech segment; the speech segments are obtained by dividing the first speech according to the silent segments in the first speech.

[0128] In this embodiment, each first syllable of the first speech includes a second syllable, and the second syllable is a syllable of each first syllable of the first speech except the first stressed syllable in each speech segment, the first syllable in the first speech segment, and the last syllable in the last speech segment. Through the previous process, the timestamps of the first stressed syllable in each speech segment, the first syllable in the first speech segment, and the last syllable in the last speech segment have been stretched, so the timestamp of the second syllable needs to be stretched here.

[0129] In this embodiment, a stretching order of each second syllable is determined, and according to the stretching order, the timestamp of the second syllable that comes first in the stretching order is stretched, and when stretching the second syllable that comes later in the stretching order, the timestamp of the second syllable that comes later in the stretching order is stretched based on the stretching amount of the timestamp of the second syllable that comes first in the stretching order and the above-mentioned stretching order. The stretching amount of the timestamp of the second syllable is equal to the length of the timestamp interval between the timestamp before and after the stretching of the second syllable. For example, according to the stretching amount of the timestamp of the second syllable that comes first in the stretching order, the timestamp of the second syllable that comes later in the stretching order is changed, and the timestamp after the change is the timestamp before the stretching of the second syllable that comes later in the stretching order, and according to the timestamp before the stretching of the second syllable that comes later in the stretching order, the timestamp of the second syllable that comes later in the stretching order is stretched.

[0130] by Figure 7 For example, each second syllable includes syllables 3, 4, 5, and 6, and each syllable is stretched one by one in a stretching order, such as the stretching order is 4, 3, 5, and 6. When each second syllable, such as syllable 4, is stretched, the timestamp of the second syllable before syllable 4, that is, syllable 3, and the timestamps of the second syllable after syllable 4, that is, syllables 5 and 6, are updated as the timestamp of syllable 4 is stretched.

[0131] from Figure 7 It can be seen that the timestamps of syllables 1, 2, and 7 that have been stretched before will not change with the change of syllable 4. The timestamps of syllables 3, 5, and 6 will change with the stretching of the timestamp of syllable 4. The change principle is: the change amount of the timestamp of each syllable is the same as the stretching amount of the timestamp of syllable 4. After the timestamp of syllable 4 is stretched, the timestamp of syllable 3 after the change is used as the timestamp of syllable 3 before the stretching.

[0132] During the stretching of the timestamp of syllable 3, the timestamp of syllable 4 does not change because the stretching has been completed. During the stretching of the timestamp of syllable 6, the timestamps of syllables 4 and 3 do not change because the stretching has been completed. Moreover, during the stretching of the timestamp of syllable 6, the timestamp of syllable 5 may also change, and the timestamp of syllable 5 after the change is stretched as the timestamp of syllable 5 before the stretching.

[0133] It can be seen that through this embodiment, the timestamp of the second syllable that comes first in the stretching order can be stretched according to the stretching order of each second syllable, and the timestamp of the second syllable that comes later in the stretching order can be stretched based on the stretching amount and the stretching order of the timestamp of the second syllable that comes first in the stretching order, so that the second syllable that is stretched later can be stretched in combination with the stretching amount of the timestamp of the first stretched second syllable, thereby improving the accuracy of the timestamp of the stretched second syllable.

[0134] In one embodiment, the method further includes:

[0135] The stretching order of each second syllable is determined according to the audio energy of each second syllable in the first speech and whether each second syllable is a stressed syllable.

[0136] In this embodiment, for each second syllable in the first speech, the audio energy of each second syllable is calculated, and it is determined whether each second syllable is a stressed syllable. According to the audio energy of each second syllable in the first speech and the determination result of whether each second syllable is a stressed syllable, the stretching order of each second syllable is determined. The audio energy of the second syllable can be obtained by performing root mean square processing on the sampled values ​​of the audio energy of the second syllable.

[0137] In one embodiment, in each second syllable in the first speech, it is determined that the stressed syllable is stretched before the unstressed syllable, and each stressed syllable in each second syllable and each unstressed syllable in each second syllable respectively form a syllable set, and in each syllable set, the stretching order of each syllable in the syllable set is determined according to the audio energy of each syllable in the syllable set. For example, each stressed syllable in all the second syllables can be combined into a syllable combination, and each unstressed syllable in all the second syllables can be combined into a syllable set. The stretching order of any stressed syllable is ahead of the stretching order of any unstressed syllable. In each syllable set, the stretching order of each syllable in the syllable set from first to last is determined according to the order of the audio energy of each syllable in the syllable set from high to low.

[0138] It can be seen that through this embodiment, the stretching order of each second syllable in the first speech can be determined according to the audio energy of each second syllable and whether each second syllable is a stressed syllable, thereby achieving the effect of orderly stretching the second syllables.

[0139] In one embodiment, after the timestamp stretching of each first syllable in the first speech is completed, the occurrence time of the volume peak of each stretched first syllable in the first speech may also be adjusted.

[0140] Figure 8 A schematic diagram of the volume peak of the first syllable provided by an embodiment of the present disclosure, Figure 8 In the example of the first syllable, the horizontal axis is time and the vertical axis is volume. Figure 8As shown, for each first syllable, it has a start playback timestamp and a volume peak occurrence timestamp. Usually, the start playback timestamp is before the volume peak occurrence timestamp. In this embodiment, the volume peak occurrence timestamp of each stretched first syllable can be adjusted forward to shorten the distance between the start playback timestamp and the volume peak occurrence timestamp, so that the volume peak is reached quickly after the sound of the stretched first syllable appears, thereby improving the user's listening experience.

[0141] In this embodiment, after the timestamp of each first syllable in the first speech is stretched, before the first audio track is added to the second audio track, the pitch of the stretched first speech can be adjusted. For example, the pitch of the second audio track is taken as a target, and the pitch of the stretched first speech is adjusted to be the same as the pitch of the second audio track, so that the stretched first syllable sounds musical and matches the second audio track.

[0142] In this embodiment, after the timestamp stretching of each first syllable in the first speech is completed, the sound effect of the first speech can also be optimized, such as compressing or equalizing the human voice in the first speech to achieve a better human voice effect.

[0143] In summary, the various embodiments of the present disclosure provide a data processing method that can convert speech into music with a certain musical style. Since the first speech will not be greatly damaged during the processing of the first speech, such as cutting off some sentences or repeating some sentences, only the timestamp of the first syllable in the first speech is adjusted, so the entire semantics of the first speech can be retained, ensuring that the user's semantic information is not lost. Since the timestamp of the stretched first syllable in the stretched first speech is the same as the timestamp of the beat in the background music, and the timestamp of each frame image in the first image set in the second video is the same as the timestamp of the corresponding stretched first syllable in the stretched first speech, the stretched first syllable in the stretched first speech in the second video can be displayed and played synchronously with the corresponding frames of images, thereby improving the user's experience of watching the second video.

[0144] An embodiment of the present disclosure further provides a data processing device, which may be located inside a terminal device and is used to implement the above-mentioned data processing method. Fig. 9 A structural diagram of a data processing device provided by an embodiment of the present disclosure is shown in FIG. Fig. 9 As shown, the device comprises:

[0145] An acquisition unit 91 is configured to acquire a first video in response to a first operation performed by a user, wherein the first video includes a first image set and a first voice, and a timestamp of each frame image in the first image set is the same as a timestamp of a corresponding first syllable in the first voice;

[0146] The playback unit 92 is used to play a second video in response to a second operation performed by the user; the second video includes the first image set, the stretched first voice and background music, the timestamp of the stretched first syllable in the stretched first voice is the same as the timestamp of the beat in the background music, and the timestamp of each frame image in the first image set in the second video is the same as the timestamp of the corresponding stretched first syllable in the stretched first voice.

[0147] Optionally, the acquiring unit 91 is specifically configured to:

[0148] In response to a video upload operation performed by the user, acquiring the first video; or

[0149] In response to a recording operation performed by the user, the first video is acquired, where the first video includes the first voice recorded by the user.

[0150] Optionally, the playback unit 92 is specifically configured to:

[0151] In response to the user selecting one of a plurality of music style controls, the second video is played; and the background music has the music style.

[0152] Optionally, the background music includes a first audio track and a second audio track; the first audio track includes an instrument audio track, and the second audio track includes an audio track other than the instrument audio track in the background music;

[0153] The timestamp of the stretched first syllable in the stretched first speech is the same as the timestamp of the instrumental sound in the first audio track, and the timestamp of the stretched first syllable in the stretched first speech is the same as the timestamp of the beat in the second audio track.

[0154] Optionally, the timestamp of the first stressed syllable in each speech segment in the stretched first speech is the same as the timestamp of the first beat of a measure in the second audio track, or the same as the timestamp of a middle beat of a measure in the second audio track;

[0155] The speech segmentation is obtained by dividing the first speech according to the silent segments in the first speech; the beats include the first beat of each measure in the second audio track and the middle beat of each measure in the second audio track.

[0156] Optionally, the device further comprises an accent determination unit, configured to:

[0157] Determining the audio energy of each of the first syllables in the first speech;

[0158] for each of the first syllables in the first speech, judging whether the first syllable is a stressed syllable according to the audio energy of the first syllable and the audio energy of the first syllable adjacent to the first syllable;

[0159] The first stressed syllable is determined in each of the speech segments according to whether each of the first syllables is a stressed syllable.

[0160] Optionally, the device further comprises a loop playing unit, configured to:

[0161] Determine a start timestamp and an end timestamp of the second audio track according to a timestamp of the first stressed syllable stretched in the first speech segment and a timestamp of the first stressed syllable stretched in the last speech segment;

[0162] The music in the second audio track is played in a loop according to the start timestamp and the end timestamp.

[0163] Optionally, the device further includes a first beat determination unit, configured to:

[0164] For a second syllable in the first speech, selecting a matching first beat for the second syllable from the two beats according to audio energy of two beats in the second audio track whose timestamps are closest to the timestamp of the second syllable before stretching and the timestamps of the two beats;

[0165] stretching the time stamp of the second syllable to be the same as the time stamp of the first beat;

[0166] Among them, the second syllables are syllables in each of the first syllables of the first speech except the first stressed syllable in each speech segment, the first syllable in the first speech segment and the last syllable in the last speech segment; the speech segments are obtained by dividing the first speech according to the silent segments in the first speech.

[0167] Optionally, the first beat determination unit is specifically used to:

[0168] A matching first beat is selected for the second syllable in the two beats based on the audio energy of the two beats, the first time length required for the timestamp of the second syllable before stretching to be stretched to the timestamps of the two beats respectively, and the change in the second time length; the second time length includes the time length between the timestamp of the second syllable before stretching and the first timestamp.

[0169] Optionally, the first beat is a beat selected from the two beats according to the order of audio energy size, so that the first time length is less than a length threshold and the change in the second time length meets the change requirement; the first timestamp includes the timestamp of the stretched first syllable before the second syllable and / or the timestamp of the stretched first syllable after the second syllable.

[0170] Optionally, the device further comprises a stretching unit, for:

[0171] For the second syllables in the first speech, according to the stretching order of each of the second syllables, stretching the timestamp of the second syllable with the earlier stretching order;

[0172] Based on the stretching amount of the timestamp of the second syllable that comes first in the stretching order and the stretching order, stretching the timestamp of the second syllable that comes later in the stretching order;

[0173] Among them, the second syllables are syllables in each of the first syllables of the first speech except the first stressed syllable in each speech segment, the first syllable in the first speech segment and the last syllable in the last speech segment; the speech segments are obtained by dividing the first speech according to the silent segments in the first speech.

[0174] Optionally, the device further comprises a sequence determination unit, configured to:

[0175] The stretching order of each second syllable is determined according to the audio energy of each second syllable in the first speech and whether each second syllable is a stressed syllable.

[0176] The data processing device in the embodiment of the present disclosure can implement each process of the above-mentioned data processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0177] An embodiment of the present disclosure further provides an electronic device, Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, such as Fig.10As shown, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 1001 and memory 1002, and one or more applications or data may be stored in the memory 1002. Among them, the memory 1002 may be a short-term storage or a persistent storage. The application stored in the memory 1002 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 1001 may be configured to communicate with the memory 1002 and execute a series of computer executable instructions in the memory 1002 on the electronic device. The electronic device may also include one or more power supplies 1003, one or more wired or wireless network interfaces 1004, one or more input or output interfaces 1005, one or more keyboards 1006, etc.

[0178] In a specific embodiment, the electronic device includes a processor; and a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor implements the following process:

[0179] In response to a first operation performed by a user, a first video is acquired, wherein the first video includes a first image set and a first voice, and a timestamp of each frame image in the first image set is the same as a timestamp of a corresponding first syllable in the first voice;

[0180] In response to a second operation performed by the user, a second video is played; the second video includes the first image set, the stretched first voice and background music, the timestamp of the stretched first syllable in the stretched first voice is the same as the timestamp of the beat in the background music, and the timestamp of each frame image in the first image set in the second video is the same as the timestamp of the corresponding stretched first syllable in the stretched first voice.

[0181] The electronic device in the embodiment of the present disclosure can implement each process of the above-mentioned data processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0182] Another embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented:

[0183] In response to a first operation performed by a user, a first video is acquired, wherein the first video includes a first image set and a first voice, and a timestamp of each frame image in the first image set is the same as a timestamp of a corresponding first syllable in the first voice;

[0184] In response to a second operation performed by the user, a second video is played; the second video includes the first image set, the stretched first voice and background music, the timestamp of the stretched first syllable in the stretched first voice is the same as the timestamp of the beat in the background music, and the timestamp of each frame image in the first image set in the second video is the same as the timestamp of the corresponding stretched first syllable in the stretched first voice.

[0185] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned data processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0186] In various embodiments of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0187] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0188] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0189] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0190] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0191] Those skilled in the art will appreciate that one or more embodiments of the present disclosure may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0192] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0193] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0194] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0195] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0196] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0197] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0198] The above description is only an embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. A data processing method, characterized in that: include: In response to a first operation performed by a user, a first video is acquired, wherein the first video includes a first image set and a first voice, and a timestamp of each frame image in the first image set is the same as a timestamp of a corresponding first syllable in the first voice; In response to a second operation performed by the user, a second video is played; the second video includes the first image set, the stretched first voice and background music, the timestamp of the stretched first syllable in the stretched first voice is the same as the timestamp of the beat in the background music, and the timestamp of each frame image in the first image set in the second video is the same as the timestamp of the corresponding stretched first syllable in the stretched first voice.

2. The method according to claim 1, characterized in that The step of acquiring a first video in response to a first operation performed by a user includes: In response to a video upload operation performed by the user, acquiring the first video; or In response to a recording operation performed by the user, the first video is acquired, where the first video includes the first voice recorded by the user.

3. The method according to claim 1, characterized in that The step of playing the second video in response to the second operation performed by the user includes: In response to the user selecting one of a plurality of music style controls, the second video is played; and the background music has the music style.

4. The method according to any one of claims 1 to 3, characterized in that The background music includes a first audio track and a second audio track; the first audio track includes an instrument audio track, and the second audio track includes an audio track other than the instrument audio track in the background music; The timestamp of the stretched first syllable in the stretched first speech is the same as the timestamp of the instrumental sound in the first audio track, and the timestamp of the stretched first syllable in the stretched first speech is the same as the timestamp of the beat in the second audio track.

5. The method according to claim 4, characterized in that The timestamp of the first stressed syllable in each speech segment in the stretched first speech is the same as the timestamp of the first beat of the bar in the second audio track, or the same as the timestamp of the middle beat of the bar in the second audio track; The speech segmentation is obtained by dividing the first speech according to the silent segments in the first speech; the beats include the first beat of each measure in the second audio track and the middle beat of each measure in the second audio track.

6. The method according to claim 5, characterized in that The method further comprises: Determining the audio energy of each of the first syllables in the first speech; for each of the first syllables in the first speech, judging whether the first syllable is a stressed syllable according to the audio energy of the first syllable and the audio energy of the first syllable adjacent to the first syllable; The first stressed syllable is determined in each of the speech segments according to whether each of the first syllables is a stressed syllable.

7. The method according to claim 6, characterized in that The method further comprises: Determine a start timestamp and an end timestamp of the second audio track according to a timestamp of the first stressed syllable stretched in the first speech segment and a timestamp of the first stressed syllable stretched in the last speech segment; The music in the second audio track is played in a loop according to the start timestamp and the end timestamp.

8. The method according to claim 4, characterized in that The method further comprises: For a second syllable in the first speech, selecting a matching first beat for the second syllable from the two beats according to audio energy of two beats in the second audio track whose timestamps are closest to the timestamp of the second syllable before stretching and the timestamps of the two beats; stretching the time stamp of the second syllable to be the same as the time stamp of the first beat; Among them, the second syllables are syllables in each of the first syllables of the first speech except the first stressed syllable in each speech segment, the first syllable in the first speech segment and the last syllable in the last speech segment; the speech segments are obtained by dividing the first speech according to the silent segments in the first speech.

9. The method according to claim 8, characterized in that The step of selecting a matching first beat for the second syllable from the two beats according to the audio energy of two beats whose timestamps in the second audio track are closest to the timestamp before the stretching of the second syllable and the timestamps of the two beats comprises: A matching first beat is selected for the second syllable in the two beats based on the audio energy of the two beats, the first time length required for the timestamp of the second syllable before stretching to be stretched to the timestamps of the two beats respectively, and the change in the second time length; the second time length includes the time length between the timestamp of the second syllable before stretching and the first timestamp.

10. The method according to claim 9, characterized in that The first beat is a beat selected from the two beats according to the order of audio energy size, so that the first time length is less than a length threshold and the change in the second time length meets the change requirement; the first timestamp includes the timestamp of the stretched first syllable before the second syllable and / or the timestamp of the stretched first syllable after the second syllable.

11. The method according to claim 4, characterized in that The method further comprises: For the second syllables in the first speech, according to the stretching order of each of the second syllables, stretching the timestamp of the second syllable with the earlier stretching order; Based on the stretching amount of the timestamp of the second syllable that comes first in the stretching order and the stretching order, stretching the timestamp of the second syllable that comes later in the stretching order; Among them, the second syllables are syllables in each of the first syllables of the first speech except the first stressed syllable in each speech segment, the first syllable in the first speech segment and the last syllable in the last speech segment; the speech segments are obtained by dividing the first speech according to the silent segments in the first speech.

12. The method according to claim 11, characterized in that The method further comprises: The stretching order of each second syllable is determined according to the audio energy of each second syllable in the first speech and whether each second syllable is a stressed syllable.

13. A data processing device, characterized in that: include: an acquisition unit, configured to acquire a first video in response to a first operation performed by a user, wherein the first video includes a first image set and a first voice, wherein a timestamp of each frame image in the first image set is the same as a timestamp of a corresponding first syllable in the first voice; A playback unit is used to play a second video in response to a second operation performed by the user; the second video includes the first image set, a stretched first voice and background music, the timestamp of a stretched first syllable in the stretched first voice is the same as the timestamp of the beat in the background music, and the timestamp of each frame image in the first image set in the second video is the same as the timestamp of the corresponding stretched first syllable in the stretched first voice.

14. An electronic device, characterized in that: include: processor; as well as, A memory configured to store computer executable instructions, which, when executed, cause the processor to implement the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method described in any one of claims 1 to 12.

Images