Data processing method and apparatus

Abstract

Embodiments of the present disclosure provide a data processing method, which includes, in response to a first operation performed by a user, acquiring a first video including a first image set and a first audio, wherein the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding first syllable in the first audio, and, in response to a second operation performed by the user, playing a second video including the first image set, an expanded first audio, and background music, wherein in the expanded first audio, the timestamp of the expanded first syllable is the same as the timestamp of the beat in the background music, and in the second video, the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding expanded first syllable in the expanded first audio.

Description

Technical Field

[0001] (Cross-reference to Related Applications) This application claims priority based on Chinese Application No. 202310829915.2, filed on July 7, 2023, with the invention title "Data Processing Method, Apparatus, Electronic Device, and Storage Medium", and all the content described in the application is incorporated herein by reference.

[0002] The present disclosure relates to the field of audio and video processing, and in particular, to a data processing method, apparatus, electronic device, and storage medium.

Background Art

[0003] Currently, there are various audio and video processing applications in the market, and users can use such applications to process recorded audio and video, for example, adding props to a video or adding background music to a video.

Summary of the Invention

[0004] Embodiments of the present disclosure provide a data processing method, apparatus, electronic device, and storage medium that can convert audio into music having a specific music style.

[0005] According to a first aspect, embodiments of the present disclosure include: obtaining a first video including a first image set and first audio in response to a first operation performed by a user, wherein a time stamp of each frame image in the first image set is the same as a time stamp of a corresponding first syllable in the first audio; playing a second video including the first image set, the extended first audio, and background music in response to a second operation performed by the user. The present invention provides a data processing method in which, in the expanded first audio, the timestamp of the expanded first syllable is the same as the timestamp of the beat in the background music, and in the second video, the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding expanded first syllable in the expanded first audio.

[0006] According to a second aspect, embodiments of the present disclosure are An acquisition unit for acquiring a first video including a first image set and a first audio in response to a first operation performed by a user, wherein the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding first syllable in the first audio, The system includes a playback unit for playing a second video, which includes the first image set, an expanded first audio, and background music, in response to a second operation performed by the user, The present invention provides a data processing device in which, in the expanded first audio, the timestamp of the expanded first syllable is the same as the timestamp of the beat in the background music, and in the second video, the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding expanded first syllable in the expanded first audio.

[0007] According to a third aspect, an embodiment of the present disclosure provides an electronic device comprising a processor and a memory configured to store computer-executable instructions, wherein when a computer-executable instruction is executed, the processor is instructed to perform the steps of the method described in the first aspect.

[0008] According to a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium used for storing computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the steps of the method described in the first aspect are performed.

[0009] In one or more embodiments of this disclosure, a second video can be played by user operation, and the second video includes not only the first image set but also an expanded first audio and background music, compared to the first video. Since the timestamp of the expanded first syllable in the expanded first audio is the same as the timestamp of the beat in the background music, the expanded first audio and background music are combined as music having a specific musical style, producing the technical effect of converting audio into music. Furthermore, in the second video, since the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding expanded first syllable in the expanded first audio, during playback, each frame image is displayed and played in sync with the corresponding expanded first syllable, improving the user's video viewing experience. [Brief explanation of the drawing]

[0010] To more clearly illustrate the embodiments of this disclosure or the prior art solutions, the accompanying drawings necessary for describing the embodiments or prior art will be briefly described below, but it will be obvious to those skilled in the art that other drawings can be obtained from these accompanying drawings without requiring any creative ingenuity. [Figure 1] This is a schematic diagram showing the flow of a data processing method according to one embodiment of the present disclosure. [Figure 2a] This is a schematic diagram illustrating how to acquire a first video according to one embodiment of the present disclosure. [Figure 2b] This is a schematic diagram illustrating how to acquire a first video according to one embodiment of the present disclosure. [Figure 3] This is a schematic diagram illustrating the expansion of a first audio signal according to one embodiment of the present disclosure. [Figure 4a] This is a schematic diagram illustrating how a user selects a music style according to one embodiment of the present disclosure. [Figure 4b] This is a schematic diagram illustrating a user selecting a music style according to another embodiment of the present disclosure. [Figure 5] This is a schematic diagram of the first syllable in each of the speech segments into which the first speech is extended according to one embodiment of the present disclosure. [Figure 6]This is a schematic diagram of a first audio energy according to one embodiment of the present disclosure. [Figure 7] This is a schematic diagram illustrating the expansion of the timestamp of the second syllable according to one embodiment of the present disclosure. [Figure 8] This is a schematic diagram of the volume peak of the first syllable according to one embodiment of the present disclosure. [Figure 9] This is a schematic diagram showing the configuration of a data processing device according to one embodiment of the present disclosure. [Figure 10] This is a schematic diagram showing the configuration of an electronic device according to one embodiment of the present disclosure. [Modes for carrying out the invention]

[0011] To enable a person skilled in the art to better understand the technical solutions in one or more embodiments of this disclosure, the technical solutions in one or more embodiments of this disclosure are described clearly and completely with reference to the accompanying drawings of one or more embodiments of this disclosure, although it is clear that the embodiments described are only a part of the embodiments of this disclosure, and not all embodiments. All other embodiments that a person skilled in the art could obtain without creative ingenuity based on the embodiments of this disclosure are within the scope of this disclosure.

[0012] The disclosed embodiments provide a data processing method that can convert audio into music having a specific musical style.

[0013] Figure 1 is a schematic diagram showing the flow of a data processing method according to one embodiment of the present disclosure, and as shown in Figure 1, the method includes the following steps: Step S102, in response to a first operation performed by the user, to acquire a first video including a first image set and a first audio, wherein the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding first syllable in the first audio, Step S104, in response to a second operation performed by the user, a second video is played, the second video including a first image set, an expanded first audio track, and background music, wherein the timestamp of the expanded first syllable in the expanded first audio track is the same as the timestamp of the beat in the background music, and in the second video, the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding expanded first syllable in the expanded first audio track.

[0014] In this embodiment, a second video can be played by user operation, and the second video includes not only the first image set but also the expanded first audio and background music, compared to the first video. Furthermore, since the timestamp of the expanded first syllable in the expanded first audio is the same as the timestamp of the beat in the background music, the expanded first audio and background music are combined as music with a specific musical style, producing the technical effect of converting audio into music. In the second video, since the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding expanded first syllable in the expanded first audio, during playback, each frame image is displayed and played in sync with the corresponding expanded first syllable, improving the user's video viewing experience.

[0015] The method of this embodiment may be executed by a terminal device that runs an audio-video processing application. The terminal device includes, but is not limited to, electronic devices such as mobile phones, computers, tablet computers, in-vehicle computers, wearable devices, etc. In this embodiment, the user can execute a first operation in the audio-video processing application, and the audio-video processing application obtains a first video in response to the first operation. The first video includes a first image set and a first audio. The time stamp of each frame image in the first image set is the same as the time stamp of the corresponding first syllable in the first audio. For example, each frame image in the first image set has a display time stamp, each first syllable in the first audio has a playback start time stamp, and the display time stamp of each frame image in the first image set is the same as the playback start time stamp of the corresponding first syllable in the first audio. That is, at the same time point, a certain frame image in the first image set is displayed and played synchronously with the corresponding first syllable in the first audio. By the cooperation of the first image set and the first audio, the experience of the user watching the first video can be improved. Taking the example that the first video is a story video, a certain frame image in the first image set is displayed and played synchronously with the corresponding first syllable in the first audio at the same time point, so that the effect of explaining each frame image in the first image set by using the first audio is achieved.

[0016] Before using the technical solutions disclosed in each embodiment of this disclosure, it should be understood that according to relevant laws and regulations, it is necessary to notify the user in an appropriate manner of the types, scope of use, usage scenarios, etc. of personal information related to this disclosure and obtain the user's permission.

[0017] For example, in response to receiving a voluntary request from a user, presentation information is sent to the user so as to clearly present to the user that the requested operation requires the acquisition and use of the user's personal information. Thereby, based on the presentation information, the user can automatically select whether to provide personal information to software or hardware such as an electronic device, an application, a server, or a storage medium that executes the operation of the disclosed technical solution.

[0018] Exemplarily, in a preferred embodiment, in response to receiving a voluntary request from a user, the form of sending presentation information to the user may be, for example, in the form of a pop-up window that can represent the presentation information in text form. Further, the pop-up window may include a selection control for the user to select to "agree" or "disagree" to provide personal information to the electronic device.

[0019] It should be understood that the above-described process of obtaining the notice and the user's permission is merely an exemplification and does not limit the embodiments of the present disclosure, and other forms that comply with relevant laws and regulations are also applicable to the embodiments of the present disclosure.

[0020] It should be understood that the data related to the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the corresponding laws and regulations and related provisions.

[0021] In this embodiment, the first voice may be a voice uttered by a human, for example, a human speech voice, a greeting voice, a singing voice, a voice reading a nursery rhyme, etc. The first voice may also be a sound emitted by a device that mimics a human voice, for example, a speech sound, a greeting sound, a singing sound, a voice reading a nursery rhyme emitted by the device. It should be understood that in this embodiment, when obtaining the user's permission and consent, the first voice of the user is acquired, and particularly, when obtaining the user's permission and consent, the first voice uttered by a human is acquired.

[0022] According to one embodiment, in step S102 above, in response to a first operation performed by the user, the first video is acquired. To retrieve a first video in response to a video upload operation performed by the user, or This includes obtaining a first video containing a first audio recording made by the user in response to a recording operation performed by the user.

[0023] In one embodiment, the first operation includes a video upload operation. In response to a video upload operation performed by the user, the audio-video processing application retrieves a first video, which includes a first image set and a first audio, uploaded by the user. Figure 2a is a schematic diagram illustrating the acquisition of a first video according to one embodiment of the present disclosure. As shown in Figure 2a, the user can select the video to be uploaded on the terminal device, click the "Upload" button, and upload the selected video as the first video to the audio-video processing application.

[0024] In another embodiment, the first operation includes a recording operation. In response to the recording operation being performed by the user, the audio-video processing application acquires a first video, the first video including not only the first image set described above but also the first audio recorded by the user. Figure 2b is a schematic diagram of acquiring a first video according to one embodiment of the present disclosure. As shown in Figure 2b, after the user uploads the first image set to the audio-video processing application, the user clicks the record button in the audio-video processing application to record a first audio to match the first image set. The audio-video processing application then acquires the first audio and combines the first audio with the previously acquired first image set to form the first video. Here, the process of the user uploading the first image set to the audio-video processing application may involve the user selecting the first image set to be uploaded on a terminal device and clicking the "Upload" button to upload the first image set to the audio-video processing application.

[0025] As can be seen, according to this embodiment, the first video may be uploaded directly by the user, the first audio of the first video is obtained by recording it for the first image set by the user, and the user's need to upload the first video in different scenes is met.

[0026] In this embodiment, a second video can also be played in response to a second operation performed by the user. The second video includes a first image set, an expanded first audio track, and background music. Expanded first audio means that the timestamp of the first syllable in the first audio track has been expanded, and the timestamp of the first syllable includes, but is not limited to, the playback start timestamp; therefore, expanded first audio includes an expanded first syllable.

[0027] If the first audio does not have a silent segment, the playback end timestamp of the previous first syllable is the playback start timestamp of the next first syllable. If the first audio has a silent segment, the playback end time of the previous first syllable is not the playback start time of the next first syllable, but rather the playback end time of the previous first syllable can be obtained by detecting the presence or absence of sound. In this embodiment, stretching refers to moving the playback start timestamp of each first syllable, for example, moving the playback start timestamp forward or backward. A silent segment is a segment that contains no sound and is detected from the first audio by the speech detection technique.

[0028] In the expanded first audio, the timestamp of the expanded first syllable is the same as the timestamp of the beat in the background music, and the timestamp of the beat includes, but is not limited to, the start timestamp of the beat. Because the timestamp of the expanded first syllable in the expanded first audio is the same as the timestamp of the beat in the background music, when the second video is played, the expanded first audio is played in sync with the beat in the background music, thereby improving the musicality of the second video.

[0029] As can be seen from the above explanation, before extending the timestamp of the first syllable, by displaying and playing a frame image in the first image set in sync with the corresponding first syllable in the first audio at the same time, the first image set and the first audio can work together to improve the user experience when viewing the first video.

[0030] To ensure that the first image set and the expanded first syllable cooperate with each other, in this embodiment, in the second video, the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding expanded first syllable in the expanded first audio. In other words, if a frame image in the first image set is displayed and played in sync with the corresponding first syllable in the first audio at the same time, then after expansion, a frame image in the first image set will be displayed and played in sync with the corresponding expanded first syllable in the first audio at the same time. This allows the first image set and the expanded first audio to cooperate with each other, improving the user's experience of watching the second video. In a specific example, in the first video, image A in the first image set is displayed and played in sync with the first syllable 1 in the first audio, so in the second video, image A in the first image set is displayed and played in sync with the expanded first syllable 1 in the first audio.

[0031] In one embodiment, the first correspondence between each frame image in the first image set and the first syllable in the first audio is defined as the second correspondence between each frame image in the first image set and the expanded first syllable in the first audio. That is, if a frame image corresponds to a first syllable in the first audio, then the frame image also corresponds to the expanded first syllable. In accordance with the second correspondence, in the second video, the timestamp of each frame image in the first image set is set to be the same as the timestamp of the corresponding expanded first syllable in the expanded first audio. Therefore, in the second video, each frame image in the first image set is displayed and played in synchronization with the corresponding expanded first syllable in the expanded first audio.

[0032] Figure 3 is a schematic diagram illustrating the expansion of a first sound according to one embodiment of the present disclosure, in which the first syllable and the expanded first syllable are shown as dots, and the time signature in the background music is shown as a grid. As shown in Figure 3, the essence of expansion is to change the playback start timestamp of the first syllable in the first sound, so that the playback timestamp of each of the first syllables is changed by expansion, and the playback timestamp of the expanded first syllable is the same as the timestamp of the time signature in the background music. The playback timestamp of the first syllable may also be the playback start timestamp of the first syllable. The timestamp of the time signature in the background music may also be the start timestamp of the time signature in the background music.

[0033] According to one embodiment, in step S104 above, in response to a second operation performed by the user, the second video is played. The process includes playing a second video in response to the user selecting one of several music style controls, wherein the background music has a music style corresponding to the selected music style control.

[0034] In this embodiment, the user accesses the video processing interface of the audio / video processing application. The audio / video processing application acquires the user's first video, acquires accompaniment music according to the default music style, expands the first audio based on the timestamp of the beat in the accompaniment music, acquires background music by adding instrument sounds to the accompaniment music according to the expanded first audio, and generates and plays a second video based on the expanded first audio, background music, and first image set. If the user decides that they do not like the default music style based on the second video that has been played, they can switch the music style of the accompaniment music by selecting a music style control from among several music style controls. The audio / video processing application then acquires the accompaniment music again according to the switched music style, expands the first audio based on the timestamp of the beat in the accompaniment music, acquires background music by adding instrument sounds to the accompaniment music according to the expanded first audio, and generates and plays a second video based on the expanded first audio, background music, and first image set.

[0035] Since background music is obtained by adding instrumental sounds to accompaniment music, it can be understood that background music has the musical style of the accompaniment music. Because the musical style of the accompaniment music is related to the time signature timestamps in the accompaniment music, switching the musical style of the accompaniment music has the effect of transforming the accompaniment music and changing the time signature timestamps in the accompaniment music. Therefore, the audio / video processing application needs to re-acquire the accompaniment music according to the musical style switched by the user, re-expand the first audio, and play the second video.

[0036] Therefore, in this embodiment, the audio / video processing application not only acquires accompaniment music according to the default music style and expands the first audio and plays the second video based on the timestamp of the beat in the accompaniment music, but also, in response to the user selecting one of several music style controls, re-determines the music style of the accompaniment music, acquires new accompaniment music according to the re-determined music style of the accompaniment music, and re-expands the first audio and plays the second video based on the timestamp of the beat in the new accompaniment music, thereby satisfying the user's need to independently select the music style of the accompaniment music. Since background music also has the music style of the accompaniment music, it also satisfies the user's need to independently select the music style of the background music.

[0037] Figure 4a is a schematic diagram illustrating a user selecting a music style according to one embodiment of the present disclosure. As shown in Figure 4a, in one specific embodiment, after the audio-video processing application acquires a first video, the user first clicks the control from audio to music. The audio-video processing application then acquires the accompaniment music according to the default accompaniment music style, expands the first audio based on the time stamps in the accompaniment music, adds instrument sounds to the accompaniment music based on the expanded first audio to acquire background music, and generates and plays a second video based on the expanded first audio, background music, and first image set. If the user does not like the default music style, they can slide the screen horizontally to switch between selected music style controls from multiple music style controls, thereby switching the accompaniment music style. The audio-video processing application then acquires new accompaniment music according to the switched accompaniment music style, re-expands the first audio based on the time stamps in the new accompaniment music, and plays the second video.

[0038] Figure 4b is a schematic diagram illustrating a user selecting a music style according to another embodiment of the present disclosure. As shown in Figure 4b, in another specific embodiment, after the audio-video processing application acquires a first video, the user first clicks the "Audio to Music" control. The audio-video processing application then acquires the accompaniment music according to the default accompaniment music style, expands the first audio based on the time signatures in the accompaniment music, adds instruments to the accompaniment music based on the expanded first audio to acquire background music, and generates and plays a second video based on the expanded first audio, background music, and first image set. If the user does not like the default music style, they can switch the accompaniment music style by clicking a button to switch between selected music style controls from multiple music style controls. The audio-video processing application then acquires new accompaniment music according to the switched accompaniment music style, and re-expands the first audio based on the time signatures in the new accompaniment music to play the second video.

[0039] As can be seen, according to this embodiment, the user can independently select a desired music style from various music styles, and by playing the second video according to the selected music style, the background music will have the selected music style, thus satisfying the user's need to independently select the music style of the background music.

[0040] In one embodiment, the background music includes a first audio track and a second audio track, the first audio track includes an instrument audio track, and the second audio track includes audio tracks from the background music other than the instrument audio track.

[0041] In the expanded first voice, the timestamp of the expanded first syllable is the same as the timestamp of the instrument sound in the first voice track, and in the expanded first voice, the timestamp of the expanded first syllable is the same as the timestamp of the beat in the second voice track.

[0042] In this embodiment, the background music includes a first audio track and a second audio track, the first audio track being an instrument audio track corresponding to the instrument sounds added to the accompanying music, and the second audio track being an accompanying audio track corresponding to the accompanying music. Since the background music consists of the first audio track and the second audio track, the second audio track may also be referred to as an audio track other than the instrument audio track in the background music.

[0043] For the expanded first syllable, the first audio track, and the second audio track, since the first syllable is expanded according to the timestamp of the beat of the second audio track, the timestamp of the expanded first syllable in the expanded first audio is the same as the timestamp of the beat in the second audio track. When adding the first audio track to the second audio track, it is necessary to add an instrument sound at the timestamp of the expanded first syllable, for example, it is necessary to add a drum sound at the playback start timestamp of the expanded first syllable. Therefore, the timestamp of the expanded first syllable in the expanded first audio is the same as the timestamp of the instrument sound in the first audio track, and the timestamp of the instrument sound may be the start timestamp of the instrument sound.

[0044] In one specific embodiment, after extending the playback start timestamp of each first syllable in the first audio in accordance with the timestamp of the beat in the second audio track, a representative pitch is determined for each of the extended first syllables in the extended first audio. The representative pitch of each extended first syllable may be the pitch that appears most frequently among the pitches of the extended first syllable. An instrument sound with the same pitch as each representative pitch of the extended first syllable is generated. For example, a drum sound with the same pitch as each representative pitch of the extended first syllable is generated. By adding the instrument sound with the same pitch at the playback start timestamp of each extended first syllable, the extended first syllable in the extended first audio is played in sync with the instrument sound, and the added instrument sound forms the first audio track.

[0045] As can be seen from this, according to this embodiment, the background music includes a first audio track and a second audio track, the first audio track includes an instrument sound track, the second audio track includes audio tracks other than the instrument sound track in the background music, the timestamp of the expanded first syllable in the expanded first audio is the same as the timestamp of the instrument sound in the first audio track, the timestamp of the expanded first syllable in the expanded first audio is the same as the timestamp of the beat in the second audio track, and by adding an instrument sound track and setting the timestamp of the instrument sound, the expanded first syllable in the expanded first audio is played in sync with the instrument sound in the first audio track, the human voice in the expanded first audio sounds more rhythmic, and the musicality of the background music can be enhanced.

[0046] In one specific embodiment, the above method flow may also select accompanying music corresponding to the first audio track from a database based on the music style and the time signature information of the first audio track. The music style includes the default music style of the audio / video processing application and music styles selected by the user, such as R&B and RAP. The time signature information of the first audio track includes the BPM (Beats Per Minute) of the first audio track, and the accompanying music is used to form the second audio track.

[0047] In this embodiment, first, the playback start timestamp of each first syllable is detected. For example, a DSP (Digital Signal Processing) algorithm is used to detect the playback start timestamp of each first syllable, and the difference between the playback start timestamps of two adjacent first syllables is taken to determine the duration of each first syllable. Next, the beat information of the first sound is determined based on the duration of each first syllable. The median duration is selected from the durations of each first syllable, and the beat information of the first sound is calculated based on this median. For example, by dividing this median by 60, the BPM of the first sound can be obtained as the beat information of the first sound.

[0048] The database stores multiple accompaniment tracks, each corresponding to a musical style and time signature information. For example, the database stores 16 accompaniment tracks in four different musical styles, and each musical style has four accompaniment tracks with different BPMs. Based on the musical style and the time signature information of the first voice, the system selects from the database a piece of music whose musical style is consistent with the aforementioned musical style and whose time signature information matches the time signature information of the first voice, as the accompaniment track corresponding to the first voice.

[0049] In this embodiment, first, each accompanying music track whose musical style is consistent with the above musical style is selected from the database. Each selected accompanying music track corresponds to one BPM range, and each selected accompanying music track corresponds to a different BPM range. It is determined whether the BPM of the first voice falls within the BPM range corresponding to any of the selected accompanying music tracks. If it does not, the BPM of the first voice is processed, for example, by multiplying it by 2 or dividing it by 2. It is then determined again whether the BPM of the first voice falls within the BPM range corresponding to any of the selected accompanying music tracks. The BPM of the first voice is processed until it falls within the BPM range corresponding to a selected accompanying music track. In this case, the accompanying music that falls within the BPM range is designated as the accompanying music corresponding to the first voice.

[0050] In the database, each piece of accompaniment music corresponds to a BPM range, and this BPM range may be a fine-tuned version of the BPM of the accompaniment music. For example, if the BPM of a certain piece of accompaniment music is 70, the corresponding BPM range may be [70-2.5, 70+2.5]. By fine-tuning the BPM of the accompaniment music to a certain range, the efficiency of selecting the accompaniment music corresponding to the first voice from the database can be improved.

[0051] As can be seen, according to this embodiment, the time signature information of the first speech can be efficiently and quickly determined based on the duration of each first syllable in the first speech, and appropriate accompaniment music from the database can be accurately matched to the first speech based on the musical style and the time signature information of the first speech.

[0052] In one embodiment, the timestamp of the first tonic syllable in each audio segment of the expanded first audio is the same as the timestamp of the first beat of the measure in the second audio track, or the same as the timestamp of the middle beat of the measure in the second audio track.

[0053] The audio segments are obtained by dividing the first audio based on the silent segments in the first audio, and the time signatures include the first time signature of each measure in the second audio track and the time signatures in the middle of each measure in the second audio track.

[0054] In this embodiment, the first audio has a silent segment, and the first audio can be divided based on the silent segment to obtain multiple audio segments, with two adjacent audio segments separated by one silent segment. A silent segment refers to a segment that the DSP algorithm determines does not contain any audio information, and an audio segment refers to a segment that the DSP algorithm determines contains audio.

[0055] Each audio segment of the first audio contains at least one tonic syllable, the tonic syllable being the first syllable whose audio energy is greater than the audio energy of the two adjacent first syllables before and after it. In this embodiment, the first tonic syllable is identified in each audio segment, and the first tonic syllable is the tonic syllable with the oldest timestamp. The time signature in the second audio track described above includes the first time signature of each measure in the second audio track and the time signatures in the middle of each measure in the second audio track. For example, if one measure in the second audio track contains four time signatures, then the time signature in the second audio track described above includes the first time signature of each measure and the time signatures corresponding to the middle positions of each measure.

[0056] In this embodiment, for the first tonic syllable in each audio segment of the first audio, the playback start timestamp of the first tonic syllable is extended based on the playback start timestamp of the first beat of each measure in the second audio track and the playback start timestamp of the middle beat of each measure, so that the extended playback start timestamp of the first tonic syllable is the same as the playback start timestamp of the first beat of the measure in the second audio track, or the same as the playback timestamp of the middle beat of the measure in the second audio track.

[0057] In a specific example, first, for the first tonic syllable in the first audio segment of the first audio, the playback start timestamp of that syllable is extended to match the playback start timestamp of the first beat of the first measure in the second audio track. Next, for the first tonic syllable in the second audio segment of the first audio, if the timestamp interval between it and each first syllable in the same audio segment is not changed, the playback start timestamp of the first tonic syllable in the second audio segment is extended to match the playback start timestamp of the first beat of a particular measure in the second audio track, or to match the playback start timestamp of a beat in the middle of a particular measure. In this way, the playback start timestamps of the first tonic syllables in each subsequent audio segment are extended.

[0058] Figure 5 is a schematic diagram of the first tonic syllable in each of the audio segments that extend the first audio according to one embodiment of the present disclosure. As shown in Figure 5, $ indicates a tonic syllable, and * indicates a non-tonic syllable. For the first tonic syllable in each of the audio segments of the first audio, if the timestamp interval between each first syllable in the same audio segment is not changed, the playback start timestamp of the first tonic syllable in each audio segment is extended to match the playback start timestamp of the first beat of a particular measure in the second audio track, or to match the playback start timestamp of a beat in the middle of a particular measure. In Figure 5, if the timestamp interval between each first syllable in the same audio segment is not changed, the length of the first audio segment in the figure is long, so the first tonic syllable of the second audio segment in the figure is shifted to the beat in the middle of the fourth measure in the figure. Considering that the first beat of the fifth measure falls after the middle beat of the fourth measure, transitioning to the position of the first beat of the fifth measure would delay the playback of the first tonic syllable of the second audio segment in the diagram. Therefore, transitioning to the position of the first beat of the fifth measure is not chosen here.

[0059] As can be seen, according to this embodiment, by stretching the timestamp of the first tonic syllable in each audio segment of the first audio track to match the timestamp of the first beat of the measure in the second audio track, or to match the timestamp of the middle beat of the measure in the second audio track, the timestamp of the stretched first tonic syllable in each audio segment can be aligned with the rhythm of the second audio track, thereby improving the user's viewing experience of the second video.

[0060] In one embodiment, the above method is To determine the audio energy of each first syllable in the first speech, For each first syllable in the first speech, it is determined whether the first syllable is the tonic syllable based on the audio energy of the first syllable and the audio energy of the adjacent first syllables. This further includes identifying the first tonic syllable in each phonetic segment based on whether or not each first syllable is a tonic syllable.

[0061] In this embodiment, for each first syllable in each first speech, since the first syllable has a certain duration of pronunciation, a sample value of the magnitude of the audio energy of each first syllable within the duration of pronunciation is obtained, and the audio energy of the first syllable is obtained by applying the root mean square to the sample value of the magnitude of each audio energy. For each first syllable in each first speech, based on the audio energy of the first syllable and the audio energy of adjacent first syllables in the first speech (adjacent first syllables may be located in different speech segments), the first syllable whose audio energy is greater than the audio energy of the two adjacent first syllables before and after it is identified as the tonic syllable. After identifying each tonic syllable in the first speech, the first tonic syllable in each speech segment can be identified, and the first tonic syllable is the syllable with the oldest timestamp in each speech segment.

[0062] Figure 6 is a schematic diagram of the first audio energy according to one embodiment of the present disclosure. As shown in Figure 6, in the first speech, each first syllable has audio energy. The horizontal axis of Figure 6 represents the syllable number, and the vertical axis represents the magnitude of the audio energy. When the audio energy of a first syllable is greater than the audio energies of the two adjacent first syllables, the audio energy of that first syllable indicates a local peak state, and therefore that first syllable is the tonic syllable.

[0063] As can be seen, according to this embodiment, by calculating the audio energy of the first syllable, each tonic syllable in the first speech can be quickly identified, and the first tonic syllable in each speech segment can be identified.

[0064] In one embodiment, the above method is Based on the timestamp of the first extended tonic syllable in the first audio segment and the timestamp of the first extended tonic syllable in the last audio segment, the start and end timestamps of the second audio track are determined. This further includes looping the music in the second audio track based on the start and end timestamps of the second audio track.

[0065] In this embodiment, if it is guaranteed that the timestamp interval between the first syllable in the first voice segment and the first syllable in the first voice segment will not change after the timestamp of the first syllable in the first voice segment has finished being stretched, for example, the playback start timestamp, then the timestamp of the first syllable in the first voice segment is stretched, that is, the timestamp of the first syllable in the first voice segment is shifted based on the timestamp of the stretched first syllable in the first voice segment.

[0066] After the timestamp of the first tonic syllable in each speech segment has been stretched, if it is guaranteed that the timestamp interval between the first tonic syllable in the last speech segment and the last syllable in the last speech segment will not change based on the timestamp of the stretched first tonic syllable in the last speech segment of the first speech, then stretch the timestamp of the last syllable in the last speech segment, that is, shift the timestamp of the last syllable in the last speech segment based on the timestamp of the stretched first tonic syllable in the last speech segment.

[0067] In this embodiment, the accompaniment music in the other second audio track is played from the first tonic syllable in the first audio segment to the last syllable in the last audio segment by setting the start timestamp of the second audio track to be equal to the timestamp of the first expanded tonic syllable in the first audio segment, and the end timestamp of the second audio track to be equal to the timestamp of the last expanded syllable in the last audio segment. Since the accompaniment music stored in the database is loopable music, the music in the second audio track is looped based on the start and end timestamps of the second audio track, and the music in the second audio track is accompaniment music. For example, if the length of the accompaniment music retrieved from the database is 50 seconds, and the time difference between the start and end timestamps of the second audio track is 150 seconds, the accompaniment music is looped 3 times.

[0068] As can be seen, according to this embodiment, the start and end timestamps of the second audio track are determined, and based on the start and end timestamps of the second audio track, the music in the second audio track is looped, that is, the accompanying music is looped, thereby determining the playback time of the accompanying music based on the duration of the expanded first audio. This has the effect of ensuring that the meaning of the user's words in the expanded first audio is not lost, and if the user's complete meaning is preserved, a second video is generated.

[0069] In one embodiment, first, based on the musical style and time signature information of the first voice, accompaniment music is selected from a database. Then, the timestamp of the first tonic syllable in each voice segment of the first voice is stretched so that it matches the timestamp of the first beat of the measure in the accompaniment music, or the timestamp of the middle beat of the measure in the accompaniment music. If the duration of the accompaniment music is too short during the stretching process, the accompaniment music is looped until the timestamp of the first tonic syllable in the last voice segment has been stretched to completion.

[0070] Then, based on the timestamp of the first extended tonic syllable in the first audio segment and the timestamp of the first extended tonic syllable in the last audio segment, the start and end timestamps of the accompaniment music are determined, and based on the start and end timestamps of the accompaniment music, the accompaniment music is looped to form the second audio track.

[0071] In one embodiment, the above method is For the second syllable in the first audio track, based on the audio energy of the two beats closest to the timestamp of the second syllable before it was stretched in the second audio track, and the timestamps of those two beats, the first beat to match for the second syllable is selected from those two beats. This further includes extending the timestamp of the second syllable to be the same as the timestamp of the first beat, Here, the second syllable is a syllable from each first syllable of the first speech that is not the first tonic syllable in each speech segment, the first syllable in the first speech segment, or the last syllable in the last speech segment. The speech segments are obtained by dividing the first speech based on the silent segments in the first speech.

[0072] In this embodiment, each first syllable of the first speech includes a second syllable, and the second syllable is a syllable from each first syllable of the first speech that is not the first dative syllable in each speech segment, the first syllable in the first speech segment, or the last syllable in the last speech segment. Because the previous processing extended the timestamps of the first dative syllable in each speech segment, the first syllable in the first speech segment, and the last syllable in the last speech segment, it is now necessary to extend the timestamp of the second syllable.

[0073] In this embodiment, the second audio track has multiple time signatures, and a time signature may be the first time signature of each measure in the second audio track, the last time signature of each measure, or a time signature at a predetermined position such as the position of each time signature, the position of each half-beat, or the position of each quarter-beat. The timestamp of a time signature may be the playback start timestamp of that time signature.

[0074] In this embodiment, each beat of the second audio track has a corresponding audio energy. For example, for each beat of the second audio track, the audio energy of the second audio track in that beat can be determined based on the sampling value of the magnitude of the audio energy of the second audio track in that beat. If there are multiple sampling values ​​of the magnitude of the audio energy of the second audio track in that beat, the audio energy of the second audio track in that beat can be obtained by applying the root mean square to the multiple sampling values. As an example, the second audio track is divided into multiple quarter beats. The first beat is the start of the first measure in the second audio track, the second beat is the quarter beat position of the first measure in the second audio track, and the third beat is the half beat position of the first measure in the second audio track. For the first beat, the sampling value of each sampling point in the time interval from the start of the first measure in the second audio track to the quarter beat position of the first measure is obtained, and the magnitude of the audio energy of the second audio track in the first beat is determined by applying the root mean square to each sampling value. For the second beat, the sampling value of each sampling point in the time interval from the quarter beat position of the first measure in the second audio track to the half beat position of the first measure is obtained, and the magnitude of the audio energy of the second audio track in the second beat is determined by applying the root mean square to each sampling value.

[0075] Of course, an audio energy diagram can also be used to show the magnitude of the audio energy of the second audio track. For example, the horizontal axis could represent the playback timestamp of each beat, and the vertical axis could represent the magnitude of the audio energy. The diagram would then contain corresponding bar patterns for each beat, with the color of the bar pattern indicating the magnitude of the audio energy for each beat.

[0076] In this embodiment, for each second syllable, the two beats closest to the timestamp before the second syllable is stretched are determined from among the beats of the second audio track, that is, in chronological order, for example in the chronological order of playback, the beat located before the second syllable before stretching and the beat located after the second syllable before stretching are determined, and based on the audio energy of these beats and the timestamps of these beats, for example, the playback start timestamp, the first beat to match for the second syllable is selected from among these beats, and the timestamp of the second syllable is stretched to be the same as the timestamp of the first beat.

[0077] As can be seen, according to this embodiment, for the second syllable in the first audio, based on the audio energy of the two beats in the second audio track whose timestamp is closest to the timestamp of the second syllable before it is stretched, and the timestamps of those two beats, the first beat to match the second syllable is selected from the two beats, and the timestamp of the second syllable is stretched to match the timestamp of the first beat, thereby producing the effect of stretching the timestamp of the second syllable in accordance with the rhythm of the second audio track.

[0078] In one embodiment, based on the audio energy of the two time signatures in the second audio track whose timestamp is closest to the timestamp before the second syllable is stretched, and the timestamps of those two time signatures, the first time signature to be matched for the second syllable is selected from among the two time signatures. The process involves selecting one of the two time signatures to align with the second syllable, based on the audio energy of the two time signatures, a first time length to be stretched so that the timestamp before the second syllable is stretched to the timestamp of each of the two time signatures, and the amount of change in the second time length, wherein the second time length includes the time between the timestamp before the second syllable is stretched and the first timestamp.

[0079] First, obtain the audio energy of the two time signatures mentioned above, obtain the timestamp of the second syllable before it is stretched, and obtain the timestamp of these time signatures. Determine the first time length that the timestamp of the second syllable before it is stretched should be stretched to the respective timestamp of each of these time signatures. Each time signature corresponds to the first time length, and the first time length is equal to the difference between the timestamp of the second syllable before it is stretched and the timestamp of each time signature.

[0080] Next, we obtain the change in the second time length, which includes the time between the timestamp of the second syllable before it was stretched and the first timestamp. Since the timestamp of the second syllable was stretched, the second time length may be changed, and the change in the second time length is equal to the difference between the second time length of the second syllable before it was stretched and the stretched second time length of the second syllable. The second time length of the second syllable before it was stretched is the time between the timestamp of the second syllable before it was stretched and the first timestamp. The stretched second time length of the second syllable is the time between the timestamp of the stretched second syllable and the first timestamp after the timestamp of the second syllable has been stretched to each of the two time signatures above, i.e., the time between the timestamp of each of the two time signatures above and the first timestamp. Each time signature corresponds to the changed second time length and also to the change in the second time length.

[0081] Finally, based on the audio energy of the two time signatures mentioned above, the change in the first time length corresponding to each time signature, and the change in the second time length corresponding to each time signature, select the first time signature to match the second syllable of the two time signatures mentioned above.

[0082] As can be seen, this embodiment achieves the technical effect of selecting the first time signature to match the second syllable of the two time signatures, and precisely determining the first time signature, based on the audio energy of the two time signatures, the first time length to be stretched so that the timestamp before the stretching of the second syllable is stretched to the timestamp of the two time signatures, and the amount of change in the second time length.

[0083] In one embodiment, the first beat is selected from the two beats described above in order of the magnitude of the audio energy such that the first time length is less than a length threshold and the amount of change in the second time length satisfies the change amount requirement, and the first timestamp includes the timestamp of the stretched first syllable located before the second syllable, and / or the timestamp of the stretched first syllable located after the second syllable.

[0084] In this embodiment, first, it is determined whether the beat with the higher audio energy among the two most recent beats is the first beat. The specific determination process involves determining whether the first time length corresponding to the beat with the higher audio energy is less than a length threshold, and whether the amount of change in the second time length corresponding to the beat with the higher audio energy satisfies the change amount requirement. The change amount requirement may be that the amount of change in the second time length is less than the change amount threshold, or that the ratio obtained by dividing the amount of change in the second time length by the second time length before the second syllable is stretched is less than the ratio threshold.

[0085] If the first time length corresponding to the beat with high audio energy is less than the length threshold, and the amount of change in the corresponding second time length satisfies the change requirement, then that beat is designated as the first beat. Conversely, it is determined whether the beat with the lowest audio energy among the two most recent beats is the first beat. If the beat with the lowest audio energy is also not the first beat, the second audio track can be split again to obtain finer beats, and from these finer beats, the first beat can be selected for the second syllable.

[0086] The process of selecting the first beat for the second syllable from the finer beats is the same as the previous process, and similarly, the first beat to match for the second syllable is selected from among the two finer beats based on the audio energy of the two finer beats whose timestamps in the second audio track are closest to the timestamp before the second syllable is stretched, and the timestamps of those two finer beats.

[0087] In this embodiment, the first timestamp includes one or all of the timestamps of the stretched first syllable located before the second syllable, and the timestamps of the stretched first syllable located after the second syllable. The timestamp of the stretched first syllable does not change during the stretching process, and after stretching the second syllable to any beat, the degree of change in the time length between the second syllable and the first timestamp can be evaluated to estimate whether the distance the second syllable is stretched too much, thereby avoiding overstretching of the second syllable. For the same reason, overstretching of the second syllable can also be avoided by evaluating the relationship between the first time length and the length threshold.

[0088] Figure 7 is a schematic diagram of the expansion of the timestamp of the second syllable according to one embodiment of the present disclosure, where the numbers of each second syllable are 1-7 in order, where syllable 1 is the first syllable of the speech segment and its timestamp is fixed, syllable 2 is the first tonic syllable of the speech segment and its timestamp is also fixed and is the same as the timestamp of the first beat of the first measure in the second audio track in the figure, syllable 7 is the first tonic syllable of the next speech segment and its timestamp is also fixed and is the same as the timestamp of the first beat of the third measure in the second audio track in the figure.

[0089] In Figure 7, the second syllables are 3, 4, 5, and 6, respectively. For syllable 4, its two closest corresponding time signatures are the third time signature of the first measure and the first time signature of the second measure in the figure. As a result of the above process, the first time signature of the second measure is determined to be the first time signature, and the timestamp of syllable 4 is stretched to be the same as the timestamp of the first time signature of the second measure. In Figure 7, the first timestamp includes the stretched timestamps of syllables 2 and 7. The second time change includes the change in time between syllable 4 and syllable 2, and the change in time between syllable 4 and syllable 7, before and after the timestamp of syllable 4 is stretched.

[0090] Furthermore, as explained above, the second audio track is played from the first tonic syllable in the first audio segment to the last syllable in the last audio segment. Therefore, if the first audio segment in the figure is the first audio segment of the first audio, then for the syllable between the first syllable and the first tonic syllable in the first audio segment in the figure, after the timestamps of the first syllable and the first tonic syllable in the first audio segment in the figure are fixed, the timestamp of the syllable between these two syllables may be extended because the syllable between these two syllables does not have a corresponding second audio track.

[0091] As can be seen, according to the embodiment, the first beat is selected from the two beats closest in order of audio energy magnitude such that the first time length is less than the length threshold and the change amount requirement for the second time length is met. Therefore, the first beat can ensure that the amount of expansion of the timestamp of the second syllable stays within an appropriate range, thus avoiding over-expansion of the second syllable.

[0092] As described above, the second audio track has multiple time signatures, and the time signatures may be the first time signature of each measure in the second audio track, the last time signature of each measure, or the time signatures at predetermined positions such as each time signature, each half-beat, or each quarter-beat. The time signature timestamp may be the playback start timestamp of the time signature.

[0093] In one embodiment, the time signature in the second audio track is obtained by dividing the second audio track according to a division rule, which consists of a plurality of sub-rules having a hierarchical relationship. If a sub-rule in the next higher hierarchy does not satisfy the selection criteria for the first time signature of the second syllable, the second audio track is divided according to a sub-rule in the next lower hierarchy, and the first time signature corresponding to the second syllable is re-selected.

[0094] Specifically, taking Figure 7 as an example, first, according to the sub-rule that each beat in the second audio track is one beat, the second audio track is divided into multiple beats, and the two closest beats corresponding to syllable 4 are the third beat of the first measure and the first beat of the second measure in the figure. If, after the above process, neither the third beat of the first measure nor the first beat of the second measure is the first beat, then according to the sub-rule of the next lower level, namely the sub-rule that each half beat in the second audio track is one beat, the second audio track is divided into multiple beats, and the first beat is re-selected for syllable 4. If syllable 4 corresponds to the 3.8 beat of the first measure of the second audio track before it is extended, then after dividing each half beat in the second audio track as one beat, the two closest beats corresponding to syllable 4 are the 3.5 beat of the first measure and the first beat of the second measure in the figure. In this way, by dividing the time signature in the second audio track into multiple layers multiple times, each second syllable can be aligned with the appropriate first time signature.

[0095] In one embodiment, the above method is Regarding the second syllable in the first speech, the timestamp of the second syllable that comes first in the extension order is extended according to the extension order of each second syllable, This further includes extending the timestamp of the second syllable that is extended later in the extension order, according to the amount and order of extension of the timestamp of the second syllable that is extended earlier in the extension order, Here, the second syllable is a syllable from each first syllable of the first speech that is not the first tonic syllable in each speech segment, the first syllable in the first speech segment, or the last syllable in the last speech segment. The speech segments are obtained by dividing the first speech based on the silent segments in the first speech.

[0096] In this embodiment, each first syllable of the first speech includes a second syllable, and the second syllable is a syllable from each first syllable of the first speech that is not the first dative syllable in each speech segment, the first syllable in the first speech segment, or the last syllable in the last speech segment. Because the timestamps of the first dative syllable in each speech segment, the first syllable in the first speech segment, and the last syllable in the last speech segment have been extended by the previous processing, it is now necessary to extend the timestamp of the second syllable.

[0097] In this embodiment, the order in which each second syllable is expanded is determined, and according to this expansion order, the timestamp of the second syllable that is earlier in the expansion order is expanded, and when expanding the second syllable that is later in the expansion order, the timestamp of the second syllable that is later in the expansion order is expanded according to the amount of expansion of the timestamp of the second syllable that is earlier in the expansion order and the above expansion order. Here, the amount of expansion of the timestamp of the second syllable is equal to the timestamp interval length between the timestamp of the second syllable before expansion and the timestamp after expansion. For example, based on the amount of expansion of the timestamp of the second syllable that is earlier in the expansion order, the timestamp of the second syllable that is later in the expansion order is changed, the changed timestamp is the timestamp of the second syllable that is later in the expansion order before expansion, and based on the timestamp of the second syllable that is later in the expansion order before expansion, the timestamp of the second syllable that is later in the expansion order is expanded.

[0098] Taking Figure 7 as an example, each second syllable contains syllables 3, 4, 5, and 6, and each syllable is stretched one by one according to the stretching order, for example, the stretching order being 4, 3, 5, and 6. When each second syllable, for example syllable 4, is stretched, the timestamp of the second syllable before syllable 4 (i.e., syllable 3) and the timestamps of the second syllables after syllable 4 (i.e., syllables 5 and 6) are updated as the timestamp of syllable 4 is stretched.

[0099] As can be seen from Figure 7, the timestamps of syllables 1, 2, and 7, which were expanded first, do not change as syllable 4 is modified. However, the timestamps of syllables 3, 5, and 6 change as the timestamp of syllable 4 is expanded. The rule for this change is that the amount of change in the timestamp of each syllable is the same as the amount of expansion of the timestamp of syllable 4. After the timestamp of syllable 4 has been expanded, the timestamp of syllable 3 after the modification is expanded and used as the timestamp of syllable 3 before it was expanded.

[0100] During the process of extending the timestamp of syllable 3, the timestamp of syllable 4 remains unchanged because the extension of syllable 4 is completed. During the process of extending the timestamp of syllable 6, the timestamps of syllables 4 and 3 remain unchanged because the extension of syllables 4 and 3 is completed. Furthermore, if there is a possibility that the timestamp of syllable 5 may change during the process of extending the timestamp of syllable 6, the timestamp of syllable 5 after the change will be extended to the timestamp of syllable 5 before the extension.

[0101] As can be seen, according to this embodiment, the timestamp of the second syllable that is extended earlier in the extension order is extended according to the extension order of each second syllable, and the timestamp of the second syllable that is extended later in the extension order is extended according to the amount of extension and extension order of the timestamp of the second syllable that is extended earlier, thereby extending the second syllable that is extended later and improving the accuracy of extending the timestamp of the second syllable.

[0102] In one embodiment, the above method is The method further includes determining the order of extension of each second syllable based on the audio energy of each second syllable in the first speech and whether or not each second syllable is a tonic syllable.

[0103] In this embodiment, for each second syllable in the first speech, the audio energy of each second syllable is calculated, and it is determined whether or not each second syllable is a tonic syllable. Based on the audio energy of each second syllable in the first speech and the determination of whether or not each second syllable is a tonic syllable, the extension order of each second syllable is determined. The audio energy of the second syllable is obtained by applying the root mean square to the sampled values ​​of the magnitude of the audio energy of the second syllable.

[0104] In one embodiment, for each of the second syllables in the first speech, it is determined that the tonic syllable is extended before the non-tonic syllables. Each tonic syllable and each non-tonic syllable of the second syllable constitute a syllable set, and within each syllable set, the order of extension of each syllable in the syllable set is determined based on the audio energy of each syllable in the syllable set. For example, each tonic syllable of all second syllables can be constituted as one syllable set, and each non-tonic syllable of all second syllables can be constituted as one syllable set. The extension order of any tonic syllable precedes the extension order of any non-tonic syllable. Within each syllable set, the extension order of each syllable in the syllable set is determined from front to back in order of decreasing audio energy of each syllable in the syllable set.

[0105] As can be seen, according to this embodiment, the order of extension of each second syllable can be determined based on the audio energy of each second syllable in the first speech and whether or not each second syllable is a tonic syllable, thereby producing the effect of extending the second syllables in an orderly manner.

[0106] In one embodiment, after the timestamp extension of each first syllable in the first audio is completed, the timing of the occurrence of each volume peak of the extended first syllable in the first audio can also be adjusted.

[0107] Figure 8 is a schematic diagram of the volume peak of a first syllable according to one embodiment of the present disclosure. In Figure 8, taking one first syllable as an example, the horizontal axis represents time and the vertical axis represents volume. As shown in Figure 8, each first syllable has a playback start timestamp and a volume peak occurrence timestamp. Typically, the playback start timestamp is located before the volume peak occurrence timestamp. In this embodiment, the volume peak occurrence timestamps of each volume peak of the stretched first syllable are adjusted forward, shortening the distance between the playback start timestamp and the volume peak occurrence timestamp. This allows for a rapid arrival of the volume peak after the appearance of the sound of the stretched first syllable, improving the user's listening experience.

[0108] In this embodiment, after the timestamps of each first syllable in the first audio have been expanded, the pitch level of the expanded first audio can also be adjusted before adding the first audio track to the second audio track. For example, by adjusting the pitch of the expanded first audio to match the pitch level of the second audio track, the expanded first syllables will sound musical and align with the second audio track.

[0109] In this embodiment, after the expansion of the timestamps of each first syllable in the first audio is complete, the acoustic effect of the first audio can also be optimized. For example, a better human voice effect can be achieved by compressing or equalizing the human voice in the first audio.

[0110] In summary, each embodiment of this disclosure provides a data processing method capable of converting audio into music having a specific musical style. In the process of processing the first audio, by simply adjusting the timestamp of the first syllable in the first audio, all semantics of the first audio can be preserved so as not to cause significant damage to the first audio, such as cutting out or repeating some sentences, and so as not to lose the user's semantic information. Since the timestamp of the expanded first syllable in the expanded first audio is the same as the timestamp of the beat in the background music, and the timestamp of each frame image in the first image set of the second video is the same as the timestamp of the corresponding expanded first syllable in the expanded first audio, the expanded first syllable in the expanded first audio of the second video is displayed and played in sync with each corresponding frame image, thereby improving the user's experience of watching the second video.

[0111] One embodiment of the present disclosure further provides a data processing device that can be placed inside a terminal device for carrying out the data processing method described above. Figure 9 is a schematic diagram showing the configuration of a data processing device according to one embodiment of the present disclosure, and as shown in Figure 9, the device is An acquisition unit 91 for acquiring a first video including a first image set and a first audio in response to a first operation performed by a user, wherein the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding first syllable in the first audio, The system includes a playback unit 92 for playing a second video, which includes the first image set, an expanded first audio, and background music, in response to a second operation performed by the user. In the expanded first audio, the timestamp of the expanded first syllable is the same as the timestamp of the beat in the background music, and in the second video, the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding expanded first syllable in the expanded first audio.

[0112] Preferably, the acquisition unit 91 is, in particular, To acquire the first video in response to a video upload operation performed by the user, or Used to acquire the first video containing the first audio recorded by the user in response to a recording operation performed by the user.

[0113] Preferably, the regeneration unit 92 is, in particular, In response to the user selecting one of a plurality of music style controls, the background music is used to play the second video and has the aforementioned music style.

[0114] Preferably, the background music includes a first audio track and a second audio track, the first audio track includes an instrument audio track, and the second audio track includes audio tracks other than the instrument audio track in the background music. The timestamp of the expanded first syllable in the expanded first audio is the same as the timestamp of the instrument sound in the first audio track, and the timestamp of the expanded first syllable in the expanded first audio is the same as the timestamp of the beat in the second audio track.

[0115] Preferably, the timestamp of the first tonic syllable in each audio segment of the extended first audio is the same as the timestamp of the first beat of the measure in the second audio track, or the same as the timestamp of the middle beat of the measure in the second audio track. The audio segment is obtained by dividing the first audio according to the silent segments in the first audio, and the time signature includes the first time signature of each measure in the second audio track and the time signatures in the middle of each measure in the second audio track.

[0116] Preferably, the device further, Determining the audio energy of each of the first syllables in the first speech, For each of the first syllables in the first sound, it is determined whether the first syllable is a tonic syllable based on the audio energy of the first syllable and the audio energy of the adjacent first syllables. Each of the aforementioned first syllables is provided with a dynastic identification unit for identifying the first dynastic syllable in each of the aforementioned speech segments, based on whether or not each of the aforementioned first syllables is a dynastic syllable.

[0117] Preferably, the device further, Based on the timestamp of the expanded first tonic syllable in the first audio segment and the timestamp of the expanded first tonic syllable in the last audio segment, the start timestamp and end timestamp of the second audio track are determined. The system includes a loop playback unit for looping the music in the second audio track based on the start timestamp and the end timestamp.

[0118] Preferably, the device further, For the second syllable in the first audio, a first time signature is selected from the two time signatures to match the second syllable, based on the audio energies of the two time signatures whose timestamps in the second audio track are closest to the timestamp of the second syllable before it is stretched, and the timestamps of the two time signatures. The system includes a first beat identification unit for extending the timestamp of the second syllable to be the same as the timestamp of the first beat, Here, the second syllable is a syllable from each of the first syllables of the first speech, other than the first tonic syllable in each speech segment, the first syllable in the first speech segment, and the last syllable in the last speech segment, and the speech segments are obtained by dividing the first speech according to the silent segments in the first speech.

[0119] Preferably, the first beat-specific unit is, in particular, Based on the audio energy of the two time signatures, the first time length to be stretched so that the timestamp before the stretching of the second syllable is stretched to the timestamp of each of the two time signatures, and the amount of change in the second time length, a first time signature to be fitted for the second syllable is selected from the two time signatures, the second time length including the time between the timestamp before the stretching of the second syllable and the first timestamp.

[0120] Preferably, the first beat is selected from the two beats in order of the magnitude of the audio energy, such that the first time length is less than a length threshold and satisfies the change amount requirement of the second time length, and the first timestamp includes the timestamp of the stretched first syllable located before the second syllable, and / or the timestamp of the stretched first syllable located after the second syllable.

[0121] Preferably, the apparatus further comprises, For the second syllable in the first sound, the timestamp of the second syllable that comes first in the extension order is extended according to the extension order of each syllable in the second syllable. The system includes an expansion unit for expanding the timestamp of the second syllable that is later in the expansion order, according to the amount of expansion of the timestamp of the second syllable that is earlier in the expansion order and the expansion order of the second syllable that is earlier in the expansion order, Here, the second syllable is a syllable from each of the first syllables of the first speech, other than the first tonic syllable in each speech segment, the first syllable in the first speech segment, and the last syllable in the last speech segment, and the speech segments are obtained by dividing the first speech according to the silent segments in the first speech.

[0122] Preferably, the apparatus further comprises, The system includes an order determination unit for determining the extension order of each of the second syllables based on the audio energy of each of the second syllables in the first sound and whether or not each of the second syllables is a tonic syllable.

[0123] The data processing apparatus of the embodiments of this disclosure can perform each process of the embodiments of the data processing method described above and achieve the same effects and functions, and redundant explanations will not be repeated here.

[0124] One embodiment of the present disclosure further provides an electronic device. Figure 10 is a schematic diagram showing the configuration of an electronic device according to one embodiment of the present disclosure, and as shown in Figure 10, the electronic device may vary relatively greatly depending on its configuration or performance, and may include one or more processors 1001 and memory 1002, the memory 1002 may store one or more applications or data. Here, the memory 1002 may be temporary memory or persistent memory. The applications stored in the memory 1002 may include one or more modules (not shown), each module may include a set of computer-executable instructions for the electronic device. Furthermore, the processor 1001 may be configured to communicate with the memory 1002, and the electronic device executes a set of computer-executable instructions in the memory 1002. The electronic device may include one or more power supplies 1003, one or more wired or wireless network interfaces 1004, one or more input or output interfaces 1005, one or more keyboards 1006, and the like.

[0125] In one specific embodiment, the electronic device comprises a processor and a memory configured to store computer-executable instructions, and when the computer-executable instructions are executed, In response to a first operation performed by a user, to obtain a first video including a first image set and a first audio, wherein the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding first syllable in the first audio, In response to a second operation performed by the user, the processor plays a second video including the first image set, an expanded first audio, and background music, wherein the timestamp of the expanded first syllable in the expanded first audio is the same as the timestamp of the beat in the background music, and the timestamp of each frame image in the first image set in the second video is the same as the timestamp of the corresponding expanded first syllable in the expanded first audio.

[0126] The electronic devices of the embodiments of this disclosure can perform each of the processes of the data processing method embodiments described above and achieve the same effects and functions, and therefore, redundant explanations will not be repeated here.

[0127] Another embodiment of the present disclosure further provides a computer-readable storage medium used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, In response to a first operation performed by a user, to obtain a first video including a first image set and a first audio, wherein the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding first syllable in the first audio, In response to a second operation performed by the user, the system plays a second video including the first image set, an expanded first audio, and background music, wherein the timestamp of the expanded first syllable in the expanded first audio is the same as the timestamp of the beat in the background music, and the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding expanded first syllable in the expanded first audio in the second video.

[0128] The storage medium of the embodiment of this disclosure can perform each of the processes of the data processing method embodiment described above and achieve the same effects and functions, and therefore, redundant explanations will not be repeated here.

[0129] In each embodiment of the present disclosure, the computer-readable storage medium may be read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0130] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (improvements to circuit structures such as diodes, transistors, and switches) or software improvements (improvements to methods and processes). However, with the evolution of technology, many of the method and process improvements to date have come to be considered direct improvements to hardware circuit structures. Designers often obtain the corresponding hardware circuit structure by programming the improved method and process into the hardware circuit. Therefore, it cannot be said that improvements to method and process cannot be implemented in physical hardware entity modules. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is an integrated circuit in which the logic function is determined by the user's programming of the device. Digital systems are "integrated" into PLDs through the designer's own programming, without chip manufacturers designing and manufacturing dedicated integrated circuit chips. Furthermore, nowadays, such programming is increasingly being implemented using software called a "logic compiler" rather than manually on IC chips, similar to the software compilers used for program development and writing. In order to compile the original code, it needs to be written in a specific programming language called Hardware Description Language (HDL).There isn't just one HDL; there are many types, including ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java® Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Furthermore, it will be apparent to those skilled in the art that hardware circuits implementing logical method processes can be easily obtained by simply programming the method processes into integrated circuits using one of the aforementioned hardware description languages.

[0131] The controller may be implemented in any suitable manner, for example, a microprocessor or processor and a computer-readable medium storing executable computer-readable program code for the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller may be implemented as part of the memory control logic. In addition to implementing the controller purely with computer-readable program code, it is also known to those skilled in the art that the same functions can be fully realized in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc., by logically programming method steps. Thus, such controllers can be considered hardware components, and the devices for realizing their various functions can also be considered structures within the hardware component. Furthermore, the devices for realizing various functions may be software modules for implementing the methods, or they may be configurations within hardware components.

[0132] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having specific functions. An exemplary implementation is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a mobile phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0133] For the sake of clarity, the above-described devices are described by dividing them into various functional units. Of course, when implementing embodiments of this disclosure, it is possible to implement the functions of various units in the same or multiple software and / or hardware.

[0134] Those skilled in the art will understand that one or more embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or a combination of software and hardware. Furthermore, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-readable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) containing computer-readable program code.

[0135] This disclosure will be described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of this disclosure. It should be understood that each process and / or box in the flowcharts and / or block diagrams, and any combination of processes and / or boxes in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor in a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, so that the instructions executed by the processor in the computer or other programmable data processing device may form equipment that generates a device for performing one or more steps in a flowchart and / or one or more blocks in a box diagram.

[0136] These computer program instructions may be stored in computer-readable memory capable of guiding a computer or other programmable data processing device to operate in a particular manner, thereby causing the instructions stored in computer-readable memory to generate a product including an instruction unit, which performs one or more steps in a flowchart and / or one or more blocks in a block diagram.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are executed on the computer or other programmable device, generating processing performed by the computer, so that instructions executed on the computer or other programmable device provide steps to realize one or more steps in a flowchart and / or one or more blocks in a block diagram.

[0138] It should be noted that the terms “includes,” “equipped with,” or other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or device containing a set of elements includes not only those elements but also other elements not explicitly listed, or elements specific to those processes, methods, products, or devices. Furthermore, unless further limitations are given, an element defined by the expression “includes one…” does not preclude the presence of another identical element in a process, method, product, or device containing that element.

[0139] One or more embodiments of this disclosure may be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, a program module includes routines, programs, objects, components, data structures, etc., that perform a particular task or realize a particular abstract data type. One or more embodiments of this disclosure may also be implemented in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may reside on local and remote computer storage media, including memory.

[0140] The various embodiments of this disclosure are described in a progression manner, and any identical or similar parts between each embodiment should refer to one another. Each embodiment will primarily describe the differences from the other embodiments. In particular, the system embodiments are essentially similar to the method embodiments and will therefore be described more simply; relevant points should be referred to the description of the method embodiments.

[0141] The above description is merely an example of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will be able to implement various modifications and variations of the present disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present disclosure shall be included in the claims of the present disclosure.

Claims

1. In response to a first operation performed by a user, a first video containing a first image set and a first audio is obtained, wherein the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding first syllable in the first audio. In response to a second operation performed by the user, the system plays a second video including the first image set, the expanded first audio, and background music, and includes: In the expanded first audio, the timestamp of the expanded first syllable is the same as the timestamp of the beat in the background music, and in the second video, the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding expanded first syllable in the expanded first audio. Data processing method.

2. To acquire a first video in response to a first operation performed by the user, To acquire the first video in response to a video upload operation performed by the user, or The process includes obtaining the first video containing the first audio recorded by the user in response to a recording operation performed by the user, The method according to claim 1.

3. Playing the second video in response to a second operation performed by the user is: The process includes playing the second video in response to the user selecting one of several music style controls, The aforementioned background music has the aforementioned musical style, The method according to claim 1.

4. The background music includes a first audio track and a second audio track, the first audio track includes an instrument audio track, and the second audio track includes audio tracks other than the instrument audio track in the background music. In the expanded first audio, the timestamp of the expanded first syllable is the same as the timestamp of the instrument sound in the first audio track, and in the expanded first audio, the timestamp of the expanded first syllable is the same as the timestamp of the beat in the second audio track. The method according to any one of claims 1 to 3.

5. The timestamp of the first tonic syllable in each audio segment of the expanded first audio is the same as the timestamp of the first beat of the measure in the second audio track, or the same as the timestamp of the middle beat of the measure in the second audio track. The audio segment is obtained by dividing the first audio according to the silent segments in the first audio, and the time signature includes the first time signature of each measure in the second audio track and the time signatures in the middle of each measure in the second audio track. The method according to claim 4.

6. Determining the audio energy of each first syllable in the first sound, For each of the first syllables in the first sound, it is determined whether or not the first syllable is a tonic syllable based on the audio energy of the first syllable and the audio energy of the first syllable adjacent to the first syllable, The further includes identifying the first syllable in each of the phonetic segments based on whether each of the first syllables is a syllable, The method according to claim 5.

7. The start and end timestamps of the second audio track are determined based on the timestamp of the expanded first tonic syllable in the first audio segment and the timestamp of the expanded first tonic syllable in the last audio segment. The further includes looping the music in the second audio track based on the start timestamp and the end timestamp, The method according to claim 6.

8. For the second syllable in the first audio, the audio energy of the two beats in the second audio track whose timestamp is closest to the timestamp before the second syllable is stretched, and the timestamp of the two beats, the first beat to be matched for the second syllable from the two beats, The method further includes extending the timestamp of the second syllable so that it is the same as the timestamp of the first beat, The second syllable is a syllable from each of the first syllables of the first speech, other than the first syllable in each speech segment, the first syllable in the first speech segment, and the last syllable in the last speech segment, and the speech segment is obtained by dividing the first speech according to the silent segments in the first speech. The method according to claim 4.

9. Based on the audio energies of the two time signatures in the second audio track whose timestamps are closest to the timestamp before the second syllable is stretched, and the timestamps of the two time signatures, selecting the first time signature from the two time signatures to match the second syllable is: This includes selecting the first time signature from the two time signatures to match the second syllable, based on the audio energy of the two time signatures, the first time length to be stretched so that the timestamp before the second syllable is stretched to the timestamp of each of the two time signatures, and the amount of change in the second time length, The second time length includes the time between the timestamp before the second syllable is stretched and the first timestamp. The method according to claim 8.

10. The first beat is a beat selected from the two beats in order of the magnitude of the audio energy such that the first time length is less than the length threshold and the amount of change in the second time length satisfies the change amount requirement, and the first timestamp includes the timestamp of the stretched first syllable located before the second syllable, and / or the timestamp of the stretched first syllable located after the second syllable. The method according to claim 9.

11. For the second syllable in the first sound, the timestamp of the second syllable that comes first in the extension order is extended according to the extension order of each second syllable, The method further includes extending the timestamp of the second syllable that is later in the extension order, according to the amount of extension of the timestamp of the second syllable that is earlier in the extension order and the extension order of the second syllable that is earlier in the extension order, The second syllable is a syllable from each of the first syllables of the first speech, other than the first syllable in each speech segment, the first syllable in the first speech segment, and the last syllable in the last speech segment, and the speech segment is obtained by dividing the first speech according to the silent segments in the first speech. The method according to claim 4.

12. The method further includes determining the order of extension of each second syllable based on the audio energy of each second syllable in the first sound and whether or not each second syllable is a tonic syllable. The method according to claim 11.

13. An acquisition unit for acquiring a first video including a first image set and a first audio in response to a first operation performed by a user, wherein the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding first syllable in the first audio, The system includes a playback unit for playing a second video, which includes the first image set, an expanded first audio, and background music, in response to a second operation performed by the user, In the expanded first audio, the timestamp of the expanded first syllable is the same as the timestamp of the beat in the background music, and in the second video, the timestamp of each frame image in the first image set is the same as the timestamp of the corresponding expanded first syllable in the expanded first audio. Data processing device.

14. Processor and A memory configured to store computer-executable instructions, When the computer-executable instruction is executed, the processor is made to perform a step of the method according to any one of claims 1 to 12. electronic equipment.

15. A computer-readable storage medium used to store computer-executable instructions, When the computer-executable instruction is executed by the processor, a step of the method according to any one of claims 1 to 12 is performed. A computer-readable storage medium.

16. A device that is tangibly stored on a non-temporary computer-readable medium and includes an executable instruction, wherein when the executable instruction is executed, the device is made to perform the method according to any one of claims 1 to 12. Computer program products.

Images