Audio / video processing method and device

Abstract

An audio / video processing method and device. The method comprises: acquiring an audio / video to be processed, the audio / video to be processed comprising: a first human voice audio; then, determining a target accompaniment audio from among at least one accompaniment audio; and finally, according to the first human voice audio and the target accompaniment audio, determining a second human voice audio, the rhythm of the second human voice audio being matched with the rhythm of the target accompaniment audio. The present application converts the rhythm of the first human voice audio on the basis of the rhythm of the accompaniment audio, so as to simulate the length of each song note to achieve the purpose of converting the human voice audio into a song audio.

Description

Audio and video processing method and device

[0001] This application claims priority to Chinese Patent Application No. 202311204599.6 filed on September 18, 2023, the entire text of which is incorporated herein by reference as a part of this application. Technical Field

[0002] The present disclosure relates to an audio and video processing method and device. Background Art

[0003] With the continuous development of computer technology, audio voice modification technology has also been continuously developing. For example, it can convert a user's voice into a voice with a specific sound color. However, with the increasing demand of users, people also hope to convert audio into audio with a musical color. For example, a user's speech can be converted into music, that is, the content of everyday expression can be "musicalized".

[0004] Therefore, how to convert vocal audio into singing audio becomes a technical problem that needs to be solved urgently.

[0005] Summary of the Invention

[0006] The present disclosure provides an audio and video processing method and device to solve the problem of audio and video processing.

[0007] In a first aspect, an embodiment of the present disclosure provides an audio and video processing method, comprising:

[0008] Acquire audio and video to be processed, where the audio and video to be processed include: first human voice audio;

[0009] determining a target accompaniment audio from at least one accompaniment audio;

[0010] A second vocal audio is determined based on the first vocal audio and the target accompaniment audio, wherein the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

[0011] In a second aspect, an embodiment of the present disclosure provides an audio and video processing device, the device comprising:

[0012] An acquisition module is used to acquire audio and video to be processed, wherein the audio and video to be processed includes: first human voice audio;

[0013] A first determining module is configured to determine a target accompaniment audio from at least one accompaniment audio;

[0014] The second determining module is used to determine a second vocal audio according to the first vocal audio and the target accompaniment audio, wherein the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

[0015] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0016] The memory stores computer-executable instructions;

[0017] The processor executes the computer-executable instructions stored in the memory to implement the audio and video processing method as described in the first aspect above.

[0018] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the audio and video processing method described in the first aspect above.

[0019] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the audio and video processing method described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] FIG1 is a schematic diagram of device interaction for implementing audio and video processing according to an embodiment of the present disclosure;

[0022] FIG2 is a flowchart of a method for processing audio and video according to an embodiment of the present disclosure;

[0023] FIG3 is a second flow chart of the method for processing audio and video according to an embodiment of the present disclosure;

[0024] FIG4 is a third flow chart of the audio and video processing method provided by an embodiment of the present disclosure;

[0025] FIG5 is a fourth flow chart of the method for processing audio and video according to an embodiment of the present disclosure;

[0026] FIG6 is a schematic diagram of reverse speed change according to an embodiment of the present disclosure;

[0027] FIG7 is a schematic diagram of the structure of an audio and video processing device provided by an embodiment of the present disclosure;

[0028] FIG8 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

[0029] The above drawings illustrate specific embodiments of the present disclosure, which will be described in more detail below. These drawings and textual descriptions are not intended to limit the scope of the present disclosure in any way, but rather to illustrate the concepts of the present disclosure to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0031] Before introducing the embodiments of the present disclosure, the terms and application background involved in the embodiments of the present disclosure are first explained:

[0032] With the continuous development of computer technology, audio voice modification technology has also been continuously developing. For example, it can convert a user's voice into a voice with a specific sound color. However, with the increasing demand of users, people also hope to convert audio into audio with a musical color. For example, a user's speech can be converted into music, that is, the content of everyday expression can be "musicalized".

[0033] For example, on some voice-related software, music-related audio often receives higher submissions and consumption within the website. However, for most ordinary users, the high barrier to entry for music production prevents them from easily transforming their daily expressions into music. Furthermore, some users prefer the highly popular voice-changing module, but this template currently lacks a "musicalization" feature, preventing it from further enhancing the user experience.

[0034] Therefore, how to provide a method for converting audio into audio with melody becomes a technical problem that needs to be solved urgently.

[0035] Furthermore, in response to the above technical problems, the inventor's technical conception is as follows: part of the difference between speaking and singing lies in the difference in pitch. If some pre-prepared accompaniment can be used, and the accompaniment can contain speed change information points and corresponding speed changes, and then the speech to be processed is divided into speech segments with corresponding speed change intervals, and then the corresponding segments are speed-changed, the audio corresponding to the speech can be converted into audio of a song with a different pitch.

[0036] The technical solution of the present disclosure is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0037] It is worth noting that the application fields of the audio and video processing method and device disclosed herein are not limited and may include: the field of audio and video processing technology. Among them, the execution subject of the present disclosure is an electronic device, specifically a mobile phone, tablet, computer, server and other devices.

[0038] Before introducing the embodiment of the method disclosed herein, the implementation architecture process involved in the method disclosed herein is first described, as shown in Figure 1 (Figure 1 is a schematic diagram of the interaction between audio and video processing implementation devices provided in the embodiment of the present invention). The schematic diagram includes: a client 11, a video cloud 12, a first server 13 and a second server 14, and an audio software development kit (Audio Software Development Kit, Audio SDK) 15.

[0039] In one possible implementation, the client 11 obtains user input or audio (i.e., human voice audio) obtained from some location, and then requests permission to access the video cloud 12 (slice upload and download) (e.g., requesting an AuthKey) from the first server 13. After obtaining authorization from the first server 13, the client 11 uploads the audio to the video cloud 12 and simultaneously sends the audio identifier (e.g., vid) to the real-time speech development toolkit (English: Audio SDK) 15. The Audio SDK 15 sends the audio identifier to the second server 14.

[0040] The second server 14 downloads the corresponding audio from the video cloud 12 according to the audio identifier, and performs relevant processing of the audio into a song (this part is given in the following embodiment and is the core of the audio and video processing method of the present disclosure), obtains the processed audio (i.e., the song-made audio), and returns the identifier of the processed audio to the Audio SDK 15. The client 11 downloads the identifier of the processed audio from the second server 14 through the Audio SDK 15 according to the identifier of the processed audio, and then applies the speech-made song.

[0041] In this implementation, the client 11 extracts and transcodes the audio through the Jianying middle layer, and finally the speech is converted into music in the Jianying middle layer.

[0042] Among them, the first server 13 can be a Jianying server; the second server can be a server that realizes the function of converting audio (for example, speech) into music, such as a SAMI server, that is, it can be the electronic device that executes the embodiment of the present disclosure.

[0043] It should be understood that the embodiment shown in FIG1 is only an example of implementation, and details of undisclosed contents are given in the following embodiments.

[0044] FIG2 is a flow chart of a method for processing audio and video according to an embodiment of the present disclosure. As shown in FIG2 , the method may include the following steps:

[0045] Step 21: Obtain the audio and video to be processed;

[0046] The audio and video to be processed include: first human voice audio.

[0047] In this step, the audio that needs to be processed and converted into music (hereinafter referred to as audio-to-song sts) is first obtained, that is, the first vocal audio.

[0048] Optionally, the first human voice audio can be input by the user, for example, clicking the recording button on the above-mentioned client, and a passage of speech spoken by the user is recorded into the client's audio from other files; it can also be the audio obtained by performing audio extraction processing on the audio and video, for example, extracting the segment audio from a section of audio and video to obtain the first human voice audio.

[0049] In one possible implementation, the client sends a request to the first server corresponding to the client for permission to upload the first human voice audio to the cloud (for example, a video cloud), and then the first server sends the permission to the client. The client uploads the first human voice audio to the cloud and obtains the identifier of the first human voice audio in the cloud, for example, vid, and then sends the speech-to-song instruction and vid to the electronic device through the AudioSDK interface through the client. After the electronic device obtains the instruction, it uses vid to download the first human voice audio from the cloud, that is, obtains the first human voice audio.

[0050] The length of the first vocal audio can also be limited (this can be a configuration item), for example, to 1 minute. During subsequent song creation, if the audio is extended to more than 1 minute, the song creation process is canceled. If it is extended but within 1 minute, the server is requested to retrieve the audio again. If it is within 1 minute, the audio is trimmed. If it is shortened, it is not re-uploaded and is simply trimmed.

[0051] In addition, in other implementations, in addition to integrating the functions of the first server into the electronic device, the storage function of the cloud can also be integrated into the electronic device.

[0052] Optionally, after step 21, voice conversion (VC) processing may be performed on the first human voice audio to obtain processed first human voice audio.

[0053] Under this implementation, before the speech-to-music function is realized, sound conversion processing can also be performed on the first human voice audio, for example, the voice input by the user is converted into the voice of other targets (for example, cartoon characters), and then, the audio of the voice of other targets is converted into music, that is, the audio of the music of other targets can be obtained.

[0054] Among them, following the above implementation, you can first upload the first human voice audio through AudioSDK, obtain the first human voice audio after VC voice change, and then perform the following operations on the first human voice audio processed by VC through AudioSDK.

[0055] In a possible implementation, the audio effects may be VC voice changing, audio to music, general timbre, general sound effect, music style, etc. Functionally, the execution order may be arbitrarily specified.

[0056] Step 22: Determine a target accompaniment audio from at least one accompaniment audio.

[0057] In this step, a plurality of accompaniment audios are pre-set in the electronic device, and one accompaniment audio can be selected from at least one accompaniment audio as the target accompaniment audio for subsequent audio composition.

[0058] Optionally, a selection method may be: the melody line in each accompaniment audio is different, the melody line is extracted from the first vocal audio, and the accompaniment audio with the closest melody line is determined among the accompaniment audios, that is, the target accompaniment audio.

[0059] This implementation ensures that the pitch contour of the original first voice audio is followed as much as possible after processing the first voice audio, so that the subsequently generated audio sounds natural and avoids excessive modification that may cause loss of sound quality.

[0060] Optionally, another selection method may be: different accompaniment audios may correspond to different speaking speeds, and the target accompaniment audio may be selected based on the speaking speed of the first vocal audio, that is, the accompaniment audio with the closest speaking speed may be selected as the target accompaniment audio.

[0061] For example, four accompaniment audios are preset with speaking speeds of 1.2, 1.3, 1.4, and 1.5 respectively, and the speaking speed of the first vocal audio is 1.51, then the accompaniment audio corresponding to 1.5 can be selected as the target accompaniment audio.

[0062] Step 23: Determine the second vocal audio according to the first vocal audio and the target accompaniment audio.

[0063] The rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

[0064] In this step, the first vocal audio is processed using the rhythm information of the target accompaniment audio to obtain the second vocal audio that matches the rhythm information.

[0065] Furthermore, in some implementations, during step 23 , the melody of the first vocal audio may be adjusted according to the melody information in the target accompaniment audio.

[0066] Under this implementation, the melody information in the target accompaniment audio can be used to fine-tune the melody of the first vocal audio to better fit the melody of the set accompaniment audio and increase the melodic sense of the subsequently generated audio.

[0067] The detailed processing process under this implementation is shown in the following embodiment and will not be repeated here.

[0068] Optionally, the target accompaniment audio includes background audio, and the following steps may be performed:

[0069] Generate new second voice audio based on the second voice audio and the background audio.

[0070] In this implementation, background music can be added to the a cappella vocals to generate a more experiential song.

[0071] Furthermore, based on the above embodiment, the audio and video to be processed also includes: a first video.

[0072] Optionally, in a video editing scenario, the first video and the first human voice audio are aligned on a timeline corresponding to the video editing, that is, the starting point and the ending point of the first video and the first human voice audio are consistent.

[0073] Furthermore, when the audio and video to be processed include the first human voice audio and the first video, the target audio and video needs to be generated in the end. There are the following three processing situations:

[0074] The first type is that when the duration of the second voice audio is different from that of the first video, the second voice audio is subjected to inverse speed processing to obtain a third voice audio of the same length as the first video, and then the target audio and video are generated based on the third voice audio and the first video.

[0075] In this implementation, due to the rhythm adjustment of the first vocal audio, the duration of the first vocal audio (for example, 60s) may be different from the duration of the generated second vocal audio (for example, 57s). Here, it is necessary to make the duration of the second vocal audio consistent with the duration of the first vocal audio, both 60s.

[0076] For example, the uniform speed change rate is 60 / 57 times, and the second voice audio of 60 / 57 times is inversely speeded down to 60s to obtain the third voice audio, which is then combined with the original first video to generate the target audio and video.

[0077] In one possible implementation, if you want to perform inverse speed processing on the first target audio to make its length consistent with the audio and video to be processed, you can use the following solution:

[0078] Use step-type anchor points to approximate uniform speed change. For example, the time range [0, 0.5] changes to 2 times the speed, and [0.5, 1] ​​changes to 0.5 times the speed. The anchor points of the curve speed change can be set to [0, 2], [0.5, 2], [0.501, 0.5], [1, 0.5] to simulate and reduce the error.

[0079] The second type is that when the duration of the second vocal audio is different from that of the first video, the first video is speed-changed using the rhythm information corresponding to the target accompaniment audio to obtain a second video with the same duration as the second vocal audio, and the target audio and video is generated based on the second vocal audio and the second video.

[0080] In this implementation, since the rhythm adjustment is performed on the first vocal audio, the duration of the first vocal audio (for example, 60s) may be different from the duration of the generated second vocal audio (for example, 57s). Here, the rhythm information corresponding to the target accompaniment audio can be used to change the speed of the first video so that the duration of the second video after the speed change is the same as the duration of the second vocal audio, and then the target audio and video are generated based on the second vocal audio and the second video.

[0081] The third type is when the duration of the second human voice audio is the same as the duration of the first video, and the target audio and video are generated based on the second human voice audio and the first video.

[0082] Under this implementation, the first human voice audio in the audio and video to be processed can be replaced with the second human voice audio.

[0083] It should be understood that in the above-mentioned methods, the implementation principle of the speed change processing for the rhythm is the same as or similar to the following embodiments, and reference can be made to the following embodiments.

[0084] Afterwards, the generated target audio and video is sent to the user equipment for use by the user equipment.

[0085] The disclosed embodiments provide a method and device for processing audio and video. The method obtains the audio and video to be processed, which includes a first vocal audio, then identifies a target accompaniment audio from at least one accompaniment audio, and finally determines a second vocal audio based on the first vocal audio and the target accompaniment audio, where the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio. In this technical solution, the rhythm of the first vocal audio is transformed using the rhythm of the accompaniment audio, thereby simulating the length of each note during singing, thereby achieving the purpose of converting the vocal audio into singing audio, thereby improving the user experience.

[0086] In the above embodiment, the target accompaniment audio includes rhythm information, which includes a plurality of speed change information points and the speed change corresponding to each speed change information point. FIG3 is a second flow chart of the audio and video processing method provided by the embodiment of the present disclosure. As shown in FIG3 , the above step 23 may include the following steps:

[0087] Step 31: Segment the first human voice audio into multiple audio segments according to the multiple speed change information points;

[0088] In this step, a plurality of speed change information points are parsed from the rhythm information, and the first vocal audio is segmented according to the speed change information points to obtain each audio segment that needs to be speed-changed.

[0089] In one possible implementation, for example, the format recorded in the rhythm information can be [1, 1.2], [2, 2.4], [3.1, 0.5], and the multiple speed change information points parsed out are the points corresponding to 1s, 2s, and 3.1s in the first vocal audio.

[0090] Furthermore, the audio and video to be processed are segmented according to 1s, 2s, and 3.1s to obtain the first audio segment 0-1s; the second audio segment 1s-2s; and the third audio segment 2s-3.1s.

[0091] Step 32: For each audio segment, perform speed change processing on the audio segment according to the corresponding speed change to obtain a second human voice audio.

[0092] In this step, each speed change information point corresponds to a speed change value for the corresponding audio segment. At this time, speed change processing is performed on the audio segment.

[0093] Continuing with the above possible implementation, the first audio segment 0-1s is sped up by 1.2, resulting in 0-1.2s; the second audio segment 1s-2s is sped up by 2.4, resulting in 1.2s-3.6s; and the third audio segment 2s-3.1s is sped up by 0.5, resulting in 3.6s-4.15s.

[0094] Furthermore, the above audio clips are spliced ​​together, and the original 0-3s first vocal audio is spliced ​​to obtain 0-4.15s second vocal audio.

[0095] The audio and video processing method provided by the disclosed embodiments divides a first vocal audio file into multiple audio segments based on multiple speed change information points. Each audio segment is speed-shifted according to the corresponding speed change to produce a second vocal audio file. In this technical solution, the first vocal audio file is segmented using the speed change information points, and each segment is speed-shifted to simulate the length of the corresponding musical note, thereby producing the second vocal audio file.

[0096] In the above embodiment, FIG4 is a flow chart of the third method for processing audio and video provided in the embodiment of the present disclosure. As shown in FIG4 , the above step 22 may include the following steps:

[0097] Step 41: In response to a user's selection operation, a target music genre is determined from at least one preset music genre.

[0098] Each music style corresponds to at least one accompaniment audio.

[0099] The target music genre is one of the preset music genres, each music genre corresponds to at least one accompaniment audio, and the accompaniment audios in each music genre are different;

[0100] In this step, before composing the audio song, the user can also select the corresponding music style according to needs, such as folk, drama, pop, etc.

[0101] In one possible implementation, the user clicks on a page on the client and selects a music genre to be generated from at least one music genre option. At this time, the electronic device determines the target music genre in response to the selection.

[0102] The accompaniment audios of different music styles represent different rhythm information of the respective music styles.

[0103] Step 42: Determine a target accompaniment audio that is closest to the melody line of the first vocal audio from at least one accompaniment audio in the target music style.

[0104] In this step, based on the melody line in the first vocal audio, an accompaniment audio closest to the melody line in the first vocal audio is determined in the target music style, and the accompaniment audio is used as the target accompaniment audio.

[0105] The audio and video processing method provided by the disclosed embodiments determines a target musical genre from at least one pre-set musical genre in response to a user's selection, and then determines the target accompaniment audio that most closely matches the melody line of the first vocal audio from at least one accompaniment audio within the target musical genre. In this technical solution, the user can freely select the musical genre of the audio to be generated, and then the accompaniment audio that most closely matches the melody line of the first vocal audio is determined from within that musical genre, providing the user with a basis for simulating the desired musical genre.

[0106] In the above embodiment, FIG5 is a fourth flow chart of the audio and video processing method provided by the embodiment of the present disclosure. As shown in FIG5 , this embodiment briefly describes a possible application:

[0107] Step 1: The user uses the audio-to-song function;

[0108] Step 2: Check the existing voice changing effect;

[0109] For example, you can change VC voice, turn audio into music, etc.

[0110] Step 3: If the user input is a video, you need to extract the segment audio from the video first;

[0111] Step 4: Process the functions in the order of voice change;

[0112] This step 4 may include:

[0113] 1) Perform VC voice change on the audio;

[0114] 2) Convert the audio after VC voice change into music;

[0115] The audio-to-song conversion process includes: cache check, calling the audioSDK, downloading the audio, splicing the spliced ​​audio with the speed-shifted video, and finally exporting the video; the action parameter is returned.

[0116] Step 5. Splice action parameters;

[0117] Step 6. Use the VE interface to replace the original audio and apply the regular voice change filter;

[0118] Step 7. Apply the audio to video effect.

[0119] Among them, step 6 and step 7 are executed according to the needs.

[0120] In this embodiment, as an example, there is a situation where the duration of the audio clip returned by the audio song is inconsistent with the original video duration, the audio song and VC voice change can be superimposed, and the fixed superposition order is VC voice change first, and then audio song. There are the following solutions:

[0121] First, how to link the generated audio with the video clip: it supports binding an additional audio to the clip, so as to replace the original sound of the video;

[0122] Secondly, the solution to the inconsistency between the audio clip duration and the original video duration is as follows:

[0123] 1. Generate a new video to replace the original video: create a new sub-draft; add the original video; first split the video into multiple segments based on the speed change information points returned by the server, and apply the corresponding speed change to each segment; mute the video; import the audio returned by the server as a new audio segment; export a new video, use the new video to return to the old draft, and replace the original video. The principle is similar to destrobing.

[0124] 2. According to the speed change points provided by the sami end, reverse the speed of the audio to the same length as the video: According to the speed change information returned by the server, calculate the position of each speed change node corresponding to the new audio segment; for the new audio segment, reverse the speed of each node, and finally make the new audio length consistent with the original video length; then perform curve speed change on the video segment.

[0125] Sami processes audio speed information in a format similar to [1, 1.2], [2, 2.4], [3.3, 100], etc. This means that the original audio segment (0-1s) is evenly stretched to 1.2s, the original video segment (1-2s) is evenly stretched to 1.2-2.4s, and so on. The speed is uniform within this interval. To reverse the speed of the new segment to match the original audio segment, you can use the following solutions:

[0126] Use stepped anchor points to approximate the simulation of uniform speed change. For example, Figure 6 is a schematic diagram of reverse speed change provided by an embodiment of the present disclosure. As shown in Figure 6, for example, the time range [0, 0.5] is accelerated to 2 times the speed, and [0.5, 1] ​​is 0.5 times the speed. The anchor points of the curve speed change can be set to [0, 2], [0.5, 2], [0.501, 0.5], and [1, 0.5] for simulation. This can reduce errors, retain the speed change information of the original video, and not regenerate a new video. It does not conflict with other algorithms and has a fast processing speed.

[0127] In addition, assemble the parm of action, configure the voice change configuration of the video vc+sts+regular voice change (that is, the duration of the audio clip returned by audio song is inconsistent with the original video duration; audio song and VC voice change can be superimposed, and the fixed order of superposition is VC voice change first, then audio song). When action is executed, update the parameters of VideoSegment and save it to the draft. When exector is executed later, it is sorted according to the voice change configuration. When only V sound effect is used, the original video interface is replaced with ve. When there is sts, the video path is replaced with the process of updating the video algorithm, and other voice change configurations are updated.

[0128] In another implementation, assemble the parm of action, configure the voice change configuration vc+sts+regular voice change of the video (that is, the duration of the audio clip returned by the audio song is inconsistent with the original video duration; the audio song and VC voice change can be superimposed, and the fixed order of superposition is VC voice change first, and then apply audio song), and merge a curve speed change; update the parameters of VideoSegment during execution and save it to the draft; when exector is executed, sort according to the voice change configuration, replace the audio of sts with the new ve interface to the video, and update other voice change configurations.

[0129] In addition, the external processing sound effect service interface is a unified interface for services such as VC voice changing and audio to song that require external processing to generate audio replacement: the processing order list is configured according to LVVEAudioEffectType (the above audio effects are sorted according to demand or default)

[0130] ExtraAudioEffectServer:

[0131] Responsible for managing and allocating calls to external sound effects services; responsible for extracting segment audio; responsible for assembling the action parm of the external sound effects service, and then calling it to the middle layer to process other effects;

[0132] ExtraAudioEffectBusiness:

[0133] Responsible for the sound effect processing of each independent business; returns the action parameters and the processed sound effect address; requires good caching within the business.

[0134] The technical solutions involved in the embodiments of the present disclosure are similar to those in the above embodiments, and the effects are also similar, so they will not be repeated here.

[0135] The following are embodiments of the apparatus disclosed herein, which can be used to implement the method embodiments disclosed herein. For details not disclosed in the apparatus embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0136] FIG7 is a schematic diagram of the structure of an audio and video processing device provided by an embodiment of the present disclosure. As shown in FIG7 , the device includes:

[0137] The acquisition module 71 is used to acquire audio and video to be processed, where the audio and video to be processed include: first human voice audio;

[0138] A first determining module 72 is configured to determine a target accompaniment audio from at least one accompaniment audio;

[0139] The second determining module 73 is configured to determine a second vocal audio according to the first vocal audio and the target accompaniment audio, wherein the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

[0140] In one or more embodiments, the audio and video to be processed further includes: a first video;

[0141] Accordingly, when the duration of the second human voice audio is different from the duration of the first video, the second determining module 73 is further configured to:

[0142] The second human voice audio is subjected to inverse speed conversion processing to obtain a third human voice audio having the same duration as the first video.

[0143] or,

[0144] The first video is speed-changed using the rhythm information corresponding to the target accompaniment audio to obtain a second video of the same length as the second vocal audio.

[0145] In one or more embodiments, in a video editing scenario, the first video and the first vocal audio are aligned on a timeline corresponding to the video editing.

[0146] In one or more embodiments, the second determining module 73 is further configured to:

[0147] Generate target audio and video according to the third person audio and the first video;

[0148] or,

[0149] Generate target audio and video according to the second human voice audio and the second video.

[0150] In one or more embodiments, the target accompaniment audio includes: rhythm information, the rhythm information including: a plurality of speed change information points and a speed change corresponding to each speed change information point;

[0151] Accordingly, the first determining module 72 is further configured to:

[0152] Segmenting the first vocal audio into multiple audio segments according to the multiple speed change information points;

[0153] For each audio segment, the audio segment is speed-changed according to the corresponding speed change to obtain a second human voice audio.

[0154] In one or more embodiments, the target accompaniment audio further includes: melody information;

[0155] Accordingly, the second determining module 73 is configured to:

[0156] The melody of the first vocal audio is adjusted according to the melody information to obtain the adjusted second vocal audio.

[0157] In one or more embodiments, the first determining module 72 is specifically configured to:

[0158] In response to a user's selection operation, a target music genre is determined from at least one preset music genre, each music genre corresponding to at least one accompaniment audio;

[0159] A target accompaniment audio closest to the melody line of the first vocal audio is determined from at least one accompaniment audio in the target music genre.

[0160] In one or more embodiments, the target accompaniment audio includes background audio. The first determination module 72 is further configured to:

[0161] Generate new second vocal audio based on the second vocal audio and the background audio.

[0162] In one or more embodiments, after obtaining the first human voice audio, the first determining module 72 is further configured to:

[0163] Perform sound conversion processing on the first human voice audio to obtain the processed first human voice audio.

[0164] The audio and video processing device provided in the embodiments of the present disclosure can be used to execute the audio and video processing method in any of the above embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0165] It should be noted that it should be understood that the division of the various modules of the above devices is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. Moreover, these modules can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. In addition, these modules can be fully or partially integrated together or implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed by hardware integrated logic circuits in the processor element or instructions in the form of software.

[0166] FIG8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. FIG8 shows a schematic diagram of the structure of an electronic device suitable for implementing an embodiment of the present disclosure, wherein the electronic device may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers.

[0167] The electronic device shown in FIG8 is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0168] As shown in FIG8 , the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 81, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 82 or a program loaded from a storage device 88 into a random access memory (RAM) 83. Various programs and data required for the operation of the electronic device are also stored in RAM 83. The processing device 81, ROM 82, and RAM 83 are connected to each other via a bus 84. An input / output (I / O) interface 85 is also connected to the bus 84.

[0169] Typically, the following devices may be connected to the I / O interface 85: an input device 86 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 87 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 88 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 89. The communication device 89 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although FIG8 illustrates an electronic device having various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may be implemented or present instead.

[0170] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 89, or installed from the storage device 88, or installed from the ROM 82. When the computer program is executed by the processing device 81, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0171] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0172] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0173] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0174] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0175] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0176] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0177] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0178] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0179] The electronic device provided in the embodiments of the present disclosure can be used to execute the audio and video processing method provided in any of the above method embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0180] An embodiment of the present disclosure provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the above-mentioned audio and video processing method.

[0181] The computer-readable storage medium mentioned above may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The computer-readable storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0182] Optionally, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0183] An embodiment of the present disclosure also provides a computer program product, which includes a computer program. The computer program is stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, the above-mentioned audio and video processing method can be implemented.

[0184] also:

[0185] In a first aspect, a method for processing audio and video is provided, comprising:

[0186] Acquire audio and video to be processed, where the audio and video to be processed include: first human voice audio;

[0187] determining a target accompaniment audio from at least one accompaniment audio;

[0188] A second vocal audio is determined based on the first vocal audio and the target accompaniment audio, wherein the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

[0189] In one or more embodiments, the audio and video to be processed further includes: a first video;

[0190] Accordingly, when the duration of the second human voice audio is different from the duration of the first video, the method further includes:

[0191] The second human voice audio is subjected to inverse speed conversion processing to obtain a third human voice audio having the same duration as the first video.

[0192] or,

[0193] The first video is speed-changed using the rhythm information corresponding to the target accompaniment audio to obtain a second video having the same duration as the second vocal audio.

[0194] In one or more embodiments, in a video editing scenario, the first video and the first vocal audio are aligned on a timeline corresponding to the video editing.

[0195] In one or more embodiments, the method further comprises:

[0196] generating a target audio and video according to the third human voice audio and the first video;

[0197] or,

[0198] Generate the target audio and video according to the second human voice audio and the second video.

[0199] In one or more embodiments, the target accompaniment audio includes: rhythm information, the rhythm information including: a plurality of speed change information points and a speed change corresponding to each speed change information point;

[0200] Accordingly, determining the second vocal audio according to the first vocal audio and the target accompaniment audio includes:

[0201] dividing the first human voice audio into a plurality of audio segments according to the plurality of speed change information points;

[0202] For each audio segment, speed change processing is performed on the audio segment according to the corresponding speed change to obtain the second human voice audio.

[0203] In one or more embodiments, the target accompaniment audio further includes: melody information;

[0204] Accordingly, determining the second vocal audio according to the first vocal audio and the target accompaniment audio includes:

[0205] The first vocal audio is melody-adjusted according to the melody information to obtain an adjusted second vocal audio.

[0206] In one or more embodiments, determining the target accompaniment audio from the at least one accompaniment audio includes:

[0207] In response to a user's selection operation, a target music genre is determined from at least one preset music genre, each music genre corresponding to at least one accompaniment audio;

[0208] A target accompaniment audio closest to the melody line of the first vocal audio is determined from at least one accompaniment audio in the target music genre.

[0209] In one or more embodiments, the target accompaniment audio includes background audio, and the method further includes:

[0210] Generate new second vocal audio based on the second vocal audio and the background audio.

[0211] In one or more embodiments, after obtaining the first human voice audio, the method further includes:

[0212] Perform sound conversion processing on the first human voice audio to obtain the processed first human voice audio.

[0213] In a second aspect, an audio and video processing device is provided, the device comprising:

[0214] An acquisition module is used to acquire audio and video to be processed, wherein the audio and video to be processed includes: first human voice audio;

[0215] A first determining module is configured to determine a target accompaniment audio from at least one accompaniment audio;

[0216] The second determining module is used to determine a second vocal audio according to the first vocal audio and the target accompaniment audio, wherein the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

[0217] In one or more embodiments, the audio and video to be processed further includes: a first video;

[0218] Correspondingly, when the duration of the second human voice audio is different from the duration of the first video, the second determining module is further configured to:

[0219] The second human voice audio is subjected to inverse speed conversion processing to obtain a third human voice audio having the same duration as the first video.

[0220] or,

[0221] The first video is speed-changed using the rhythm information corresponding to the target accompaniment audio to obtain a second video having the same duration as the second vocal audio.

[0222] In one or more embodiments, in a video editing scenario, the first video and the first vocal audio are aligned on a timeline corresponding to the video editing.

[0223] In one or more embodiments, the second determining module is further configured to:

[0224] generating a target audio and video according to the third human voice audio and the first video;

[0225] or,

[0226] Generate the target audio and video according to the second human voice audio and the second video.

[0227] In one or more embodiments, the target accompaniment audio includes: rhythm information, the rhythm information including: a plurality of speed change information points and a speed change corresponding to each speed change information point;

[0228] Accordingly, the first determining module is further configured to:

[0229] dividing the first human voice audio into a plurality of audio segments according to the plurality of speed change information points;

[0230] For each audio segment, speed change processing is performed on the audio segment according to the corresponding speed change to obtain the second human voice audio.

[0231] In one or more embodiments, the target accompaniment audio further includes: melody information;

[0232] Accordingly, the second determining module is configured to:

[0233] The first vocal audio is melody-adjusted according to the melody information to obtain an adjusted second vocal audio.

[0234] In one or more embodiments, the first determining module is specifically configured to:

[0235] In response to a user's selection operation, a target music genre is determined from at least one preset music genre, each music genre corresponding to at least one accompaniment audio;

[0236] A target accompaniment audio closest to the melody line of the first vocal audio is determined from at least one accompaniment audio in the target music genre.

[0237] In one or more embodiments, the target accompaniment audio includes: background audio, and the first determination module is further configured to:

[0238] Generate new second vocal audio based on the second vocal audio and the background audio.

[0239] In one or more embodiments, after obtaining the first human voice audio, the first determination module is further configured to:

[0240] Perform sound conversion processing on the first human voice audio to obtain the processed first human voice audio.

[0241] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0242] The memory stores computer-executable instructions;

[0243] The processor executes the computer-executable instructions stored in the memory to implement the audio and video processing method as described in the first aspect above.

[0244] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the audio and video processing method described in the first aspect above.

[0245] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the audio and video processing method described in the first aspect above.

[0246] The above description is merely an optional embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0247] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0248] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A method for processing audio and video, comprising: Acquire audio and video to be processed, wherein the audio and video to be processed includes: first human voice audio; determining a target accompaniment audio from at least one accompaniment audio; According to the first vocal audio and the target accompaniment audio, a second vocal audio is determined, and the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

2. The method according to claim 1, wherein: The audio and video to be processed also includes: a first video; Correspondingly, when the duration of the second human voice audio is different from the duration of the first video, the method further includes: Performing inverse speed conversion processing on the second human voice audio to obtain a third human voice audio having the same duration as the first video; or, The first video is speed-changed by using the rhythm information corresponding to the target accompaniment audio to obtain a second video with the same duration as the second vocal audio.

3. The method according to claim 2, further comprising: Generate target audio and video according to the third human voice audio and the first video; or, The target audio and video is generated according to the second human voice audio and the second video.

4. The method according to claim 2, wherein: In the video editing scenario, the first video and the first vocal audio are aligned on a timeline corresponding to the video editing.

5. The method according to any one of claims 1 to 4, wherein: The target accompaniment audio includes: rhythm information, the rhythm information includes: a plurality of speed change information points and the speed change corresponding to each speed change information point; Accordingly, determining the second vocal audio according to the first vocal audio and the target accompaniment audio includes: According to the multiple speed change information points, the first human voice audio is divided into multiple audio segments; For each audio segment, the audio segment is speed-changed according to the corresponding speed change to obtain the second human voice audio.

6. The method according to claim 5, wherein: The target accompaniment audio also includes: melody information; Accordingly, determining the second vocal audio according to the first vocal audio and the target accompaniment audio includes: The first vocal audio is melody-adjusted according to the melody information to obtain an adjusted second vocal audio.

7. The method according to any one of claims 1 to 4, wherein: The step of determining a target accompaniment audio from at least one accompaniment audio comprises: In response to a selection operation by a user, a target music genre is determined from at least one preset music genre, each music genre corresponding to at least one accompaniment audio; A target accompaniment audio closest to the melody line of the first vocal audio is determined from at least one accompaniment audio in the target music genre.

8. The method according to any one of claims 1 to 4, wherein: The target accompaniment audio includes: background audio, and the method further includes: Generate new second vocal audio based on the second vocal audio and the background audio.

9. The method according to any one of claims 1 to 4, wherein: After acquiring the first human voice audio, the method further includes: The first human voice audio is subjected to sound conversion processing to obtain the processed first human voice audio.

10. An audio and video processing device, comprising: An acquisition module is used to acquire audio and video to be processed, wherein the audio and video to be processed includes: first human voice audio; A first determining module, configured to determine a target accompaniment audio from at least one accompaniment audio; The second determination module is used to determine the second vocal audio according to the first vocal audio and the target accompaniment audio, and the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

11. An electronic device, comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images