Audio and video processing method and device

Abstract

The invention provides an audio and video processing method and equipment, and the method comprises the steps: obtaining a to-be-processed audio and video, determining a target accompaniment audio in at least one accompaniment audio, determining a second audio according to the first audio and the target accompaniment audio, and processing the second audio according to the second audio. The rhythm of the second voice audio is matched with the rhythm of the target accompaniment audio. In the technical scheme, the rhythm of the first human voice audio is converted according to the rhythm in the accompaniment audio, so that the length of each voice during singing is simulated, and the purpose of converting the human voice audio into the singing audio is achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an audio and video processing method and device. Background Art

[0002] With the continuous development of computer technology, audio voice changing technology is also constantly developing, for example, converting the user's voice into a sound indicated by a specific sound color. However, with the increasing needs of users, people also hope to convert audio into audio with musical colors, for example, a user's speech can be converted into music, that is, the content of daily expression can be "musicalized".

[0003] Therefore, how to convert vocal audio into singing audio becomes a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides an audio and video processing method and device to solve the problem of audio and video processing in the prior art.

[0005] In a first aspect, an embodiment of the present disclosure provides an audio and video processing method, including:

[0006] Acquire audio and video to be processed, wherein the audio and video to be processed includes: first human voice audio;

[0007] determining a target accompaniment audio from at least one accompaniment audio;

[0008] According to the first vocal audio and the target accompaniment audio, a second vocal audio is determined, and the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

[0009] In a second aspect, an embodiment of the present disclosure provides an audio and video processing device, the device comprising:

[0010] An acquisition module is used to acquire audio and video to be processed, wherein the audio and video to be processed includes: first human voice audio;

[0011] A first determining module, configured to determine a target accompaniment audio from at least one accompaniment audio;

[0012] The second determination module is used to determine the second vocal audio according to the first vocal audio and the target accompaniment audio, and the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0014] The memory stores computer-executable instructions;

[0015] The processor executes the computer-executable instructions stored in the memory to implement the audio and video processing method as described in the first aspect above.

[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the audio and video processing method described in the first aspect above.

[0017] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the audio and video processing method as described in the first aspect above.

[0018] The present disclosure provides an audio and video processing method and device. The method obtains the audio and video to be processed, and the audio and video to be processed includes: a first vocal audio, then determines a target accompaniment audio in at least one accompaniment audio, and finally determines a second vocal audio based on the first vocal audio and the target accompaniment audio, and the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio. In this technical solution, the rhythm of the first vocal audio is transformed with the rhythm in the accompaniment audio, so as to simulate the length of each note when singing, so as to achieve the purpose of converting the vocal audio into the singing audio, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0020] Figure 1 A schematic diagram of the interaction between the audio and video processing implementation device provided in the embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram of the process of the audio and video processing method provided in the embodiment of the present disclosure Figure 1 ;

[0022] Figure 3 A schematic diagram of the process of the audio and video processing method provided in the embodiment of the present disclosure Figure 2 ;

[0023] Figure 4 A schematic diagram of the process of the audio and video processing method provided in the embodiment of the present disclosure Figure 3 ;

[0024] Figure 5 A schematic diagram of the process of the audio and video processing method provided in the embodiment of the present disclosure Figure 4 ;

[0025] Figure 6 A schematic diagram of reverse speed change provided in an embodiment of the present disclosure;

[0026] Figure 7 A schematic diagram of the structure of an audio and video processing device provided in an embodiment of the present disclosure;

[0027] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure.

[0028] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0030] Before introducing the embodiments of the present disclosure, the terms and application backgrounds involved in the embodiments of the present disclosure are first explained:

[0031] With the continuous development of computer technology, audio voice changing technology is also constantly developing, for example, converting the user's voice into a sound indicated by a specific sound color. However, with the increasing needs of users, people also hope to convert audio into audio with musical colors, for example, a user's speech can be converted into music, that is, the content of daily expression can be "musicalized".

[0032] For example, on some voice-related software, music audio usually receives higher submissions and consumption on the website. However, for most ordinary users, the threshold for music production is high, and they cannot "musicalize" their daily expressions with a low threshold. At the same time, some users prefer to use the voice changer module, which is a very popular module, but this template does not have the "musicalization" function yet, and cannot further provide users with a better user experience.

[0033] Therefore, how to provide a method for converting audio into audio with melody becomes a technical problem that needs to be solved urgently.

[0034] Furthermore, in response to the above technical problems, the inventor's technical conception is as follows: part of the difference between speaking and singing lies in the difference in pitch. If some pre-prepared accompaniment can be used, and the accompaniment can contain speed change information points and corresponding speed changes, and then the speech to be processed is divided into speech segments with corresponding speed change intervals, and then the corresponding segments are speed-changed, the audio corresponding to the speech can be converted into audio of a song with a different pitch.

[0035] The technical solution of the present disclosure is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0036] It is worth noting that the application fields of the audio and video processing method and device disclosed in the present invention are not limited and may include: the technical field of audio and video processing. Among them, the execution subject of the present invention is an electronic device, which may be a mobile phone, a tablet, a computer, a server and other devices.

[0037] Before introducing the method embodiment of the present invention, the implementation architecture process involved in the method of the present invention is described. Figure 1 ( Figure 1 As shown in the diagram of the interaction between the audio and video processing implementation device provided in the embodiment of the present disclosure, the diagram includes: a client 11, a video cloud 12, a first server 13 and a second server 14, and an audio software development kit (Audio Software Development Kit, Audio SDK) 15.

[0038] In one possible implementation, the client 11 obtains user input or audio (i.e., human voice audio) obtained from some location, and then requests the first server 13 for permission to access the video cloud 12 (slice upload and download) (e.g., requesting AuthKey). After obtaining authorization from the first server 13, the audio is uploaded to the video cloud 12, and at the same time, the identifier of the audio (e.g., vid) is sent to the real-time speech development toolkit (English: Audio SDK) 15, and the Audio SDK 15 sends the identifier of the audio to the second server 14.

[0039] The second server 14 downloads the corresponding audio from the video cloud 12 according to the audio identifier, and performs relevant processing of the audio into a song (this part is given in the following embodiment and is the core of the audio and video processing method of the present invention), obtains the processed audio (i.e., the song-made audio), and returns the identifier of the processed audio to the Audio SDK 15. The client 11 downloads the identifier of the processed audio from the second server 14 through the Audio SDK 15 according to the identifier of the processed audio, and then applies the speech to make a song.

[0040] In this implementation, the client 11 implements audio extraction and transcoding through the Jianying middle layer, and finally the speech is converted into music in the Jianying middle layer.

[0041] Among them, the first server 13 can be a Jianying server; the second server can be a server that implements the function of composing audio (for example, speaking) into music, such as a SAMI server, that is, it can be the electronic device that executes the embodiment of the present disclosure.

[0042] It should be understood that: Figure 1 The embodiment shown is only an example of implementation, and the undisclosed contents are detailed in the following embodiments.

[0043] Figure 2 A schematic diagram of the process of the audio and video processing method provided in the embodiment of the present disclosure Figure 1 ,like Figure 2 As shown, the method may include the following steps:

[0044] Step 21, obtaining the audio and video to be processed;

[0045] Among them, the audio and video to be processed include: the first human voice audio.

[0046] In this step, the audio that needs to be processed and converted into music (hereinafter referred to as audio-to-song sts) is first obtained, that is, the first vocal audio.

[0047] Optionally, the first human voice audio may be input by the user, for example, by clicking the recording button on the above-mentioned client and recording a passage spoken by the user into the client's audio from other files; it may also be the audio obtained by performing audio extraction processing on the audio and video, for example, extracting the segment audio from a section of audio and video to obtain the first human voice audio.

[0048] In one possible implementation, the client sends a request to the first server corresponding to the client for permission to upload the first human voice audio to the cloud (for example, a video cloud), and then the first server sends the permission to the client. The client uploads the first human voice audio to the cloud and obtains an identifier of the first human voice audio in the cloud, for example, vid, and then sends the speech-to-song instruction and vid to the electronic device through the AudioSDK interface through the client. After the electronic device obtains the instruction, it uses vid to download the first human voice audio from the cloud, that is, obtains the first human voice audio.

[0049] The duration of the first human voice audio can also be limited (which can be a configuration item), for example, 1 minute. When the audio is subsequently processed into a song, if it is extended to more than 1 minute, the audio song is cancelled; if it is extended, but within 1 minute, the server is requested to obtain the audio again; it is trimmed within 1 minute, and if it is shortened, it is not re-uploaded and only trimmed.

[0050] In addition, in other implementations, the functions of the first server may be integrated into the electronic device, and the storage functions of the cloud may also be integrated into the electronic device.

[0051] Optionally, after step 21, voice conversion (VC) processing may be performed on the first human voice audio to obtain the processed first human voice audio.

[0052] Under this implementation, before the speech-to-music function is realized, sound conversion processing can also be performed on the first human voice audio, for example, the voice input by the user is converted into the voice of other targets (for example, cartoon characters), and then, the audio of the voice of other targets is converted into music, that is, the audio of the music of other targets can be obtained.

[0053] Among them, following the above implementation, you can first upload the first voice audio through AudioSDK, obtain the first voice audio after VC voice change, and then perform the following operations on the first voice audio processed by VC through AudioSDK.

[0054] In a possible implementation, the audio effect can be VC voice change, audio to music, general timbre, general sound effect, music style, etc. Functionally, the execution order can be arbitrarily specified.

[0055] Step 22: Determine a target accompaniment audio from at least one accompaniment audio.

[0056] In this step, a plurality of accompaniment audios are pre-set in the electronic device, and one accompaniment audio can be selected from at least one accompaniment audio as the target accompaniment audio for subsequent audio composition.

[0057] Optionally, a selection method may be: the melody line in each accompaniment audio is different, the melody line is extracted from the first vocal audio, and the accompaniment audio with the closest melody line is determined among the accompaniment audios, that is, the target accompaniment audio.

[0058] This implementation ensures that the pitch contour of the original first voice audio is followed as much as possible after the first voice audio is processed, so that the subsequently generated audio sounds natural and avoids excessive modification that may cause loss of sound quality.

[0059] Optionally, another selection method may be: different accompaniment audios may correspond to different speaking speeds, and the target accompaniment audio may be selected based on the speaking speed of the first vocal audio, that is, the accompaniment audio with the closest speaking speed may be selected as the target accompaniment audio.

[0060] For example, four accompaniment audios are preset, with speaking speeds of 1.2, 1.3, 1.4, and 1.5 respectively, and the speaking speed of the first human voice audio is 1.51, then the accompaniment audio corresponding to 1.5 can be selected as the target accompaniment audio.

[0061] Step 23: Determine the second vocal audio according to the first vocal audio and the target accompaniment audio.

[0062] Among them, the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

[0063] In this step, the first vocal audio is processed using the rhythm information of the target accompaniment audio to obtain the second vocal audio matching the rhythm information.

[0064] Furthermore, in some implementations, during step 23, the melody of the first vocal audio may be adjusted according to the melody information in the target accompaniment audio.

[0065] In this implementation, the melody information in the target accompaniment audio can be used to fine-tune the melody of the first vocal audio to better fit the melody of the set accompaniment audio and increase the sense of melody of the subsequently generated audio.

[0066] The detailed processing process under this implementation is shown in the following embodiment and will not be repeated here.

[0067] Optionally, the target accompaniment audio includes: background audio, and the following steps may also be performed:

[0068] Generate new second voice audio according to the second voice audio and the background audio.

[0069] In this implementation, background music can be added to the a cappella singing to generate a more experiential song.

[0070] Further, based on the above embodiment, the audio and video to be processed also includes: a first video.

[0071] Optionally, in a video editing scenario, the first video and the first vocal audio are aligned on a timeline corresponding to the video editing, that is, the starting point and the ending point of the first video and the first vocal audio are consistent.

[0072] Furthermore, when the audio and video to be processed include the first human voice audio and the first video, the target audio and video needs to be generated in the end, and there are the following three processing situations:

[0073] The first type is that when the duration of the second voice audio is different from that of the first video, the second voice audio is subjected to inverse speed processing to obtain a third voice audio of the same length as the first video, and then the target audio and video are generated based on the third voice audio and the first video.

[0074] In this implementation, since the rhythm adjustment is performed on the first vocal audio, the duration of the first vocal audio (for example, 60s) may be different from the duration of the generated second vocal audio (for example, 57s). Here, it is necessary to make the duration of the second vocal audio consistent with the duration of the first vocal audio, both of which are 60s.

[0075] For example, the uniform speed change rate is 60 / 57 times, and the second voice audio with a speed of 60 / 57 times is inversely changed to 60s to obtain the third voice audio, which is then combined with the original first video to generate the target audio and video.

[0076] In a possible implementation, if the first target audio needs to be processed at an inverse speed so that the length is consistent with the audio and video to be processed, the following solution can be used:

[0077] Use step-type anchor points to approximate uniform speed changes. For example, the time range [0, 0.5] changes to 2 times the speed, and [0.5, 1] ​​changes to 0.5 times the speed. The anchor points of the curve speed change can be set to [0, 2], [0.5, 2], [0.501, 0.5], [1, 0.5] to simulate and reduce errors.

[0078] The second type is that when the duration of the second vocal audio is different from that of the first video, the first video is speed-changed using the rhythm information corresponding to the target accompaniment audio to obtain a second video of the same length as the second vocal audio, and the target audio and video is generated based on the second vocal audio and the second video.

[0079] In this implementation, since the rhythm adjustment is performed on the first vocal audio, the duration of the first vocal audio (for example, 60s) may be different from the duration of the generated second vocal audio (for example, 57s). Here, the rhythm information corresponding to the target accompaniment audio can be used to change the speed of the first video so that the duration of the second video after the speed change is the same as the duration of the second vocal audio, and then the target audio and video is generated based on the second vocal audio and the second video.

[0080] The third type is when the duration of the second voice audio is the same as the duration of the first video, and the target audio and video are generated based on the second voice audio and the first video.

[0081] In this implementation, the first human voice audio in the audio and video to be processed can be replaced by the second human voice audio.

[0082] It should be understood that in the above-mentioned methods, the implementation principle of the speed change processing for the rhythm is the same as or similar to the following embodiments, and reference may be made to the following embodiments.

[0083] Afterwards, the generated target audio and video is sent to the user equipment for use by the user equipment.

[0084] The present disclosure provides an audio and video processing method and device. The method obtains the audio and video to be processed, and the audio and video to be processed includes: a first vocal audio, then determines a target accompaniment audio in at least one accompaniment audio, and finally determines a second vocal audio based on the first vocal audio and the target accompaniment audio, and the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio. In this technical solution, the rhythm of the first vocal audio is transformed with the rhythm in the accompaniment audio, so as to simulate the length of each note when singing, so as to achieve the purpose of converting the vocal audio into the singing audio, thereby improving the user experience.

[0085] In the above embodiment, the target accompaniment audio includes: rhythm information, the rhythm information includes: a plurality of speed change information points and the speed change corresponding to each speed change information point, then Figure 3 A schematic diagram of the process of the audio and video processing method provided in the embodiment of the present disclosure Figure 2 ,like Figure 3 As shown, the above step 23 may include the following steps:

[0086] Step 31: dividing the first human voice audio into multiple audio segments according to multiple speed change information points;

[0087] In this step, a plurality of speed change information points are parsed from the rhythm information, and the first vocal audio is segmented according to the speed change information points to obtain each audio segment that needs to be speed-changed.

[0088] In one possible implementation, for example, the format recorded in the rhythm information can be [1, 1.2], [2, 2.4], [3.1, 0.5], and then the multiple speed change information points parsed out are the points corresponding to 1s, 2s, and 3.1s in the first vocal audio.

[0089] Furthermore, the audio and video to be processed are segmented according to 1s, 2s, and 3.1s to obtain the first audio segment 0-1s; the second audio segment 1s-2s; and the third audio segment 2s-3.1s.

[0090] Step 32: for each audio segment, perform speed change processing on the audio segment according to the corresponding speed change to obtain a second vocal audio.

[0091] In this step, each speed change information point corresponds to a speed change value for the corresponding audio segment. At this time, speed change processing is performed on the audio segment.

[0092] In the above possible implementation, the first audio segment 0-1s is sped up by 1.2, resulting in 0-1.2s; the second audio segment 1s-2s is sped up by 2.4, resulting in 1.2s-3.6s; the third audio segment 2s-3.1s is sped up by 0.5, resulting in 3.6s-4.15s.

[0093] Furthermore, the above audio clips are spliced ​​together, and the original 0-3s first vocal audio is speed-changed to obtain 0-4.15s second vocal audio.

[0094] The audio and video processing method provided by the embodiment of the present disclosure divides the first human voice audio into multiple audio segments according to multiple speed change information points, and for each audio segment, performs speed change processing on the audio segment according to the corresponding speed change to obtain the second human voice audio. In this technical solution, the first human voice audio is divided by using the speed change information points, and then each divided segment is subjected to speed change processing to simulate the length of the corresponding music, that is, to obtain the second human voice audio.

[0095] In the above embodiment, Figure 4 A schematic diagram of the process of the audio and video processing method provided in the embodiment of the present disclosure Figure 3 ,like Figure 4 As shown, the above step 22 may include the following steps:

[0096] Step 41: In response to a selection operation by a user, a target music genre is determined from at least one preset music genre.

[0097] Each music style corresponds to at least one accompaniment audio.

[0098] The target music genre is one of the preset music genres, each music genre corresponds to at least one accompaniment audio, and the accompaniment audios in each music genre are different;

[0099] In this step, before composing the audio song, the user can also select the corresponding style of the song according to needs, such as folk, drama, pop, etc.

[0100] In one possible implementation, the user clicks on a page on the client and selects a music genre to be generated from at least one music genre option. At this time, the electronic device determines the target music genre in response to the selection.

[0101] The accompaniment audios of different music styles represent different rhythm information of the respective music styles.

[0102] Step 42: Determine a target accompaniment audio that is closest to the melody line of the first vocal audio from at least one accompaniment audio in the target music style.

[0103] In this step, based on the melody line in the first vocal audio, an accompaniment audio closest to the melody line in the first vocal audio is determined in the target music style, and the accompaniment audio is used as the target accompaniment audio.

[0104] The audio and video processing method provided by the embodiment of the present disclosure determines a target music style from at least one pre-set music style in response to a user's selection operation, and determines a target accompaniment audio closest to the melody line of the first vocal audio from at least one accompaniment audio in the target music style. In this technical solution, the user can freely select the music style of the audio to be generated, and then determine the accompaniment audio closest to the melody line of the first vocal audio from the music style, providing the user with a basis for the simulation effect of the desired music style.

[0105] In the above embodiment, Figure 5 A schematic diagram of the process of the audio and video processing method provided in the embodiment of the present disclosure Figure 4 ,like Figure 5 As shown, this embodiment briefly describes a possible application:

[0106] Step 1: The user uses the function of composing audio into a song;

[0107] Step 2: Check the existing voice changing effect;

[0108] For example, there can be VC voice change, audio to music, etc.

[0109] Step 3: If the user input is a video, you need to extract the segment audio from the video first;

[0110] Step 4: Process the functions in the order of voice changer.

[0111] This step 4 may include:

[0112] 1) Perform VC voice change on the audio;

[0113] 2) Convert the audio after the VC voice change into a song;

[0114] The execution process of audio-to-song conversion is as follows: cache check, calling audioSDK, downloading audio, splicing the audio after the speed change video, and finally exporting the video; returning the action parameter.

[0115] Step 5: Concatenate action parameters;

[0116] Step 6. Use the VE interface to replace the original audio and apply the regular voice change filter;

[0117] Step 7. Apply audio to video effect.

[0118] Among them, step 6 and step 7 are executed one by one according to the needs.

[0119] In this embodiment, as an example, there is a situation where the duration of the audio segment returned by the audio song is inconsistent with the duration of the original video, the audio song and the VC voice change can be superimposed, and the fixed order of superposition is first VC voice change, and then the audio song is applied. There are the following solutions:

[0120] First, how to link the generated audio with the video clip: it supports binding an additional audio to the clip, so as to replace the original sound of the video;

[0121] Secondly, the solution for the inconsistency between the duration of the audio clip and the original video:

[0122] 1. Generate a new video to replace the original video: create a new sub-draft; add the original video; first divide the video into multiple segments according to the speed change information points sent back by the server, and apply the corresponding speed change to each segment; mute the video; import the audio sent back by the server as a new audio segment; export a new video, use the new video, go back to the old draft, and replace the original video. The principle is similar to destrobing.

[0123] 2. According to the speed change points given by the sami end, reverse the speed of the audio to the same length as the video: according to the speed change information returned by the server, calculate the position of each speed change node corresponding to the new audio segment; for the new audio segment, reverse the speed of each node, and finally make the new audio length consistent with the original video length; then perform curve speed change on the video segment.

[0124] Sami processes audio speed change information in a format similar to [1, 1.2], [2, 2.4], [3.3, 100]... This means that the original audio 0-1s is evenly stretched to 1.2s, the original video 1-2s is evenly stretched to 1.2-2.4s, and so on. Here, the speed is evenly changed within the interval. If you want to reverse the speed of the new segment to make the length consistent with the original audio, you can use the following solutions:

[0125] Use step-shaped anchor points to approximate uniform speed changes, for example, Figure 6 A schematic diagram of the reverse speed change provided by the embodiment of the present disclosure, such as Figure 6 As shown, for example, the time range [0, 0.5] is sped up to 2 times, and [0.5, 1] ​​is sped up to 0.5 times. The anchor points of the curve speed change can be set to [0, 2], [0.5, 2], [0.501, 0.5], [1, 0.5] for simulation. This can reduce the error, retain the speed change information of the original video, and not regenerate a new video. It does not conflict with other algorithms and has a fast processing speed.

[0126] In addition, assemble the parm of action, configure the voice change configuration of the video vc+sts+regular voice change (that is, the duration of the audio segment returned by the audio song is inconsistent with the original video duration; the audio song and VC voice change can be superimposed, and the fixed order of superposition is VC voice change first, and then apply audio song). When the action is executed, the parameters of VideoSegment are updated and saved to the draft. When the exector is executed later, it is sorted according to the voice change configuration. When only V sound effect is used, ve is used to replace the original video interface. When there is sts, the video path is replaced with the process of updating the video algorithm, and other voice change configurations are updated.

[0127] In another implementation, assemble the parm of action, configure the voice change configuration vc+sts+regular voice change of the video (that is, the duration of the audio segment returned by the audio-to-song is inconsistent with the duration of the original video; the audio-to-song and VC voice change can be superimposed, and the fixed order of superposition is VC voice change first, and then apply the audio-to-song), and merge a curve speed change; update the parameters of VideoSegment during execution and save them to the draft; when exector is executed, sort according to the voice change configuration, replace the audio of sts with the video with the new interface of ve, and update other voice change configurations.

[0128] In addition, the external processing audio effect service interface: a unified interface for services such as VC voice change and audio to music that require external processing to generate audio replacement: according to the LVVEAudioEffectType configuration processing order list (the above audio effects are sorted according to demand or default)

[0129] ExtraAudioEffectServer:

[0130] Responsible for managing and allocating the calls of external sound effect services; responsible for extracting the audio of the segment; responsible for assembling the action parm of the external sound effect service, and then calling it to the middle layer to process other effects;

[0131] ExtraAudioEffectBusiness:

[0132] Responsible for the sound effect processing of each independent business; returns the action parameters and the processed sound effect address; requires good cache processing within the business.

[0133] The technical solutions involved in the embodiments of the present disclosure are similar to those in the above embodiments, and the effects are also similar, which will not be repeated here.

[0134] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0135] Figure 7This is a schematic diagram of the structure of the audio and video processing device provided by the embodiment of the present disclosure. Figure 7 As shown, the device includes:

[0136] The acquisition module 71 is used to acquire the audio and video to be processed, and the audio and video to be processed include: first human voice audio;

[0137] A first determination module 72, configured to determine a target accompaniment audio from at least one accompaniment audio;

[0138] The second determination module 73 is used to determine the second vocal audio according to the first vocal audio and the target accompaniment audio, and the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

[0139] In one or more embodiments, the audio and video to be processed further includes: a first video;

[0140] Correspondingly, when the duration of the second human voice audio is different from the duration of the first video, the second determining module 73 is further used to:

[0141] The second human voice audio is processed at an inverse speed to obtain a third human voice audio having the same length as the first video.

[0142] or,

[0143] The first video is speed-changed using the rhythm information corresponding to the target accompaniment audio to obtain a second video of the same length as the second vocal audio.

[0144] In one or more embodiments, in a video editing scenario, the first video and the first vocal audio are aligned on a timeline corresponding to the video editing.

[0145] In one or more embodiments, the second determining module 73 is further configured to:

[0146] Generate target audio and video according to the third person audio and the first video;

[0147] or,

[0148] Generate target audio and video according to the second human voice audio and the second video.

[0149] In one or more embodiments, the target accompaniment audio includes: rhythm information, the rhythm information includes: a plurality of speed change information points and a speed change corresponding to each speed change information point;

[0150] Accordingly, the first determining module 72 is further configured to:

[0151] According to the multiple speed change information points, the first vocal audio is divided into multiple audio segments;

[0152] For each audio segment, the audio segment is speed-changed according to the corresponding speed change to obtain the second vocal audio.

[0153] In one or more embodiments, the target accompaniment audio further includes: melody information;

[0154] Accordingly, the second determining module 73 is used to:

[0155] According to the melody information, the melody of the first vocal audio is adjusted to obtain the adjusted second vocal audio.

[0156] In one or more embodiments, the first determination module 72 is specifically configured to:

[0157] In response to a selection operation by a user, a target music genre is determined from at least one preset music genre, each music genre corresponding to at least one accompaniment audio;

[0158] A target accompaniment audio closest to the melody line of the first vocal audio is determined from at least one accompaniment audio in the target music genre.

[0159] In one or more embodiments, the target accompaniment audio includes: background audio, and the first determination module 72 is further used to:

[0160] Generate new second vocal audio based on the second vocal audio and the background audio.

[0161] In one or more embodiments, after obtaining the first human voice audio, the first determination module 72 is further configured to:

[0162] Perform sound conversion processing on the first human voice audio to obtain the processed first human voice audio.

[0163] The audio and video processing device provided in the embodiments of the present disclosure can be used to execute the audio and video processing method in any of the above embodiments. The implementation principles and technical effects are similar and will not be repeated here.

[0164] It should be noted that it should be understood that the division of the various modules of the above devices is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. In addition, all or part of these modules can be integrated together or implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0165] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as Figure 8 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure, and the electronic device may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc.

[0166] Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0167] like Figure 8 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 81, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 82 or a program loaded from a storage device 88 to a random access memory (RAM) 83. Various programs and data required for the operation of the electronic device are also stored in the RAM 83. The processing device 81, the ROM 82, and the RAM 83 are connected to each other via a bus 84. An input / output (I / O) interface 85 is also connected to the bus 84.

[0168] Typically, the following devices may be connected to the I / O interface 85: input devices 86 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 87 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 88 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 89. The communication device 88 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0169] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 88, or installed from a storage device 88, or installed from a ROM 82. When the computer program is executed by the processing device 81, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0170] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0171] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0172] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0173] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0174] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0175] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0176] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0177] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0178] The electronic device provided in the embodiments of the present disclosure may be used to execute the audio and video processing method provided in any of the above method embodiments, and its implementation principle and technical effects are similar and will not be described in detail here.

[0179] An embodiment of the present disclosure provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the above-mentioned audio and video processing method.

[0180] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0181] Optionally, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0182] The embodiments of the present disclosure also provide a computer program product, which includes a computer program. The computer program is stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, the above-mentioned audio and video processing method can be implemented.

[0183] also:

[0184] In a first aspect, a method for processing audio and video is provided, comprising:

[0185] Acquire audio and video to be processed, wherein the audio and video to be processed includes: first human voice audio;

[0186] determining a target accompaniment audio from at least one accompaniment audio;

[0187] According to the first vocal audio and the target accompaniment audio, a second vocal audio is determined, and the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

[0188] In one or more embodiments, the audio and video to be processed further includes: a first video;

[0189] Correspondingly, when the duration of the second human voice audio is different from the duration of the first video, the method further includes:

[0190] The second human voice audio is subjected to inverse speed change processing to obtain a third human voice audio having the same duration as the first video.

[0191] or,

[0192] The first video is speed-changed by using the rhythm information corresponding to the target accompaniment audio to obtain a second video with the same duration as the second vocal audio.

[0193] In one or more embodiments, in a video editing scenario, the first video and the first vocal audio are aligned on a timeline corresponding to the video editing.

[0194] In one or more embodiments, the method further comprises:

[0195] Generate target audio and video according to the third human voice audio and the first video;

[0196] or,

[0197] The target audio and video is generated according to the second human voice audio and the second video.

[0198] In one or more embodiments, the target accompaniment audio includes: rhythm information, the rhythm information includes: a plurality of speed change information points and a speed change corresponding to each speed change information point;

[0199] Accordingly, determining the second vocal audio according to the first vocal audio and the target accompaniment audio includes:

[0200] According to the multiple speed change information points, the first human voice audio is divided into multiple audio segments;

[0201] For each audio segment, the audio segment is speed-changed according to the corresponding speed change to obtain the second human voice audio.

[0202] In one or more embodiments, the target accompaniment audio further includes: melody information;

[0203] Accordingly, determining the second vocal audio according to the first vocal audio and the target accompaniment audio includes:

[0204] The first vocal audio is melody-adjusted according to the melody information to obtain an adjusted second vocal audio.

[0205] In one or more embodiments, determining the target accompaniment audio from at least one accompaniment audio comprises:

[0206] In response to a selection operation by a user, a target music genre is determined from at least one preset music genre, each music genre corresponding to at least one accompaniment audio;

[0207] A target accompaniment audio closest to the melody line of the first vocal audio is determined from at least one accompaniment audio in the target music genre.

[0208] In one or more embodiments, the target accompaniment audio includes: background audio, and the method further includes:

[0209] Generate new second vocal audio based on the second vocal audio and the background audio.

[0210] In one or more embodiments, after acquiring the first human voice audio, the method further includes:

[0211] The first human voice audio is subjected to sound conversion processing to obtain the processed first human voice audio.

[0212] In a second aspect, an audio and video processing device is provided, the device comprising:

[0213] An acquisition module is used to acquire audio and video to be processed, wherein the audio and video to be processed includes: first human voice audio;

[0214] A first determining module, configured to determine a target accompaniment audio from at least one accompaniment audio;

[0215] The second determination module is used to determine the second vocal audio according to the first vocal audio and the target accompaniment audio, and the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

[0216] In one or more embodiments, the audio and video to be processed further includes: a first video;

[0217] Correspondingly, when the duration of the second human voice audio is different from the duration of the first video, the second determining module is further used to:

[0218] The second human voice audio is subjected to inverse speed change processing to obtain a third human voice audio having the same duration as the first video.

[0219] or,

[0220] The first video is speed-changed by using the rhythm information corresponding to the target accompaniment audio to obtain a second video with the same duration as the second vocal audio.

[0221] In one or more embodiments, in a video editing scenario, the first video and the first vocal audio are aligned on a timeline corresponding to the video editing.

[0222] In one or more embodiments, the second determining module is further configured to:

[0223] Generate target audio and video according to the third human voice audio and the first video;

[0224] or,

[0225] The target audio and video is generated according to the second human voice audio and the second video.

[0226] In one or more embodiments, the target accompaniment audio includes: rhythm information, the rhythm information includes: a plurality of speed change information points and a speed change corresponding to each speed change information point;

[0227] Accordingly, the first determining module is further used for:

[0228] According to the multiple speed change information points, the first human voice audio is divided into multiple audio segments;

[0229] For each audio segment, the audio segment is speed-changed according to the corresponding speed change to obtain the second human voice audio.

[0230] In one or more embodiments, the target accompaniment audio further includes: melody information;

[0231] Accordingly, the second determining module is used to:

[0232] The first vocal audio is melody-adjusted according to the melody information to obtain an adjusted second vocal audio.

[0233] In one or more embodiments, the first determining module is specifically configured to:

[0234] In response to a selection operation by a user, a target music genre is determined from at least one preset music genre, each music genre corresponding to at least one accompaniment audio;

[0235] A target accompaniment audio closest to the melody line of the first vocal audio is determined from at least one accompaniment audio in the target music genre.

[0236] In one or more embodiments, the target accompaniment audio includes: background audio, a first determination module, and is further used to:

[0237] Generate new second vocal audio based on the second vocal audio and the background audio.

[0238] In one or more embodiments, after acquiring the first human voice audio, the first determination module is further configured to:

[0239] The first human voice audio is subjected to sound conversion processing to obtain the processed first human voice audio.

[0240] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0241] The memory stores computer-executable instructions;

[0242] The processor executes the computer-executable instructions stored in the memory to implement the audio and video processing method as described in the first aspect above.

[0243] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the audio and video processing method described in the first aspect above.

[0244] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the audio and video processing method as described in the first aspect above.

[0245] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0246] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0247] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for processing audio and video, characterized in that: include: Acquire audio and video to be processed, wherein the audio and video to be processed includes: first human voice audio; determining a target accompaniment audio from at least one accompaniment audio; According to the first vocal audio and the target accompaniment audio, a second vocal audio is determined, and the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

2. The method according to claim 1, characterized in that The audio and video to be processed also includes: a first video; Correspondingly, when the duration of the second human voice audio is different from the duration of the first video, the method further includes: Performing inverse speed conversion processing on the second human voice audio to obtain a third human voice audio having the same duration as the first video; or, The first video is speed-changed by using the rhythm information corresponding to the target accompaniment audio to obtain a second video with the same duration as the second vocal audio.

3. The method according to claim 2, characterized in that The method further comprises: Generate target audio and video according to the third human voice audio and the first video; or, The target audio and video is generated according to the second human voice audio and the second video.

4. The method according to claim 2, characterized in that: In the video editing scenario, the first video and the first vocal audio are aligned on a timeline corresponding to the video editing.

5. The method according to any one of claims 1 to 4, characterized in that: The target accompaniment audio includes: rhythm information, the rhythm information includes: a plurality of speed change information points and the speed change corresponding to each speed change information point; Accordingly, determining the second vocal audio according to the first vocal audio and the target accompaniment audio includes: According to the multiple speed change information points, the first human voice audio is divided into multiple audio segments; For each audio segment, the audio segment is speed-changed according to the corresponding speed change to obtain the second human voice audio.

6. The method according to claim 5, characterized in that The target accompaniment audio also includes: melody information; Accordingly, determining the second vocal audio according to the first vocal audio and the target accompaniment audio includes: The first vocal audio is melody-adjusted according to the melody information to obtain an adjusted second vocal audio.

7. The method according to any one of claims 1 to 4, characterized in that: The step of determining a target accompaniment audio from at least one accompaniment audio comprises: In response to a selection operation by a user, a target music genre is determined from at least one preset music genre, each music genre corresponding to at least one accompaniment audio; A target accompaniment audio closest to the melody line of the first vocal audio is determined from at least one accompaniment audio in the target music genre.

8. The method according to any one of claims 1 to 4, characterized in that: The target accompaniment audio includes: background audio, and the method further includes: Generate new second vocal audio based on the second vocal audio and the background audio.

9. The method according to any one of claims 1 to 4, characterized in that: After acquiring the first human voice audio, the method further includes: The first human voice audio is subjected to sound conversion processing to obtain the processed first human voice audio.

10. An audio and video processing device, characterized in that: The device comprises: An acquisition module is used to acquire audio and video to be processed, wherein the audio and video to be processed includes: first human voice audio; A first determining module, configured to determine a target accompaniment audio from at least one accompaniment audio; The second determination module is used to determine the second vocal audio according to the first vocal audio and the target accompaniment audio, and the rhythm of the second vocal audio matches the rhythm of the target accompaniment audio.

11. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images