Audio import method, device and electronic device

Abstract

The present application discloses an audio import method, device and electronic device, belonging to the field of image processing technology. The method comprises: obtaining a first video containing a first audio; obtaining each object action in the first video and the audio segment corresponding to each object action in the first audio; in the case of obtaining a second video to be imported with audio, identifying the target action in the second video; determining the first object action in each object action that matches the target action and the first audio segment corresponding to the first object action; importing the first audio segment into the part of the target action in the second video.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an audio import method, device and electronic device. Background Art

[0002] With the rapid development of self-media, the requirements for the shooting functions of mobile terminals are getting higher and higher. Mobile terminals are not only required to be able to shoot pictures or record videos, but also to process the pictures or videos.

[0003] Currently, when adding music to user videos, the audio often needs to be imported according to the time sequence of the template video. Subsequently, the user performs the same actions in the template video in a timely manner to adapt to the audio. Often the user's actions are fast or slow, and the shot video will have audio and picture out of sync. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an audio import method, device and electronic device, which can solve the problem of the user's own movements and audio being out of sync when composing music for a video.

[0005] In a first aspect, an embodiment of the present application provides an audio import method, the method comprising: obtaining a first video containing a first audio; obtaining each object action in the first video, and an audio segment corresponding to each object action in the first audio; when a second video into which audio is to be imported is obtained, identifying a target action in the second video; determining a first object action among the each object action that matches the target action and a first audio segment corresponding to the first object action; and importing the first audio segment into the portion of the target action in the second video.

[0006] In a second aspect, an embodiment of the present application provides a device for audio importing, which includes: a video acquisition module, used to acquire a first video containing a first audio; an audio determination module, used to acquire each object action in the first video, and the audio segments corresponding to the each object action in the first audio; a target action recognition module, used to identify the target action in the second video when a second video to which audio is to be imported is obtained; an audio segment determination module, used to determine the first object action among the each object action that matches the target action and the first audio segment corresponding to the first object action; an audio segment importing module, used to import the first audio segment into the part of the target action in the second video.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0011] In the embodiment of the present application, a video with audio synchronization imported by a user is obtained, and each action in the video and the corresponding audio segment are determined. When the user follows a similar video with a fast or slow rhythm, by identifying actions similar to each action in the similar video, the corresponding audio segment is imported, so that the audio segment can be imported along with the user's fast or slow actions, so as to achieve the effect of audio and video synchronization of the captured video, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flowchart of an audio import method in an embodiment of the present application;

[0013] Figure 2 is a schematic diagram of a virtual key of an audio import method in an embodiment of the present application;

[0014] Figure 3 is a schematic diagram of a second prompt page of an audio import method in an embodiment of the present application;

[0015] Figure 4 is a schematic diagram of a first prompt page of an audio import method in an embodiment of the present application;

[0016] Figure 5 It is a flowchart of an audio import device in an embodiment of the present application;

[0017] Figure 6 is a structural block diagram of an electronic device in an embodiment of the present application;

[0018] Figure 7It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0020] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0021] In the following, in combination with the accompanying drawings, an audio import method, device and electronic device provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0022] right Figure 1 The processing flow shown is described in detail, and the content can be as follows:

[0023] Step 101, obtaining a first video including a first audio.

[0024] In implementation, a user can upload a video with audio in a corresponding shooting software, or select a video with audio in the corresponding shooting software. After the user uploads or selects, a video synchronized with the audio can be obtained, that is, the audio and video match each other. The video can be the first video, and the audio can be the first audio.

[0025] In some specific embodiments, in order to provide users with more selectable videos with audio, step 101 includes the following contents: generating order-adjusted options corresponding to the template videos through one or more template videos containing audio selected by the user in a preset operation interface; based on the adjustment request output by the user for the order-adjusted options, using the template video after the order is adjusted as the first video and the audio as the first audio, to obtain a first video containing the first audio.

[0026] Among them, a corresponding operation interface can be preset, and the operation interface can be a UI interface and multiple template videos are displayed in the UI interface, and each template video is pre-set with corresponding audio. Among them, the way of displaying the template video on the operation interface can be to display the template video in the form of playback, or to display a certain frame of the preset template video, which is not limited here. Accordingly, the corresponding audio song name and other information can be displayed below the screen of the template video being played or a certain frame of the template video.

[0027] Specifically, when a user clicks on a certain template video in the operation interface, the template video and the audio corresponding to the template video can be played at the same time, and a corresponding first option is generated in the blank space below the played screen. The option can be, but is not limited to, a virtual button that the user can click to move. The specific virtual button is as follows: Figure 2 The "Template 1", "Template 2", "Template 3", and "Template 4" shown. The user can continue to return to the operation interface, and when clicking to select other template videos, the template video and the audio corresponding to the template video are played at the same time, and the corresponding second option is generated in the blank space below the playback screen. And the second option can be arranged after the first option according to the time sequence of the user selecting the two template videos. By analogy, after the user selects the required template video, a plurality of options arranged in sequence can be generated in the blank space below. Among them, the plurality of options arranged in sequence can be options to be adjusted in sequence. The adjustment request output by the user for the options to be adjusted in sequence, wherein the adjustment request output by the user can be, but not limited to, adjusting the sequence of multiple options by dragging a certain option. The user can press the corresponding confirmation button to use the template video after the adjustment as the first video and the audio as the first audio, and obtain the first video containing the first audio. In other words, more videos with audio that can be selected by the user are provided, and reference resources are provided for the user's video soundtrack, thereby improving the user's experience.

[0028] In addition, when selecting multiple template videos containing audio, since the multiple template videos correspond to a variety of different audios, the user can insert the audio from different videos into the current video by changing the action, effectively reducing the situation where the user needs to switch the audio to match the action.

[0029] Step 102: Obtain each object action in the first video and the audio segment corresponding to each object action in the first audio.

[0030] In implementation, after obtaining the first video synchronized with the first audio, the object in the video can be identified by human body recognition, and the various object actions of the object in the first video can be detected by action detection. Specific examples of various actions include: swinging arms, squatting, bending over, etc. The action time of the object action in the first video is determined. Since the first video and the first audio are synchronized in time, the audio segment corresponding to the same action time in the first audio is determined, that is, the audio segment corresponding to the object action in the first audio is determined by the same action time.

[0031] In some specific embodiments, in order to split the activities of an object in a video into multiple actions, the specific process may include the following: splitting the first video into multiple first video segments with a preset accuracy; extracting the object actions and action time periods in each first video segment; and obtaining the audio segment in the first audio that has the same action time period as each object action.

[0032] In implementation, after obtaining the first video with the first audio, the first video is split into multiple video segments according to a preset accuracy. For example: the first video is split into multiple video segments based on a preset amount of bytes, or the first video is split into a preset number of video segments, which is not limited here. Through human body recognition and motion detection, different object actions in each first video segment are extracted, and each object action corresponds to a time period for the execution of an action. Since the first video and the first audio are synchronized, it is possible to determine a segment of audio in the first audio that is consistent with the time period for the execution of the object action, that is, the audio segment paired with the object action in the first audio. Among them, the time period for the execution of the object action can be an action time period. In other words, the audio segment corresponding to any object action in the video can be determined through the time period of synchronization of the video and audio.

[0033] Furthermore, in order to split the video more reasonably to improve the fluency of subsequent video playback, the process of splitting the video into multiple video segments specifically includes the following process: splitting the first video into multiple first video segments according to at least one of the related attributes of the first video and the pause point in the first audio. The pause point represents any one or more of the interruption point, extreme value point or inflection point of the audio signal of the first audio.

[0034] Among them, when a break point, extreme point or inflection point appears in the audio signal, it can be considered that there is a break, pause, etc. in the audio. For example, when the lyrics in the audio are interrupted or end, the dance movements will stop accordingly.

[0035] In implementation, the splitting method of the first video can be set manually or by system default. The specific splitting method can be splitting by the relevant attributes of the first video, for example: splitting into multiple first video segments with the same byte amount according to the byte amount of the first video; or splitting by the pause point of the synchronized first audio, for example: when any of the pause points, extreme points or inflection points appear in the synchronized first audio, the first video is split into multiple first video segments according to the time nodes corresponding to the pause points. Among them, the above-mentioned splitting by relevant attributes and the splitting by pause points in the first audio can be based on the existing video splitting algorithm, which will not be elaborated here.

[0036] Specifically, the first video can be split into n first video segments, each of which corresponds to m object actions and m audio segments, where n≥1 and m≥1. The first video has n×m object actions and n×m audio segments. By dynamically adjusting n and m, n and m can be increased or decreased according to the complexity of the action in the current scene, so as to reduce the hardware load and achieve a better value in power consumption and final product quality.

[0037] Specific as Figure 3 As shown, the first video is represented as a visual track, and the first video can be divided into multiple video segments, each of which corresponds to an object action. Accordingly, each object action displayed above the visual track is represented as sn1, sn2, sn3, and snm. Each audio segment displayed above the audio is represented as n1, vn2, vn3, and vnm. The first video has m object actions and corresponding m audio segments.

[0038] In one embodiment, in order to display the split actions to the user, the following contents are also included: according to the playback length of the first audio and the first video, a first prompt page is generated and displayed, and the first prompt page is used to display the visual track of the first video and the audio track of the first audio; according to the object action and the paired audio segment, a second prompt page is generated and displayed, and the second prompt page is used to display the corresponding position of the object action on the visual track, the corresponding position of the audio segment on the audio track, and the corresponding relationship between the object action and the audio segment.

[0039] In implementation, after obtaining the first video with the first audio, the corresponding playing lengths of the first audio and the first video can be obtained, and a corresponding first prompt page can be generated and displayed to the user. Specifically, the first prompt page is as follows: Figure 4As shown, the first prompt page displays the visual track of the corresponding first video. The user can click a certain point in the visual track to start playing the video segment starting from that point, or click a certain point in the audio track to start playing the audio segment starting from that point, so that the user can check whether the video or audio meets the needs. Accordingly, after determining the object action and the paired audio segment of the first video, a corresponding second prompt page is generated according to the object action and the paired audio segment, and the second prompt page is jumped from the first prompt page to be displayed to the user. Specifically, the second prompt page is as follows: Figure 3 As shown, the position of each object action is displayed above the visual track, where sn1, sn2, sn3 and snm represent different object actions. The position of each audio segment is displayed above the audio, where vn1, vn2, vn3 and vnm represent different audio segments. And each object action corresponds to an audio segment in a first direction, where the first direction can be the length direction of the prompt page.

[0040] Step 103, when a second video to be imported with audio is obtained, identifying a target action in the second video;

[0041] In implementation, the user can directly upload the second video that has been recorded, or click the corresponding recording button to start recording the second video. Accordingly, the second video to be recorded with audio can be obtained, and then the target action of the object in the second video can be identified through human body recognition and action detection.

[0042] Step 104 : determining a first object action that matches the target action among the object actions and a first audio segment corresponding to the first object action.

[0043] In implementation, after the target action is identified, a first object action that matches the target action is searched from among the obtained object actions, and then a first audio segment corresponding to the first object action is determined.

[0044] In one embodiment, the motion matching includes having the same number and position of motion points, and the motion amplitude deviation value of the motion points at the same position does not exceed a preset deviation threshold.

[0045] Among them, corresponding deviation thresholds can be pre-set for different parts of the human body. For example: if the action point is a hand joint, the deviation threshold corresponding to the hand joint can be an angle deviation of 5°; if the action point is an arm, the deviation threshold corresponding to the arm can be an angle deviation of 10°, and so on.

[0046] Specifically, the number of action points of the target action and the position of each action point are determined, and the object action with the same number of action points and the same position of each action point is found from each object action as the second object action, and then the deviation value of the action amplitude between each action point in the second object action and the action point corresponding to the target action is compared. When the deviation value of the first object action in the second object action does not exceed the preset deviation threshold, the first object action and the target action can be considered to be consistent in action. In other words, by finding an object action that is consistent with the target action based on the number, position and action amplitude of the action points, it helps to improve the accuracy of the obtained first object action.

[0047] In one embodiment, before extracting the object action and action period in each first video segment, the following contents are also included: obtaining the action change information of the object in each first video segment, the action change information includes the object action point and the action amplitude corresponding to the object action; judging the action complexity of each first video segment based on the action change information. The process of extracting the object action and action period in each first video segment specifically includes: if the action complexity of the second video segment in the first video segment exceeds a preset threshold, then splitting each action of the object in the second video segment into multiple actions as different object actions, and determining the action period corresponding to each object action; otherwise, taking the action of the object in the second video segment as the object action, and determining the action period corresponding to the object action.

[0048] Among them, the complexity of the action can be judged according to the number of action points and the movement amplitude of each action point, wherein the movement amplitude can be the speed of the action point movement. Different numerical values ​​can be set for different action complexities by artificial setting. For example: set the threshold of the number of action points to 5, when the number of action points does not exceed 5, set the action complexity to 1, and when the number of action points exceeds 5, set the action complexity to 2. Similarly, when the movement speed of the action point exceeds the preset speed threshold, the action complexity is set to 2. For example, the speed threshold of the arm extension speed is set to 4m / s, and the action change information is that the arm is stretched upward and the arm extension speed is 5m / s, then the corresponding action complexity is 2. "1" can indicate that the action is relatively simple, and "2" can indicate that the action is relatively complex. Of course, different numerical values ​​can also be set to indicate the complexity of the action by combining the number of action points and the movement amplitude of each action point.

[0049] In implementation, before extracting the object actions and action periods in each first video segment, the action change information of the object in each first video segment is obtained by means of action detection, and the action change information includes the object action point and the corresponding action amplitude. Based on the action change information, the action complexity of each first video segment is judged by setting different values ​​for different action complexities as described above. When the action complexity of the second video segment in the first video segment exceeds a preset threshold, each action of the object in the second video segment is split into multiple actions as object actions, wherein each action can be split into multiple segments. And determine the action periods corresponding to each object action. When the action complexity of the second video segment in the first video segment does not exceed the preset threshold, the action of the object in the second video segment is taken as an object action as a whole, and the action period corresponding to the object action is determined. In other words, by splitting complex actions, the hardware load can be further reduced.

[0050] Step 104: import the first audio segment into the portion of the target action in the second video.

[0051] In implementation, after determining the first audio segment corresponding to the first object action, the time of the first audio segment is aligned with the action time corresponding to the target action in the second video, that is, the first audio segment is imported into the corresponding part of the second video.

[0052] In one embodiment, when there are multiple actions in the first video that are similar to or identical to the actions in the second video, the process of determining the first object action among the various object actions that matches the target action specifically includes the following contents: if there are at least two object actions among the various object actions that match the target action, then the action with the smallest timestamp is selected from the at least two object actions as the first object action.

[0053] In implementation, when there are at least two object actions that match the target action, the action with the smallest timestamp is selected from the at least two object actions as the first object action. In other words, selecting the action with the smallest timestamp for insertion can effectively reduce the situation where the insertion of actions with later timestamps causes the audio and video to be out of sync.

[0054] In one embodiment, when the second video does not have an action similar to or identical to the action in the first video, step 104 may specifically include: if there is no action in the object action that matches the target action, then selecting a second audio segment that matches the action time of the target action in the audio segment. Accordingly, step 105 may specifically include: importing the second audio segment into the part of the target action in the second video.

[0055] During implementation, when there is no action in the object action that matches the target action, a second audio segment that matches the action time of the target action is selected. The matching action time can mean that the action time of the first video and the action time of the second video are the same or similar. For example, if the action time of the first video is 3s and the action time of the second video is 3.1s, then the two action times are consistent. In other words, when the action performed by the user is different from any action in the first video, an audio segment of the same length can be imported by determining the duration of the action performed by the user, thereby reducing the situation where a certain section of the audio is silent when the second playback is imported, thereby improving the user experience.

[0056] The embodiment of the present application provides an audio import method, and the execution subject may be a device for the audio import method. The embodiment of the present application takes an audio import method device executing an audio import method as an example to illustrate the device for the audio import method provided by the embodiment of the present application.

[0057] like Figure 5 As shown, a device 500 for importing audio includes: a video acquisition module 501, used to obtain a first video containing a first audio; an audio determination module 502, used to obtain each object action in the first video, and the audio segment corresponding to each object action in the first audio; a target action recognition module 503, used to identify the target action in the second video when a second video to which the audio is to be imported is obtained; an audio segment determination module 504, used to determine the first object action in each object action that matches the target action and the first audio segment corresponding to the first object action; an audio segment importing module 505, used to import the first audio segment into the part of the target action in the second video.

[0058] Optionally, the audio determination module 502 is further used to split the first video into multiple first video segments with a preset accuracy; extract the object action and action period in each first video segment; and obtain the audio segment in the first audio with the same action period as each object action.

[0059] Optionally, the audio determination module 502 is further configured to split the first video into a plurality of first video segments according to at least one of a relevant attribute of the first video and a pause point in the first audio.

[0060] Optionally, the action change acquisition module is used to obtain the action change information of the object in each first video segment; the complexity judgment module is used to judge the action complexity of each first video segment based on the action change information; the audio determination module 502 is also used to, when the action complexity of the second video segment in the first video segment exceeds a preset threshold, split each action of the object in the second video segment into multiple actions as different object actions; determine the action time period corresponding to each object action; when the action complexity of the second video segment in the first video segment does not exceed the preset threshold, regard the action of the object in the second video segment as the object action; and determine the action time period corresponding to the object action.

[0061] Optionally, the audio segment determination module 504 is further configured to select a motion with a smallest timestamp from the at least two object motions as the first object motion when there are at least two object motions that match the target motion among the object motions.

[0062] Optionally, the audio segment determination module 504 is also used to select a second audio segment that matches the action time of the target action in the audio segment when there is no action in the object action that matches the target action; the audio segment import module 505 is also used to import the second audio segment into the part of the target action in the second video.

[0063] Optionally, the display module is used to generate and display a first prompt page based on the playback length of the first audio and the first video, the first prompt page being used to display the visual track of the first video and the audio track of the first audio; and to generate and display a second prompt page based on the object action and the paired audio segment.

[0064] Optionally, the video acquisition module 501 is also used to generate order-adjusted options corresponding to the template video through one or more template videos containing audio selected by the user in a preset operation interface; based on the adjustment request output by the user for the order-adjusted options, the template video after the order is adjusted is used as the first video and the audio is used as the first audio, to obtain the first video containing the first audio.

[0065] An audio import device in an embodiment of the present application may be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices other than a terminal. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It may also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiments of the present application.

[0066] In the embodiment of the present application, an audio import device may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0067] The audio import device provided in the embodiment of the present application can realize Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0068] Alternatively, if Figure 6 As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601 and a memory 602, wherein the memory 602 stores a program or instruction that can be executed on the processor 601, and when the program or instruction is executed by the processor 601, each step of the above-mentioned audio import method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0069] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0070] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0071] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0072] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0073] The input unit 1004 is used to obtain a first video including a first audio.

[0074] Processor 1010 is used to obtain each object action in the first video and the audio segment corresponding to each object action in the first audio; when a second video to be imported into the audio is obtained, identify the target action in the second video; determine the first object action that matches the target action among the object actions and the first audio segment corresponding to the first object action; and import the first audio segment into the part of the target action in the second video.

[0075] In the embodiment of the present application, a video with audio synchronization imported by a user is obtained, and each action in the video and the corresponding audio segment are determined. When the user follows a similar video with a fast or slow rhythm, by identifying actions similar to each action in the similar video, the corresponding audio segment is imported, so that the audio segment can be imported along with the user's fast or slow actions, so as to achieve the effect of audio and video synchronization of the captured video, thereby improving the user experience.

[0076] Optionally, the processor 1010 is further used to split the first video into multiple first video segments with a preset accuracy; extract the object action and action time period in each first video segment; and obtain the audio segment in the first audio with the same action time period as each object action.

[0077] Optionally, the processor 1010 is further configured to split the first video into a plurality of first video segments according to at least one of the related attributes of the first video and a pause point in the first audio, wherein the pause point represents any one or more of a discontinuity point, an extreme value point or an inflection point of the audio signal of the first audio.

[0078] Optionally, the interface unit 1008 is further configured to obtain motion change information of an object in each first video segment.

[0079] Processor 1010 is also used to determine the complexity of the movements of each first video segment based on the movement change information; if the complexity of the movement of the second video segment of the first video segment exceeds a preset threshold, each movement of the object in the second video segment is split into multiple movements as different object movements, and the movement time periods corresponding to the object movements are determined; otherwise, the movement of the object in the second video segment is taken as the object movement, and the movement time period corresponding to the object movement is determined.

[0080] Optionally, the processor 1010 is further configured to select an action with a smallest timestamp from the at least two object actions as the first object action if there are at least two object actions that match the target action among the object actions.

[0081] Optionally, the processor 1010 is further configured to select a second audio segment that matches the action time of the target action in the audio segment if there is no action in the object action that matches the target action; and import the second audio segment into the part of the target action in the second video.

[0082] Optionally, the display unit 1006 is used to display a first prompt page based on the playback length of the first audio and the first video, the first prompt page being used to display the visual track of the first video and the audio track of the first audio; and to display a second prompt page based on the object action and the paired audio segment.

[0083] Optionally, processor 1010 is also used to generate order-adjusted options corresponding to the template video through one or more template videos containing audio selected by a user in a preset operation interface; based on an adjustment request output by the user for the order-adjusted options, the template video after the order is adjusted is used as the first video and the audio is used as the first audio, to obtain a first video containing the first audio.

[0084] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0085] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0086] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0087] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned audio import method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0088] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0089] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned audio import method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0090] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0091] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned audio import method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0092] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0094] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An audio import method, characterized in that: The method comprises: Obtain a first video including a first audio; Acquire each object action in the first video and an audio segment corresponding to each object action in the first audio; When a second video to be imported with audio is obtained, identifying a target action in the second video; Determine a first object action among the object actions that matches the target action and a first audio segment corresponding to the first object action; The first audio segment is introduced into a portion of the target action in the second video.

2. The audio import method according to claim 1, characterized in that: The obtaining of each object action in the first video includes: Splitting the first video into a plurality of first video segments with a preset accuracy; Extracting object actions and action time periods in each of the first video segments; The obtaining of the audio segments corresponding to the respective object actions in the first audio includes: An audio segment in the first audio having the same action period as the actions of each object is obtained.

3. The audio import method according to claim 2, characterized in that: The step of splitting the first video into a plurality of first video segments with a preset accuracy includes: The first video is split into a plurality of first video segments according to at least one of the relevant attributes of the first video and a pause point in the first audio; wherein the pause point represents any one or more of an interruption point, an extreme point or an inflection point of an audio signal of the first audio.

4. The audio import method according to claim 2, characterized in that: Before extracting the object actions and action periods in each of the first video segments, the method further includes: Acquire motion change information of an object in each of the first video segments, where the motion change information includes an object motion point and a motion amplitude corresponding to the object motion point; Based on the action change information, determining the action complexity of each of the first video segments; The extracting the object action and action period in each of the first video segments includes: If the complexity of the action of the second video segment in the plurality of first video segments exceeds a preset threshold, splitting each action of the object in the second video segment into multiple actions as different object actions; Determine the action time periods corresponding to the respective object actions; otherwise, taking the action of the object in the second video segment as the object action; An action period corresponding to the object action is determined.

5. The audio import method according to claim 4, characterized in that: The matching of actions includes having the same number and position of action points, and the action amplitude deviation value of the action points at the same position does not exceed the preset deviation threshold.

6. The audio import method according to claim 1, characterized in that: The determining a first object action among the object actions that matches the target action includes: If there are at least two object actions among the object actions that match the target action, then an action with the smallest timestamp is selected from the at least two object actions as the first object action.

7. The audio import method according to claim 1, characterized in that: The determining a first object action among the object actions that matches the target action and a first audio segment corresponding to the first object action includes: If there is no action in the object action that matches the target action, selecting a second audio segment that matches the action time of the target action in the audio segment; The step of importing the first audio segment into a portion of the target action in the second video includes: The second audio segment is introduced into a portion of the target action in the second video.

8. The audio import method according to claim 1, characterized in that: Before obtaining each object action in the first video and the audio segment corresponding to each object action in the first audio, the method further includes: Generate and display a first prompt page according to the play length of the first audio and the first video, where the first prompt page is used to display the visual track of the first video and the audio track of the first audio; After obtaining each object action in the first video and the audio segment corresponding to each object action in the first audio, the method further includes: Based on the object action and the paired audio segment, a second prompt page is generated and displayed, wherein the second prompt page is used to display the corresponding position of the object action on the visual track, the corresponding position of the audio segment on the audio track, and the corresponding relationship between the object action and the audio segment.

9. The audio import method according to claim 1, characterized in that: The first video containing the first audio includes: Generate order-adjustable options corresponding to the template video by using one or more template videos containing audio selected by the user in a preset operation interface; Based on the adjustment request output by the user for the order-to-be-adjusted options, the template video after the order is adjusted is used as the first video and the audio is used as the first audio, so as to obtain the first video including the first audio.

10. An audio import device, characterized in that: The device comprises: A video acquisition module, used for acquiring a first video including a first audio; An audio determination module, used to obtain each object action in the first video and an audio segment corresponding to each object action in the first audio; A target action recognition module, configured to recognize a target action in a second video to be imported with audio when the second video is obtained; An audio segment determination module, used to determine a first object action among the various object actions that matches the target action and a first audio segment corresponding to the first object action; An audio segment importing module is used to import the first audio segment into the portion of the target action in the second video.

11. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the audio import method as described in any one of claims 1 to 9 are implemented.

Images