Video generation method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present disclosure provide a video generation method and apparatus, an electronic device, and a storage medium. The method comprises: obtaining action data and beat data on the basis of a target video, wherein the action data represents action characteristics of a target object in the target video, and the beat data represents beat characteristics of background music in the target video; generating a special effect sequence of the target video on the basis of the action data and the beat data, wherein the special effect sequence is used for triggering a corresponding video special effect at a special effect time point of the target video, and the special effect time point is jointly determined by the action data and the beat data; and generating a special effect video on the basis of the special effect sequence and the target video.

Description

Video generation method, device, electronic device and storage medium

[0001] This application claims priority to the Chinese invention patent application entitled “Video Generation Method, Device, Electronic Device and Storage Medium” and application number 2023109223201, filed on July 25, 2023. The entire contents of that application are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present disclosure relate to the field of Internet technology, and in particular to a video generation method, device, electronic device, and storage medium. Background Art

[0003] Editing videos with special effects at specific points, also known as video effects card points or card point video editing, involves setting corresponding sound and visual effects based on the content of the video, thereby matching the video content with the effects and better integrating the effects into the video.

[0004] In the prior art, the process of setting card point special effects for a video is usually to manually determine the special effect insertion point by the video editor user, and insert the special effect based on the special effect insertion point to generate a special effect video with the card point special effect.

[0005] However, the methods in the prior art have the problems of low efficiency in generating special effects videos and mismatch between the special effects at certain points and the video content.

[0006] Summary of the Invention

[0007] The embodiments of the present disclosure provide a video generation method, device, electronic device, and storage medium to overcome the problems of low efficiency in generating special effect videos and mismatch between the special effects at certain points and the video content.

[0008] In a first aspect, an embodiment of the present disclosure provides a video generation method, comprising: obtaining action data and beat data based on a target video, wherein the action data represents the action characteristics of a target object in the target video, and the beat data represents the beat characteristics of background music in the target video; generating a special effects sequence of the target video based on the action data and the beat data, wherein the special effects sequence is used to trigger corresponding video special effects at a special effects time point of the target video, and the special effects time point is jointly determined by the action data and the beat data; and generating a special effects video based on the special effects sequence and the target video.

[0009] In a second aspect, an embodiment of the present disclosure provides a video generating device, comprising: an acquisition module for obtaining action data and beat data based on a target video, wherein the action data represents the action characteristics of a target object in the target video, and the beat data represents the beat characteristics of background music in the target video; a sequence module for generating a special effects sequence of the target video based on the action data and the beat data, wherein the special effects sequence is used to trigger corresponding video special effects at a special effects time point of the target video, and the special effects time point is jointly determined by the action data and the beat data; a rendering module for generating a special effects video based on the special effects sequence and the target video.

[0010] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the video generation method described in the first aspect and various possible designs of the first aspect.

[0011] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the video generation method described in the first aspect and various possible designs of the first aspect is implemented.

[0012] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the video generation method described in the first aspect and various possible designs of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0014] FIG1 is a diagram of an application scenario of the video generation method provided by an embodiment of the present disclosure;

[0015] FIG2 is a flow chart of a video generation method according to an embodiment of the present disclosure;

[0016] FIG3 is a flowchart of a specific implementation of step S101 in the embodiment shown in FIG2 ;

[0017] FIG4 is a flowchart of a specific implementation method of step S1013 in the embodiment shown in FIG3 ;

[0018] FIG5 is a schematic diagram of determining a key action provided by an embodiment of the present disclosure;

[0019] FIG6 is a flowchart of another specific implementation of step S101 in the embodiment shown in FIG2 ;

[0020] FIG7 is a flowchart of a specific implementation method of step S1016 in the embodiment shown in FIG6 ;

[0021] FIG8 is a schematic diagram of a beat pattern provided by an embodiment of the present disclosure;

[0022] FIG9 is a second flow chart of a video generation method according to an embodiment of the present disclosure;

[0023] FIG10 is a flowchart of a specific implementation of step S203 in the embodiment shown in FIG9 ;

[0024] FIG11 is a schematic diagram of a process for determining a second special effect time point according to an embodiment of the present disclosure;

[0025] FIG12 is a flowchart of a specific implementation of step S204 in the embodiment shown in FIG9 ;

[0026] FIG13 is a structural block diagram of a video generating device provided by an embodiment of the present disclosure;

[0027] FIG14 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure;

[0028] FIG15 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0030] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0031] The following explains the application scenarios of the embodiments of the present disclosure:

[0032] FIG1 is a diagram of an application scenario of the video generation method provided by an embodiment of the present disclosure. The video generation method provided by an embodiment of the present disclosure can be applied to a video editing scenario. More specifically, it can be applied to an application scenario of adding card point special effects to a video. The method provided by an embodiment of the present disclosure can be applied to a terminal device or a server. As shown in FIG1 , taking a terminal device as an example, after obtaining a target video to be processed, the terminal device processes the target video using the video generation method provided by an embodiment of the present disclosure, extracts the motion data and beat data in the target video, and then generates a corresponding video special effect, i.e., a card point special effect, for the target video at time P based on the motion data and beat data that matches the content of the target video. The video special effect is then added to the target video to generate a special effect video with the above-mentioned card point special effect.

[0033] Before setting a card point special effect for a video, it is necessary to first determine the special effect time point of the card point special effect based on the video content. The above steps in the prior art are usually manually determined by the video editing user. Therefore, the setting process of the card point special effect is time-consuming and the generation efficiency of the special effect video is low. In other related technical solutions, the corresponding special effect time point is set by a single feature dimension such as the music rhythm or video picture in the target video, but the video content is composed of multiple content dimensions, such as music, background, task action, screen objects, etc., so that the generated card point special effect can only match one feature dimension corresponding to the (card point) target video, but cannot match in other dimensions, resulting in the problem of poor matching between the card point special effect and the video content. The embodiment of the present disclosure provides a video generation method to solve the above problems.

[0034] Referring to FIG2 , FIG2 is a flow chart of a video generation method provided by an embodiment of the present disclosure. The method of this embodiment can be applied to electronic devices such as terminal devices and servers. The video generation method includes:

[0035] Step S101: obtaining motion data and beat data according to a target video, wherein the motion data represents the motion characteristics of a target object in the target video, and the beat data represents the beat characteristics of background music in the target video.

[0036] For example, referring to the application scenario diagram shown in FIG1 , the execution subject of the method provided in the embodiment of the present disclosure is, for example, a terminal device, more specifically, a smartphone, tablet computer, etc. The terminal device obtains the target video by locally producing it or obtaining it from a network or other device. The target video includes an audio track and a video track, which are used to carry the sound and picture of the target video respectively. The specific video format of the target video is not limited here and can be configured as needed.

[0037] Furthermore, after obtaining a target video, the target video is parsed to obtain at least motion data and tempo data representing the target video's video content. The motion data represents the motion characteristics of the target subject in the target video, and the tempo data represents the tempo characteristics of the background music in the target video. Specifically, for example, the target video's content is a "street dance video" containing dancing video footage and synchronously playing background music. After processing the target video, the motion data and tempo data of the target video are obtained. The motion data represents the motion characteristics of the dancing character (target subject) in the "street dance video." More specifically, the motion data includes, for example, an image or video segment of the body movement pose_1 performed by character A in the "street dance video," as well as the video timestamp corresponding to the image or video segment. The tempo data, on the other hand, represents the melodic and rhythmic characteristics of the background music in the "street dance video," which may be the melodic beat. More specifically, the tempo data includes, for example, the strength and corresponding tempo of several beats in the background music of the "street dance video."

[0038] In a possible implementation, as shown in FIG3 , the specific implementation of obtaining the action data in step S101 includes:

[0039] Step S1011: Obtain at least two consecutive frames of video screenshots from the target video.

[0040] Step S1012: Obtain key structural points of the target object based on the video screenshot.

[0041] Step S1013: obtaining motion data according to the position coordinates of key structural points in at least two consecutive frames of video screenshots.

[0042] Exemplarily, after the target video is parsed, a multi-frame orderly video screenshot can be obtained, and then the target video is extracted and cut off based on the needs, for example, the original (all) video screenshot corresponding to a frame of the target video is extracted every 2 frames, while reducing the amount of data, ensuring the continuity in the content, and then, every two frames are used as a video screenshot group for subsequent processing, and the above-mentioned video screenshot group is a video screenshot of at least two frames of continuous content. Wherein, it is understandable that, based on the original (all) video screenshot corresponding to the target video, the larger the extraction interval is, the worse the continuity in the picture content is, and the specific extraction interval can be set as needed. Of course, in other possible implementations, it is also possible not to extract the video screenshot of the target video, and directly intercept at least two frames in all the video screenshots of the target video to form a video screenshot group, which is not specifically limited here. Afterwards, target recognition is performed on each of the at least two consecutive frames of video screenshots (video screenshot group) obtained above to obtain the key structural points of the target object in each video screenshot. For example, if the target object is a person, the key structural points may be the facial contour, limb contours, torso center point, etc. of the person. Afterwards, motion data is obtained based on the coordinates of the key structural points in each video screenshot in the video screenshot group. Since the position coordinates of the key structural points in each video frame will also change accordingly when the target object is in motion, the motion data representing the motion characteristics of the target object can be obtained based on the change vectors between the position coordinates of the key structural points in each video frame.

[0043] Exemplarily, the target object may include at least one of the following: torso, hand, head. That is, the motion features of the target object in the target video, such as at least one of body posture, hand movement, and head movement, can be characterized by the motion data. More specifically, for example, when the target object is the torso, the corresponding motion features include forward and backward movement, left and right movement, left and right tilting, etc. Among them, the motion data representing the forward and backward movement of the torso can be achieved by calculating the change in the area enclosed by the contour points (key structural points) of the torso (target object) on two adjacent frames of video screenshots. For example, the motion data includes a data sequence, and each sequence point on the data sequence represents the area enclosed by the contour points in a video screenshot. The change in each sequence point can be used to characterize the motion features of the forward and backward movement of the torso. Similarly, for the motion data representing left and right movement (motion features), it can be achieved by calculating the change in the center point (key structural point) of the torso (target object) on two adjacent frames of video screenshots. For motion data that characterizes left and right tilt (motion features), this can be achieved by calculating the change in the angle of the line connecting the shoulder center point (key structural point) and the hip center point (key structural point) of the torso (target object) in two adjacent frames of video screenshots. The specific steps are described one by one.

[0044] Furthermore, for example, when the target object is a head, the corresponding motion features include the appearance and disappearance of the head outline, etc. Among them, the motion data also includes the confidence corresponding to the key structural points, and the confidence is information that characterizes the credibility of the key structural points predicted based on the model. The specific implementation of the confidence is a prior art and will not be repeated here. The motion data characterizing the appearance and disappearance of the head outline can be achieved by calculating the confidence of the corresponding facial features and contour points (key structural points) of the head (target object) and the changes in the two adjacent frames of video screenshots. For another example, when the target object is a hand (arm), the corresponding motion features include stretching the hands left and right, opening and closing the arms, etc. Among them, the motion data characterizing the left and right stretching of the hands can be achieved by calculating the distance between the center point of the torso and the center point of the hand (key structural point). The motion data characterizing the opening and closing of the arms can be achieved by calculating the distance between the center points of the arms (key structural points), which will not be repeated here.

[0045] Furthermore, exemplarily, as shown in FIG4 , the specific implementation of step S1013 includes:

[0046] Step S1013A: Determine key actions based on the change characteristics of the position coordinates of key structural points in at least two consecutive frames of video screenshots.

[0047] Step S1013B: Obtain the action time point according to the video timestamp corresponding to the target video screenshot in the video screenshots with at least two consecutive frames of content.

[0048] Step S1013C: Generate action data based on the key actions and corresponding action time points.

[0049] Exemplarily, according to the change of the position coordinates of the key structure points in at least two frames of continuous video screenshots, the key action in the video screenshot can be determined. The key action can include a static action or a dynamic action, wherein the dynamic action has a corresponding action duration. Figure 5 is a schematic diagram of a method for determining a key action provided by an embodiment of the present disclosure. As shown in the figure, taking two frames of video screenshots as an example, in video screenshot A, key structure point P1 and key structure point P2 (shown as P1 and P2 in the figure) represent the left hand center point and right hand center point of the character, respectively. The coordinates of key structure point P1 and key structure point P2 in the video screenshot are located at pos_1 and Pos_2, respectively. Video screenshot B is the adjacent image frame after video screenshot A. In video screenshot B, the coordinates of key structure point P1 and key structure point P2 are located at pos_3 and Pos_4, respectively, that is, key structure point P1 moves from pos_1 to pos_3, and key structure point P2 moves from pos_2 to pos_4. Then the vector vec_1 representing the movement of the left hand is (pos_1, pos_3); the vector vec_2 representing the movement of the hand is (pos_2, pos_4); the set of vectors vec_1 and vec_2 is the vector representing the limb movement. Furthermore, when at least one of the vectors vec_1 and vec_2 is similar to the preset target action vector (the vector distance is less than the threshold), the above vector set (the limb movement represented) is determined to be a key action. Afterwards, the action time point can be obtained based on the video timestamp corresponding to the video screenshot A or the video screenshot B. In one possible implementation, when the video screenshots used to determine the action data are greater than N (N is, for example, an integer greater than 1), the video timestamp of the last frame of the N frames of video screenshots is used as the action time point. Finally, the key action (such as the set of vectors vec_1 and vec_2) and the corresponding action time point are used to generate action data.

[0050] Among them, it should be noted that when obtaining the position coordinates of key structural points, it is necessary to normalize the coordinate system of the position coordinates of the key structural points in different video screenshots, so that the position coordinates of the key structural points in different video screenshots are in the same coordinate system, avoiding the misdetection of key actions due to changes in video camera movement and improving the accuracy of special effects time points.

[0051] In this embodiment, the key action and the corresponding action time point are determined by combining the changing characteristics of the position coordinates of the key structural points with the preset target action vector, so as to capture the specific key action in the target video and obtain the action data. Then, the card point special effects are matched based on the action data so that the obtained card point special effects (special effects time points) can be accurately matched with the key action, thereby improving the effect of the card point special effects.

[0052] In another possible implementation, as shown in FIG6 , the specific implementation of obtaining the beat data in step S101 includes:

[0053] Step S1014: Obtain the audio track data of the target video.

[0054] Step S1015: Obtaining a beat time point and a corresponding beat value of at least one beat in the background music according to the audio track data, wherein the beat value represents the strength of the beat when the beat is triggered.

[0055] Step S1016: Obtaining beat data according to the beat time point and the corresponding beat value of at least one beat.

[0056] For example, after first parsing the target video, the target video's audio track data can be obtained. The specific implementation is based on existing technology and will not be repeated here. Then, the audio track data of the background music carrying the target video is processed based on the beat detection model to obtain the beats in the background music. Specifically, the beat time point and the corresponding beat value of the beat are obtained. The beat time point is the time point when the beat is triggered and is consistent with the video timestamp coordinate system of the target video. The beat value represents the strength and type of the beat. Then, based on the beat time point and corresponding beat value of at least one beat, the corresponding beat data representing the beat characteristics of the background music is determined.

[0057] Furthermore, in a possible implementation, as shown in FIG7 , a specific implementation of step S1016 includes:

[0058] Step S1016A: Acquire at least one set of beat sequences, where the beat sequence includes at least two beats with the same beat value.

[0059] Step S1016B: Obtain the beat pattern of the beat sequence according to the time intervals between the beat time points of each beat in the beat sequence.

[0060] Step S1016C: Obtain beat data based on the beat sequence having the target beat pattern.

[0061] Exemplarily, in the steps of this embodiment, by processing the audio track data, at least two beats can be obtained, and by combining the at least two beats, at least one group of beat sequences (patterns) can be obtained, wherein the beat sequence includes at least two beats with the same beat value, such as continuous drum beats. Afterwards, based on the time interval between the beat time points of each beat in the beat sequence, a beat pattern can be obtained, wherein the beat pattern is an implementation method for characterizing beat characteristics, and a beat sequence composed of multiple beats has a personalized beat pattern. The beat data contains multiple beat sequences, and different beat sequences can correspond to different beat patterns. Afterwards, based on a preset target beat pattern, a corresponding set of beat sequences with a target beat pattern is determined as beat data.

[0062] FIG8 is a schematic diagram of a beat pattern provided by an embodiment of the present disclosure. As shown in FIG8 , based on the processing of the audio track data, within an exemplary time interval T1-T2, beats B1, B2, B3, B4, B5, B6, and B7 are included. Among them, beats B1, B3, B5, and B7 are a group of beat sequences list_1 with the same beat values. Beats B2, B4, and B6 are another group of beat sequences list_2 with the same beat values. Afterwards, based on the intervals between the beats in the beat sequence, it is determined that the beat sequence list_2 is a beat sequence with a target beat pattern, and then based on the beat values ​​and corresponding beat time points of beats B2, B4, and B6 in the beat sequence list_2, beat data is generated.

[0063] In the steps of this embodiment, the corresponding beat sequence in the audio track data is filtered from the beat pattern dimension to obtain a beat sequence with a target beat pattern, and then in the subsequent process, the special effect time points of the generated card point special effect are made consistent with the beat time points in the beat sequence of the target beat pattern, thereby improving the visual effect of the card point special effect.

[0064] Step S102: Generate a special effects sequence of the target video based on the motion data and the beat data, wherein the special effects sequence is used to trigger corresponding video special effects at the special effects time point of the target video, and the special effects time point is determined by the motion data and the beat data.

[0065] Step S103: Generate a special effects video based on the special effects sequence and the target video.

[0066] For example, after obtaining the action data and beat data, the key action represented by the action data and the beat represented by the beat data are used to find the overlap point between the two. That is, the time period when the key action and beat are triggered simultaneously, i.e., the special effect time point. This allows the triggering moment of the generated card point special effect to simultaneously superimpose the key action in the target video and the important beat in the background music, so that the card point special effect matches the target video in multiple dimensions (image dimension and music dimension). Afterwards, based on the special effect time point represented by the special effect sequence, the corresponding special effect template is called, and special effects rendering is performed at the special effect time point position in the target video, generating a special effects video with the above-mentioned card point special effect.

[0067] In this embodiment, motion data and tempo data are obtained based on a target video, wherein the motion data represents the motion characteristics of a target object in the target video, and the tempo data represents the tempo characteristics of the background music in the target video. Based on the motion data and tempo data, a special effects sequence is generated for the target video. The special effects sequence is used to trigger corresponding video special effects at special effect time points in the target video, where the special effect time points are determined by both the motion data and tempo data. A special effects video is generated based on the special effects sequence and the target video. The motion data and tempo data extracted from the target video are used to represent the content of the target video in terms of both motion and tempo. Furthermore, the special effects sequence is determined by both the motion data and tempo data, and a special effects video is automatically generated based on the special effects sequence. The special effects time points in the special effects video that trigger the special effects match the motion effects and tempo characteristics of the target video. This automatically inserts special effects that match the video content, improving the efficiency of special effects video generation. Furthermore, it improves the matching degree between the special effects in the video and the video content, enhancing the visual quality of the special effects.

[0068] Referring to FIG9 , FIG9 is a second flow chart of the video generation method provided by the embodiment of the present disclosure. Based on the embodiment shown in FIG2 , this embodiment further refines step S102 , and the video generation method includes:

[0069] S201: obtaining motion data and beat data according to a target video, wherein the motion data represents motion characteristics of a target object in the target video, and the beat data represents beat characteristics of background music in the target video.

[0070] S202: Obtain at least one beat sequence according to the beat data, where the beat sequence includes at least one beat.

[0071] S203: Obtaining special effect time points according to the beat time points of the beats in the beat sequence and the action time points of the key actions in the action data.

[0072] For example, the special effect timing can be determined based on the beat timing of overlapping beats, where overlapping beats are the same beat as the action timing. Specifically, the beat timing corresponding to each beat in the beat data and the action timing of the key action in the action data are used to determine the time point where the beat timing and the action timing coincide as the special effect timing point. Subsequently, a card point special effect is added at this special effect timing point, so that the triggering of the card point special effect matches the key action and beat of the target object.

[0073] In a possible implementation, as shown in FIG10 , the special effect time point includes a first special effect time point, and a specific implementation of step S203 includes:

[0074] Step S2031: Obtain overlapping beats and non-overlapping beats in the beat sequence, where an overlapping beat is a beat with the same beat time point as the action time point.

[0075] Step S2032: Obtaining a first special effect time point corresponding to the first type of special effect according to the beat time point of the overlapping beat.

[0076] Step S2033: Obtain the first special effect time point corresponding to the second type of special effect according to the beat time point of the non-overlapping beats.

[0077] Exemplarily, the first special effect time point is a special effect time point that matches the beat and the corresponding static key action. For example, the key action is the action of "clapping" of a character in the target video. More specifically, it refers to the static action of the character "clapping" (touching both hands) in the target video. The corresponding beat is, for example, a sound effect such as drum beats in the background music. Therefore, the first special effect time point corresponds to a moment at which the card point special effect is triggered. Further, based on the previous introduction, the beat that is triggered simultaneously with the key action is a coincident beat, and correspondingly, the beat that is not triggered simultaneously with the key action is a non-coinciding beat. For the coincident beat, the beat time point of the coincident beat is set as the first special effect time point corresponding to the first type of special effect, and the beat time point of the non-coinciding beat is set as the first special effect time point corresponding to the second type of special effect. Exemplarily, the first type of special effect and the second type of special effect have different special effects. Specifically, for example, the first type of special effect is a full-screen special effect, and the second type of special effect is a local special effect. By setting the first type of special effect at the beat points where the action and beat are consistent, and setting the first type of special effect at the non-coincident beat points where the action and beat are inconsistent, we can differentiate the types of special effects, further highlighting the special effects at the beat points where the action and beat are consistent, and making the card point special effects present a layered visual effect.

[0078] For example, after obtaining the first special effects time point corresponding to the first type of special effects and the first special effects time point corresponding to the second type of special effects, the special effects sequence of the target video can be generated directly based on the set of the first special effects time point corresponding to the first type of special effects and the first special effects time point corresponding to the second type of special effects.

[0079] In another possible implementation, the special effect time point includes a second special effect time point, and a specific implementation of step S203 includes:

[0080] Step S2034: Obtain an overlapping beat sequence according to the beat time points of the beats in the beat sequence and the action time points of the key action in the action data. The overlapping beat sequence is a beat sequence whose sequence duration is the same as the action duration of the key action.

[0081] Step S2035: obtaining a second special effect time point according to the beat time point of the target beat in the overlapping beat sequence, wherein the target beat is the first beat or the last beat of the overlapping beat sequence.

[0082] Exemplarily, in another implementation, the special effect time point includes a second special effect time point, which is a special effect time point that matches a set of multiple beats and a corresponding dynamic key action. For example, the key action is the "somersault" action of the character in the target video, which lasts for 0.5 seconds. More specifically, the target action refers to the dynamic action of the character in the target video from "take-off-somersault-landing". The corresponding overlapping beat sequence is a beat sequence in the background music that lasts for 0.5 seconds at one end, such as a sound effect of continuous drum beats for 0.5 seconds. The overlapping beat sequence coincides with the duration of the key action. Afterwards, the beat time point of the target beat in the overlapping beat sequence, such as the beat time point of the first beat and the last beat in the overlapping beat sequence, is obtained as the second special effect time point. At the second special effect time point, the corresponding card point special effect is triggered.

[0083] Figure 11 is a schematic diagram of a process for determining the second special effect time point provided by an embodiment of the present disclosure. As shown in Figure 11, combined with the introduction in the previous steps, based on the same video timestamp, multiple beat sequences are obtained through beat data, such as the beat sequence list_1, beat sequence list_2 and beat sequence list_3 shown in the figure. Each beat sequence may include several beats with different numbers of beats. The start time point P1 and the end time point P2 corresponding to the key action are obtained through the action data. Afterwards, based on the beat time points of each beat and the start and end time points corresponding to the key action, a comparison is performed to determine that the beat sequence list_2 is a coincident beat sequence. Afterwards, the beat time point P1 corresponding to the first beat of the beat sequence list_2 is determined as the second special effect time point.

[0084] In this embodiment, the steps of the key action and beat sequence of the action are matched, and the overlapping beat sequence is determined. Then, based on the overlapping beat sequence, a second special effect time point is determined. At this second special effect time point, a corresponding card point special effect is applied, allowing the card point special effect to match the music or action. Compared to a single-point special effect matching scheme (single-point card point), the card point special effect has a higher degree of match with the target video and a better visual effect. At the same time, through subsequent steps, the special effect parameters of the card point special effect can be further set so that the duration of the special effect triggered at the second special effect time point is consistent with the duration of the key action and the overlapping beat sequence, further improving the match and visual effect of the card point special effect with the target video content.

[0085] In addition, in this embodiment, the steps of generating the first special effect time point and the second special effect time point can be performed separately or together. In the case of performing them together, they can be performed synchronously or asynchronously, and are not specifically limited here. After obtaining the first special effect time point and the second special effect time point, the first special effect time point and the second special effect time point are combined, and duplicate special effect time points are removed to generate a special effect sequence for the target video.

[0086] S204: Generate special effect parameters according to the motion data and / or beat data, where the special effect parameters represent the special effect of the video special effect triggered at the special effect time point.

[0087] For example, after obtaining the motion data and beat data, the special effect parameters of the stuttering effect can be further set based on the motion data and / or beat data. The special effect parameters represent the special effect effects of the stuttering effect triggered at the special effect time point, such as the duration of the stuttering effect, special effect size, special effect amplitude, special effect color, etc. The specific implementation method can be set as needed.

[0088] For example, as shown in FIG12 , the specific implementation of step S204 includes:

[0089] Step S2041: Generate a first special effect parameter according to the beat interval and corresponding beat value of at least two beats in the beat data.

[0090] Step S2042: Generate second special effect parameters according to the action amplitude and corresponding duration of the key action in the action data.

[0091] Step S2043: Generate special effect parameters according to the first special effect parameters and / or the second special effect parameters.

[0092] For example, first, based on the introduction in the previous embodiment, the beat data can include beat time points and corresponding beat values. Based on the beat intervals and corresponding beat values ​​of the beats in the beat data, a first special effect parameter for controlling the card point special effect can be generated. For example, the larger the beat interval, the larger the area of ​​the card point special effect; conversely, the smaller the beat interval, the smaller the area of ​​the card point special effect. Similarly, the action data includes the movement amplitude and corresponding duration of the key action. Based on the movement amplitude and corresponding duration of the key action, a second special effect parameter for controlling the card point special effect can be generated. For example, the longer the duration of the key action, the longer the duration of the card point special effect. The larger the movement amplitude of the key action, the warmer the color of the card point special effect. Subsequently, the first special effect parameter and / or the second special effect parameter are used as special effect parameters to set subsequent card point special effects, so that the special effect of the card point special effect matches the content (key action, beat) of the target video, thereby further improving the matching degree of the card point special effect with the target video and enhancing the visual expression. Of course, it is understandable that the special effect parameters may also include other implementation methods. The specific implementation method is set based on needs and will not be repeated here.

[0093] S205: Generate a special effects sequence for the target video according to the special effects time points and corresponding special effects parameters.

[0094] Afterwards, based on the generated special effects parameters and special effects time points, a special effects sequence of the target video is jointly generated, so that the special effects sequence can control the triggering time points and triggering effects of the card point special effects in the target video, thereby improving the matching degree and consistency between the card point special effects and the target video.

[0095] S206: Generate a special effects video based on the special effects sequence and the target video.

[0096] In this embodiment, the specific implementation methods of step S201 and step S206 are consistent with those of step S101 and step S103 in the above embodiment. For detailed discussion, please refer to the discussion of steps S101 and S103, which will not be repeated here.

[0097] Corresponding to the video generation method of the above embodiment, FIG13 is a structural block diagram of the video generation device provided by the embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Referring to FIG13, the video generation device 3 includes:

[0098] The acquisition module 31 is used to obtain motion data and beat data according to the target video, wherein the motion data represents the motion characteristics of the target object in the target video, and the beat data represents the beat characteristics of the background music in the target video.

[0099] The sequence module 32 is used to generate a special effects sequence of the target video based on the motion data and the beat data, wherein the special effects sequence is used to trigger corresponding video special effects at the special effects time point of the target video, and the special effects time point is determined by the motion data and the beat data.

[0100] The rendering module 33 is used to generate a special effects video based on the special effects sequence and the target video.

[0101] In one embodiment of the present disclosure, the sequence module 32 is specifically configured to: obtain at least one beat sequence based on the beat data, wherein the beat sequence includes at least one beat; obtain special effect time points based on the beat time points of the beats in the beat sequence and the action time points of the key action in the action data; and generate a special effect sequence for the target video based on the special effect time points.

[0102] In one embodiment of the present disclosure, the special effect time point includes a first special effect time point. When the sequence module 32 obtains the special effect time point based on the beat time points of the beats in the beat sequence and the action time points of the key action in the action data, it is specifically configured to: obtain overlapping beats in the beat sequence, where the overlapping beats are the same as the action time points; and obtain the first special effect time point corresponding to the beat sequence based on the beat time points of the overlapping beats.

[0103] In one embodiment of the present disclosure, the sequence module 32 is further used to: obtain non-overlapping beats in the beat sequence. When the sequence module 32 obtains the first special effect time point corresponding to the beat sequence based on the beat time point of the overlapping beat, it is specifically used to: obtain the first special effect time point corresponding to the first type of special effect based on the beat time point of the overlapping beat. Obtain the first special effect time point corresponding to the second type of special effect based on the beat time point of the non-overlapping beat. When the sequence module 32 generates the special effects sequence of the target video based on the special effects time point, it is specifically used to: generate the special effects sequence of the target video based on the set of the first special effects time point corresponding to the first type of special effect and the first special effects time point corresponding to the second type of special effect.

[0104] In one embodiment of the present disclosure, the special effect time point includes a second special effect time point. When the sequence module 32 obtains the special effect time point based on the beat time point of the beat in the beat sequence and the action time point of the key action in the action data, it is specifically used to: obtain a coincident beat sequence based on the beat time point of the beat in the beat sequence and the action time point of the key action in the action data, and the coincident beat sequence is a beat sequence whose sequence duration is the same as the action duration of the key action. According to the beat time point of the target beat in the coincident beat sequence, the second special effect time point is obtained, wherein the target beat is the first beat or the last beat of the coincident beat sequence.

[0105] In one embodiment of the present disclosure, the sequence module 32 is further configured to generate special effect parameters based on the motion data and / or beat data, where the special effect parameters represent the special effect of the video special effect triggered at the special effect time point. The sequence module 32 is specifically configured to generate a special effect sequence for the target video based on the special effect time points and the corresponding special effect parameters.

[0106] In one embodiment of the present disclosure, the special effect parameters include at least one of the following: special effect duration, special effect size, special effect amplitude, and special effect color.

[0107] In one embodiment of the present disclosure, when generating special effect parameters based on motion data and / or beat data, the sequence module 32 is specifically configured to: generate a first special effect parameter based on the beat interval and corresponding beat value of at least two beats in the beat data, and / or generate a second special effect parameter based on the motion amplitude and corresponding duration of a key motion in the motion data. The special effect parameter is generated based on the first special effect parameter and / or the second special effect parameter.

[0108] In one embodiment of the present disclosure, when obtaining motion data from a target video, the acquisition module 31 is specifically configured to: obtain at least two consecutive frames of video screenshots from the target video; obtain key structural points of the target object based on the video screenshots; and obtain the motion data based on the position coordinates of the key structural points in the at least two consecutive frames of video screenshots.

[0109] In one embodiment of the present disclosure, when obtaining action data based on the position coordinates of key structural points in at least two consecutive frames of video screenshots, the acquisition module 31 is specifically configured to: determine a key action based on the changing characteristics of the position coordinates of the key structural points in the at least two consecutive frames of video screenshots; obtain an action time point based on the video timestamp corresponding to a target video screenshot in the at least two consecutive frames of video screenshots; and generate action data based on the key action and the corresponding action time point.

[0110] In one embodiment of the present disclosure, when obtaining beat data based on a target video, the acquisition module 31 is specifically configured to: obtain audio track data of the target video; obtain the beat time point and corresponding beat value of at least one beat in the background music based on the audio track data, wherein the beat value represents the strength of the beat when the beat is triggered; and obtain beat data based on the beat time point and corresponding beat value of the at least one beat.

[0111] In one embodiment of the present disclosure, when obtaining beat data based on the beat time points and corresponding beat values ​​of at least one beat, the acquisition module 31 is specifically configured to: obtain at least one set of beat sequences, each of which includes at least two beats with the same beat value; obtain a beat pattern of the beat sequence based on the time intervals between the beat time points of each beat in the beat sequence; and obtain beat data based on the beat sequence having a target beat pattern.

[0112] The acquisition module 31, sequence module 32 and rendering module 33 are connected in sequence. The video generation device 3 provided in this embodiment can implement the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0113] FIG14 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. As shown in FIG14 , the electronic device 4 includes:

[0114] A processor 41 and a memory 42 communicatively connected to the processor 41 .

[0115] Memory 42 stores computer-executable instructions.

[0116] The processor 41 executes the computer-executable instructions stored in the memory 42 to implement the video generation method in the embodiments shown in FIG. 2 to FIG. 12 .

[0117] Optionally, the processor 41 and the memory 42 are connected via a bus 43 .

[0118] The relevant explanations can be understood by referring to the relevant descriptions and effects corresponding to the steps in the embodiments corresponding to Figures 2 to 12, and no further details will be given here.

[0119] An embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the video generation method provided in any of the embodiments corresponding to Figures 2 to 12 of the present disclosure.

[0120] The present disclosure provides a computer program product, including a computer program. When the computer program is executed by a processor, the video generation method shown in the embodiments of FIG. 2 to FIG. 12 is implemented.

[0121] Referring to FIG15 , there is shown a schematic diagram of the structure of an electronic device 900 suitable for implementing an embodiment of the present disclosure. The electronic device 900 may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG15 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0122] As shown in FIG15 , the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the electronic device 900 are also stored in the RAM 903. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0123] Typically, the following devices can be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc. An output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc. A storage device 908 including, for example, a magnetic tape, a hard disk, etc. And a communication device 909. The communication device 909 can allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 15 shows an electronic device 900 with various devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0124] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 909, or installed from the storage device 908, or installed from the ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0125] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0126] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0127] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0128] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0131] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0132] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] In a first aspect, according to one or more embodiments of the present disclosure, a video generation method is provided, comprising:

[0134] According to the target video, motion data and beat data are obtained, wherein the motion data represents the motion characteristics of the target object in the target video, and the beat data represents the beat characteristics of the background music in the target video; according to the motion data and beat data, a special effects sequence of the target video is generated, wherein the special effects sequence is used to trigger corresponding video special effects at the special effects time point of the target video, and the special effects time point is jointly determined by the motion data and the beat data; based on the special effects sequence and the target video, a special effects video is generated.

[0135] According to one or more embodiments of the present disclosure, generating the special effects sequence of the target video based on the action data and the beat data includes: obtaining at least one beat sequence based on the beat data, the beat sequence including at least one beat; obtaining special effects time points based on the beat time points of the beats in the beat sequence and the action time points of the key actions in the action data; and generating the special effects sequence of the target video based on the special effects time points.

[0136] According to one or more embodiments of the present disclosure, the special effect time point includes a first special effect time point, and the special effect time point is obtained according to the beat time point of the beat in the beat sequence and the action time point of the key action in the action data, including: obtaining the overlapping beats in the beat sequence, the overlapping beats being the beats whose beat time point is the same as the action time point; and obtaining the first special effect time point corresponding to the beat sequence according to the beat time point of the overlapping beats.

[0137] According to one or more embodiments of the present disclosure, the method further includes: obtaining non-overlapping beats in the beat sequence; obtaining the first special effect time point corresponding to the beat sequence based on the beat time point of the overlapping beats, including: obtaining the first special effect time point corresponding to the first type of special effect based on the beat time point of the overlapping beats; obtaining the first special effect time point corresponding to the second type of special effect based on the beat time point of the non-overlapping beats; generating the special effects sequence of the target video based on the special effects time points, including: generating the special effects sequence of the target video based on the set of the first special effects time point corresponding to the first type of special effect and the first special effects time point corresponding to the second type of special effect.

[0138] According to one or more embodiments of the present disclosure, the special effect time point includes a second special effect time point; the special effect time point is obtained according to the beat time point of the beat in the beat sequence and the action time point of the key action in the action data, including: obtaining an overlapping beat sequence according to the beat time point of the beat in the beat sequence and the action time point of the key action in the action data, the overlapping beat sequence being a beat sequence whose sequence duration is the same as the action duration of the key action; obtaining the second special effect time point according to the beat time point of the target beat in the overlapping beat sequence, wherein the target beat is the first beat or the last beat of the overlapping beat sequence.

[0139] According to one or more embodiments of the present disclosure, the method further includes: generating special effect parameters based on the action data and / or beat data, the special effect parameters representing the special effect effect of the video special effect triggered at the special effect time point; generating the special effect sequence of the target video based on the special effect time point includes: generating the special effect sequence of the target video based on the special effect time point and the corresponding special effect parameters.

[0140] According to one or more embodiments of the present disclosure, the special effect parameters include at least one of the following: special effect duration, special effect size, special effect amplitude, and special effect color.

[0141] According to one or more embodiments of the present disclosure, generating special effect parameters based on the action data and / or beat data includes: generating a first special effect parameter based on the beat interval and the corresponding beat value of at least two beats in the beat data; and / or generating a second special effect parameter based on the action amplitude and the corresponding duration of the key action in the action data; generating the special effect parameter based on the first special effect parameter and / or the second special effect parameter.

[0142] According to one or more embodiments of the present disclosure, obtaining action data based on a target video includes: obtaining video screenshots with at least two continuous frames of content from the target video; obtaining key structural points of a target object based on the video screenshots; and obtaining the action data based on the position coordinates of the key structural points in the video screenshots with at least two continuous frames of content.

[0143] According to one or more embodiments of the present disclosure, obtaining the action data based on the position coordinates of the key structural points in the video screenshots of the at least two continuous frames of content includes: determining the key action based on the change characteristics of the position coordinates of the key structural points in the video screenshots of the at least two continuous frames of content; obtaining the action time point based on the video timestamp corresponding to the target video screenshot in the video screenshots of the at least two continuous frames of content; and generating the action data based on the key action and the corresponding action time point.

[0144] According to one or more embodiments of the present disclosure, beat data is obtained based on a target video, including: obtaining audio track data of the target video; obtaining the beat time point and corresponding beat value of at least one beat in the background music based on the audio track data, wherein the beat value represents the strength of the beat when the beat is triggered; and obtaining the beat data based on the beat time point and corresponding beat value of at least one beat.

[0145] According to one or more embodiments of the present disclosure, obtaining the beat data based on the beat time point and the corresponding beat value of at least one of the beats includes: obtaining at least one group of beat sequences, the beat sequence including at least two beats with the same beat value; obtaining the beat pattern of the beat sequence based on the time interval between the beat time points of each beat in the beat sequence; and obtaining the beat data based on the beat sequence having a target beat pattern.

[0146] In a second aspect, according to one or more embodiments of the present disclosure, a video generating apparatus is provided, including:

[0147] An acquisition module, configured to obtain motion data and beat data according to a target video, wherein the motion data represents motion characteristics of a target object in the target video, and the beat data represents beat characteristics of background music in the target video;

[0148] a sequence module, configured to generate a special effects sequence for the target video based on the motion data and the beat data, wherein the special effects sequence is configured to trigger corresponding video special effects at special effects time points in the target video, the special effects time points being determined by both the motion data and the beat data;

[0149] A rendering module is used to generate a special effects video based on the special effects sequence and the target video.

[0150] According to one or more embodiments of the present disclosure, the sequence module is specifically used to: obtain at least one beat sequence based on the beat data, wherein the beat sequence includes at least one beat; obtain special effect time points based on the beat time points of the beats in the beat sequence and the action time points of the key actions in the action data; and generate a special effect sequence of the target video based on the special effect time points.

[0151] According to one or more embodiments of the present disclosure, the special effect time point includes a first special effect time point. When the sequence module obtains the special effect time point based on the beat time point of the beat in the beat sequence and the action time point of the key action in the action data, it is specifically used to: obtain the overlapping beats in the beat sequence, the overlapping beats being the beats whose beat time point is the same as the action time point; and obtain the first special effect time point corresponding to the beat sequence based on the beat time point of the overlapping beats.

[0152] According to one or more embodiments of the present disclosure, the sequence module is also used to: obtain non-overlapping beats in the beat sequence; when the sequence module obtains the first special effect time point corresponding to the beat sequence according to the beat time point of the overlapping beat, the sequence module is specifically used to: obtain the first special effect time point corresponding to the first type of special effect according to the beat time point of the overlapping beat; obtain the first special effect time point corresponding to the second type of special effect according to the beat time point of the non-overlapping beat; when the sequence module generates the special effects sequence of the target video according to the special effects time point, the sequence module is specifically used to: generate the special effects sequence of the target video according to the set of the first special effects time point corresponding to the first type of special effect and the first special effects time point corresponding to the second type of special effect.

[0153] According to one or more embodiments of the present disclosure, the special effect time point includes a second special effect time point; when the sequence module obtains the special effect time point based on the beat time point of the beat in the beat sequence and the action time point of the key action in the action data, it is specifically used to: obtain an overlapping beat sequence based on the beat time point of the beat in the beat sequence and the action time point of the key action in the action data, and the overlapping beat sequence is a beat sequence whose sequence duration is the same as the action duration of the key action; obtain the second special effect time point based on the beat time point of the target beat in the overlapping beat sequence, wherein the target beat is the first beat or the last beat of the overlapping beat sequence.

[0154] According to one or more embodiments of the present disclosure, the sequence module is further used to: generate special effect parameters based on the action data and / or beat data, and the special effect parameters represent the special effect effect of the video special effect triggered at the special effect time point; the sequence module is specifically used to: generate the special effect sequence of the target video based on the special effect time point and the corresponding special effect parameters.

[0155] According to one or more embodiments of the present disclosure, the special effect parameters include at least one of the following: special effect duration, special effect size, special effect amplitude, and special effect color.

[0156] According to one or more embodiments of the present disclosure, when the sequence module generates special effect parameters based on the action data and / or beat data, it is specifically used to: generate a first special effect parameter based on the beat interval and the corresponding beat value of at least two beats in the beat data; and / or generate a second special effect parameter based on the action amplitude and the corresponding duration of the key action in the action data; generate the special effect parameter based on the first special effect parameter and / or the second special effect parameter.

[0157] According to one or more embodiments of the present disclosure, when the acquisition module obtains action data based on the target video, it is specifically used to: obtain video screenshots with at least two continuous frames of content from the target video; obtain key structural points of the target object based on the video screenshots; and obtain the action data based on the position coordinates of the key structural points in the video screenshots with at least two continuous frames of content.

[0158] According to one or more embodiments of the present disclosure, when the acquisition module obtains the action data based on the position coordinates of the key structural points in the video screenshots with at least two continuous frames of content, it is specifically used to: determine the key action based on the change characteristics of the position coordinates of the key structural points in the video screenshots with at least two continuous frames of content; obtain the action time point based on the video timestamp corresponding to the target video screenshot in the video screenshots with at least two continuous frames of content; and generate the action data based on the key action and the corresponding action time point.

[0159] According to one or more embodiments of the present disclosure, when the acquisition module obtains beat data based on the target video, it is specifically used to: obtain the audio track data of the target video; obtain the beat time point and corresponding beat value of at least one beat in the background music based on the audio track data, wherein the beat value represents the strength of the beat when the beat is triggered; obtain the beat data based on the beat time point and corresponding beat value of at least one beat.

[0160] According to one or more embodiments of the present disclosure, when the acquisition module obtains the beat data based on the beat time point and the corresponding beat value of at least one of the beats, it is specifically used to: obtain at least one group of beat sequences, the beat sequence including at least two beats with the same beat value; obtain the beat pattern of the beat sequence based on the time interval between the beat time points of each beat in the beat sequence; and obtain the beat data based on the beat sequence with a target beat pattern.

[0161] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0162] The memory stores computer-executable instructions;

[0163] The processor executes the computer-executable instructions stored in the memory to implement the video generation method described in the first aspect and various possible designs of the first aspect.

[0164] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the video generation method described in the first aspect and various possible designs of the first aspect is implemented.

[0165] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the video generation method described in the first aspect and various possible designs of the first aspect.

[0166] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0167] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0168] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A video generation method, comprising: According to the target video, motion data and beat data are obtained, wherein the motion data represents the motion characteristics of the target object in the target video, and the beat data represents the beat characteristics of the background music in the target video; Generate a special effects sequence of the target video according to the motion data and the beat data, wherein the special effects sequence is used to trigger a corresponding video special effect at a special effects time point of the target video, and the special effects time point is determined by the motion data and the beat data; A special effects video is generated based on the special effects sequence and the target video.

2. The method according to claim 1, wherein generating a special effects sequence of the target video according to the motion data and the beat data comprises: According to the beat data, obtaining at least one beat sequence, wherein the beat sequence includes at least one beat; Obtaining a special effect time point according to the beat time point of the beat in the beat sequence and the action time point of the key action in the action data; A special effects sequence of the target video is generated according to the special effects time points.

3. The method according to claim 2, wherein the special effect time point includes a first special effect time point, and obtaining the special effect time point according to the beat time point of the beat in the beat sequence and the action time point of the key action in the action data comprises: Obtaining the coincident beats in the beat sequence, wherein the coincident beats are beats whose beat time points are the same as the action time points; According to the beat time point of the overlapping beats, the first special effect time point corresponding to the beat sequence is obtained.

4. The method according to claim 3, wherein the method further comprises: Obtaining non-overlapping beats in the beat sequence; The step of obtaining the first special effect time point corresponding to the beat sequence according to the beat time point of the overlapping beats includes: Obtaining a first special effect time point corresponding to a first type of special effect according to the beat time point of the overlapping beats; According to the beat time point of the non-coincident beats, the first special effect time corresponding to the second type of special effect is obtained. interval; Generating the special effects sequence of the target video according to the special effects time point includes: A special effects sequence of the target video is generated according to a set of first special effects time points corresponding to the first type of special effects and first special effects time points corresponding to the second type of special effects.

5. The method according to claim 2, wherein the special effect time point comprises a second special effect time point; The obtaining of special effect time points according to the beat time points of the beats in the beat sequence and the action time points of the key actions in the action data comprises: According to the beat time points of the beats in the beat sequence and the action time points of the key action in the action data, a coincident beat sequence is obtained, wherein the coincident beat sequence is a beat sequence whose sequence duration is the same as the action duration of the key action; The second special effect time point is obtained according to the beat time point of the target beat in the overlapping beat sequence, wherein the target beat is the first beat or the last beat of the overlapping beat sequence.

6. The method according to claim 2, wherein the method further comprises: Generate special effect parameters according to the motion data and / or beat data, wherein the special effect parameters represent the special effect of the video special effect triggered at the special effect time point; Generating the special effects sequence of the target video according to the special effects time point includes: A special effects sequence of the target video is generated according to the special effects time points and corresponding special effects parameters.

7. The method according to claim 6, wherein the special effect parameter comprises at least one of the following: Special effect duration, special effect size, special effect amplitude, and special effect color.

8. The method according to claim 6, wherein generating special effect parameters according to the motion data and / or beat data comprises: Generate a first special effect parameter according to a beat interval and a corresponding beat value of at least two beats in the beat data; and / or, generating a second special effect parameter according to the action amplitude and the corresponding duration of the key action in the action data; The special effect parameter is generated according to the first special effect parameter and / or the second special effect parameter.

9. The method according to claim 1, wherein obtaining motion data according to the target video comprises: Acquire at least two video screenshots with continuous content from the target video; According to the video screenshot, obtain key structural points of the target object; The action data is obtained according to the position coordinates of the key structural points in the video screenshots of at least two continuous frames of content.

10. The method according to claim 9, wherein obtaining the motion data according to the position coordinates of the key structural points in the video screenshots of the at least two consecutive frames of content comprises: Determine the key action according to the change characteristics of the position coordinates of the key structural points in the video screenshots of at least two consecutive frames; Obtaining an action time point according to a video timestamp corresponding to a target video screenshot in the at least two video screenshots with continuous content; The action data is generated according to the key action and the corresponding action time point.

11. The method according to claim 1, wherein obtaining the beat data according to the target video comprises: Obtaining the audio track data of the target video; According to the audio track data, a beat time point and a corresponding beat value of at least one beat in the background music are obtained, wherein the beat value represents the strength of the beat when the beat is triggered; The beat data is obtained according to a beat time point of at least one of the beats and a corresponding beat value.

12. The method according to claim 11, wherein obtaining the beat data according to the beat time point and the corresponding beat value of at least one of the beats comprises: Acquire at least one set of beat sequences, wherein the beat sequence includes at least two beats with the same beat value; Obtaining a beat pattern of the beat sequence according to the time intervals between the beat time points of each beat in the beat sequence; The beat data is obtained based on a beat sequence having a target beat pattern.

13. A video generating device, comprising: An acquisition module, used to obtain motion data and beat data according to a target video, wherein the motion data represents motion characteristics of a target object in the target video, and the beat data represents beat characteristics of background music in the target video; A sequence module is used to generate a special effect sequence of the target video according to the action data and the beat data, wherein the special effect sequence is used to trigger the corresponding special effect time point of the target video Video special effects, the special effects time point is determined by the action data and the beat data; A rendering module is used to generate a special effects video based on the special effects sequence and the target video.

14. An electronic device, comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the video generating method according to any one of claims 1 to 12.

15. A computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the video generation method according to any one of claims 1 to 12 is implemented.

16. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 12.

Images