An action template generation method, device, equipment and storage medium

Abstract

The present disclosure provides a motion template generation method, device and equipment and storage medium. The method comprises the following steps: first, determining a key point position sequence corresponding to a target video, wherein the key point position sequence is used to represent the position information of a preset key point on a video frame picture in the target video with a time sequence relationship. Then, based on the angle change of the included angle formed by the preset key point in the key point position sequence, a target sequence frame is determined from the key point position sequence. Further, a motion template corresponding to the target video is generated based on the target sequence frame. It can be seen that the motion template can be automatically generated based on the angle change of the included angle in the key point position sequence of the target video, and the generation efficiency and accuracy of the motion template are improved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing, and in particular to a method, apparatus, device and storage medium for generating action templates. Background Technology

[0002] As living standards improve, people's demand for entertainment is becoming more and more diversified. For example, dance-following interactive systems have emerged as one form of entertainment. These systems can display the timing and posture descriptions of key movements based on pre-acquired motion templates and music, enabling users to follow along.

[0003] Currently, the motion templates in dance follow-up interactive systems are generated manually, which not only results in low generation efficiency but also compromises the accuracy of the motion template configuration. Summary of the Invention

[0004] To address or at least partially address the aforementioned technical problems, this disclosure provides an action template generation method that can automatically generate action templates based on the angle changes in the key point position sequence of a target video, thereby improving the efficiency and accuracy of action template generation.

[0005] In a first aspect, this disclosure provides a method for generating action templates, the method comprising:

[0006] Determine the key point location sequence corresponding to the target video; wherein, the key point location sequence is used to characterize the location information of preset key points on video frames with a time sequence relationship in the target video;

[0007] Based on the angle change of the included angle formed by the preset key points in the key point position sequence, the target sequence frame is determined from the key point position sequence;

[0008] Based on the target sequence frames, an action template corresponding to the target video is generated.

[0009] In one optional implementation, determining the target sequence frame from the key point position sequence based on the angular change of the angle formed by the preset key points in the key point position sequence includes:

[0010] Determine the first included angle in the sequence of key point positions; wherein the first included angle is determined by at least three of the preset key points;

[0011] The sequence frame corresponding to the point where the angular velocity of the first included angle crosses zero is determined from the sequence of key point positions and used as the target sequence frame.

[0012] In an optional implementation, before generating the motion template corresponding to the target video based on the target sequence frames, the method further includes:

[0013] Extract the replay time sequence from the audio of the target video; wherein, the replay time sequence is used to characterize the time point corresponding to the replay in the audio;

[0014] Based on the replay time series, the angle time series composed of the target sequence frames is filtered to obtain a filtered angle time series; wherein, the filtered angle time series includes the target sequence frames in the angle time series that are within a preset time interval where the replay in the replay time series is located;

[0015] Accordingly, generating the motion template corresponding to the target video based on the target sequence frames includes:

[0016] Based on the target sequence frames in the filtered angle time series, an action template corresponding to the target video is generated.

[0017] In one optional implementation, determining the sequence frame corresponding to the zero-crossing point of the angular velocity of the first included angle from the key point position sequence, as the target sequence frame, includes:

[0018] Determine whether the directions of the angular velocities of adjacent frames in the key point position sequence with respect to the first included angle are consistent;

[0019] If it is determined that the directions of the angular velocities of the adjacent frames with respect to the first included angle are inconsistent, then the previous or next frame in the adjacent frames is determined as the target sequence frame.

[0020] In one optional embodiment, the first included angle includes at least one of the following: the included angle between the neck and the left shoulder, the included angle between the neck and the right shoulder, the included angle between the right upper arm and the torso, the included angle between the left upper arm and the torso, the included angle between the right thigh and the torso, the included angle between the left thigh and the torso, the included angle between the right calf and the right thigh, the included angle between the left calf and the left thigh, and the included angle between the right thigh and the left thigh.

[0021] In one alternative implementation, the method is applied to a dance follow-up interactive system.

[0022] Secondly, this disclosure provides an action template generation apparatus, the apparatus comprising:

[0023] The first determining module is used to determine the key point position sequence corresponding to the target video; wherein, the key point position sequence is used to characterize the position information of preset key points on video frames with a time sequence relationship in the target video;

[0024] The second determining module is used to determine the target sequence frame from the key point position sequence based on the angle change of the included angle formed by the preset key points;

[0025] The generation module is used to generate an action template corresponding to the target video based on the target sequence frames.

[0026] Thirdly, this disclosure provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to implement the above-described method.

[0027] Fourthly, this disclosure provides an apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described above.

[0028] Fifthly, this disclosure provides a computer program product comprising a computer program / instruction that, when executed by a processor, implements the method described above.

[0029] The technical solution provided in this disclosure has at least the following advantages compared with the prior art:

[0030] This disclosure provides a method for generating motion templates. First, a keypoint position sequence corresponding to a target video is determined. This keypoint position sequence represents the positional information of preset keypoints on video frames with a temporal sequence relationship in the target video. Then, based on the angular changes of the angles formed by the preset keypoints in the keypoint position sequence, a target sequence frame is determined from the keypoint position sequence. Finally, a motion template corresponding to the target video is generated based on the target sequence frame. Therefore, this disclosure can automatically generate motion templates based on the angular changes of the angles in the keypoint position sequence of the target video, improving the efficiency and accuracy of motion template generation. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0032] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1A flowchart of an action template generation method provided in this embodiment of the disclosure;

[0034] Figure 2 A flowchart of another action template generation method provided in this disclosure embodiment;

[0035] Figure 3 This is a schematic diagram of the structure of an action template generation device provided in an embodiment of the present disclosure;

[0036] Figure 4 This is a schematic diagram of the structure of an action template generation device provided in an embodiment of the present disclosure. Detailed Implementation

[0037] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0038] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0039] As living standards improve, people's demand for entertainment is becoming more and more diversified. For example, dance-following interactive systems have emerged as one form of entertainment. These systems can display the timing and posture descriptions of key movements based on pre-acquired motion templates and music, enabling users to follow along.

[0040] Currently, the motion templates in dance follow-along interactive systems are generated manually, which is not only inefficient but also lacks accuracy. Therefore, improving the efficiency and accuracy of motion template generation is a pressing technical problem that needs to be solved.

[0041] To address this, this disclosure provides a method for generating motion templates. First, a keypoint position sequence corresponding to a target video is determined. This keypoint position sequence represents the positional information of preset keypoints on video frames with a temporal sequence relationship within the target video. Then, based on the angular changes of the angles formed by the preset keypoints in the keypoint position sequence, a target sequence frame is determined from the keypoint position sequence. Finally, a motion template corresponding to the target video is generated based on the target sequence frame. Therefore, this disclosure can automatically generate motion templates based on the angular changes of the angles in the keypoint position sequence of the target video, improving the efficiency and accuracy of motion template generation.

[0042] Based on this, the present disclosure provides a method for generating action templates, referring to... Figure 1 The flowchart below illustrates an action template generation method provided in this disclosure, which includes:

[0043] S101: Determine the sequence of key point locations corresponding to the target video.

[0044] Among them, the key point location sequence is used to characterize the location information of preset key points on video frames with time sequence relationships in the target video.

[0045] In this embodiment of the disclosure, the target video can be a video containing a series of actions, such as a dance video, a martial arts video, a yoga video, etc. A target video can include multiple video frames, wherein the multiple video frames have a temporal sequence relationship. For example, a target video includes 1 to N video frames arranged in chronological order.

[0046] In this embodiment of the disclosure, the subject performing a series of actions in the target video may include preset key points. For example, if the subject performing a series of actions in the target video is the human body, the preset key points may be human joints, such as the head, neck, right shoulder, left shoulder, right elbow, left elbow, spine, right hand, left hand, right knee, left knee, right foot, left foot, and other human joints.

[0047] In this embodiment, image processing is performed on each frame of the target video to obtain the position information of all preset key points on each frame. The position information may include the two-dimensional coordinates of the preset key points on the video frame. After obtaining the position information of all preset key points on each frame, for each preset key point, a key point position sequence is determined according to the time sequence of each frame in the target video. Furthermore, a key point position sequence corresponding to the target video is determined, whereby the key point position sequence represents the position information of preset key points on video frames with a time sequence relationship in the target video.

[0048] For example, taking a dance video as the target video, the dance video includes 1 to N frames with a time sequence relationship. Preset key points on each frame include human joint point a, human joint point b, and human joint point c. First, image processing is performed on the first frame to obtain the position information of human joint point a, human joint point b, and human joint point c. Image processing is then performed on the second frame to obtain the same position information. This process continues for the Nth frame. Then, according to the time sequence of frames 1 to N in the dance video, the position sequence A of human joint point a is determined; the position sequence B of human joint point b is determined; and the position sequence C of human joint point c is determined. Furthermore, position sequences A, B, and C can be identified as the key point position sequences corresponding to the dance video.

[0049] S102: Determine the target sequence frame from the key point position sequence based on the angle change of the included angle formed by preset key points in the key point position sequence.

[0050] In this embodiment of the disclosure, the preset key points can form several angles, and each angle corresponds to a local action. For example, if the preset key points include three human joints: head, neck, and left shoulder, then the three human joints of head, neck, and left shoulder can form an angle between the neck and left shoulder, and the angle between the neck and left shoulder corresponds to the local limb action of the neck and left shoulder.

[0051] In this embodiment of the disclosure, a target sequence frame is determined from the key point position sequence based on the angular change of the angle formed by preset key points in the key point position sequence. The target sequence frame is a video frame containing a key action. The angular change of the angle can characterize whether the local action corresponding to the preset key point forming the angle is a key action. If it is a key action, the video frame containing the key action can be determined as the target sequence frame. The target sequence frame is a collection of multiple video frames containing key actions.

[0052] In one alternative implementation, a first included angle in the sequence of key point positions is determined, wherein the first included angle is determined by at least three preset key points.

[0053] In this embodiment of the disclosure, an included angle is determined by at least three preset key points. Based on the key point position sequence determined in S101 above, at least three preset key points are selected and determined as the first included angle. For example, taking human joint points as the preset key points, based on the key point position sequence, if the head, neck, and left shoulder are selected as three human joint points, the included angle between the neck and the left shoulder can be determined as the first included angle in the key point position sequence; if the head, neck, and right shoulder are selected as three human joint points, the included angle between the neck and the right shoulder can be determined as the first included angle in the key point position sequence; and so on.

[0054] In one optional embodiment, the first included angle may include at least one of the following: the included angle between the neck and the left shoulder, the included angle between the neck and the right shoulder, the included angle between the right upper arm and the torso, the included angle between the left upper arm and the torso, the included angle between the right thigh and the torso, the included angle between the left thigh and the torso, the included angle between the right calf and the right thigh, the included angle between the left calf and the left thigh, and the included angle between the right thigh and the left thigh.

[0055] In one optional implementation, the sequence frame corresponding to the point where the angular velocity of the first included angle crosses zero is determined from the key point position sequence and used as the target sequence frame.

[0056] In this embodiment, the first included angle determined from the key point position sequence presents a changing curve on the time axis, and the rate of change of the first included angle can be obtained by differentiating it with respect to time. The zero-crossing point of the angular velocity of the included angle can characterize the change of the included angle. Specifically, the zero-crossing point of the angular velocity of the included angle means that if the rate of change of the included angle before a certain time point is opposite in sign to the rate of change of the included angle after that time point, then that time point is the zero-crossing point of the angular velocity. For example, if the angular velocity of the first included angle changes from positive to negative or from negative to positive, then the angular velocity of the first included angle has crossed zero. Furthermore, the sequence frame corresponding to the zero-crossing point of the angular velocity of the first included angle can be determined as the target sequence frame.

[0057] In this embodiment of the disclosure, after determining the target sequence frame corresponding to the first included angle, the second included angle is determined by at least three preset key points. Then, the target sequence frame corresponding to the zero point of the angular velocity of the second included angle is determined from the key point position sequence. In this way, the target sequence frames corresponding to each included angle are determined respectively, and the union of the target sequence frames corresponding to each included angle is taken as the final target sequence frame.

[0058] In one optional implementation, it is determined whether the directions of the angular velocities of adjacent frames with respect to the first included angle are consistent. If it is determined that the directions of the angular velocities of adjacent frames with respect to the first included angle are inconsistent, then the previous or next frame in the adjacent frames is determined as the target sequence frame.

[0059] In this embodiment of the disclosure, after determining the first included angle in the key point position sequence, it is determined whether the directions of the angular velocities of adjacent frames in the key point position sequence with respect to the first included angle are consistent. For example, if the direction of the angular velocity of the included angle changes from positive to negative or from negative to positive, then it is determined that the directions of the angular velocities of the included angle are inconsistent. If it is determined that the directions of the angular velocities of adjacent frames with respect to the first included angle are inconsistent, then the previous or next frame in the adjacent frames is determined as the target sequence frame.

[0060] For example, taking the angle between the neck and left shoulder as the first angle determined in the key point position sequence, in the first frame of the key point position sequence, the angular velocity direction of the angle between the neck and left shoulder is positive; in the second frame, the angular velocity direction of the angle between the neck and left shoulder is positive; in the third frame, the angular velocity direction of the angle between the neck and left shoulder is positive; and in the fourth frame, the angular velocity direction of the angle between the neck and left shoulder is negative. Therefore, the direction of the angular velocity of the angle between the neck and left shoulder is inconsistent between the adjacent third and fourth frames. Consequently, the third or fourth frame can be determined as the target sequence frame corresponding to the first angle.

[0061] S103: Generate an action template corresponding to the target video based on the target sequence frames.

[0062] In this embodiment of the disclosure, based on the union of the target sequence frames corresponding to each included angle determined in S102 above, an action template corresponding to the target video can be generated, wherein the action template corresponding to the target video includes the key actions of each included angle at the corresponding time point.

[0063] As can be seen, in the motion template generation method provided in this embodiment, firstly, a keypoint position sequence corresponding to the target video is determined, wherein the keypoint position sequence is used to characterize the position information of preset keypoints on video frames with a time sequence relationship in the target video. Then, based on the angle changes of the angles formed by the preset keypoints in the keypoint position sequence, a target sequence frame is determined from the keypoint position sequence. Furthermore, a motion template corresponding to the target video is generated based on the target sequence frame. Therefore, this embodiment can automatically generate motion templates based on the angle changes of the angles in the keypoint position sequence of the target video, improving the generation efficiency and accuracy of motion templates.

[0064] Since the target sequence frames are the union of all target sequence frames corresponding to the included angles, the motion templates generated based on these frames may contain too many key movements, leading to a poor user experience. Therefore, to make the motion templates for the target video more in sync with the music, some target sequence frames can be filtered out using the repetition times in the target video's audio. This simplifies the motion templates while maintaining their accuracy, making them more suitable for a wider range of users.

[0065] Based on the above embodiments, this disclosure also provides a method for generating action templates. For example... Figure 2 The diagram shows another action template generation method provided in this disclosure, which includes the following steps:

[0066] S201: Determine the sequence of key point locations corresponding to the target video.

[0067] Among them, the key point location sequence is used to characterize the location information of preset key points on video frames with time sequence relationships in the target video.

[0068] S202: Determine the first included angle in the key point position sequence.

[0069] The first included angle is determined by at least three preset key points.

[0070] S203: Determine the sequence frame corresponding to the point where the angular velocity of the first included angle crosses zero from the key point position sequence, and use it as the target sequence frame.

[0071] The specific processes of S201-S203 described above have been explained in detail in the above embodiments and will not be repeated here.

[0072] S204: Extract the replay time sequence from the audio of the target video.

[0073] Among them, the repeat time series is used to represent the time point corresponding to the repeat in the audio.

[0074] In this embodiment of the disclosure, the target video may include not only a series of actions, but also audio. The audio of the target video includes one or more replay time points. When the audio includes multiple replay time points, the multiple replay time points are arranged in chronological order to form a replay time sequence. The replay time sequence is used to characterize the time point corresponding to the replay in the audio.

[0075] In this embodiment of the disclosure, there are various ways to extract the replay time points in the audio of the target video. For example, based on the time domain energy density detection method, the time period (or time point) with high energy density is determined as the replay.

[0076] It should be noted that the execution order of S201-S203 and S204 described above is not limited in this embodiment.

[0077] S205: Based on the replay time series, filter the angle time series composed of target sequence frames to obtain the filtered angle time series.

[0078] The filtered angle time series includes target sequence frames within a preset time interval where the retake in the retake time series is located.

[0079] In this embodiment of the disclosure, the included angle time series composed of target sequence frames includes a set of target sequence frames corresponding to each included angle in the time series as determined based on the above embodiments. Based on the replay time series, the set of target sequence frames is filtered to obtain a filtered included angle time series. The filtered included angle time series includes target sequence frames within a preset time interval where the replay occurs in the replay time series.

[0080] It should be noted that the above-mentioned preset time interval may include within 1 second, within 2 seconds, and within 3 seconds. The specific range of the above-mentioned preset time interval is not limited in the embodiments of this disclosure.

[0081] For example, taking a dance video as the target video, the target video contains an audio segment with multiple stressed time points (such as stressed time point A, stressed time point B, stressed time point C, etc.). Based on stressed time point A, the set of target sequence frames is filtered. If target sequence frame a corresponding to the first angle is within a preset time interval (such as within 2 seconds) of stressed time point A, then target sequence frame a can be identified as a filtered target sequence frame; if target sequence frame b corresponding to the first angle is not within the preset time interval (such as within 2 seconds) of stressed time point A, then target sequence frame b does not belong to the filtered target sequence frames; and so on, the filtered target sequence frames can be obtained, forming a filtered angle time sequence.

[0082] S206: Generate an action template corresponding to the target video based on the target sequence frames in the filtered angle time series.

[0083] In this embodiment of the disclosure, based on the target sequence frames in the filtered angle time series obtained in S205 above, an action template corresponding to the target video can be generated, wherein the action template corresponding to the target video is only included in the target sequence frames near the retake time point.

[0084] The embodiments disclosed herein can combine audio replay time series to filter the angle time series formed by the target sequence frames to obtain filtered target sequence frames, and then generate an action template corresponding to the target video. This makes the generated action template more focused on audio replay, improves the rationality of the action template, and makes the action template corresponding to the target video more suitable for general users, thereby improving the user experience.

[0085] In one alternative implementation, the motion template generation method can be applied to a dance follow-up interactive system.

[0086] In this embodiment of the disclosure, the target video can be a dance video. Based on the above embodiments, a dance follow-along template is generated by combining the time series of the angular velocity crossing zero point and the audio replay time series. Users (such as users of the dance follow-along interactive system) can follow along with the dance according to the dance follow-along template and view real-time scores.

[0087] In this embodiment of the disclosure, the dance-following interaction system can be applied to a smart terminal, which may include smartphones, tablets, desktop computers, smartwatches, etc. The smart terminal may include devices such as a screen, camera, and microphone.

[0088] In this embodiment, when the generated dance choreography template is played on the screen of a smart terminal, the smart terminal's camera captures and records the user's choreography and detects the user's posture in real time. If detection fails, the user can be prompted, for example, by displaying the text "Sorry, you have gone off-frame, please start again!" on the screen. For each frame of the movement, the smart terminal compares the user's movement with the movement of the dance choreography template by angle similarity, obtaining an overall posture similarity score, which is then included in the real-time score. When the user follows the dance choreography template, the key movements of the template are displayed in a scrolling graphic format in the movement preview area on the smart terminal's screen, providing prompts to the user. For example, key movements are illustrated with stick figures in the movement preview area. Furthermore, during the user's choreography, the smart terminal's screen can display the user's score and the quality rating of the key movements in real time. After the choreography is finished, the user can export the choreography video to local storage.

[0089] In this embodiment of the disclosure, users can also import local videos (such as their favorite dance videos) into the dance follow-along interactive system, and the system will automatically generate corresponding follow-along video templates. Users can then follow along with the dance videos based on their favorite dance videos, which improves the user experience.

[0090] Based on the above method embodiments, this disclosure also provides an action template generation device 300, see reference. Figure 3 This is a schematic diagram of the structure of an action template generation device provided in an embodiment of the present disclosure. The action template generation device 300 includes:

[0091] The first determining module 301 is used to determine the key point position sequence corresponding to the target video; wherein, the key point position sequence is used to characterize the position information of preset key points on video frames with a time sequence relationship in the target video;

[0092] The second determining module 302 is used to determine the target sequence frame from the key point position sequence based on the angle change of the included angle formed by the preset key points;

[0093] The generation module 303 is used to generate an action template corresponding to the target video based on the target sequence frames.

[0094] In one optional implementation, the second determining module 302 includes:

[0095] A first determining submodule is used to determine a first included angle in the key point position sequence; wherein the first included angle is determined by at least three preset key points;

[0096] The second determining submodule is used to determine the sequence frame corresponding to the zero point of the angular velocity of the first included angle from the key point position sequence, as the target sequence frame.

[0097] In one optional embodiment, the apparatus further includes:

[0098] An extraction module is used to extract the replay time sequence from the audio of the target video; wherein the replay time sequence is used to characterize the time point corresponding to the replay in the audio;

[0099] A filtering module is used to filter the angled time sequence composed of the target sequence frames based on the replay time sequence to obtain a filtered angled time sequence; wherein, the filtered angled time sequence includes the target sequence frames in the angled time sequence that are within a preset time interval of the replay in the replay time sequence;

[0100] Accordingly, the generation module 303 includes:

[0101] A generation submodule is used to generate an action template corresponding to the target video based on the target sequence frames in the filtered angle time series.

[0102] In one optional implementation, the second determining submodule is specifically used for:

[0103] Determine whether the directions of the angular velocities of adjacent frames in the key point position sequence with respect to the first included angle are consistent;

[0104] If it is determined that the directions of the angular velocities of the adjacent frames with respect to the first included angle are inconsistent, then the previous or next frame in the adjacent frames is determined as the target sequence frame.

[0105] In one optional embodiment, the first included angle includes at least one of the following: the included angle between the neck and the left shoulder, the included angle between the neck and the right shoulder, the included angle between the right upper arm and the torso, the included angle between the left upper arm and the torso, the included angle between the right thigh and the torso, the included angle between the left thigh and the torso, the included angle between the right calf and the right thigh, the included angle between the left calf and the left thigh, and the included angle between the right thigh and the left thigh.

[0106] In one alternative implementation, the method is applied to a dance follow-up interactive system.

[0107] In the motion template generation apparatus provided in this embodiment, firstly, a keypoint position sequence corresponding to the target video is determined. This keypoint position sequence represents the position information of preset keypoints on video frames with a temporal sequence relationship in the target video. Then, based on the angular changes of the angles formed by the preset keypoints in the keypoint position sequence, a target sequence frame is determined from the keypoint position sequence. Furthermore, a motion template corresponding to the target video is generated based on the target sequence frame. Therefore, this embodiment can automatically generate motion templates based on the angular changes of the angles in the keypoint position sequence of the target video, improving the efficiency and accuracy of motion template generation.

[0108] In addition to the methods and apparatus described above, this disclosure also provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to implement the action template generation method described in this disclosure.

[0109] This disclosure also provides a computer program product, which includes a computer program / instruction. When the computer program / instruction is executed by a processor, it implements the action template generation method described in this disclosure.

[0110] In addition, this disclosure also provides an action template generation device 400, see [link to relevant documentation]. Figure 4 As shown, it may include:

[0111] The device includes a processor 401, a memory 402, an input device 403, and an output device 404. The number of processors 401 in the motion template generation device can be one or more. Figure 4Taking a processor as an example. In some embodiments of this disclosure, the processor 401, memory 402, input device 43, and output device 404 can be connected via a bus or other means, wherein, Figure 4 Taking the example of a connection between China and Israel via a bus.

[0112] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing of the action template generation device by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The input device 403 can be used to receive input digital or character information, and to generate signal inputs related to user settings and function control of the action template generation device.

[0113] Specifically in this embodiment, the processor 401 will load the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 will run the applications stored in the memory 402 to realize the various functions of the above-mentioned action template generation device.

[0114] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0115] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating action templates, characterized in that, The method includes: Determine the key point location sequence corresponding to the target video; wherein, the key point location sequence is used to characterize the location information of all preset key points on video frames with a time sequence relationship in the target video; Based on the angular change of the included angle formed by the preset key points in the key point position sequence, the target sequence frame is determined from the key point position sequence; wherein, the zero point of the angular velocity of the included angle is used to characterize the angular change of the included angle. Based on the target sequence frames, an action template corresponding to the target video is generated.

2. The method according to claim 1, characterized in that, The step of determining the target sequence frame from the key point position sequence based on the angular change of the included angle formed by the preset key points in the key point position sequence includes: Determine the first included angle in the sequence of key point positions; wherein the first included angle is determined by at least three of the preset key points; The sequence frame corresponding to the point where the angular velocity of the first included angle crosses zero is determined from the sequence of key point positions and used as the target sequence frame.

3. The method according to claim 1 or 2, characterized in that, Before generating the motion template corresponding to the target video based on the target sequence frames, the method further includes: Extract the replay time sequence from the audio of the target video; wherein, the replay time sequence is used to characterize the time point corresponding to the replay in the audio; Based on the replay time series, the angle time series composed of the target sequence frames is filtered to obtain a filtered angle time series; wherein, the filtered angle time series includes the target sequence frames in the angle time series that are within a preset time interval where the replay in the replay time series is located; Accordingly, generating the motion template corresponding to the target video based on the target sequence frames includes: Based on the target sequence frames in the filtered angle time series, an action template corresponding to the target video is generated.

4. The method according to claim 2, characterized in that, The step of determining the sequence frame corresponding to the point where the angular velocity of the first included angle crosses zero from the key point position sequence, as the target sequence frame, includes: Determine whether the directions of the angular velocities of adjacent frames in the key point position sequence with respect to the first included angle are consistent; If it is determined that the directions of the angular velocities of the adjacent frames with respect to the first included angle are inconsistent, then the previous or next frame in the adjacent frames is determined as the target sequence frame.

5. The method according to claim 2, characterized in that, The first included angle includes at least one of the following: the angle between the neck and the left shoulder, the angle between the neck and the right shoulder, the angle between the right upper arm and the torso, the angle between the left upper arm and the torso, the angle between the right thigh and the torso, the angle between the left thigh and the torso, the angle between the right lower leg and the right thigh, the angle between the left lower leg and the left thigh, and the angle between the right thigh and the left thigh.

6. The method according to claim 1, characterized in that, The method is applied to a dance follow-up interactive system.

7. An action template generation device, characterized in that, The device includes: The first determining module is used to determine the key point position sequence corresponding to the target video; wherein, the key point position sequence is used to characterize the position information of all preset key points on the video frames with a time sequence relationship in the target video; The second determining module is used to determine the target sequence frame from the key point position sequence based on the angle change of the included angle formed by the preset key points; wherein, the zero point of the angular velocity of the included angle is used to characterize the angle change of the included angle. The generation module is used to generate an action template corresponding to the target video based on the target sequence frames.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method as described in any one of claims 1-6.

9. A device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program / instruction that, when executed by a processor, implements the method as described in any one of claims 1-6.

Images