An action video synthesis method and system, a storage medium and a terminal device

Abstract

Embodiments of the present application disclose a motion video synthesis method and system, a storage medium and a terminal device, and are applied to the technical field of information processing based on artificial intelligence. The motion video synthesis system divides a piece of audio data into multiple audio data subsegments, respectively obtains first feature information corresponding to each of the multiple audio data subsegments, determines second feature information of preset motion subsegments, selects multiple motion subsegments from the preset motion subsegments according to the first feature information and the second feature information to form a motion sequence to be synthesized, sorts the multiple motion subsegments in a certain order, and then synthesizes the multiple motion subsegments in the motion sequence to be synthesized into a motion synthesis video. The motion sequence to be synthesized is formed by the constraint of the fine-grained audio data subsegments, which is used to form the motion synthesis video, effectively controls the final motion synthesis video, and thus makes the motion synthesis video meet the diversified needs of users.

Description

Technical Field

[0001] This invention relates to the field of information processing technology based on artificial intelligence, and in particular to a method, system, storage medium, and terminal device for synthesizing motion videos. Background Technology

[0002] Music-driven dance synthesis tasks primarily focus on learning the underlying connections between music and dance, mapping the features of the music input based on these connections, and generating corresponding high-quality dance movements. This allows any piece of music to be presented to the user in a vivid way, providing a great user experience.

[0003] Existing dance synthesis methods mainly include generating a dance video to music based on a generative model (such as a dance synthesis model based on autoencoders and graph convolutional networks). However, this method can only effectively generate short-duration dance videos. Its performance is often unsatisfactory for generating long-duration (more than ten seconds) dance videos, and the generated dance movements always degenerate into simple repetitions or frozen states.

[0004] Another existing dance synthesis method may include a music-driven dance synthesis method based on motion unit graphs, which can combine sub-segments of multiple dance videos into a whole dance video. However, in this method, the construction of motion unit graphs depends on the annotation of data by professional artists, which requires a lot of human and material resources, thus limiting its ability to expand dance data and making it difficult to improve the diversity of dance.

[0005] In summary, existing dance synthesis methods cannot effectively control the resulting dance videos, making it impossible for synthesized dances to meet the diverse needs of users. Summary of the Invention

[0006] This invention provides a method, system, storage medium, and terminal device for synthesizing motion videos, enabling fine-grained control over the formation of synthesized motion videos.

[0007] One embodiment of the present invention provides a motion video synthesis method, comprising:

[0008] The audio data to be processed is divided into multiple audio data segments;

[0009] The first feature information of the plurality of audio data segments is obtained respectively, and the second feature information of the preset action segments is determined;

[0010] Based on the first feature information of the plurality of audio data segments and the second feature information of the preset action segments, a plurality of action segments are selected from the preset action segments to form an action sequence to be synthesized. The action sequence to be synthesized includes action segments arranged in a certain order and associated with the plurality of audio data segments respectively.

[0011] Multiple action segments from the action sequence to be synthesized are combined to obtain the action-synthesized video of the audio data to be processed.

[0012] Another aspect of this invention provides a motion video synthesis system, comprising:

[0013] An audio segment unit is used to divide the audio data to be processed into multiple audio data segments.

[0014] The feature acquisition unit is used to acquire the first feature information of the plurality of audio data segments respectively, and to determine the second feature information of the preset action sub-segment;

[0015] The sequence determination unit is used to select multiple action segments from the preset action segments based on the first feature information of the multiple audio data segments and the second feature information of the preset action segments to form an action sequence to be synthesized. The action sequence to be synthesized includes action segments arranged in a certain order and associated with the multiple audio data segments respectively.

[0016] The synthesis unit is used to synthesize multiple action sub-segments in the action sequence to be synthesized, so as to obtain the action-synthesized video of the audio data to be processed.

[0017] Another aspect of the present invention provides a computer-readable storage medium storing a plurality of computer programs adapted for loading by a processor and executing the motion video synthesis method as described in one aspect of the present invention.

[0018] Another embodiment of the present invention provides a terminal device, including a processor and a memory;

[0019] The memory is used to store multiple computer programs, which are loaded and executed by a processor as described in one aspect of the motion video synthesis method of the present invention; the processor is used to implement each of the multiple computer programs.

[0020] As can be seen, in the method of this embodiment, the motion video synthesis system divides an audio data segment into multiple audio data segments, obtains the first feature information corresponding to each of the multiple audio data segments to be processed, and determines the second feature information of preset motion sub-segments. Then, based on the first and second feature information, it selects multiple motion sub-segments from the preset motion sub-segments to form a sequence of actions to be synthesized. These multiple motion sub-segments are sorted in a certain order, and then the multiple motion sub-segments in the sequence of actions to be synthesized are synthesized into a motion-synthesized video. In this process, an audio data segment is divided into fine-grained audio data segments, and then the sequence of actions to be synthesized can be formed through the constraints of the fine-grained audio data segments to form a motion-synthesized video. This achieves effective control over the final motion-synthesized video, thereby enabling the motion-synthesized video to meet the diverse needs of users. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of a motion video synthesis method provided in an embodiment of the present invention;

[0023] Figure 2 This is a flowchart of a motion video synthesis method provided in one embodiment of the present invention;

[0024] Figure 3 This is a flowchart of a method for determining a sequence of actions to be synthesized in one embodiment of the present invention;

[0025] Figure 4 This is a flowchart of a method for training a feature extraction model in one embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of PC-Syn in one application embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of M2D-Align in one application embodiment of the present invention;

[0028] Figure 7 This is a flowchart of a method for generating a dance synthesis video in one application embodiment of the present invention;

[0029] Figure 8 This is a schematic diagram illustrating the determination of the action sequence to be synthesized in one application embodiment of the present invention;

[0030] Figure 9 This is a schematic diagram of anchor point posture information and dance synthesis video in one application embodiment of the present invention;

[0031] Figure 10 This is a schematic diagram of a distributed system to which the motion video synthesis method of this invention is applied in another application embodiment of the invention;

[0032] Figure 11 This is a schematic diagram of the block structure in another application embodiment of the present invention;

[0033] Figure 12 This is a schematic diagram of the logical structure of an action video synthesis system provided in an embodiment of the present invention;

[0034] Figure 13 This is a schematic diagram of the logical structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] This invention provides a method for synthesizing motion videos, which mainly involves taking any audio data segment and obtaining a corresponding motion-synthesized video, such as a dance video or a martial arts video, etc., specifically as follows: Figure 1 As shown in this embodiment of the invention, the motion video synthesis system can perform the following steps:

[0038] The audio data to be processed is divided into multiple audio data segments;

[0039] The first feature information of the plurality of audio data segments is obtained respectively, and the second feature information of the preset action segments is determined;

[0040] Based on the first feature information of the plurality of audio data segments and the second feature information of the preset action segments, a plurality of action segments are selected from the preset action segments to form an action sequence to be synthesized. The action sequence to be synthesized includes action segments arranged in a certain order and associated with the plurality of audio data segments respectively.

[0041] Multiple action segments from the action sequence to be synthesized are combined to obtain the action-synthesized video of the audio data to be processed.

[0042] In practical applications, the above-mentioned motion video synthesis system can be used in, but is not limited to, any of the following types of terminal devices: mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc.

[0043] It should be noted that the first feature information of the audio data segment and the second feature information of the action segment can be obtained through a pre-set feature extraction model. This pre-set feature extraction model is a machine learning model based on artificial intelligence. Artificial intelligence (AI) is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0044] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0045] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0046] In this process, an audio data segment is divided into fine-grained audio data segments. The constraints of these fine-grained audio data segments can then be used to form a sequence of actions to be synthesized, which is then used to create an action-synthesized video. This achieves effective control over the final action-synthesized video, thereby enabling the action-synthesized video to meet the diverse needs of users.

[0047] One embodiment of the present invention provides a motion video synthesis method, mainly a method executed by a motion video synthesis system, the flowchart of which is shown below. Figure 2 As shown, it includes:

[0048] Step 101: Divide the audio data to be processed into multiple audio data segments.

[0049] It is understandable that a user can input a piece of audio data (such as a piece of music) into the motion video synthesis system, and the motion video synthesis system can initiate the process of this embodiment based on the audio data, dividing the audio data into multiple audio data sub-segments to be processed.

[0050] Step 102: Obtain the first feature information of multiple audio data segments and determine the second feature information of preset action segments.

[0051] Specifically, when obtaining the first feature information of any audio data segment, the first feature information can be obtained through a pre-set feature extraction model. The feature extraction model can be trained in advance using a certain method, and the running logic of the trained feature extraction model can be stored in the motion video synthesis system.

[0052] In addition, regarding the determination of the second feature information, the motion video synthesis system can extract the second feature information of each motion sub-segment in advance through a certain feature extraction model and store the second feature information in the system. When the process of this embodiment is initiated, the second feature information stored in the system can be directly extracted.

[0053] Multiple action segments can come from multiple segments in the same action video, and an action construction graph can be generated based on each action segment. The information of the action construction graph is stored in the system in advance. The action construction graph can include multiple nodes and the edges between nodes. Each node is used to represent an action segment, and the directed edge between any two nodes is used to indicate that there is a relationship between the action segments represented by the two nodes, such as action continuity, time continuity, etc.

[0054] Step 103: Based on the first feature information of multiple audio data segments and the second feature information of preset action segments, select multiple action segments from the preset action segments to form an action sequence to be synthesized. The action sequence to be synthesized includes action segments arranged in a certain order and associated with multiple audio data segments respectively.

[0055] The order of multiple action segments in the action sequence to be synthesized is consistent with the temporal order of the corresponding audio data segments in the entire audio data to be processed. For example, if multiple audio data segments are divided into m1, m2, ..., mn according to time, and the corresponding action segments are v1, v2, ..., vn, the order of these action segments is consistent with the temporal order of the audio data segments. For example, the audio data segment m1 corresponding to action segment v1 is always listed first.

[0056] Specifically, when constructing all action sub-segments in a certain order within the action sequence to be synthesized:

[0057] (1) For the first action segment in the sequence of actions to be synthesized, the action segment that is first in the sequence is...

[0058] Since there are no action segments in the action sequence to be synthesized when determining the first action segment, there will be no previous action segment. The first feature information M of any audio data segment will be used directly. in The second feature information v of each action sub-segment i The first action segment is selected from a set of preset action segments based on the first similarity S0 between the two segments. Specifically, the first action segment can be selected using the following formula 1: when the calculated first similarity is greater than or equal to a certain preset value θ, the action segment is selected as the first action segment; otherwise, the action segment is not selected.

[0059]

[0060] In practical applications, when initiating the process of this embodiment for multiple audio data segments corresponding to a 60-second music piece, the efficiency improvement and motion similarity of determining the action sequence to be synthesized can be as shown in Table 1 below when the preset value θ is different. It can be seen that as the preset value increases, the calculation efficiency improvement increases, and the motion similarity decreases accordingly. Considering all factors, preset values ​​of 0.9 and 0.95 can be selected.

[0061] θ Time efficiency gain Motion similarity 0 0 1.00 0.1 17％ 1.00 0.5 31％ 0.99 0.9 59％ 0.97 0.95 106％ 0.95 0.99 210％ 0.58

[0062] Table 1

[0063] (2) For the second action segment in the action sequence to be synthesized that is not in the first position, specifically as follows: Figure 3 As shown, this can be achieved by following these steps:

[0064] A. Calculate the first similarity between the first feature information of any audio data segment and the second feature information of each action segment. The first similarity is used to represent the consistency of style and rhythm between the audio data and the action.

[0065] It should be noted that, in a specific embodiment, for any audio data segment, the motion video synthesis system can first select all motion sub-segments that have directed edges between their corresponding nodes and the nodes of the preceding motion sub-segments in the motion construction graph generated based on multiple preset motion sub-segments, and then calculate the first similarity between any audio data segment and each of these selected motion sub-segments. In this way, by selecting a subset of motion sub-segments to calculate the first similarity, without needing to consider the first similarity between any audio data segment and all preset motion sub-segments in the system, the computational load in determining the motion sequence to be synthesized is significantly reduced.

[0066] B. Determine the attribute information of each action sub-segment.

[0067] Specifically, the attribute information of any action sub-segment may include the attribute information of the action sub-segment itself, or the association attributes between the action sub-segment and other action sub-segments. Specifically, it may include, but is not limited to, at least one of the following information: the smoothness of action transfer between the action sub-segment and the previous action sub-segment in the determined action sequence to be synthesized, the range of action changes between the previous action sub-segment and the action sub-segment, the action repeatability, and the action range of the action sub-segment.

[0068] Among them, the smoothness of motion transition between motion sub-segments and the previous motion sub-segment is used to indicate the smoothness of the transition from the motion in the previous motion sub-segment to the motion in any other motion sub-segment; the range of motion variation is used to indicate the positional change of the motion when the motion in the previous motion sub-segment is transferred to the motion in any other motion sub-segment; the repeatability of motion is used to indicate the degree to which the motion in the previous motion sub-segment is the same as the motion in any other motion sub-segment; and the range of motion of the motion sub-segment is used to indicate the degree of extension of the motion contained in the motion sub-segment itself.

[0069] C. Based on the first similarity and the attribute information of the action sub-segments, calculate the cost function for selecting each action sub-segment as the action sub-segment corresponding to any audio data segment.

[0070] Specifically, when calculating the cost function, the cost parameter values ​​corresponding to each attribute information can be determined, and the cost function of each action sub-segment can be calculated based on the first similarity and the determined cost parameter values.

[0071] Furthermore, in a specific embodiment, to further meet the diverse needs of users, users can also input other information into the motion video synthesis system, such as inputting at least one set of pose information, the position of these pose information sets in the final motion-synthesized video, and the number of repetitions of these pose information sets. Thus, when calculating the cost function, in addition to considering the aforementioned first similarity and attribute information, other factors can also be considered. Specifically, before executing step 103, the motion video synthesis system can also acquire at least one set of pose information and acquire the third feature information corresponding to each set of pose information, and then calculate the second similarity between the third feature information of each set of pose information and the second feature information of the motion sub-segment.

[0072] Here, any set of posture information refers to the posture information of key body parts (i.e., anchor points) in a certain action state. The third feature information can be obtained based on the aforementioned pre-set feature extraction model.

[0073] In this scenario, the motion video synthesis system can calculate the cost function based on the first similarity, the second similarity, and the attribute information of the motion sub-segments. Specifically, it can determine the cost parameter values ​​corresponding to each attribute information, and then calculate the cost function based on the first similarity, the second similarity, and the determined cost parameter values. The cost function can be expressed using the following formula 2:

[0074] C over (i)=λ3*C over (i-1)+λ4*C tran (i-1,i)

[0075] +λ5*C cons (i)+λ6*C rang (i)+λ7*C repe (i)

[0076] +λ8*C stre (i)+λ9*S ap (P in ,i) (2)

[0077] Among them, C over (i-1) represents the sum of cost functions calculated for each of the determined action sub-segments in the action sequence to be synthesized; C tran (i-1,i) represents the cost parameter value for the smoothness of action transition between any action sub-segment i and the previous action sub-segment i-1; C cons (i) represents the first similarity between the second feature information of the action sub-segment and the first feature information of the audio data sub-segment; C rang (i) represents the cost parameter value indicating the range of action changes compared to the previous action segment; C repe (i) represents the cost parameter value for the repetition of the action segment with the previous action segment; C stre (i) represents the cost parameter value indicating the range of motion of the action sub-segment; S ap The third feature information P represents at least one set of the above-mentioned attitude information. in The second similarity between the second feature information of action segment i and the second feature information of action segment i.

[0078] Specifically, this second similarity can be constrained by the following formula 3, which is to obtain the second feature information of a certain frame p(l) in action sub-segment i. The third feature information of the j-th group of pose information Maximize the similarity between them:

[0079]

[0080] D. Select the second action segment corresponding to any audio data segment from multiple action segments according to the cost function, and put the second action segment into the action sequence to be synthesized.

[0081] Specifically, when the cost function of a certain action segment is greater than or equal to a preset value, the certain action segment is selected as the second action segment corresponding to any audio data segment, wherein the preset value is dynamically changing.

[0082] It should be noted that by performing steps A to D above, a second action sub-segment in the action sequence to be synthesized can be determined. By repeating steps A to D, all second action sub-segments can be determined. If at least one set of pose information is considered when calculating the cost function, fine-grained constraints based on at least one set of pose information will ensure that the action sequence to be synthesized includes action sub-segments corresponding to these pose information. This achieves fine-grained control over the final synthesized video and improves the controllability of the synthesized video.

[0083] Step 104: Combine multiple action segments from the action sequence to be synthesized to obtain an action-synthesized video of the audio data to be processed.

[0084] Specifically, in one case, multiple action segments can be directly spliced ​​together in the aforementioned order, and then smoothed to obtain a composite video.

[0085] In another scenario, if the individual action segments do not include audio data, when obtaining the action-synthesized video, multiple action segments from the action sequence to be synthesized can be spliced ​​together in a certain order to obtain spliced ​​action information. Then, the spliced ​​action information can be directly synthesized synchronously with the aforementioned audio data to be processed to obtain the action-synthesized video.

[0086] As can be seen, in the method of this embodiment, the motion video synthesis system divides an audio data segment into multiple audio data segments, obtains the first feature information corresponding to each of the multiple audio data segments to be processed, and determines the second feature information of preset motion sub-segments. Then, based on the first and second feature information, it selects multiple motion sub-segments from the preset motion sub-segments to form a sequence of actions to be synthesized. These multiple motion sub-segments are sorted in a certain order, and then the multiple motion sub-segments in the sequence of actions to be synthesized are synthesized into a motion-synthesized video. In this process, an audio data segment is divided into fine-grained audio data segments, and then the sequence of actions to be synthesized can be formed through the constraints of the fine-grained audio data segments to form a motion-synthesized video. This achieves effective control over the final motion-synthesized video, thereby enabling the motion-synthesized video to meet the diverse needs of users.

[0087] In a specific embodiment, the first feature information and the second feature information in step 102 above can both be extracted by a preset feature extraction model. The training of this feature extraction model can be implemented according to the following self-supervised training method, as shown in the flowchart below. Figure 4 As shown, it includes:

[0088] Step 201: Determine the initial model for feature extraction.

[0089] It is understandable that when determining the initial feature extraction model, the motion video synthesis system will determine the initial values ​​of the multi-layer structure and parameters in each layer of the initial feature extraction model. The parameters of the initial feature extraction model refer to the fixed parameters used in the calculation process of each layer of the initial feature extraction model that do not need to be reassigned at any time, such as parameter size, number of network layers, and user vector length.

[0090] Specifically, the initial feature extraction model is used to extract feature information from any segment of audio data. This can be any neural network structure used for feature extraction, such as a convolutional neural network (CNN). Furthermore, in one scenario, the action video synthesis system will also define a classification module, connected to the initial feature extraction model. This module is used to classify the file to be processed (e.g., an audio segment or action clip) based on the feature information extracted by the initial feature extraction model, thus determining the type of the file to be processed.

[0091] Step 202: Determine the training samples, which include multiple sample audios and their corresponding sample action segments.

[0092] Furthermore, the training samples can also include type labeling information for each sample audio, such as belonging to jazz music, classical music, etc., and type labeling information for sample action segments, such as modern dance or Latin dance, etc.

[0093] Step 203: Obtain the feature information corresponding to the sample audio and sample action segments respectively through the initial feature extraction model.

[0094] Step 204: Adjust the initial feature extraction model based on the feature information obtained from the initial feature extraction model to obtain a preset feature extraction model.

[0095] Specifically, the motion video synthesis system first calculates a loss function related to the initial feature extraction model based on the feature information obtained from the initial model in step 203 above. Then, it adjusts the parameters in the initial feature extraction model according to this loss function. The training process of the initial feature extraction model aims to minimize the value of this loss function. This training process continuously optimizes the parameter values ​​in the initial feature extraction model determined in step 201 above through a series of mathematical optimization techniques such as backpropagation differentiation and gradient descent, thereby minimizing the calculated value of the aforementioned loss function.

[0096] Specifically, in this embodiment, the motion video synthesis system can calculate a first loss function, a second loss function, and a third loss function based on the feature information corresponding to the sample audio and the feature information of the sample motion segments obtained from the initial feature extraction model. Then, it adjusts the parameter values ​​in the initial feature extraction model based on the first, second, and third loss functions. For example, it calculates an overall loss function related to the initial feature extraction model based on the first, second, and third loss functions, and adjusts the parameter values ​​based on this overall loss function. Wherein:

[0097] (1) The first loss function includes: the difference between the feature information of any sample audio and the feature information of its corresponding sample action segment, and the difference between the feature information of any sample audio and the feature information of the sample action segment corresponding to other sample audio.

[0098] The first loss function can be directly calculated based on the feature information obtained from the initial feature extraction model. During the training of the feature extraction model, the constraint of the first loss function ensures that the feature information between a sample audio and its corresponding sample action segment is similar, while the feature information between a sample audio and the sample action segments corresponding to other sample audios is not similar.

[0099] (2) The second loss function includes the difference between the classification information determined based on the feature information of the sample audio and sample action segments and the type labeling information of the corresponding sample audio and sample action segments.

[0100] The second loss function can be calculated based on the classification information obtained by classifying sample audio or sample action segments according to the feature information obtained from the initial model of feature extraction, and the corresponding type labeling information, for the further added classification module mentioned above.

[0101] Specifically, the first loss function and the second loss function L u This can be represented by the following formula 4, where, This represents the feature information of a certain audio sample a(i). This represents the feature information of the sample action segment p(i) corresponding to a sample audio a(i). The feature information representing the action segment n(i) corresponding to another sample audio:

[0102]

[0103] During the training of the feature extraction model, the constraint of the second loss function can minimize the difference between the classification information determined by the feature information of the sample audio and sample action segments obtained based on the feature extraction model and the actual type.

[0104] (3) The third loss function includes the distance between the sample audio and sample action segments and their pseudo-classification information, which is determined based on the feature information of the sample audio and sample action segments.

[0105] The pseudo-classification information here is not the classification information obtained according to the above classification module, but rather the type obtained by reclassifying each sample audio and each sample action segment using another method. Specifically, in this embodiment, the sample audio and sample action segments can be clustered according to their feature information to determine the cluster center features, which are then used as the pseudo-classification information. Next, the fitting features of the rhythm features of the sample audio and the rhythm features of the sample action segments are calculated, and the distances between these fitting features and the cluster center features of the sample audio and sample action segments are calculated. Finally, the third loss function is calculated based on these calculated distances.

[0106] Specifically, the third loss function can be represented by the following formula 5, where Y p R represents the cluster center features corresponding to the sample audio and sample action segments, respectively. md This is a joint representation of the rhythmic features of the sample audio and the rhythmic features of the sample action segments. It is a feature transformation mapping function, which can be implemented by a multilayer perceptron. CrossEntropy is the cross-entropy loss function.

[0107] L r =CrossEntropy(R md ,Y p (5)

[0108] Among them, R md This can be expressed by the following formula 6:

[0109]

[0110] During the training of the feature extraction model, the constraint of the third loss function can ensure that the rhythm features of a sample audio obtained by the feature extraction model are consistent and similar with the rhythm features of the corresponding sample action segments, and that the rhythm features of a sample audio are not similar with the corresponding sample action segments of other sample audios.

[0111] It should be noted that steps 203 to 204 above are adjustments to the parameter values ​​in the initial feature extraction model based on the feature information of each sample audio and sample action segment detected by the initial feature extraction model. In practical applications, steps 203 to 204 above need to be executed repeatedly until the adjustment of the parameter values ​​meets certain stopping conditions.

[0112] Therefore, after executing steps 201 to 204 of the above embodiment, the motion video synthesis system also needs to determine whether the current adjustment of the parameter values ​​meets the preset stopping conditions. If it does, the process ends, and the parameter values ​​of the initial feature extraction model adjusted in step 204 are used as the parameters of the finally trained feature extraction model. If it does not meet the conditions, the system returns to executing steps 203 to 204 for the initial feature extraction model after adjusting the parameter values. The preset stopping conditions include, but are not limited to, any one of the following: the difference between the currently adjusted parameter value and the previously adjusted parameter value is less than a threshold, i.e., the adjusted parameter value has converged; and the number of parameter value adjustments is equal to the preset number, etc.

[0113] The following is a specific application example to illustrate the motion video synthesis method in this embodiment of the invention. The motion video synthesis system in this embodiment mainly synthesizes a dance video from music data (i.e., audio data), such as... Figure 5 As shown, a piece of music data and at least one set of posture information (the figure illustrates this with two sets of anchor point postures as an example) can be input into a Posture-controllable Dance Synthesis (PC-Syn) network to output a corresponding synthesized dance video. The method in this embodiment can include the following two parts:

[0114] (1) Training the feature extraction model

[0115] In this embodiment, the feature extraction model can be trained using a self-supervised music and dance rhythm alignment (SSRA) method. During this process, the consistency of rhythm between music data and dance segments can be constrained by calculated loss functions (such as the first loss function, the second loss function, and the third loss function mentioned above). After the feature extraction model is trained, its operating logic can be pre-set into the motion video synthesis system.

[0116] In training the feature extraction model, the training method described in the above embodiments can be used. Specifically, in this embodiment, the following method can be used: Figure 6The music-to-dance alignment embedding (M2D-Align) network shown is used to train a feature extraction model. This M2D-Align network includes: a style encoder, a rhythm encoder, a style classifier, and a self-supervised rhythm alignment. There are two sets of style encoders, rhythm encoders, and style classifiers, corresponding to the sample audio and sample action segments, respectively. Specifically:

[0117] The style encoder encodes style features for both sample audio and sample action segments; the rhythm encoder encodes rhythmic features for both the superimposed features of sample audio and the superimposed features of sample action segments, where the superimposed features of the sample audio are the sum of the sample audio and its style features; the style classifier classifies the style to which the frequency bands of the sample audio and sample action belong; and the self-supervised rhythm adjustment is mainly used to calculate the first, second, and third loss functions based on the feature information of the sample audio and sample action videos to adjust the parameter values ​​in the initial feature extraction model. Thus, the trained feature extraction model includes the aforementioned style encoder and rhythm encoder.

[0118] In this context, each audio sample in the determined training samples can be a piece of music data, while the corresponding action segment of the audio sample can be a dance segment corresponding to the music data.

[0119] In practical applications, after training the feature extraction model using different loss function calculation methods (i.e., full mode, triple loss, and SSRA), multiple evaluation parameters are calculated, such as Fréchet inception distance (FID), style accuracy, and diversity. Among them, FID can represent a distance metric between the dance video synthesized based on the feature information obtained from the feature extraction model and the actual dance video, as shown in Table 2 below:

[0120]

[0121] Table 2

[0122] The full-mode approach combines the first, second, and third loss functions mentioned above. The w / o triple loss approach does not use the three loss functions mentioned above, while the w / o SSRA approach does not use the third loss function mentioned above. It can be seen that using the three loss functions greatly improves diversity and style accuracy. When FID decreases, the feature extraction model trained using the three loss functions performs better.

[0123] (2) Figure 7 As shown, in the motion video synthesis system, a dance-synthesized video corresponding to any piece of music data can be generated through the PC-Syn network via the following steps:

[0124] Step 301: Input a piece of music data and multiple sets of anchor point pose information into the PC-Syn network. The PC-Syn network will first divide the music data into multiple audio data segments.

[0125] Step 302: In the PC-Syn network, a preset feature extraction model is invoked through graph optimization. The first feature information corresponding to multiple audio data segments is obtained through the feature extraction model. The second feature information corresponding to each preset action segment (dance action segment in this embodiment) is obtained. The third feature information of each set of anchor point posture information is also obtained.

[0126] Here, the second feature information can be obtained in advance through a feature extraction model and stored in the system. When the process of this embodiment is initiated, the second feature information can be extracted directly.

[0127] Step 303: Calculate the first similarity between the first feature information of each audio data segment and the second feature information of each dance movement segment, and calculate the second similarity between the third feature information of each set of anchor point posture information and the second feature information of each dance movement segment.

[0128] Step 304: Based on the first similarity, the second similarity, and each dance movement sub-segment, the Adaptive Motion Graph (AMGC) adaptively determines the motion unit graph corresponding to the above-mentioned music data and multiple sets of anchor point pose information. The motion unit graph is used to determine the motion sequence to be synthesized, which includes multiple dance movement sub-segments arranged in a certain order.

[0129] The preset multiple dance movement segments can form a movement construction graph, which includes multiple nodes. Each node represents a dance movement segment, and the directed edges between nodes indicate that the dance movement segments are related.

[0130] Specifically, such as Figure 8 As shown, assuming a music segment is divided into n audio data segments, AMGC will first calculate the first similarity based on an audio data segment 1, and combine it with the above formula 1 to select a dance action segment 1 corresponding to a node in the motion construction graph as the first dance action segment in the sequence to be synthesized.

[0131] It should be noted that, since the user inputs multiple sets of anchor point posture information into the system in step 301 above, in another specific embodiment, when determining the first dance action sub-segment, the user can also select a dance action sub-segment 1 corresponding to a node from the action construction graph based on the first similarity calculated based on an audio data sub-segment 1 and the second similarity mentioned above, and combine it with the above formula 2, as the first dance action sub-segment arranged at the beginning in the action sequence to be synthesized.

[0132] Furthermore, AMGC will, based on the first similarity calculated from the next audio data segment 2, the aforementioned second similarity, and the previous dance action segment 1 in the action sequence to be synthesized, and in conjunction with Formula 2, select the next node from the motion construction graph. The dance action segment corresponding to the selected next node will be taken as the next dance action segment 2 of the aforementioned dance action segment 1, i.e., the second dance action segment. In this way, by repeating the method of selecting the next node, multiple nodes can be selected sequentially from the motion construction graph, and the dance action segments corresponding to these nodes will be placed into the action sequence to be synthesized.

[0133] Step 305: Combine the individual dance action segments in the sequence of actions to be synthesized determined in step 304 into a dance synthesis video corresponding to the aforementioned music data.

[0134] In this embodiment, by constraining multiple sets of anchor point pose information, the sequence of actions to be synthesized is determined to include dance movement sub-segments associated with each set of anchor point pose information, such as... Figure 9 As shown, the dance movements corresponding to the two frames in the dance synthesis video are the dance movements corresponding to the anchor point posture information. In this way, dance movements that meet the user's expectations can be smoothly generated based on the anchor point posture information input by the user. This control over the final synthesized dance video can improve the diversity of dances generated by this system and enhance the user's interactive experience and participation.

[0135] As can be seen, the following technical effects can be achieved through the methods in the above embodiments:

[0136] (1) Using a piece of music data given by the user and at least one set of anchor point pose information, a high-quality dance synthesis video can be generated in a fine-grained manner according to the user's expectations. The dance synthesis video includes dance movements related to the anchor point pose.

[0137] (2) A self-supervised rhythm alignment method is adopted in music-dance relationship mapping learning, so that the feature extraction model can be trained in a self-supervised manner without the annotation of professional artists.

[0138] (3) In this embodiment, when training the feature extraction model, there is no need for professional artists to annotate the sample audio. This makes it easy to expand the database and enrich the diversity of generating dance synthesis videos based on a piece of music data.

[0139] (4) Before generating dance synthesis videos with controllable anchor point posture, the adaptive motion unit graph construction scheme (AMGC) can improve the efficiency of graph-based optimization and maintain the diversity of dance motion units.

[0140] In practical applications, after generating dance synthesis videos using different methods (such as DanceNet, Learning2dance, AIChoreographer, and the method in this embodiment), multiple evaluation parameters are calculated, such as FID, style accuracy, and diversity, as shown in Table 3 below:

[0141]

[0142] Table 3

[0143] It is evident that the diversity and style accuracy are greatly improved when using the method of the present invention embodiment. High diversity indicates that the method of this embodiment can better generate dance synthesis videos for different music data; while the FID is reduced when using the method of the present invention embodiment, indicating that the distribution of the dance synthesis video generated by the system is more similar to that of real dance.

[0144] The following uses another specific application example to illustrate the motion video synthesis method of the present invention. The motion video synthesis system in this embodiment of the present invention is mainly a distributed system 100. The distributed system may include a client 300 and multiple nodes 200 (any form of computing device in the network, such as a server or user terminal). The client 300 and the nodes 200 are connected through network communication.

[0145] Taking a distributed system as an example, see blockchain system. Figure 10This is an optional structural diagram of the distributed system 100 provided in this embodiment of the invention applied to a blockchain system. It consists of multiple nodes 200 (any form of computing device in the network, such as servers or user terminals) and clients 300. The nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). In the distributed system, any machine, such as a server or terminal, can join and become a node. A node includes a hardware layer, a middleware layer, an operating system layer, and an application layer.

[0146] See Figure 10 The functions of each node in the blockchain system shown include:

[0147] 1) Routing: A basic function of nodes used to support communication between nodes.

[0148] In addition to routing capabilities, nodes can also have the following functions:

[0149] 2) Applications are deployed in the blockchain to implement specific business needs. They record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system. When other nodes successfully verify the source and integrity of the record data, they add the record data to the temporary block.

[0150] For example, the application's business logic includes code that implements motion video compositing functionality, which mainly includes:

[0151] The audio data to be processed is divided into multiple audio data segments; first feature information of each of the multiple audio data segments is obtained, and second feature information of preset action segments is determined; based on the first feature information of the multiple audio data segments and the second feature information of the preset action segments, multiple action segments are selected from the preset action segments to form an action sequence to be synthesized, the action sequence to be synthesized includes action segments arranged in a certain order and associated with the multiple audio data segments respectively; the multiple action segments in the action sequence to be synthesized are synthesized to obtain the action-synthesized video of the audio data to be processed.

[0152] 3) A blockchain consists of a series of blocks that are sequentially generated. Once a new block is added to the blockchain, it will not be removed. The blocks contain the data submitted by the nodes in the blockchain system.

[0153] See Figure 11 This is an optional schematic diagram of the block structure provided in an embodiment of the present invention. Each block includes the hash value of the transaction records stored in this block (the hash value of this block) and the hash value of the previous block. The blocks are connected through their hash values ​​to form a blockchain. Additionally, the block may include information such as a timestamp when it was generated. A blockchain is essentially a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains relevant information used to verify the validity of the information (anti-counterfeiting) and to generate the next block.

[0154] This invention also provides an action video synthesis system, the structural schematic of which is shown below. Figure 12 As shown, it can specifically include:

[0155] The audio segment unit 10 is used to divide the audio data to be processed into multiple audio data segments.

[0156] The feature acquisition unit 11 is used to acquire the first feature information of multiple audio data segments divided by the audio segment unit 10, and to determine the second feature information of preset action segments.

[0157] The feature acquisition unit 11 is specifically used to acquire the first feature information of the multiple audio data segments according to the preset feature extraction model.

[0158] The sequence determination unit 12 is used to select multiple action sub-segments from the preset action sub-segments based on the first feature information of multiple audio data sub-segments obtained by the feature acquisition unit 11 and the second feature information of preset action sub-segments to form an action sequence to be synthesized. The action sequence to be synthesized includes action sub-segments arranged in a certain order and respectively associated with the multiple audio data sub-segments.

[0159] The synthesis unit 13 is used to synthesize multiple action sub-segments in the action sequence to be synthesized determined by the sequence determination unit 12 to obtain the action-synthesized video of the audio data to be processed.

[0160] Furthermore, the system in this embodiment of the invention may also include:

[0161] Training unit 14 is used to determine an initial feature extraction model; determine training samples, the training samples including multiple sample audios and their corresponding sample action segments; obtain feature information corresponding to the sample audios and sample action segments respectively through the initial feature extraction model; adjust the initial feature extraction model according to the feature information obtained by the initial feature extraction model to obtain the feature extraction model used by the feature acquisition unit 11 when acquiring each feature information.

[0162] Specifically, when adjusting the initial feature extraction model based on the feature information obtained from the initial feature extraction model, the training unit 14 calculates a first loss function, a second loss function, and a third loss function based on the feature information corresponding to the sample audio and the feature information of the sample action segment obtained by the initial feature extraction model. The first loss function includes the difference between the feature information of any sample audio and the feature information of its corresponding sample action segment, and the difference between the feature information of any sample audio and the feature information of the sample action segment corresponding to other sample audios. The second loss function includes the difference between the classification information determined based on the feature information of the sample audio and the sample action segment and the corresponding type labeling information of the sample audio and the sample action segment. The third loss function includes the distance between the pseudo-classification information of the sample audio and the sample action segment determined based on the feature information of the sample audio and the sample action segment. The parameter values ​​in the initial feature extraction model are adjusted according to the first, second, and third loss functions.

[0163] Specifically, when calculating the third loss function, the training unit 14 is used to determine the cluster center features of the sample audio and sample action segments based on the feature information of the sample audio and sample action segments, and use the cluster center features as pseudo-classification information of the sample audio and sample action segments; calculate the fitting features of the rhythm features of the sample audio and the rhythm features of the sample action segments, and the distances between them and the cluster center features of the sample audio and sample action segments; and calculate the third loss function based on the calculated distances.

[0164] The training unit 14 is also used to stop adjusting the parameter value when the number of times the parameter value is adjusted is equal to a preset number, or when the difference between the currently adjusted parameter value and the previously adjusted parameter value is less than a threshold.

[0165] Further, the sequence determination unit 12, when dealing with the second action sub-segment that is not ranked first in the action sequence to be synthesized, is specifically used to calculate the first similarity between the first feature information of any audio data segment and the second feature information of each action sub-segment; determine the attribute information of each action sub-segment; calculate the cost function for selecting each action sub-segment as the action sub-segment corresponding to any audio data segment based on the first similarity and the attribute information of the action sub-segment; select the second action sub-segment corresponding to any audio data segment from among the multiple action sub-segments based on the cost function, and put the second action sub-segment into the action sequence to be synthesized.

[0166] The audio segment unit 10 is further configured to acquire at least one set of posture information, and the feature acquisition unit 11 is further configured to acquire the third feature information corresponding to each set of posture information. The sequence determination unit 12 is further configured to calculate the second similarity between the third feature information of the set of posture information and the second feature information of the action segment. Thus, when calculating the cost function for selecting each action segment as the action segment corresponding to any audio data segment based on the first similarity and the attribute information of the action segment, the cost function is specifically calculated based on the first similarity, the second similarity, and the attribute information of the action segment.

[0167] Specifically, when the sequence determination unit 12 calculates the cost function based on the first similarity, the second similarity, and the attribute information of the action sub-segment, it is used to determine the cost parameter values ​​corresponding to the attribute information respectively, and calculate the cost function based on the first similarity, the second similarity, and the cost parameter values. The attribute information of the action sub-segment includes at least one of the following: the smoothness of action transfer, the range of action changes, the repeatability of action, and the range of motion of the action sub-segment between the action sub-segment and the previous action sub-segment in the determined action sequence to be synthesized.

[0168] Specifically, when the sequence determination unit 12 selects the second action sub-segment corresponding to any audio data sub-segment from the plurality of action sub-segments according to the cost function, it selects the certain action sub-segment as the second action sub-segment corresponding to any audio data sub-segment when the cost function of a certain action sub-segment is greater than or equal to a preset value, wherein the preset value is dynamically changing.

[0169] As can be seen, in the motion video synthesis system of this embodiment, the audio segment unit 10 divides an audio data segment into multiple audio data segments. The feature acquisition unit 11 acquires the first feature information corresponding to each of the multiple audio data segments to be processed, and determines the second feature information of the preset motion sub-segments. Then, the sequence determination unit 12 selects multiple motion sub-segments from the preset motion sub-segments according to the first and second feature information to form a sequence of motion to be synthesized. These multiple motion sub-segments are sorted in a certain order, and then the synthesis unit 13 synthesizes the multiple motion sub-segments in the sequence of motion to be synthesized into a motion-synthesized video. In this process, an audio data segment is divided into fine-grained audio data segments, and then the sequence of motion to be synthesized can be formed by the constraints of the fine-grained audio data segments to form a motion-synthesized video. This achieves effective control over the final motion-synthesized video, thereby enabling the motion-synthesized video to meet the diverse needs of users.

[0170] This invention also provides a terminal device, the structural schematic of which is shown below. Figure 13As shown, the terminal device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 20 (e.g., one or more processors) and memory 21, and one or more storage media 22 (e.g., one or more mass storage devices) for storing application programs 221 or data 222. The memory 21 and storage media 22 can be temporary or persistent storage. The program stored in the storage media 22 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the terminal device. Furthermore, the CPU 20 may be configured to communicate with the storage media 22 and execute the series of instruction operations in the storage media 22 on the terminal device.

[0171] Specifically, the application program 221 stored in the storage medium 22 includes an application program for motion video synthesis, and this program may include the audio segment unit 10, feature acquisition unit 11, sequence determination unit 12, synthesis unit 13, and training unit 14 in the aforementioned motion video synthesis system, which will not be described in detail here. Furthermore, the central processing unit 20 may be configured to communicate with the storage medium 22 and execute a series of operations corresponding to the motion video synthesis application program stored in the storage medium 22 on the terminal device.

[0172] The terminal device may also include one or more power supplies 23, one or more wired or wireless network interfaces 24, one or more input / output interfaces 25, and / or one or more operating systems 223, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0173] The steps performed by the motion video synthesis system in the above method embodiments can be based on this. Figure 13 The structure of the terminal device shown is illustrated.

[0174] Furthermore, in another aspect, embodiments of the present invention also provide a computer-readable storage medium storing a plurality of computer programs adapted for loading by a processor and executing the motion video compositing method performed by the motion video compositing system described above.

[0175] Another embodiment of the present invention provides a terminal device, including a processor and a memory;

[0176] The memory is used to store multiple computer programs, which are loaded by a processor and executed as described in the motion video compositing system above; the processor is used to implement each of the multiple computer programs.

[0177] Additionally, according to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the motion video synthesis method provided in various alternative implementations of the above aspects.

[0178] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0179] The above provides a detailed description of the motion video synthesis method, system, storage medium, and terminal device provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method of action video synthesis, the method comprising: The method comprises the following steps: dividing the to-be-processed audio data into a plurality of audio data subsegments; obtaining first feature information of the plurality of audio data subsegments respectively, and determining second feature information of preset action subsegments; selecting a plurality of action subsegments from the preset action subsegments according to the first feature information of the plurality of audio data subsegments and the second feature information of the preset action subsegments, to constitute a to-be-combined action sequence, comprising: calculating a first similarity between the first feature information of any audio data subsegment and the second feature information of each action subsegment; determining attribute information of the each action subsegment; calculating a cost function of selecting the each action subsegment as an action subsegment corresponding to any audio data subsegment according to the first similarity and the attribute information of the action subsegment; selecting a second action subsegment corresponding to the any audio data subsegment from the plurality of action subsegments according to the cost function, and putting the second action subsegment into the to-be-combined action sequence; the to-be-combined action sequence comprises action subsegments arranged in a certain order and associated with the plurality of audio data subsegments respectively; combining the plurality of action subsegments in the to-be-combined action sequence to obtain an action combined video of the to-be-processed audio data.

2. The method of claim 1, wherein, The method further comprises: determining a feature extraction initial model; 3. The method of claim 2, wherein, determining training samples, wherein the training samples comprise a plurality of sample audios and sample action segments corresponding to the sample audios respectively; obtaining feature information corresponding to the sample audios and the sample action segments respectively by using the feature extraction initial model; adjusting the feature extraction initial model according to the feature information obtained by using the feature extraction initial model, to obtain a preset feature extraction model. The method further comprises: calculating a first loss function, a second loss function and a third loss function according to the feature information of the sample audios and the feature information of the sample action segments obtained by using the feature extraction initial model; 4. The method of claim 3, wherein, wherein, the first loss function comprises a difference value between the feature information of any sample audio and the feature information of a sample action segment corresponding to the sample audio, and a difference value between the feature information of any sample audio and the feature information of a sample action segment corresponding to another sample audio; the second loss function comprises a difference value between classification information determined based on the feature information of the sample audios and the feature information of the sample action segments and type label information of the corresponding sample audios and sample action segments; the third loss function comprises a distance between the sample audios and the sample action segments and pseudo classification information of the sample audios and the sample action segments determined based on the feature information of the sample audios and the feature information of the sample action segments; wherein, the pseudo classification information is a type obtained by reclassifying each sample audio and each sample action segment. ​ ​ Adjust the parameter value in the feature extraction initial model according to the first loss function, the second loss function and the third loss function.

5. The method of claim 4, wherein, The third loss function is calculated specifically by: According to the feature information of the sample audio and the sample action segment, respectively determine the cluster center feature of the sample audio and the sample action segment, and take the cluster center feature as the pseudo-classification information of the sample audio and the sample action segment; Calculate the fitting feature of the rhythm feature of the sample audio and the rhythm feature of the sample action segment, and the distance between the cluster center feature of the sample audio and the sample action segment, respectively; According to the calculated distance, calculate the third loss function.

6. The method of claim 4, wherein, When the number of adjustments of the parameter value is equal to the preset number of times, or if the difference between the current adjusted parameter value and the last adjusted parameter value is less than a threshold value, stop adjusting the parameter value.

7. The method of claim 1, wherein, Before the action sequence to be synthesized is constituted, further comprising: Obtain at least one set of posture information, and obtain the third feature information corresponding to any set of posture information; Calculate the second similarity between the third feature information of any set of posture information and the second feature information of the action sub-segment; Then, the calculation of the cost function for selecting each action sub-segment as the action sub-segment corresponding to the any audio data sub-segment according to the first similarity and the attribute information of the action sub-segment specifically comprises: calculating the cost function according to the first similarity, the second similarity and the attribute information of the action sub-segment.

8. The method of claim 7, wherein, The calculation of the cost function according to the first similarity, the second similarity and the attribute information of the action sub-segment specifically comprises: Determine the cost parameter value corresponding to the attribute information, and calculate the cost function according to the first similarity, the second similarity and the cost parameter value; The attribute information of the action sub-segment includes at least one of the following information: action transition smoothness, action change range, action repeatability and motion range of the action sub-segment between the action sub-segment and the previous action sub-segment in the action sequence to be synthesized.

9. The method of claim 1, wherein, The selection of the second action sub-segment corresponding to the any audio data sub-segment from the plurality of action sub-segments according to the cost function specifically comprises: When the cost function of a certain action sub-segment is greater than or equal to a preset value, the certain action sub-segment is selected as the second action sub-segment corresponding to the any audio data sub-segment, wherein the preset value is dynamically changed.

10. A motion video synthesis system characterized by comprising: Comprise: An audio sub-segment unit for dividing the audio data to be processed into a plurality of audio data sub-segments; A feature acquisition unit for respectively acquiring the first feature information of the plurality of audio data sub-segments, and determining the second feature information of the preset action sub-segment; The sequence determining unit is configured to select a plurality of action sub-clips from the preset action sub-clips according to the first feature information of the plurality of audio data sub-segments and the second feature information of the preset action sub-clips, so as to constitute a to-be-combined action sequence, and includes: calculating a first similarity between the first feature information of any audio data sub-segment and the second feature information of each action sub-clip; determining attribute information of the each action sub-clip; calculating a cost function of selecting the each action sub-clip as an action sub-clip corresponding to any audio data sub-segment according to the first similarity and the attribute information of the action sub-clip; selecting a second action sub-clip corresponding to the any audio data sub-segment from the plurality of action sub-clips according to the cost function, and putting the second action sub-clip into the to-be-combined action sequence; the to-be-combined action sequence includes action sub-clips arranged in a certain order and associated with the plurality of audio data sub-segments respectively; The combining unit is configured to combine the plurality of action sub-clips in the to-be-combined action sequence, so as to obtain an action combined video of the to-be-processed audio data.

11. The system of claim 10, wherein, The system further includes: The training unit is configured to determine a feature extraction initial model, determine training samples including a plurality of sample audios and sample action clips corresponding to the sample audios respectively, acquire feature information corresponding to the sample audios and the sample action clips respectively by using the feature extraction initial model, and adjust the feature extraction initial model according to the feature information acquired by using the feature extraction initial model, so as to obtain a feature extraction model used when each feature information is acquired by using the feature acquisition unit.

12. The system of claim 11, wherein, When the feature extraction initial model is adjusted according to the feature information obtained by using the feature extraction initial model, the training unit is specifically configured to calculate a first loss function, a second loss function and a third loss function according to the feature information of the sample audios and the feature information of the sample action clips acquired by using the feature extraction initial model; the first loss function includes a difference value between the feature information of any sample audio and the feature information of the sample action clip corresponding to the sample audio, and a difference value between the feature information of any sample audio and the feature information of the sample action clip corresponding to another sample audio; the second loss function includes a difference value between classification information determined based on the feature information of the sample audio and the sample action clip and type label information of the corresponding sample audio and sample action clip; the third loss function includes a distance between pseudo classification information of the sample audio and the sample action clip determined based on the feature information of the sample audio and the sample action clip; and the parameter values in the feature extraction initial model are adjusted according to the first loss function, the second loss function and the third loss function; the pseudo classification information is a type obtained by reclassifying each sample audio and each sample action clip.

13. The system of claim 12, wherein, The training unit is specifically configured to determine clustering center features of the sample audio and the sample action segment according to feature information of the sample audio and the sample action segment, and take the clustering center features as pseudo-classification information of the sample audio and the sample action segment; calculate distances between rhythm features of the sample audio and rhythm features of the sample action segment and between the clustering center features of the sample audio and the sample action segment respectively; and calculate the third loss function according to the calculated distances.

14. The system of claim 12, wherein, The training unit is further configured to stop adjusting the parameter value when the number of times of adjusting the parameter value is equal to a preset number of times or when a difference between the current adjusted parameter value and a last adjusted parameter value is less than a threshold value.

15. The system of claim 10, wherein, The audio sub-segment unit is further configured to obtain at least one set of posture information, and the feature obtaining unit is further configured to obtain third feature information corresponding to any set of posture information; the sequence determining unit is further configured to calculate a second similarity between the third feature information of the any set of posture information and the second feature information of the action sub-segment, and calculate a cost function of selecting the respective action sub-segment as the action sub-segment corresponding to the any audio data sub-segment according to the first similarity, the second similarity and attribute information of the action sub-segment, and specifically configured to calculate the cost function according to the first similarity, the second similarity and the attribute information of the action sub-segment.

16. The system of claim 15, wherein, The sequence determining unit is specifically configured to determine cost parameter values corresponding to the attribute information respectively, and calculate the cost function according to the first similarity, the second similarity and the cost parameter values when calculating the cost function according to the first similarity, the second similarity and the attribute information of the action sub-segment; and the attribute information of the action sub-segment includes at least one of the following information: action transition smoothness between the action sub-segment and a previous action sub-segment in the to-be-synthesized action sequence, action change range, action repeatability and movement range of the action sub-segment.

17. The system of claim 10, wherein, The sequence determining unit is specifically configured to select a certain action sub-segment as the second action sub-segment corresponding to the any audio data sub-segment when the cost function of the certain action sub-segment is greater than or equal to a preset value, wherein the preset value is dynamically changed.

18. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a plurality of computer programs, and the computer programs are adapted to be loaded and executed by the processor to perform the action video synthesis method according to any one of claims 1 to 9.

19. A terminal device, comprising: The processor and the memory are included. The memory is configured to store a plurality of computer programs, and the computer programs are used to load and execute the action video synthesis method according to any one of claims 1 to 9 by the processor; and the processor is configured to implement each computer program in the plurality of computer programs.

20. A computer program product, characterised in that, The computer program product comprises computer instructions stored in a computer readable storage medium; a processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the action video synthesis method according to any one of claims 1 to 9.

Images