A pose-driven controllable video generation method based on diffusion model

Abstract

The application discloses a pose-driven controllable video generation method based on a diffusion model, which comprises the following steps: acquiring a reference appearance image set and a pose sequence, generating an observation tensor and establishing a skeleton template; calling a differentiable SoftPOSIT to obtain a soft assignment matrix, a frame-level pose initial result, a visibility weight and a matching confidence; alternately updating and applying a bone length invariance, joint amplitude limiting and motion chain consistency constraint to output a frame-level pose estimation; organizing an initial pose sequence according to the matching confidence to generate a visibility weight sequence and a soft assignment matrix sequence; establishing a cross-frame factor graph to output a time-consistent pose sequence and generate a factor graph state; integrating the foregoing quantities into a video conditional sequence; synchronously adjusting a matching temperature parameter and a noise intensity in diffusion denoising, calculating a pose consistency guide and incrementally updating the factor graph state; and generating a video frame sequence according to the pose consistency guide. The application improves pose consistency and time sequence stability and enhances controllability.

Description

Technical Field

[0001] This invention relates to the field of controllable video generation technology, and in particular to a pose-driven controllable video generation method based on a diffusion model. Background Technology

[0002] Existing pose-driven video generation methods often employ 2D keypoint guidance, followed by video reconstruction using generative adversarial networks, spatiotemporal diffusion models, and transformer-based synthesis networks. A common workflow involves first extracting identity features from a set of reference appearance images, then using the pose sequence as conditional input. Due to occlusion, depth ambiguity, and detection noise, the matching process often relies on hard assignment and non-differentiable POSIT / PnP estimation. The gradient of the pose calculation on the observation tensor cannot be fully integrated, leading to frame-level pose initial result offsets and frequent inter-frame drift and jitter.

[0003] In terms of registration and visibility processing, publicly available solutions typically use heuristic occlusion masks to correct matching confidence, lacking differentiable occlusion buffers and joint-level visibility weight modeling. Soft assignment often uses a fixed temperature constant, failing to adapt to the intensity of diffuse noise and observation confidence. Soft assignment and pose update are often loosely coupled, and the alternating strategy does not incorporate the three constraints of constant bone length, joint amplitude limiting, and kinematic chain consistency into the same optimization process, thus limiting the stability of frame-level pose estimation.

[0004] In the temporal dimension, most mainstream methods perform simple smoothing after generation, with weak cross-frame dependencies, and rarely use factor graphs to globally constrain pose across the entire sequence. The conditional construction in the diffusion stage usually only includes pose vectors and a few appearance features, failing to encode the temporally consistent pose sequence, factor graph state, visibility weight sequence, and soft assignment matrix sequence into a unified video conditional sequence. Denoising guidance relies heavily on classifier gradients and empirical weights, which are disconnected from geometric consistency and lack synchronous scheduling and time-step interaction updates with matching temperature parameters, resulting in a lack of a closed loop between the diffusion time step and geometric solution.

[0005] Therefore, how to provide a pose-driven controllable video generation method based on a diffusion model is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a pose-driven controllable video generation method based on a diffusion model. This invention employs a differentiable SoftPOSIT module to complete the soft assignment and pose joint solution between the observation tensor and the skeleton template. Based on the cross-frame factor map, a time-consistent pose sequence is obtained by optimizing the initial pose sequence. The time-consistent pose sequence, factor map state, visibility weight sequence, soft assignment matrix sequence, and observation tensor are integrated into a video conditional sequence. During the diffusion denoising process, pose consistency guidance is generated synchronously by matching temperature parameters and noise intensity, thereby outputting a controllable video consistent with the pose sequence. This method has the advantages of end-to-end differentiability, clear occlusion handling, strong temporal consistency, high controllability, and good convergence stability.

[0007] A pose-driven controllable video generation method based on a diffusion model according to an embodiment of the present invention includes the following steps:

[0008] Acquire a set of reference appearance images and pose sequences, generate observation tensors, and establish a skeleton template;

[0009] The SoftPOSIT module is called between the observation tensor and the skeleton template to construct the soft assignment matrix and estimate the initial frame-level pose results, and to calculate the visibility weights and matching confidence.

[0010] Within the differentiable SoftPOSIT module, the soft assignment matrix and the initial frame-level pose are updated alternately. Bone length invariance constraint, joint amplitude constraint and kinematic chain consistency constraint are applied to output the frame-level pose estimate.

[0011] Based on the matching confidence, the frame-level pose estimation is organized into an initial pose sequence, and a visibility weight sequence and a soft assignment matrix sequence are generated;

[0012] Establish a cross-frame factor graph, using the initial attitude sequence as nodes and applying temporal smoothing and velocity constraints, output a time-consistent attitude sequence and generate the factor graph state;

[0013] The time-consistent pose sequence, factor graph state, visibility weight sequence, soft assignment matrix sequence, and observation tensor are integrated into a video conditional sequence.

[0014] In the diffusion denoising iteration, the matching temperature parameter and diffusion noise intensity are adjusted synchronously. The attitude consistency guide is calculated based on the video condition sequence and the differentiable SoftPOSIT module, and the factor map state is updated incrementally according to the time step.

[0015] Based on the attitude consistency guide, update the denoising and generate a video frame sequence, and output a controllable video with the same attitude sequence as the time.

[0016] Optionally, the Microcontroller SoftPOSIT module includes:

[0017] The input interface unit is used to receive observation tensors, skeleton templates, matching temperature parameters, video conditional sequences, and factor graph states.

[0018] The soft assignment building unit is used to construct a soft assignment matrix based on the observation tensor, skeleton template and matching temperature parameters and output the matching confidence.

[0019] Visibility assessment unit, used to calculate visibility weight;

[0020] The joint solver unit is used to perform alternating updates between the soft assignment matrix and the initial frame-level pose result and apply bone length invariance constraints, joint amplitude limiting constraints and kinematic chain consistency constraints to output frame-level pose estimates.

[0021] The temperature parameter interface unit is used to update the matching temperature parameters on a time step basis and distribute them to the soft allocation building unit and the joint solution unit.

[0022] The factor graph interface unit is used to read the factor graph state by time step and write it back to the factor graph state.

[0023] The attitude consistency guidance calculation unit is used to generate attitude consistency guidance based on the video condition sequence, soft allocation matrix sequence, visibility weight sequence and time-consistent attitude sequence.

[0024] The output interface unit is used to output the soft assignment matrix, visibility weights, matching confidence, frame-level pose initialization results, frame-level pose estimation, and pose consistency guidance.

[0025] Optionally, the differentiable SoftPOSIT matching and initial pose estimation process specifically includes:

[0026] Set the initial frame-level pose result, with rotation as an identity matrix and translation as a zero vector;

[0027] Based on the camera intrinsic parameters, the skeleton template joints are projected as pixel positions. The key point pixel positions and confidence scores are extracted from the observation tensor. A matching cost matrix is ​​constructed and scaled and normalized according to the matching temperature parameter.

[0028] The matching temperature parameters are determined as follows: when the diffusion time step is in the high noise stage, the first set value is used; when it is in the transition stage, the second set value is used; and when it is in the low noise stage, the third set value is used. The first set value is greater than the second set value, and the second set value is greater than the third set value. When the average confidence of key points is low and the average visibility weight is low, the setting value is increased to the upper limit of the first set value. When the average confidence of key points is high and the average visibility weight is high, the setting value is decreased to the lower limit of the third set value. In other cases, the matching temperature parameters are calculated by interval linear interpolation between adjacent set values.

[0029] Under the duality of entropy-regularized optimal transport and the KKT condition, Sinkhorn fixed-point iteration is performed in the logarithmic domain to obtain the soft-allocation matrix, with the iterative formula as follows:

[0030] ;

[0031] Where K = exp(−D / τ) is the kernel matrix, D is the matching cost matrix, and τ is the matching temperature parameter. Let be the row dual potential vector. Let be the column dual potential vector. For rows and target vectors, For columns and target vectors, The above iterations are based on dual separable structures and row and column sum constraints, and are given by an exponential function.

[0032] Based on the soft assignment matrix, the first-order linearization is performed on the weighted projected energy, and the pose increment is solved by weighted least squares. The rotation and translation are updated using exponential mapping, and the initial frame-level pose result is written back.

[0033] A depth layer is generated in a differentiable occlusion buffer using the initial frame-level pose and skeleton template. The component depth is continuously compared along each line of sight to obtain the pixel-level foreground probability. The pixel-level foreground probability is weighted and averaged in the neighborhood of the key point of each joint to obtain the joint-level visibility weight. The semantic component map is used to reset the weight of the non-component region to zero. The edge distance field is used to decay the weight in the contour neighborhood according to the distance. The visibility weight is restricted to the interval between zero and one to form the visibility weight.

[0034] The initial matching confidence of the corresponding joint is obtained by statistically analyzing the total probability of each column of the soft assignment matrix. The joint-level visibility weight is used for multiplication correction and normalized across all joints. The soft assignment matrix, frame-level pose initial result, visibility weight and matching confidence are output.

[0035] Optionally, the alternating update of the soft assignment matrix and the frame-level pose initial result, along with constraint joint optimization, specifically includes:

[0036] Set an iteration counter and a stopping condition. The stopping condition includes a threshold for the residual decrease rate and an upper limit for the number of iterations.

[0037] Based on the initial frame-level pose result, the skeleton template joints are projected onto the pixel plane, the corresponding key point pixel positions are read in the observation tensor, and the pixel differences are weighted according to the visibility weight and the probability in the soft assignment matrix to form a weighted projection residual and pose parameter derivative matrix.

[0038] The pose increment is solved according to the weighted least squares rule. The frame-level pose initial result is updated by exponential mapping in the continuous parameter space of 3D rotation and translation, and the updated rotation and translation are written back.

[0039] Calculate the difference between the current inter-joint distance and the target bone segment length recorded by the skeleton template, generate bone length invariant constraint residuals and add them to the optimization;

[0040] Based on the upper and lower limits of each joint angle, joint amplitude constraint residuals are generated, and out-of-bounds angles are written into the optimization using a piecewise quadratic penalty function.

[0041] Perform positive kinematics, set the global pose of the root joint to a unit transformation, propagate local rotation and local translation in sequence according to parent-child relationship to obtain the global joint position, calculate the position difference obtained by propagation to adjacent bone segments, generate kinematic chain consistency constraint residuals and add them to optimization;

[0042] Under the condition of writing back the updated frame-level pose initial result, the matching cost matrix is ​​recalculated. Based on the matching temperature parameter, the soft allocation matrix is ​​updated by Sinkhorn fixed point iteration in the logarithmic domain. The convergence criterion is that the error between the row sum target vector and the column sum target vector is less than the preset tolerance.

[0043] The system makes a judgment based on the stopping condition. If the stopping condition is met, the frame-level pose estimate is output and the updated soft assignment matrix is ​​retained. If the stopping condition is not met, the system continues to update alternately. When the upper limit of the number of iterations is reached, the system stops and outputs the frame-level pose estimate.

[0044] Optionally, the initial pose sequence assembly and sequence generation based on matching confidence specifically includes:

[0045] Frame-level pose estimation, matching confidence, visibility weight, and soft assignment matrix are used, and the frame confidence of each time frame is formed by weighting and summing the matching confidence and visibility weight of the joints.

[0046] Set a frame confidence threshold, retain time frames with a frame confidence level not lower than the threshold, and form a time index sequence;

[0047] Set a matching confidence threshold and a visibility weight threshold. For each joint in each time frame, mark it as valid when the matching confidence is not lower than the matching confidence threshold and the visibility weight is not lower than the visibility weight threshold, and generate a joint valid mask.

[0048] Within each time frame, joint parameters are extracted from frame-level pose estimation based on the joint effective mask. For unmarked joints, the previous and next effective frames are searched in chronological order. If both exist simultaneously, linear interpolation is performed. If only the previous effective frame exists, the parameters of the previous effective frame are used. If only the next effective frame exists, the parameters of the next effective frame are used to obtain the time frame pose parameters.

[0049] Concatenate the time frame attitude parameters according to the time index sequence to generate the initial attitude sequence;

[0050] The visibility weights of each time frame are stacked according to the time index sequence to generate a visibility weight sequence;

[0051] The soft allocation matrices of each time frame are stacked according to the time index sequence to generate a soft allocation matrix sequence;

[0052] Output the initial pose sequence, visibility weight sequence, and soft assignment matrix sequence.

[0053] Optionally, the construction of the cross-frame factor map and the solution of the time-consistent pose sequence specifically include:

[0054] For each time frame of the initial attitude sequence, an attitude node is established on the time index sequence, and the node variable is a joint parameter vector, forming a node set;

[0055] Set a temporal smoothing factor, construct smooth residuals based on the differences between adjacent attitude nodes, and weight them with a weight matrix Q;

[0056] Set a velocity factor, construct velocity residuals by performing first-order differences at time intervals, and weight them with a weight matrix R;

[0057] At the current linearization point, the smoothing residual and the velocity residual are combined into a weighted least squares problem, assembled into a block tridiagonal normal equation arranged in time order, and the block diagonal terms and adjacent coupling terms are recorded.

[0058] Performing the Schur complement recursion and back substitution on the normal equations of the block tridiagonal, the recursive relation is written as:

[0059] ;

[0060] in, For time frame indexing, For time frames The block Hessian matrix obtained by linearizing the time-series smoothing factor and velocity factor is given. For time frames With time frame The block coupling matrix between them For time frames The equivalent block matrix, For time frames The first derivative term at that point, For time frames The equivalent first-order term, the above recursion is obtained based on the first-order Taylor expansion of the weighted least squares objective at the current linearization point and the block tridiagonal structure of the normal equation, because elimination in time order can obtain closed recursion without destroying the sparse structure.

[0061] The attitude increment vector is obtained by back-substitution, the joint parameter vector is updated according to the time index sequence, and a time-consistent attitude sequence is generated.

[0062] Record the state of the factor graph, which includes the pose node update results, weight matrix Q, weight matrix R and sparse connection relationship;

[0063] Output time-consistent attitude sequence and factor graph state.

[0064] Optionally, the process of constructing and standardizing the video conditional sequence specifically includes:

[0065] Establish a time-indexed alignment table, with inputs including time-consistent attitude sequences, factor graph states, visibility weight sequences, soft assignment matrix sequences, and observation tensors;

[0066] Align the pose vector, visibility weight vector, soft assignment matrix, and observation tensor slices of each time frame according to the time index to form a frame-level data set;

[0067] The observation tensor slices are expanded into one-dimensional vectors using a fixed channel order, the soft assignment matrix is ​​expanded into one-dimensional vectors using a column priority order, and the attitude vector and visibility weight vector are concatenated in a predetermined order to form a frame-level original conditional vector.

[0068] Based on the statistical values ​​of the factor weights in the factor graph state and the time-consistent pose sequence, the center value and scale value of each dimension are determined and recorded as standardized parameters;

[0069] The frame-level raw conditional vector is decentered and scaled according to the normalization parameters to obtain the frame-level normalized conditional vector while retaining the time index.

[0070] The frame-level normalized condition vector and the time index are combined to form a video condition frame, which is then concatenated in time index order to form a video condition sequence.

[0071] Optionally, the synchronization scheduling of the matching temperature parameter and the diffusion noise intensity, as well as the attitude-consistent guidance in the diffusion denoising iteration, specifically include:

[0072] Establish a time step index and load the video condition sequence and factor graph initial values, and set the lower bound constant of the matching temperature parameter, the upper bound constant of the matching temperature parameter, the upper bound constant of the diffusion noise intensity, and the curve shape coefficient.

[0073] The intensity of diffused noise is calculated based on the time step index, and a time series of diffused noise intensity is generated.

[0074] Based on the synchronous scheduling, the matching temperature parameters are calculated at each time step to generate a time series of matching temperature parameters;

[0075] At each time step, input the matching temperature parameter into the temperature parameter interface unit and input the factor graph status into the factor graph interface unit.

[0076] Establish a frame-to-step mapping, extract video condition frames from the video condition sequence according to the time step index, and input them into the differentiable SoftPOSIT module along with the matching temperature parameters and factor graph state.

[0077] Within the Differentiable SoftPOSIT module, attitude consistency guidance is generated. The processing order is as follows: extract time-consistent attitude vector, soft assignment matrix, visibility weight, and observation tensor slice; perform skeleton projection based on time-consistent attitude vector and align it with the observation tensor slice; weight and converge pixel difference and component difference according to soft assignment matrix and visibility weight to form residual field; superimpose bone length invariance constraint, joint amplitude constraint, and kinematic chain consistency constraint to obtain geometric residual set; determine guidance direction based on matching temperature parameter and diffusion noise intensity; map guidance direction into attitude consistency guidance component in denoised input space according to observation channel order; and generate factor map state increment.

[0078] The attitude-consistent guidance components are organized into an attitude-consistent guidance sequence according to the time step index, and then aligned with the time series of the matching temperature parameters and the time series of the diffuse noise intensity.

[0079] Output the time series of matched temperature parameters, the time series of diffused noise intensity, the attitude-consistent guidance sequence, and the updated factor graph state.

[0080] The beneficial effects of this invention are:

[0081] This invention establishes a gradient link between the observation tensor and pose through a differentiable SoftPOSIT module. The soft assignment matrix and the initial frame-level pose result are updated alternately within the same optimization framework. The matching temperature parameter is synchronously scheduled according to the intensity of diffused noise, and the visibility weight participates in residual construction and weight allocation at the joint level. As a result, stable frame-level pose estimation can still be obtained in the presence of occlusion and detection noise, drift and jitter are significantly reduced, pose is consistent with pixel evidence, and the initialization and convergence processes are more controllable.

[0082] Temporal smoothing and velocity constraints are applied to the initial pose sequence through cross-frame factor maps to generate a time-consistent pose sequence. This time-consistent pose sequence, factor map state, visibility weight sequence, soft assignment matrix sequence, and observation tensor are then integrated into a video conditional sequence. In the diffusion denoising stage, pose consistency guides the formation of a geometric closed loop. Matching temperature parameters and noise intensity within the time step work synergistically, and the denoising direction aligns with the geometric constraints. This results in improved motion coherence, occlusion recovery capability, and pose controllability in the generated video.

[0083] The entire process is end-to-end differentiable. Interface units and factor graph states standardize condition construction and iterative scheduling. The video condition sequence carries pose, visibility, and matching information, and the diffusion model can be corrected at each time step using reliable geometric signals. This enables precise driving of the reference appearance image set and pose sequence, outputting controllable video with time- and pose-consistent sequences, resulting in greater stability and repeatability. Attached Figure Description

[0084] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0085] Figure 1 This is a flowchart of a pose-driven controllable video generation method based on a diffusion model proposed in this invention.

[0086] Figure 2 This is a block diagram of the differentiable SoftPOSIT module structure of a pose-driven controllable video generation method based on a diffusion model proposed in this invention. Detailed Implementation

[0087] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0088] refer to Figure 1-2 A pose-driven controllable video generation method based on a diffusion model includes the following steps:

[0089] Acquire a set of reference appearance images and pose sequences, generate observation tensors, and establish a skeleton template;

[0090] The SoftPOSIT module is called between the observation tensor and the skeleton template to construct the soft assignment matrix and estimate the initial frame-level pose results, and to calculate the visibility weights and matching confidence.

[0091] Within the differentiable SoftPOSIT module, the soft assignment matrix and the initial frame-level pose are updated alternately. Bone length invariance constraint, joint amplitude constraint and kinematic chain consistency constraint are applied to output the frame-level pose estimate.

[0092] Based on the matching confidence, the frame-level pose estimation is organized into an initial pose sequence, and a visibility weight sequence and a soft assignment matrix sequence are generated;

[0093] Establish a cross-frame factor graph, using the initial attitude sequence as nodes and applying temporal smoothing and velocity constraints, output a time-consistent attitude sequence and generate the factor graph state;

[0094] The time-consistent pose sequence, factor graph state, visibility weight sequence, soft assignment matrix sequence, and observation tensor are integrated into a video conditional sequence.

[0095] In the diffusion denoising iteration, the matching temperature parameter and diffusion noise intensity are adjusted synchronously. The attitude consistency guide is calculated based on the video condition sequence and the differentiable SoftPOSIT module, and the factor map state is updated incrementally according to the time step.

[0096] Based on the attitude consistency guide, update the denoising and generate a video frame sequence, and output a controllable video with the same attitude sequence as the time.

[0097] This invention forms a geometrically consistent and statistically generated closed loop through differentiable SoftPOSIT matching, cross-frame factor graph optimization, and conditional diffusion denoising. It observes the gradient link from tensor to pose, stabilizes soft assignment and visibility weights in occluded scenes to suppress erroneous associations, suppresses jitter and maintains motion continuity in the time domain through weight matrices Q and R, and consolidates the denoising direction by guiding pose consistency and matching temperature parameters during the diffusion phase. Thus, it simultaneously improves appearance preservation, pose fit, and temporal stability.

[0098] In this embodiment, the SoftPOSIT module includes:

[0099] The input interface unit is used to receive observation tensors, skeleton templates, matching temperature parameters, video conditional sequences, and factor graph states.

[0100] The soft assignment building unit is used to construct a soft assignment matrix based on the observation tensor, skeleton template and matching temperature parameters and output the matching confidence.

[0101] Visibility assessment unit, used to calculate visibility weight;

[0102] The joint solver unit is used to perform alternating updates between the soft assignment matrix and the initial frame-level pose result and apply bone length invariance constraints, joint amplitude limiting constraints and kinematic chain consistency constraints to output frame-level pose estimates.

[0103] The temperature parameter interface unit is used to update the matching temperature parameters on a time step basis and distribute them to the soft allocation building unit and the joint solution unit.

[0104] The factor graph interface unit is used to read the factor graph state by time step and write it back to the factor graph state.

[0105] The attitude consistency guidance calculation unit is used to generate attitude consistency guidance based on the video condition sequence, soft allocation matrix sequence, visibility weight sequence and time-consistent attitude sequence.

[0106] The output interface unit is used to output the soft assignment matrix, visibility weights, matching confidence, frame-level pose initialization results, frame-level pose estimation, and pose consistency guidance.

[0107] This invention guides the hierarchical collaboration of the computational units through the input interface unit, soft assignment construction unit, visibility evaluation unit, joint solution unit, temperature parameter interface unit, factor graph interface unit, and attitude consistency of the differentiable SoftPOSIT module. This enables soft assignment, pose estimation, temperature scheduling, and factor graph state to be robustly linked within the same computational framework. The output interface unit uniformly derives key quantities, ensuring consistency between training and inference, and clear engineering integration.

[0108] In this embodiment, the differentiable SoftPOSIT matching and initial pose estimation process specifically includes:

[0109] Set the initial frame-level pose result, with rotation as an identity matrix and translation as a zero vector;

[0110] Based on the camera intrinsic parameters, the skeleton template joints are projected as pixel positions. The key point pixel positions and confidence scores are extracted from the observation tensor. A matching cost matrix is ​​constructed and scaled and normalized according to the matching temperature parameter.

[0111] The matching temperature parameters are determined as follows: when the diffusion time step is in the high noise stage, the first set value is used; when it is in the transition stage, the second set value is used; and when it is in the low noise stage, the third set value is used. The first set value is greater than the second set value, and the second set value is greater than the third set value. When the average confidence of key points is low and the average visibility weight is low, the setting value is increased to the upper limit of the first set value. When the average confidence of key points is high and the average visibility weight is high, the setting value is decreased to the lower limit of the third set value. In other cases, the matching temperature parameters are calculated by interval linear interpolation between adjacent set values.

[0112] Under the duality of entropy-regularized optimal transport and the KKT condition, Sinkhorn fixed-point iteration is performed in the logarithmic domain to obtain the soft-allocation matrix, with the iterative formula as follows:

[0113] ;

[0114] Where K = exp(−D / τ) is the kernel matrix, D is the matching cost matrix, and τ is the matching temperature parameter. Let be the row dual potential vector. Let be the column dual potential vector. For rows and target vectors, For columns and target vectors, The above iterations are based on dual separable structures and row and column sum constraints, and are given by an exponential function.

[0115] Based on the soft assignment matrix, the first-order linearization is performed on the weighted projected energy, and the pose increment is solved by weighted least squares. The rotation and translation are updated using exponential mapping, and the initial frame-level pose result is written back.

[0116] A depth layer is generated in a differentiable occlusion buffer using the initial frame-level pose and skeleton template. The component depth is continuously compared along each line of sight to obtain the pixel-level foreground probability. The pixel-level foreground probability is weighted and averaged in the neighborhood of the key point of each joint to obtain the joint-level visibility weight. The semantic component map is used to reset the weight of the non-component region to zero. The edge distance field is used to decay the weight in the contour neighborhood according to the distance. The visibility weight is restricted to the interval between zero and one to form the visibility weight.

[0117] The initial matching confidence of the corresponding joint is obtained by statistically analyzing the total probability of each column of the soft assignment matrix. The joint-level visibility weight is used for multiplication correction and normalized across all joints. The soft assignment matrix, frame-level pose initial result, visibility weight and matching confidence are output.

[0118] This invention introduces entropy regularized optimal transmission and logarithmic domain Sinkhorn fixed-point update to obtain a stable soft assignment matrix in the matching and initial pose estimation process. It also introduces dynamic matching temperature parameters to adapt to the noise stage and observation confidence, and introduces differentiable occlusion buffer to generate joint-level visibility weights and correct the matching confidence, so that the frame-level pose initial result is close to the true solution and the optimization difficulty is reduced.

[0119] In this embodiment, the alternating update and constraint joint optimization of the soft allocation matrix and the frame-level pose initial result specifically includes:

[0120] Set an iteration counter and a stopping condition. The stopping condition includes a threshold for the residual decrease rate and an upper limit for the number of iterations.

[0121] Based on the initial frame-level pose result, the skeleton template joints are projected onto the pixel plane, the corresponding key point pixel positions are read in the observation tensor, and the pixel differences are weighted according to the visibility weight and the probability in the soft assignment matrix to form a weighted projection residual and pose parameter derivative matrix.

[0122] The pose increment is solved according to the weighted least squares rule. The frame-level pose initial result is updated by exponential mapping in the continuous parameter space of 3D rotation and translation, and the updated rotation and translation are written back.

[0123] Calculate the difference between the current inter-joint distance and the target bone segment length recorded by the skeleton template, generate bone length invariant constraint residuals and add them to the optimization;

[0124] Based on the upper and lower limits of each joint angle, joint amplitude constraint residuals are generated, and out-of-bounds angles are written into the optimization using a piecewise quadratic penalty function.

[0125] Perform positive kinematics, set the global pose of the root joint to a unit transformation, propagate local rotation and local translation in sequence according to parent-child relationship to obtain the global joint position, calculate the position difference obtained by propagation to adjacent bone segments, generate kinematic chain consistency constraint residuals and add them to optimization;

[0126] Under the condition of writing back the updated frame-level pose initial result, the matching cost matrix is ​​recalculated. Based on the matching temperature parameter, the soft allocation matrix is ​​updated by Sinkhorn fixed point iteration in the logarithmic domain. The convergence criterion is that the error between the row sum target vector and the column sum target vector is less than the preset tolerance.

[0127] The system makes a judgment based on the stopping condition. If the stopping condition is met, the frame-level pose estimate is output and the updated soft assignment matrix is ​​retained. If the stopping condition is not met, the system continues to update alternately. When the upper limit of the number of iterations is reached, the system stops and outputs the frame-level pose estimate.

[0128] This invention simultaneously injects bone length invariance constraints, joint amplitude constraints, and kinematic chain consistency constraints into the alternating update of the soft assignment matrix and the initial frame-level pose results. The pose increment is solved based on the weighted projection residual and the rotation and translation are updated stably using exponential mapping. Then, the soft assignment matrix is ​​updated iteratively using a Sinkhorn method driven by the matching temperature parameter. The projection error and geometric violation are compressed in a coordinated manner, and the frame-level pose estimation converges stably.

[0129] In this embodiment, the initial pose sequence assembly and sequence generation based on matching confidence specifically include:

[0130] Frame-level pose estimation, matching confidence, visibility weight, and soft assignment matrix are used, and the frame confidence of each time frame is formed by weighting and summing the matching confidence and visibility weight of the joints.

[0131] Set a frame confidence threshold, retain time frames with a frame confidence level not lower than the threshold, and form a time index sequence;

[0132] Set a matching confidence threshold and a visibility weight threshold. For each joint in each time frame, mark it as valid when the matching confidence is not lower than the matching confidence threshold and the visibility weight is not lower than the visibility weight threshold, and generate a joint valid mask.

[0133] Within each time frame, joint parameters are extracted from frame-level pose estimation based on the joint effective mask. For unmarked joints, the previous and next effective frames are searched in chronological order. If both exist simultaneously, linear interpolation is performed. If only the previous effective frame exists, the parameters of the previous effective frame are used. If only the next effective frame exists, the parameters of the next effective frame are used to obtain the time frame pose parameters.

[0134] Concatenate the time frame attitude parameters according to the time index sequence to generate the initial attitude sequence;

[0135] The visibility weights of each time frame are stacked according to the time index sequence to generate a visibility weight sequence;

[0136] The soft allocation matrices of each time frame are stacked according to the time index sequence to generate a soft allocation matrix sequence;

[0137] Output the initial pose sequence, visibility weight sequence, and soft assignment matrix sequence.

[0138] This invention filters time frames by frame confidence and constrains the joint parameters involved in assembly by joint effective mask. For missing joints, it uses neighboring time frame interpolation and nearest neighbor filling to obtain time frame attitude parameters with complete structure. Then, it connects them in time index order to form an initial attitude sequence and simultaneously stacks the visibility weight sequence and soft assignment matrix sequence, thereby providing clean input for factor graph optimization and condition construction.

[0139] In this embodiment, the construction of the cross-frame factor map and the solution of the time-consistent pose sequence specifically include:

[0140] For each time frame of the initial attitude sequence, an attitude node is established on the time index sequence, and the node variable is a joint parameter vector, forming a node set;

[0141] Set a temporal smoothing factor, construct smooth residuals based on the differences between adjacent attitude nodes, and weight them with a weight matrix Q;

[0142] Set a velocity factor, construct velocity residuals by performing first-order differences at time intervals, and weight them with a weight matrix R;

[0143] At the current linearization point, the smoothing residual and the velocity residual are combined into a weighted least squares problem, assembled into a block tridiagonal normal equation arranged in time order, and the block diagonal terms and adjacent coupling terms are recorded.

[0144] Performing the Schur complement recursion and back substitution on the normal equations of the block tridiagonal, the recursive relation is written as:

[0145] ;

[0146] in, For time frame indexing, For time frames The block Hessian matrix obtained by linearizing the time-series smoothing factor and velocity factor is given. For time frames With time frame The block coupling matrix between them For time frames The equivalent block matrix, For time frames The first derivative term at that point, For time frames The equivalent first-order term, the above recursion is obtained based on the first-order Taylor expansion of the weighted least squares objective at the current linearization point and the block tridiagonal structure of the normal equation, because elimination in time order can obtain closed recursion without destroying the sparse structure.

[0147] The attitude increment vector is obtained by back-substitution, the joint parameter vector is updated according to the time index sequence, and a time-consistent attitude sequence is generated.

[0148] Record the state of the factor graph, which includes the pose node update results, weight matrix Q, weight matrix R and sparse connection relationship;

[0149] Output time-consistent attitude sequence and factor graph state.

[0150] This invention constructs a cross-frame factor graph and uses a weight matrix Q to weight the difference between adjacent attitude nodes and a weight matrix R to weight the discrete velocity. After linearization, a block tridiagonal normal equation is formed, and the attitude increment is solved in linear time by combining Shur complement recursion and back substitution. This maintains the sparse structure while achieving full sequence constraints, outputs a time-consistent attitude sequence, and records the factor graph state for scheduling and condition integration.

[0151] In this embodiment, the process of constructing and standardizing the video conditional sequence specifically includes:

[0152] Establish a time-indexed alignment table, with inputs including time-consistent attitude sequences, factor graph states, visibility weight sequences, soft assignment matrix sequences, and observation tensors;

[0153] Align the pose vector, visibility weight vector, soft assignment matrix, and observation tensor slices of each time frame according to the time index to form a frame-level data set;

[0154] The observation tensor slices are expanded into one-dimensional vectors using a fixed channel order, the soft assignment matrix is ​​expanded into one-dimensional vectors using a column priority order, and the attitude vector and visibility weight vector are concatenated in a predetermined order to form a frame-level original conditional vector.

[0155] Based on the statistical values ​​of the factor weights in the factor graph state and the time-consistent pose sequence, the center value and scale value of each dimension are determined and recorded as standardized parameters;

[0156] The frame-level raw conditional vector is decentered and scaled according to the normalization parameters to obtain the frame-level normalized conditional vector while retaining the time index.

[0157] The frame-level normalized condition vector and the time index are combined to form a video condition frame, which is then concatenated in time index order to form a video condition sequence.

[0158] In the process of constructing and standardizing video conditional sequences, this invention performs time alignment and vectorization expansion of pose vectors, visibility weight vectors, soft allocation matrices, and observation tensor slices. It calculates center and scale values ​​based on factor graph state statistics to complete decentering and scale adjustment. After unifying the channel order and scale, video conditional frames are formed and sequentially connected to form a video conditional sequence. The diffusion model can stably receive structured geometric signals.

[0159] In this embodiment, the synchronous scheduling of the matching temperature parameter and the diffusion noise intensity, as well as the attitude-consistent guidance generation in the diffusion denoising iteration, specifically include:

[0160] Establish a time step index and load the video condition sequence and factor graph initial values, and set the lower bound constant of the matching temperature parameter, the upper bound constant of the matching temperature parameter, the upper bound constant of the diffusion noise intensity, and the curve shape coefficient.

[0161] The intensity of diffused noise is calculated based on the time step index, and a time series of diffused noise intensity is generated.

[0162] Based on the synchronous scheduling, the matching temperature parameters are calculated at each time step to generate a time series of matching temperature parameters;

[0163] At each time step, input the matching temperature parameter into the temperature parameter interface unit and input the factor graph status into the factor graph interface unit.

[0164] Establish a frame-to-step mapping, extract video condition frames from the video condition sequence according to the time step index, and input them into the differentiable SoftPOSIT module along with the matching temperature parameters and factor graph state.

[0165] Within the Differentiable SoftPOSIT module, attitude consistency guidance is generated. The processing order is as follows: extract time-consistent attitude vector, soft assignment matrix, visibility weight, and observation tensor slice; perform skeleton projection based on time-consistent attitude vector and align it with the observation tensor slice; weight and converge pixel difference and component difference according to soft assignment matrix and visibility weight to form residual field; superimpose bone length invariance constraint, joint amplitude constraint, and kinematic chain consistency constraint to obtain geometric residual set; determine guidance direction based on matching temperature parameter and diffusion noise intensity; map guidance direction into attitude consistency guidance component in denoised input space according to observation channel order; and generate factor map state increment.

[0166] The attitude-consistent guidance components are organized into an attitude-consistent guidance sequence according to the time step index, and then aligned with the time series of the matching temperature parameters and the time series of the diffuse noise intensity.

[0167] Output the time series of matched temperature parameters, the time series of diffused noise intensity, the attitude-consistent guidance sequence, and the updated factor graph state.

[0168] In the diffusion denoising iteration, this invention synchronously calculates the diffusion noise intensity and matching temperature parameters using time step index and establishes a frame-to-step mapping. The micro-SoftPOSIT generates pose consistency guidance based on video condition frames and incrementally updates the factor map state. Noise prediction gradually fits the time-consistent pose sequence under the guidance of geometric constraints, maintaining long-range motion continuity and making the generation of occluded regions more stable.

[0169] Example 1:

[0170] To verify the feasibility of this invention in practice, it was applied to a dance motion transfer generation scenario. The reference appearance image set consisted of static images of the same person from three perspectives: front, side, and rotation. The posture sequence consisted of the key point trajectories of a professional dancer completing rotations and steps. The observation tensor included three channels: key point heatmap, edge response, and depth cues. The skeleton template used a fixed bone segment length configuration. The materials covered three types of interference: limb cross-occlusion, rapid angular velocity changes, and strong contrast lighting. The resolution was uniformly set to 1280×720, and the frame rate was uniformly set to 30 frames per second. The calibration file provided camera intrinsics to ensure projection consistency. The evaluation data was automatically statistically analyzed under a unified script to eliminate subjective bias.

[0171] In this scenario, the present invention operates in a pipeline manner as described in the claims without user intervention. First, an observation tensor is constructed using a reference appearance image set and pose sequence, and a skeleton template is established. Then, the differentiable SoftPOSIT module is called between the observation tensor and the skeleton template to construct a soft assignment matrix and provide frame-level pose initial results. At the same time, visibility weights and matching confidence are calculated. Then, within the differentiable SoftPOSIT module, the soft assignment matrix and frame-level pose initial results are alternately updated, and bone length invariance constraints, joint amplitude limiting constraints, and kinematic chain consistency constraints are applied. The initial pose sequence is organized by the matching confidence and a visibility weight sequence and a soft assignment matrix sequence are generated. A time-consistent pose sequence is obtained by applying a temporal smoothing factor and a velocity factor to the cross-frame factor map, and the factor map state is recorded. The time-consistent pose sequence, factor map state, visibility weight sequence, soft assignment matrix sequence, and observation tensor are integrated into a video conditional sequence. In diffusion denoising, the matching temperature parameter and diffusion noise intensity are adjusted synchronously, and the pose consistency guidance and factor map state increment are calculated accordingly. Finally, based on the pose consistency guidance, the denoising update is completed, and a controllable video consistent with the time-consistent pose sequence is output.

[0172] To highlight key differences, three types of publicly reproducible experimental subjects were set up for comparison. The first type was hard assignment plus POSIT solution, which was not differentiable and the occlusion depended on a fixed mask. The second type was differentiable assignment, but the matching temperature parameter was fixed and did not change with noise and confidence. The third type was cross-frame smoothing averaging without building a factor map and without introducing weight matrices Q and R. The four methods were run under the same material, the same resolution, the same frame rate, and the same number of diffusion steps, and a uniform threshold and statistical caliber were used. The evaluation indicators included the joint positioning error pixel value PCK@5px percentage, velocity jitter root mean square pixels per frame, and video quality score. There were no reference indicators, and the higher the value, the better.

[0173] Table 1 Comparison Results of Dance Movement Transfer Generation

[0174]

[0175] Table 1 shows that under the same conditions, this invention significantly reduces joint positioning error and improves PCK@5px to over 90%, while controlling the root mean square of velocity jitter to near pixel level. The video quality score is also steadily improved compared to other methods. The differentiable SoftPOSIT opens up the gradient link from the observation tensor to the pose and stabilizes occluded segments with joint-level visibility weights. The cross-frame factor map uses weight matrices Q and R to constrain the difference between adjacent pose nodes and discrete velocity throughout the entire sequence. The video conditional sequence encodes the temporally consistent pose sequence factor map state visibility weight sequence and soft allocation matrix sequence for the diffusion model to consume. The synchronous scheduling of matching temperature parameters and noise intensity makes the denoising direction converge in the same direction as geometric consistency, thereby achieving the goal of simultaneously achieving stability in appearance, motion fit, and temporal continuity.

[0176] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A pose-driven controllable video generation method based on a diffusion model, characterized in that, Includes the following steps: Acquire a set of reference appearance images and pose sequences, generate observation tensors, and establish a skeleton template; The SoftPOSIT module is called between the observation tensor and the skeleton template to construct the soft assignment matrix and estimate the initial frame-level pose results, and to calculate the visibility weights and matching confidence. Within the differentiable SoftPOSIT module, the soft assignment matrix and the initial frame-level pose are updated alternately. Bone length invariance constraint, joint amplitude constraint and kinematic chain consistency constraint are applied to output the frame-level pose estimate. Based on the matching confidence, the frame-level pose estimation is organized into an initial pose sequence, and a visibility weight sequence and a soft assignment matrix sequence are generated; Establish a cross-frame factor graph, using the initial attitude sequence as nodes and applying temporal smoothing and velocity constraints, output a time-consistent attitude sequence and generate the factor graph state; The time-consistent pose sequence, factor graph state, visibility weight sequence, soft assignment matrix sequence, and observation tensor are integrated into a video conditional sequence. In the diffusion denoising iteration, the matching temperature parameter and diffusion noise intensity are adjusted synchronously. The attitude consistency guide is calculated based on the video condition sequence and the differentiable SoftPOSIT module, and the factor map state is updated incrementally according to the time step. Based on the attitude consistency guide, update the denoising and generate a video frame sequence, and output a controllable video with a time-consistent attitude sequence; The customizable SoftPOSIT module includes: The input interface unit is used to receive observation tensors, skeleton templates, matching temperature parameters, video conditional sequences, and factor graph states. The soft assignment building unit is used to construct a soft assignment matrix based on the observation tensor, skeleton template and matching temperature parameters and output the matching confidence. Visibility assessment unit, used to calculate visibility weight; The joint solver unit is used to perform alternating updates between the soft assignment matrix and the initial frame-level pose result and apply bone length invariance constraints, joint amplitude limiting constraints and kinematic chain consistency constraints to output frame-level pose estimates. The temperature parameter interface unit is used to update the matching temperature parameters on a time step basis and distribute them to the soft allocation building unit and the joint solution unit. The factor graph interface unit is used to read the factor graph state by time step and write it back to the factor graph state. The attitude consistency guidance calculation unit is used to generate attitude consistency guidance based on the video condition sequence, soft allocation matrix sequence, visibility weight sequence and time-consistent attitude sequence. The output interface unit is used to output the soft assignment matrix, visibility weights, matching confidence, frame-level pose initialization results, frame-level pose estimation and pose consistency guidance. The construction of the cross-frame factor map and the solution of the time-consistent pose sequence specifically include: For each time frame of the initial attitude sequence, an attitude node is established on the time index sequence, and the node variable is a joint parameter vector, forming a node set; Set a temporal smoothing factor, construct smooth residuals based on the differences between adjacent attitude nodes, and weight them with a weight matrix Q; Set a velocity factor, construct velocity residuals by performing first-order differences at time intervals, and weight them with a weight matrix R; At the current linearization point, the smoothing residual and the velocity residual are combined into a weighted least squares problem, assembled into a block tridiagonal normal equation arranged in time order, and the block diagonal terms and adjacent coupling terms are recorded. Perform Schur complement recursion and back substitution on the block tridiagonal normal equation; The attitude increment vector is obtained by back-substitution, the joint parameter vector is updated according to the time index sequence, and a time-consistent attitude sequence is generated. Record the factor graph state, which includes the pose node update results, weight matrix Q, weight matrix R and sparse connection relationships.

2. The pose-driven controllable video generation method based on a diffusion model according to claim 1, characterized in that, The differentiable SoftPOSIT matching and initial pose estimation process specifically includes: Set the initial frame-level pose result, with rotation as an identity matrix and translation as a zero vector; Based on the camera intrinsic parameters, the skeleton template joints are projected as pixel positions. The key point pixel positions and confidence scores are extracted from the observation tensor. A matching cost matrix is ​​constructed and scaled and normalized according to the matching temperature parameter. The matching temperature parameters are determined as follows: when the diffusion time step is in the high noise stage, the first set value is used; when it is in the transition stage, the second set value is used; and when it is in the low noise stage, the third set value is used. The first set value is greater than the second set value, and the second set value is greater than the third set value. When the average confidence of key points is low and the average visibility weight is low, the setting value is increased to the upper limit of the first set value. When the average confidence of key points is high and the average visibility weight is high, the setting value is decreased to the lower limit of the third set value. In other cases, the matching temperature parameters are calculated by interval linear interpolation between adjacent set values. Under the duality of entropy-regularized optimal transmission and KKT conditions, Sinkhorn fixed-point iteration is performed in the logarithmic field to obtain the soft allocation matrix. Based on the soft assignment matrix, the first-order linearization is performed on the weighted projected energy, and the pose increment is solved by weighted least squares. The rotation and translation are updated using exponential mapping, and the initial frame-level pose result is written back. A depth layer is generated in a differentiable occlusion buffer using the initial frame-level pose and skeleton template. The component depth is continuously compared along each line of sight to obtain the pixel-level foreground probability. The pixel-level foreground probability is weighted and averaged in the neighborhood of the key point of each joint to obtain the joint-level visibility weight. The semantic component map is used to reset the weight of the non-component region to zero. The edge distance field is used to decay the weight in the contour neighborhood according to the distance. The visibility weight is restricted to the interval between zero and one to form the visibility weight. The initial matching confidence of the corresponding joint is obtained by statistically analyzing the total probability of each column of the soft assignment matrix. The joint-level visibility weight is used for multiplication correction and normalized across all joints. The soft assignment matrix, frame-level pose initial result, visibility weight and matching confidence are output.

3. The pose-driven controllable video generation method based on a diffusion model according to claim 1, characterized in that, The alternating update and constraint joint optimization of the soft assignment matrix and frame-level pose initial results specifically include: Set an iteration counter and a stopping condition. The stopping condition includes a threshold for the residual decrease rate and an upper limit for the number of iterations. Based on the initial frame-level pose result, the skeleton template joints are projected onto the pixel plane, the corresponding key point pixel positions are read in the observation tensor, and the pixel differences are weighted according to the visibility weight and the probability in the soft assignment matrix to form a weighted projection residual and pose parameter derivative matrix. The pose increment is solved according to the weighted least squares rule. The frame-level pose initial result is updated by exponential mapping in the continuous parameter space of 3D rotation and translation, and the updated rotation and translation are written back. Calculate the difference between the current inter-joint distance and the target bone segment length recorded by the skeleton template, generate bone length invariant constraint residuals and add them to the optimization; Based on the upper and lower limits of each joint angle, joint amplitude constraint residuals are generated, and out-of-bounds angles are written into the optimization using a piecewise quadratic penalty function. Perform positive kinematics, set the global pose of the root joint to a unit transformation, propagate local rotation and local translation in sequence according to parent-child relationship to obtain the global joint position, calculate the position difference obtained by propagation to adjacent bone segments, generate kinematic chain consistency constraint residuals and add them to optimization; Under the condition of writing back the updated frame-level pose initial result, the matching cost matrix is ​​recalculated. Based on the matching temperature parameter, the soft allocation matrix is ​​updated by Sinkhorn fixed point iteration in the logarithmic domain. The convergence criterion is that the error between the row sum target vector and the column sum target vector is less than the preset tolerance. The system makes a judgment based on the stopping condition. If the stopping condition is met, the frame-level pose estimate is output and the updated soft assignment matrix is ​​retained. If the stopping condition is not met, the system continues to update alternately. When the upper limit of the number of iterations is reached, the system stops and outputs the frame-level pose estimate.

4. The pose-driven controllable video generation method based on a diffusion model according to claim 1, characterized in that, The initial pose sequence assembly and sequence generation based on matching confidence specifically include: Frame-level pose estimation, matching confidence, visibility weight, and soft assignment matrix are used, and the frame confidence of each time frame is formed by weighting and summing the matching confidence and visibility weight of the joints. Set a frame confidence threshold, retain time frames with a frame confidence level not lower than the threshold, and form a time index sequence; Set a matching confidence threshold and a visibility weight threshold. For each joint in each time frame, mark it as valid when the matching confidence is not lower than the matching confidence threshold and the visibility weight is not lower than the visibility weight threshold, and generate a joint valid mask. Within each time frame, joint parameters are extracted from frame-level pose estimation based on the joint effective mask. For unmarked joints, the previous and next effective frames are searched in chronological order. If both exist simultaneously, linear interpolation is performed. If only the previous effective frame exists, the parameters of the previous effective frame are used. If only the next effective frame exists, the parameters of the next effective frame are used to obtain the time frame pose parameters. Concatenate the time frame attitude parameters according to the time index sequence to generate the initial attitude sequence; The visibility weights of each time frame are stacked according to the time index sequence to generate a visibility weight sequence; The soft allocation matrices of each time frame are stacked according to the time index sequence to generate a soft allocation matrix sequence.

5. The pose-driven controllable video generation method based on a diffusion model according to claim 1, characterized in that, The process of constructing and standardizing the video conditional sequence specifically includes: Establish a time-indexed alignment table, with inputs including time-consistent attitude sequences, factor graph states, visibility weight sequences, soft assignment matrix sequences, and observation tensors; Align the pose vector, visibility weight vector, soft assignment matrix, and observation tensor slices of each time frame according to the time index to form a frame-level data set; The observation tensor slices are expanded into one-dimensional vectors using a fixed channel order, the soft assignment matrix is ​​expanded into one-dimensional vectors using a column priority order, and the attitude vector and visibility weight vector are concatenated in a predetermined order to form a frame-level original conditional vector. Based on the statistical values ​​of the factor weights in the factor graph state and the time-consistent pose sequence, the center value and scale value of each dimension are determined and recorded as standardized parameters; The frame-level raw conditional vector is decentered and scaled according to the normalization parameters to obtain the frame-level normalized conditional vector while retaining the time index. The frame-level normalized condition vector and the time index are combined to form a video condition frame, which is then concatenated in time index order to form a video condition sequence.

6. The pose-driven controllable video generation method based on a diffusion model according to claim 1, characterized in that, The synchronous scheduling of the matching temperature parameter and the intensity of the diffused noise in the diffusion denoising iteration, as well as the attitude-consistent guidance for generation, specifically include: Establish a time step index and load the video condition sequence and factor graph initial values, and set the lower bound constant of the matching temperature parameter, the upper bound constant of the matching temperature parameter, the upper bound constant of the diffusion noise intensity, and the curve shape coefficient. The intensity of diffused noise is calculated based on the time step index, and a time series of diffused noise intensity is generated. Based on the synchronous scheduling, the matching temperature parameters are calculated at each time step to generate a time series of matching temperature parameters; At each time step, input the matching temperature parameter into the temperature parameter interface unit and input the factor graph status into the factor graph interface unit. Establish a frame-to-step mapping, extract video condition frames from the video condition sequence according to the time step index, and input them into the differentiable SoftPOSIT module along with the matching temperature parameters and factor graph state. Within the Differentiable SoftPOSIT module, attitude consistency guidance is generated. The processing order is as follows: extract time-consistent attitude vector, soft assignment matrix, visibility weight, and observation tensor slice; perform skeleton projection based on time-consistent attitude vector and align it with the observation tensor slice; weight and converge pixel difference and component difference according to soft assignment matrix and visibility weight to form residual field; superimpose bone length invariance constraint, joint amplitude constraint, and kinematic chain consistency constraint to obtain geometric residual set; determine guidance direction based on matching temperature parameter and diffusion noise intensity; map guidance direction into attitude consistency guidance component in denoised input space according to observation channel order; and generate factor map state increment. The attitude consistency guide components are organized into an attitude consistency guide sequence based on the time step index and aligned with the time series of the matching temperature parameters and the time series of the diffuse noise intensity.

Images