Pose-driven controllable video generation method based on diffusion model

Abstract

The invention discloses a pose-driven controllable video generation method based on a diffusion model, and the method comprises the following steps: obtaining a reference appearance image set and a pose sequence, generating an observation tensor, and building a skeleton template; calling a micro SoftPOSIT to obtain a soft distribution matrix, a frame-level pose initial result, a visibility weight and a matching confidence coefficient; alternately updating and applying bone length invariance, joint amplitude limiting and kinematic chain consistency constraint, and outputting frame-level pose estimation; organizing an initial attitude sequence according to the matching confidence, and generating a visibility weight sequence and a soft distribution matrix sequence; establishing a cross-frame factor graph, outputting a time consistent attitude sequence and generating a factor graph state; integrating the quantities into a video condition sequence; in diffusion denoising, temperature parameters and noise intensity are synchronously adjusted and matched, attitude consistent guidance is calculated, and the state of a factor graph is incrementally updated; and generating a video frame sequence according to posture consistency guidance. According to the method, the attitude consistency and the time sequence stability are improved, and the controllability is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of controllable video generation, and particularly relates to a pose-driven controllable video generation method based on a diffusion model. BACKGROUND

[0002] Existing pose-driven video generation mostly uses two-dimensional key points as guidance, and then uses generative adversarial networks, spatiotemporal diffusion models, and transformer-based synthesis networks to reconstruct videos. The common process first extracts identity features from a reference appearance image set, and then inputs a pose sequence as a condition. Due to occlusion, depth ambiguity, and detection noise, the matching link often relies on hard assignment and non-differentiable POSIT / PnP estimation, and the gradient of the pose solution to the observation tensor cannot be connected, resulting in frame-level pose initial result deviation, frequent frame-to-frame drift and jitter.

[0003] In terms of registration and visibility processing, the disclosed scheme usually corrects the matching confidence with a heuristic occlusion mask, and lacks differentiable occlusion buffering and joint-level visibility weight modeling. Soft assignment is mostly fixed with a temperature constant, and cannot adapt to the diffusion noise intensity and observation confidence. Soft assignment and pose updating are loosely coupled, and the alternating strategy does not include the three constraints of bone length invariance, joint amplitude limiting, and motion chain consistency in the same optimization process, and the frame-level pose estimation stability is limited.

[0004] In the time dimension, most mainstream methods mostly do simple smoothing after generation, have weak cross-frame dependence, and rarely use factor graphs to globally constrain poses within the full sequence. The condition construction in the diffusion stage usually only includes the pose vector and a small amount of appearance features, and does not integrate the time-consistent pose sequence, factor graph state, visibility weight sequence, and soft assignment matrix sequence into the video condition sequence. De-noising guidance mostly relies on classifier gradients and empirical weights, which are disconnected from geometric consistency, and also lacks synchronous scheduling with matching temperature parameters and time step interactive updating, and lacks a closed loop between diffusion time steps and geometric solving.

[0005] Therefore, how to provide a pose-driven controllable video generation method based on a diffusion model is a problem that those skilled in the art need to solve. SUMMARY

[0006] One purpose of the present application is to propose a pose-driven controllable video generation method based on a diffusion model, which uses a differentiable SoftPOSIT module to complete the soft assignment between the observation tensor and the skeleton template and jointly solve the pose, optimizes the time-consistent pose sequence from the initial pose sequence based on the cross-frame factor graph, integrates the time-consistent pose sequence, the factor graph state, the visibility weight sequence, the soft assignment matrix sequence, and the observation tensor into a video condition sequence, and generates a pose-consistent guide in the diffusion denoising process by synchronously scheduling the matching temperature parameter and the noise intensity to match the temperature parameter and the noise intensity, thereby outputting a controllable video consistent with the pose sequence, which has the advantages of end-to-end differentiability, clear occlusion processing, strong time consistency, high controllability, and good convergence stability.

[0007] A pose-driven controllable video generation method based on a diffusion model according to an embodiment of the present application comprises the following steps: Obtain a reference appearance image set and a pose sequence, generate an observation tensor, and establish a skeleton template; Call a differentiable SoftPOSIT module between the observation tensor and the skeleton template, construct a soft assignment matrix, and estimate the initial results of the frame-level pose, calculate the visibility weight and the matching confidence; Alternately update the soft assignment matrix and the initial results of the frame-level pose in the differentiable SoftPOSIT module, apply the constant length constraint, the joint amplitude constraint, and the motion chain consistency constraint, and output the frame-level pose estimation; Organize the frame-level pose estimation into an initial pose sequence according to the matching confidence, and generate a visibility weight sequence and a soft assignment matrix sequence; Establish a cross-frame factor graph, take the initial pose sequence as a node, and apply the time smoothing and speed constraints, output a time-consistent pose sequence, and generate a factor graph state; Integrate the time-consistent pose sequence, the factor graph state, the visibility weight sequence, the soft assignment matrix sequence, and the observation tensor into a video condition sequence; Synchronously adjust the matching temperature parameter and the diffusion noise intensity in the diffusion denoising iteration, calculate the pose-consistent guide based on the video condition sequence and the differentiable SoftPOSIT module, and update the factor graph state according to the time step increment; Update the denoising generated video frame sequence according to the pose-consistent guide, and output a controllable video consistent with the time-consistent pose sequence.

[0008] Optionally, the differentiable SoftPOSIT module comprises: An input interface unit for receiving the observation tensor, the skeleton template, the matching temperature parameter, the video condition sequence, and the factor graph state; A soft assignment construction unit for constructing a soft assignment matrix according to the observation tensor, the skeleton template, and the matching temperature parameter and outputting a matching confidence; a visibility evaluation unit configured to calculate a visibility weight; a joint solving unit configured to perform alternating update between the soft assignment matrix and the frame-level pose initial result and impose bone length invariance constraint, joint amplitude constraint and kinematic chain consistency constraint, and output the frame-level pose estimation; a temperature parameter interface unit configured to update the matching temperature parameter at time steps and distribute to the soft assignment construction unit and the joint solving unit; a factor graph interface unit configured to read the factor graph state at time steps and write back the factor graph state; a pose consistency guidance computation unit configured to generate the pose consistency guidance according to the video condition sequence, the soft assignment matrix sequence, the visibility weight sequence and the time-consistent pose sequence; an output interface unit configured to output the soft assignment matrix, the visibility weight, the matching confidence, the frame-level pose initial result, the frame-level pose estimation and the pose consistency guidance.

[0009] Optionally, the differentiable SoftPOSIT matching and initial pose estimation process specifically comprises: setting the frame-level pose initial result, the rotation as a unit matrix and the translation as a zero vector; projecting the skeleton template joints as pixel positions according to the camera intrinsic parameters, extracting the key point pixel positions and confidence from the observation tensor, constructing the matching cost matrix and completing scaling and normalization according to the matching temperature parameter; determining the matching temperature parameter, taking the first set value when the diffusion time step is in the high noise stage, taking the second set value when it is in the transition stage, and taking the third set value when it is in the low noise stage, the first set value being greater than the second set value and the second set value being greater than the third set value, increasing to the upper limit of the first set value when the average confidence of the key points is low and the average value of the visibility weight is low, decreasing to the lower limit of the third set value when the average confidence of the key points is high and the average value of the visibility weight is high, and linearly interpolating the matching temperature parameter between adjacent set values in other cases; under the dual and KKT conditions of entropy regularized optimal transport, performing Sinkhorn fixed point iteration in the logarithmic domain to obtain the soft assignment matrix, and the iteration formula is: ; wherein K = exp(−D / τ) is a kernel matrix, D is a matching cost matrix, τ is a matching temperature parameter, is a row dual potential vector, is a column dual potential vector, is a row sum target vector, is a column sum target vector, is an exponential function, and the above iteration is based on the dual separable structure and the row sum and column sum constraints; According to first-order linearization of the soft assignment matrix on weighted projection energy and solving pose increments by weighted least squares, exponential mapping is used to update rotation and translation, and frame-level pose initial results are written back; Using frame-level pose initial results and skeleton template to generate depth layers in differentiable occlusion buffer, obtaining pixel-level foreground probability by comparing component depth along each line of sight, weighting and averaging pixel-level foreground probability in key point neighborhood of each joint to obtain joint-level visibility weight, setting zero to non-component region weight using semantic component map, decaying weight by distance in contour neighborhood using edge distance field, and limiting visibility weight to the interval of zero to one to form visibility weight; Statistically obtaining initial matching confidence of corresponding joints by summing up probability of each column of the soft assignment matrix, multiplying and correcting joint-level visibility weight, and normalizing in the range of all joints, and outputting the soft assignment matrix, frame-level pose initial results, visibility weight and matching confidence.

[0010] Optionally, the alternating update and constraint joint optimization of the soft assignment matrix and the frame-level pose initial results specifically includes: Setting an iteration counter and a stop condition, the stop condition including a residual error drop amplitude threshold and an upper limit of the number of iterations; According to the frame-level pose initial results, projecting the joints of the skeleton template to the pixel plane, reading the corresponding key point pixel positions in the observation tensor, weighting the pixel difference according to the visibility weight and the probability in the soft assignment matrix, forming the weighted projection residual error and the pose parameter derivative matrix; Solving the pose increments according to the weighted least squares rule, updating the frame-level pose initial results in the continuous parameter space of three-dimensional rotation and translation using exponential mapping, and writing back the updated rotation and translation; Calculating the difference between the current joint distance and the target bone length recorded by the skeleton template, generating a bone length invariant constraint residual and adding it to the optimization; Generating joint amplitude limiting constraint residuals according to the upper and lower limits of the angle of each joint, and writing the out-of-range angles into the optimization with a piecewise quadratic penalty function; Performing forward kinematics, setting the global pose of the root joint as the unit transformation, propagating the local rotation and local translation according to the parent-child relationship to obtain the global joint position, calculating the position difference propagated by adjacent bone segments, generating a motion chain consistency constraint residual and adding it to the optimization; Recalculating the matching cost matrix under the condition of writing back the updated frame-level pose initial results, updating the soft assignment matrix by Sinkhorn fixed point iteration in the logarithmic domain according to the matching temperature parameter, and taking the error between the row sum target vector and the column sum target vector less than the preset tolerance as the convergence criterion; According to the stop condition completion judgment, when the stop condition is met, the frame-level pose estimation is output and the updated soft assignment matrix is retained, when the stop condition is not met, the alternating update is continued, and when the upper limit of the iteration number is reached, the frame-level pose estimation is stopped and output.

[0011] Optionally, the initial pose sequence assembling and sequence generating based on the matching confidence specifically includes: The frame-level pose estimation, the matching confidence, the visibility weight and the soft assignment matrix are adopted, the matching confidence and the visibility weight of the joint are weighted and summarized, and the frame confidence of each time frame is formed; A frame confidence threshold is set, time frames with frame confidence not lower than the threshold are retained, and a time index sequence is formed; A matching confidence threshold and a visibility weight threshold are set, for each joint of each time frame, when the matching confidence is not lower than the matching confidence threshold and the visibility weight is not lower than the visibility weight threshold, the joint is marked as valid, and a joint valid mask is generated; In each time frame, the joint parameters are extracted from the frame-level pose estimation according to the joint valid mask, for the unmarked joint, the previous valid frame and the next valid frame are searched in time sequence, when both exist, linear interpolation is performed, when only the previous valid frame exists, the parameter of the previous valid frame is adopted, when only the next valid frame exists, the parameter of the next valid frame is adopted, and the time frame pose parameter is obtained; The time frame pose parameters are connected according to the time index sequence, and an initial pose sequence is generated; The visibility weights of each time frame are stacked according to the time index sequence, and a visibility weight sequence is generated; The soft assignment matrices of each time frame are stacked according to the time index sequence, and a soft assignment matrix sequence is generated; The initial pose sequence, the visibility weight sequence and the soft assignment matrix sequence are output.

[0012] Optionally, the construction of the cross-frame factor graph and the solving of the time-consistent pose sequence specifically include: Pose nodes are established for each time frame of the initial pose sequence on the time index sequence, the node variable is the joint parameter vector, and a node set is formed; A timing smoothing factor is set, a smoothing residual is constructed according to the difference between adjacent pose nodes, and is weighted by a weight matrix Q; A speed factor is set, a speed residual is constructed by first-order difference according to a time interval, and is weighted by a weight matrix R; The smoothing residual and the speed residual are summarized as a weighted least squares problem at the current linearization point, assembled as a block tri-diagonal normal equation arranged in time sequence, and the block diagonal items and the adjacent coupling items are recorded; The Schur complement recursion and back substitution are performed on the block tri-diagonal normal equation, and the recursion relationship is written as: ; wherein, is the time frame index, is the time frame is the block hessian matrix linearized by the timing smoothing factor and the velocity factor at, is the time frame is the block coupling matrix between the time frames and is the equivalent block matrix of the time frame at is the first derivative term at the time frame at is the equivalent first order term of the time frame at , the above recursion is based on the first order Taylor expansion of the weighted least squares objective at the current linearization point and the block tridiagonal structure of the normal equation, which is due to the elimination in time sequence without destroying the sparsity structure to obtain the closed-form recursion; the pose increment vector is solved by back substitution, the joint parameter vector is updated according to the time index sequence, and the time consistent pose sequence is generated; the factor graph state is recorded, the factor graph state includes the pose node update result, the weight matrix Q, the weight matrix R and the sparse connection relationship; the time consistent pose sequence and the factor graph state are output.

[0013] Optionally, the composition and standardized integration process of the video condition sequence specifically includes: a time index alignment table is established, and the input is the time consistent pose sequence, the factor graph state, the visibility weight sequence, the soft assignment matrix sequence and the observation tensor; the pose vector, the visibility weight vector, the soft assignment matrix and the observation tensor slice of each time frame are aligned according to the time index to form a frame-level data set; the observation tensor slice is unfolded into a one-dimensional vector using a fixed channel order, the soft assignment matrix is unfolded into a one-dimensional vector using a column-first order, and the pose vector and the visibility weight vector are spliced according to a predetermined order to form a frame-level original condition vector; the center value and the scale value of each dimension are determined according to the factor weight in the factor graph state and the statistics of the time consistent pose sequence, and are recorded as the standardization parameters; the frame-level original condition vector is decentered and scaled according to the standardization parameters to obtain a frame-level standardized condition vector and preserve the time index; the frame-level standardized condition vector and the time index are combined to form a video condition frame, which is connected into a video condition sequence according to the time index sequence.

[0014] Optionally, the matching temperature parameter in the diffusion denoising iteration is synchronously scheduled with the diffusion noise intensity, and the pose consistent guidance generation specifically includes: Establish time step index and load video condition sequence and factor graph state initial value, set matching temperature parameter lower bound constant, matching temperature parameter upper bound constant, diffusion noise intensity upper bound constant and curve shape coefficient; According to time step index, calculate diffusion noise intensity, generate diffusion noise intensity time sequence; According to synchronous scheduling, calculate matching temperature parameter at each time step, generate matching temperature parameter time sequence; At each time step, input matching temperature parameter to temperature parameter interface unit, and input factor graph state to factor graph interface unit; Establish frame to step mapping, extract video condition frame from video condition sequence according to time step index, and input matching temperature parameter and factor graph state into differentiable SoftPOSIT module; In differentiable SoftPOSIT module, generate pose consistent guide, the processing order is to extract time consistent pose vector, soft assignment matrix, visibility weight and observation tensor slice, perform skeleton projection according to time consistent pose vector and align with observation tensor slice, weight pixel difference and component difference according to soft assignment matrix and visibility weight to form residual field, superimpose length invariant constraint, joint amplitude constraint and motion chain consistency constraint to obtain geometric residual set, determine guide direction according to matching temperature parameter and diffusion noise intensity, map guide direction to pose consistent guide component in denoising input space according to observation channel order, and generate factor graph state increment; According to time step index, arrange pose consistent guide component into pose consistent guide sequence, and align with matching temperature parameter time sequence and diffusion noise intensity time sequence; Output matching temperature parameter time sequence, diffusion noise intensity time sequence, pose consistent guide sequence and updated factor graph state.

[0015] The beneficial effects of the present application are: The present application passes through the gradient link of observation tensor to pose through differentiable SoftPOSIT module, soft assignment matrix and frame level pose initial result are updated alternately in the same optimization framework, matching temperature parameter is scheduled synchronously with diffusion noise intensity, and visibility weight participates in residual construction and weight distribution at joint level. Therefore, stable frame level pose estimation can be obtained when occlusion and detection noise exist, drift and jitter are significantly reduced, pose and pixel evidence are consistent, and initialization and convergence process are more controllable.

[0016] The initial pose sequence is subjected to temporal smoothing and speed constraint by a cross-frame factor graph to generate a time-consistent pose sequence, and the time-consistent pose sequence, factor graph state, visibility weight sequence, soft assignment matrix sequence and observation tensor are integrated into a video conditional sequence. In the diffusion denoising stage, the pose consistency is guided to form a geometric loop, the matching temperature parameter in the time step cooperates with the noise intensity, the denoising direction is consistent with the geometric constraint, and the generated video is improved in action coherence, occlusion recovery capability and pose controllability.

[0017] The overall process is end-to-end differentiable, the interface unit and the factor graph state standardize the conditional construction and iterative scheduling, the video conditional sequence carries the pose, visibility and matching information, and the diffusion model can correct at each time step using reliable geometric signals. Thus, the reference appearance image set and the pose sequence are accurately driven, a controllable video consistent with the time-consistent pose sequence is output, and the stability and repeatability are stronger. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, illustrate the application, and are used to explain the application together with the embodiments of the application, and do not constitute a limitation on the application. In the drawings:

[0019] Fig. 1 A flowchart of a pose-driven controllable video generation method based on a diffusion model according to the present application; Fig. 2 A differentiable SoftPOSIT module structure block diagram of a pose-driven controllable video generation method based on a diffusion model according to the present application. DETAILED DESCRIPTION

[0020] The application will now be described in further detail with reference to the drawings. These drawings are simplified schematic diagrams, and only illustrate the basic structure of the application in a schematic manner, and therefore only show the components related to the application.

[0021] REFERENCE Figs. 1-2 A pose-driven controllable video generation method based on a diffusion model, comprising the following steps: Obtain a reference appearance image set and a pose sequence, generate an observation tensor and establish a skeleton template; Call a differentiable SoftPOSIT module between the observation tensor and the skeleton template, construct a soft assignment matrix and estimate an initial frame-level pose result, calculate a visibility weight and a matching confidence; Alternately update the soft assignment matrix and the initial frame-level pose result in the differentiable SoftPOSIT module, apply bone length invariance constraint, joint amplitude constraint and motion chain consistency constraint, and output frame-level pose estimation; The frame-level pose estimation is organized into an initial pose sequence according to matching confidence, and a visibility weight sequence and a soft assignment matrix sequence are generated; A cross-frame factor graph is established, the initial pose sequence is taken as a node, and a time smoothing and a speed constraint are applied, so that a time-consistent pose sequence is output and a factor graph state is generated; The time-consistent pose sequence, the factor graph state, the visibility weight sequence, the soft assignment matrix sequence and the observation tensor are integrated into a video condition sequence; In the diffusion denoising iteration, the matching temperature parameter and the diffusion noise intensity are adjusted synchronously, the pose-consistent guidance is calculated based on the video condition sequence and the differentiable SoftPOSIT module, and the factor graph state is updated in time step increments; The video frame sequence is generated according to the pose-consistent guidance, and the controllable video consistent with the time-consistent pose sequence is output.

[0022] The present application forms a geometric-consistent and statistically-generated closed loop through the differentiable SoftPOSIT matching, the cross-frame factor graph optimization and the conditional diffusion denoising, the gradient link from the observation tensor to the pose is through, the soft assignment and the visibility weight in the occlusion scene are stably inhibited from error association, the weight matrix Q and R in the time domain are inhibited from jitter and maintain motion continuity, and the diffusion stage synchronously schedules the denoising direction by the pose-consistent guidance and the matching temperature parameter, so that the appearance retention, the pose fitting and the time stability are simultaneously improved.

[0023] In the embodiment, the differentiable SoftPOSIT module comprises: An input interface unit is configured to receive an observation tensor, a skeleton template, a matching temperature parameter, a video condition sequence and a factor graph state; A soft assignment construction unit is configured to construct a soft assignment matrix according to the observation tensor, the skeleton template and the matching temperature parameter, and output matching confidence; A visibility evaluation unit is configured to calculate a visibility weight; A joint solving unit is configured to perform alternating update between the soft assignment matrix and the frame-level pose initial result, and apply a bone length invariable constraint, a joint amplitude constraint and a motion chain consistency constraint, and output a frame-level pose estimation; A temperature parameter interface unit is configured to update the matching temperature parameter in time steps and distribute the matching temperature parameter to the soft assignment construction unit and the joint solving unit; A factor graph interface unit is configured to read the factor graph state in time steps and write back the factor graph state; A pose-consistent guidance calculation unit is configured to generate pose-consistent guidance according to the video condition sequence, the soft assignment matrix sequence, the visibility weight sequence and the time-consistent pose sequence; An output interface unit is configured to output the soft assignment matrix, the visibility weight, the matching confidence, the frame-level pose initial result, the frame-level pose estimation and the pose-consistent guidance.

[0024] The present application realizes the robust linkage of soft assignment, pose estimation, temperature scheduling and factor graph state in the same calculation framework through the hierarchical cooperation of the input interface unit, soft assignment construction unit, visibility evaluation unit, joint solving unit, temperature parameter interface unit, factor graph interface unit and pose consistent guidance calculation unit of the differentiable SoftPOSIT module, and the output interface unit uniformly exports key quantities, and the training and inference are consistent, and the engineering integration is clear.

[0025] In the embodiment, the differentiable SoftPOSIT matching and initial pose estimation process specifically includes: Set the frame-level pose initial result, and rotate it to a unit matrix and translate it to a zero vector; Project the skeleton template joint to a pixel position according to the camera intrinsic parameter, extract the key point pixel position and confidence from the observation tensor, construct a matching cost matrix, and complete scaling and normalization according to the matching temperature parameter; Determine the matching temperature parameter, take the first set value when the diffusion time step is in the high noise stage, take the second set value when it is in the transition stage, and take the third set value when it is in the low noise stage, the first set value is greater than the second set value and the second set value is greater than the third set value, when the average confidence of the key points is low and the average value of the visibility weight is low, it is raised to the upper limit of the first set value, when the average confidence of the key points is high and the average value of the visibility weight is high, it is reduced to the lower limit of the third set value, and the matching temperature parameter is calculated by linear interpolation between adjacent set values in the interval under other conditions; Under the dual and KKT conditions of entropy regularized optimal transport, Sinkhorn fixed point iteration is performed in the logarithmic domain to obtain the soft assignment matrix, and the iteration formula is: ; Wherein, K=exp(−D / τ) is the kernel matrix, D is the matching cost matrix, τ is the matching temperature parameter, is the row dual potential vector, is the column dual potential vector, is the row sum target vector, is the column sum target vector, is the exponential function, and the above iteration is based on the dual separable structure and the row sum and column sum constraints; First-order linearization of the soft assignment matrix on the weighted projection energy and solving the pose increment by weighted least squares, updating the rotation and translation by using exponential mapping, and writing back the frame-level pose initial result; A depth layer is generated in the differentiable occlusion buffer using the frame-level pose initial result and the skeleton template, a pixel-level foreground probability is obtained by continuously comparing the part depth along each line of sight, a joint-level visibility weight is obtained by weighted averaging the pixel-level foreground probability in the key point neighborhood of each joint, the semantic part graph is used to set the weight of the non-part region to zero, the edge distance field is used to decay the weight by distance in the contour neighborhood, and the visibility weight is limited to the interval of zero to one to form the visibility weight; The total amount of probability of each column of the soft assignment matrix is counted to obtain an initial matching confidence of the corresponding joint, the joint-level visibility weight is multiplied to correct it, and the soft assignment matrix, the frame-level pose initial result, the visibility weight, and the matching confidence are normalized in the range of all joints.

[0026] In the matching and initial pose estimation process, the present application introduces entropy regular optimal transport and log domain Sinkhorn fixed point update to obtain a stable soft assignment matrix, introduces a dynamic matching temperature parameter to adapt to the noise stage and observation confidence, introduces a differentiable occlusion buffer to generate a joint-level visibility weight and correct the matching confidence, so that the frame-level pose initial result is close to the true solution and the optimization difficulty is reduced.

[0027] In the embodiment, the alternating update and constraint joint optimization of the soft assignment matrix and the frame-level pose initial result specifically includes: An iteration counter and a stop condition are set, and the stop condition includes a residual error drop amplitude threshold and an upper limit of the number of iterations; The joint of the skeleton template is projected to the pixel plane according to the frame-level pose initial result, the pixel position of the corresponding key point is read in the observation tensor, the pixel difference is weighted according to the visibility weight and the probability in the soft assignment matrix, and the weighted heavy projection residual error and the pose parameter derivative matrix are formed; The pose increment is solved according to the weighted least square rule, the frame-level pose initial result is updated in the continuous parameter space of three-dimensional rotation and translation using exponential mapping, and the updated rotation and translation are written back; The difference between the current joint distance and the target bone length recorded by the skeleton template is calculated to generate a bone length invariant constraint residual error and add it to the optimization; Joint amplitude limiting constraint residual errors are generated according to the upper and lower limits of the angle of each joint, and the super-limit angle is written into the optimization with a piecewise quadratic penalty function; Forward kinematics is performed, the global pose of the root joint is set as a unit transformation, the global joint position is obtained by sequentially propagating the local rotation and local translation according to the parent-child relationship, the position difference propagated by the adjacent bone segments is calculated, the motion chain consistency constraint residual error is generated and added to the optimization; The matching cost matrix is recalculated under the condition of writing back the updated frame-level pose initial result, the soft assignment matrix is updated in the log domain by performing Sinkhorn fixed point iteration according to the matching temperature parameter, and the error between the row sum target vector and the column sum target vector is less than the preset tolerance as the convergence criterion. According to the stop condition completion judgment, when the stop condition is met, the frame-level pose estimation is output and the updated soft assignment matrix is retained, when the stop condition is not met, the alternating update is continued, and when the upper limit of the iteration number is reached, the frame-level pose estimation is stopped and output.

[0028] In the alternating update of the soft assignment matrix and the frame-level pose initial result, the bone length invariance constraint, the joint amplitude limiting constraint and the motion chain consistency constraint are injected at the same time, the pose increment is solved based on the weighted heavy projection residual, the rotation and translation are stably updated by exponential mapping, the soft assignment matrix is updated by Sinkhorn iteration driven by the matching temperature parameter, the projection error and the geometric violation are cooperatively compressed, and the frame-level pose estimation is stably converged.

[0029] In the embodiment, the initial pose sequence assembly and sequence generation based on the matching confidence specifically include: The frame confidence is formed by weighting and summarizing the matching confidence and the visibility weight of each joint according to the frame-level pose estimation, the matching confidence and the visibility weight, and the soft assignment matrix; The frame confidence threshold is set, the time frames with the frame confidence not lower than the threshold are retained, and the time index sequence is formed; The matching confidence threshold and the visibility weight threshold are set, for each joint of each time frame, when the matching confidence is not lower than the matching confidence threshold and the visibility weight is not lower than the visibility weight threshold, the joint is marked as valid, and the joint valid mask is generated; In each time frame, the joint parameters are extracted from the frame-level pose estimation according to the joint valid mask, for the unmarked joint, the previous valid frame and the next valid frame are searched in time sequence, when both exist, linear interpolation is performed, when only the previous valid frame exists, the parameter of the previous valid frame is used, when only the next valid frame exists, the parameter of the next valid frame is used, and the time frame pose parameter is obtained; The time frame pose parameters are connected according to the time index sequence, and the initial pose sequence is generated; The visibility weight of each time frame is stacked according to the time index sequence, and the visibility weight sequence is generated; The soft assignment matrix of each time frame is stacked according to the time index sequence, and the soft assignment matrix sequence is generated; The initial pose sequence, the visibility weight sequence and the soft assignment matrix sequence are output.

[0030] The frame confidence is used for screening the time frames, the joint parameters participating in the assembly are constrained by the joint valid mask, the missing joints are interpolated by the adjacent time frames and filled by the nearest neighbor, the time frame pose parameters with complete structure are obtained, the initial pose sequence is formed by connecting the time frame pose parameters in time index sequence, and the visibility weight sequence and the soft assignment matrix sequence are stacked synchronously, so that clean input is provided for factor graph optimization and condition construction.

[0031] In this embodiment, the construction of the cross-frame factor map and the solution of the time-consistent pose sequence specifically include: For each time frame of the initial attitude sequence, an attitude node is established on the time index sequence, and the node variable is a joint parameter vector, forming a node set; Set a temporal smoothing factor, construct smooth residuals based on the differences between adjacent attitude nodes, and weight them with a weight matrix Q; Set a velocity factor, construct velocity residuals by performing first-order differences at time intervals, and weight them with a weight matrix R; At the current linearization point, the smoothing residual and the velocity residual are combined into a weighted least squares problem, assembled into a block tridiagonal normal equation arranged in time order, and the block diagonal terms and adjacent coupling terms are recorded. Performing the Schur complement recursion and back substitution on the normal equations of the block tridiagonal, the recursive relation is written as: ; in, For time frame indexing, For time frames The block Hessian matrix obtained by linearizing the time-series smoothing factor and velocity factor is given. For time frames With time frame The block coupling matrix between them For time frames The equivalent block matrix, For time frames The first derivative term at that point, For time frames The equivalent first-order term, the above recursion is obtained based on the first-order Taylor expansion of the weighted least squares objective at the current linearization point and the block tridiagonal structure of the normal equation, because elimination in time order can obtain closed recursion without destroying the sparse structure. The attitude increment vector is obtained by back-substitution, the joint parameter vector is updated according to the time index sequence, and a time-consistent attitude sequence is generated. Record the state of the factor graph, which includes the pose node update results, weight matrix Q, weight matrix R and sparse connection relationship; Output time-consistent attitude sequence and factor graph state.

[0032] This invention constructs a cross-frame factor graph and uses a weight matrix Q to weight the difference between adjacent attitude nodes and a weight matrix R to weight the discrete velocity. After linearization, a block tridiagonal normal equation is formed, and the attitude increment is solved in linear time by combining Shur complement recursion and back substitution. This maintains the sparse structure while achieving full sequence constraints, outputs a time-consistent attitude sequence, and records the factor graph state for scheduling and condition integration.

[0033] In the embodiment, the video condition sequence construction and the standardization integration process specifically include: A time index alignment table is established, and the input is a time consistent pose sequence, a factor graph state, a visibility weight sequence, a soft assignment matrix sequence, and an observation tensor; The pose vector, the visibility weight vector, the soft assignment matrix, and the observation tensor slice of each time frame are aligned according to the time index, and a frame-level data set is formed; The observation tensor slice is unfolded into a one-dimensional vector using a fixed channel order, the soft assignment matrix is unfolded into a one-dimensional vector using a column-first order, and the pose vector and the visibility weight vector are spliced according to a predetermined order to form a frame-level original condition vector; The center value and the scale value of each dimension are determined according to the factor weight in the factor graph state and the statistical quantity of the time consistent pose sequence, and are recorded as standardization parameters; The frame-level original condition vector is subjected to center removal and scale adjustment according to the standardization parameters, and a frame-level standardized condition vector is obtained while the time index is preserved; The frame-level standardized condition vector and the time index are combined to form a video condition frame, which is connected in time index order to form a video condition sequence.

[0034] In the video condition sequence construction and the standardization integration process, the pose vector, the visibility weight vector, the soft assignment matrix, and the observation tensor slice are time-aligned and vectorized, the center value and the scale value are calculated according to the factor graph state statistics to complete center removal and scale adjustment, and the video condition frame is formed after the channel order and the scale are unified and sequentially connected to form the video condition sequence, so that the diffusion model can stably receive the structured geometric signal.

[0035] In the embodiment, the matching temperature parameter in the diffusion denoising iteration and the synchronous scheduling of the diffusion noise intensity and the pose consistent guidance generation specifically include: A time step index is established, and the video condition sequence and the factor graph state initial value are loaded, and a matching temperature parameter lower bound constant, a matching temperature parameter upper bound constant, a diffusion noise intensity upper bound constant, and a curve shape coefficient are set; The diffusion noise intensity is calculated according to the time step index, and a diffusion noise intensity time sequence is generated; The matching temperature parameter is calculated at each time step according to the synchronous scheduling, and a matching temperature parameter time sequence is generated; The matching temperature parameter is input to the temperature parameter interface unit at each time step, and the factor graph state is input to the factor graph interface unit; A frame-to-step mapping is established, and the video condition frame is extracted from the video condition sequence according to the time step index, and is input to the differentiable SoftPOSIT module together with the matching temperature parameter and the factor graph state; Generate pose consistent guidance in the differentiable SoftPOSIT module, the processing sequence is extracting time consistent pose vector, soft assignment matrix, visibility weight and observation tensor slice, project skeleton according to time consistent pose vector and align with observation tensor slice, weight pixel difference and component difference according to soft assignment matrix and visibility weight, form residual field, add bone length invariance constraint, joint amplitude constraint and motion chain consistency constraint to get geometric residual set, determine guidance direction according to matching temperature parameter and diffusion noise intensity, map guidance direction to pose consistent guidance component in denoising input space according to observation channel order, and generate factor graph state increment at the same time; Arrange pose consistent guidance component according to time step index to pose consistent guidance sequence, and align with matching temperature parameter time sequence and diffusion noise intensity time sequence; Output matching temperature parameter time sequence, diffusion noise intensity time sequence, pose consistent guidance sequence and updated factor graph state.

[0036] In the diffusion denoising iteration, the diffusion noise intensity and the matching temperature parameter are calculated synchronously according to the time step index, and the frame-to-step mapping is established, the differentiable SoftPOSIT generates pose consistent guidance according to the video condition frame and updates the factor graph state, the noise prediction gradually fits the time consistent pose sequence under the guidance of geometric constraint, the long-range action coherence is maintained, and the occlusion area is more stable.

[0037] Embodiment 1 In order to verify the feasibility of the application in implementation, the application is applied to the dance action migration generation scene, the reference appearance image set is composed of static images of the same person in front, side and rotation three kinds of view angles, the pose sequence is the key point trajectory of professional dancers completing rotation and step, the observation tensor contains three kinds of channels of key point heat map edge response and depth prompt, the skeleton template adopts fixed bone segment length configuration, the material covers three kinds of interference of limb crossing occlusion, rapid angular velocity change and strong contrast light, the resolution is unified to 1280*720, the frame rate is unified to 30 frames per second, the calibration file provides camera intrinsic parameters to ensure projection consistency, and the evaluation data is automatically counted under the unified script to eliminate subjective bias.

[0038] In this scenario, the present application operates the pipeline as given in the claims without user intervention, first constructing the observation tensor with the reference appearance image set and the pose sequence and establishing the skeleton template, then invoking the differentiable SoftPOSIT module between the observation tensor and the skeleton template to construct the soft assignment matrix and give the frame-level pose initial result, while calculating the visibility weight and the matching confidence, and then updating the soft assignment matrix and the frame-level pose initial result alternately in the differentiable SoftPOSIT module and imposing the joint amplitude constraint and the motion chain consistency constraint with the bone length invariance constraint, organizing the initial pose sequence with the matching confidence and generating the visibility weight sequence and the soft assignment matrix sequence, applying the temporal smoothing factor and the velocity factor to the cross-frame factor graph to obtain the time-consistent pose sequence and record the factor graph state, integrating the time-consistent pose sequence, the factor graph state, the visibility weight sequence and the soft assignment matrix sequence into the video conditional sequence, synchronously adjusting the matching temperature parameter and the diffusion noise intensity in the diffusion denoising and calculating the pose consistent guide and the factor graph state increment accordingly, and finally outputting the controllable video consistent with the time-consistent pose sequence and the denoising update according to the pose consistent guide.

[0039] To highlight the key differences, three types of publicly reproducible experimental objects are set as a comparison. The first type is hard assignment plus POSIT solving without differentiability and occlusion dependence fixed mask, the second type is differentiable assignment but the matching temperature parameter is fixed and does not change with noise and confidence, and the third type is cross-frame smoothing average without establishing a factor graph and introducing weight matrices Q and R. The four methods are run under the same material, the same resolution, the same frame rate and the same diffusion step number, and a unified threshold and statistical caliber are adopted. The evaluation indicators include the joint positioning error pixel value PCK@5px percentage, the speed jitter root mean square pixel per frame and the video quality score. The higher the value of the no-reference indicator is, the better it is.

[0040] Table 1 Comparison results of dance motion transfer generation

[0041] Table 1 shows that the present application significantly reduces the joint positioning error and improves the PCK@5px to more than 90% under the same conditions, while the video quality score is stably improved compared with the comparison methods. The differentiable SoftPOSIT connects the gradient link from the observation tensor to the pose and stabilizes the occlusion segment with the joint-level visibility weight, the cross-frame factor graph constrains the adjacent pose node difference and the discrete speed through the weight matrices Q and R, the video conditional sequence uniformly encodes the time-consistent pose sequence, the factor graph state, the visibility weight sequence and the soft assignment matrix sequence for the diffusion model to consume, and the synchronous scheduling of the matching temperature parameter and the noise intensity makes the denoising direction and the geometric consistency converge in the same direction, so that the appearance stability, the motion fitting and the time continuity are all achieved at the same time.

[0042] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art, according to the technical solution and inventive concept of the present application, makes equivalent replacement or change within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A pose-driven controllable video generation method based on a diffusion model, characterized in that, The method comprises the following steps: Obtain a reference appearance image set and a pose sequence, generate an observation tensor and establish a skeleton template; Call a differentiable SoftPOSIT module between the observation tensor and the skeleton template, construct a soft assignment matrix and estimate a frame-level pose initial result, and calculate a visibility weight and a matching confidence; Alternately update the soft assignment matrix and the frame-level pose initial result in the differentiable SoftPOSIT module, apply bone length invariance constraints, joint amplitude constraints and motion chain consistency constraints, and output a frame-level pose estimate; Organize the frame-level pose estimate into an initial pose sequence according to the matching confidence, and generate a visibility weight sequence and a soft assignment matrix sequence; Establish a cross-frame factor graph, take the initial pose sequence as a node and apply a timing smoothing and a speed constraint, output a time-consistent pose sequence and generate a factor graph state; Integrate the time-consistent pose sequence, the factor graph state, the visibility weight sequence, the soft assignment matrix sequence and the observation tensor into a video conditional sequence; Synchronously adjust a matching temperature parameter and a diffusion noise intensity in a diffusion denoising iteration, calculate a pose consistency guide based on the video conditional sequence and the differentiable SoftPOSIT module, and update the factor graph state according to a time step increment; Update a denoising generated video frame sequence according to the pose consistency guide, and output a controllable video consistent with the time-consistent pose sequence.

2. The pose-driven controllable video generation method based on diffusion model according to claim 1, wherein, The differentiable SoftPOSIT module comprises: An input interface unit for receiving an observation tensor, a skeleton template, a matching temperature parameter, a video conditional sequence and a factor graph state; A soft assignment construction unit for constructing a soft assignment matrix according to the observation tensor, the skeleton template and the matching temperature parameter and outputting a matching confidence; A visibility evaluation unit for calculating a visibility weight; A joint solving unit for performing an alternating update between a soft assignment matrix and a frame-level pose initial result and applying bone length invariance constraints, joint amplitude constraints and motion chain consistency constraints, and outputting a frame-level pose estimate; A temperature parameter interface unit for updating a matching temperature parameter according to a time step and distributing it to the soft assignment construction unit and the joint solving unit; A factor graph interface unit for reading a factor graph state according to a time step and writing back the factor graph state; A pose consistency guide calculation unit for generating a pose consistency guide according to a video conditional sequence, a soft assignment matrix sequence, a visibility weight sequence and a time-consistent pose sequence; An output interface unit for outputting a soft assignment matrix, a visibility weight, a matching confidence, a frame-level pose initial result, a frame-level pose estimate and a pose consistency guide.

3. The pose-driven controllable video generation method based on diffusion model according to claim 1, wherein, The differentiable SoftPOSIT matching and initial pose estimation process specifically comprises: Set a frame-level pose initial result, rotate it to a unit matrix and translate it to a zero vector; Project skeleton template joints to pixel positions according to camera intrinsic parameters, extract key point pixel positions and confidence from the observation tensor, construct a matching cost matrix and complete scaling and normalization according to a matching temperature parameter; Determine the matching temperature parameter, the first set value when the diffusion time step is in the high noise stage, the second set value when it is in the transition stage, and the third set value when it is in the low noise stage, the first set value is greater than the second set value, and the second set value is greater than the third set value, when the average confidence of the key points is low and the average value of the visibility weight is low, it is raised to the upper limit of the first set value, when the average confidence of the key points is high and the average value of the visibility weight is high, it is reduced to the lower limit of the third set value, and in other cases, the matching temperature parameter is calculated by linear interpolation between adjacent set values according to the interval; Under the duality and KKT conditions of entropy-regularized optimal transport, Sinkhorn fixed point iteration is performed in the logarithmic domain to obtain a soft assignment matrix; According to the first-order linearization of the soft assignment matrix on the weighted weight projection energy and the weighted least squares solution of the pose increment, the rotation and translation are updated using the exponential mapping, and the frame-level pose initial result is written back; Using the frame-level pose initial result and the skeleton template, a depth layer is generated in the differentiable occlusion buffer, the pixel-level foreground probability is obtained by continuously comparing the part depth along each line of sight, the joint-level visibility weight is obtained by weighted averaging the pixel-level foreground probability in the key point neighborhood of each joint, the semantic part graph is used to set the weight of the non-part region to zero, the distance attenuation weight is used in the contour neighborhood using the edge distance field, and the visibility weight is limited to the interval of zero to one to form the visibility weight; The probability total of each column of the soft assignment matrix is counted to obtain the initial matching confidence of the corresponding joint, the joint-level visibility weight is multiplied and modified, and the total joint range is normalized, and the soft assignment matrix, frame-level pose initial result, visibility weight and matching confidence are output.

4. The pose-driven controllable video generation method based on diffusion model according to claim 1, wherein, The alternating update and constraint joint optimization of the soft assignment matrix and the frame-level pose initial result specifically includes: Set the iteration counter and the stop condition, the stop condition includes the residual error drop amplitude threshold and the upper limit of the number of iterations; According to the frame-level pose initial result, the joints of the skeleton template are projected onto the pixel plane, the corresponding key point pixel positions are read in the observation tensor, the pixel difference is weighted according to the visibility weight and the probability in the soft assignment matrix, and the weighted weight projection residual error and the pose parameter derivative matrix are formed; According to the weighted least squares rule, the pose increment is solved, the exponential mapping is used to update the frame-level pose initial result in the continuous parameter space of three-dimensional rotation and translation, and the updated rotation and translation are written back; Calculate the difference between the current joint distance and the target bone length recorded by the skeleton template, generate a bone length invariant constraint residual and add it to the optimization; According to the upper and lower limits of the angle of each joint, generate a joint amplitude limiting constraint residual, and write the out-of-range angle to the optimization with a piecewise quadratic penalty function; Perform forward kinematics, set the global pose of the root joint as the unit transformation, and sequentially propagate the local rotation and local translation to obtain the global joint position according to the parent-child relationship, calculate the position difference propagated by adjacent bone segments, generate a motion chain consistency constraint residual and add it to the optimization; Recalculate the matching cost matrix under the condition of writing back the updated frame-level pose initial result, update the soft assignment matrix in the logarithmic domain according to the matching temperature parameter, and take the error of the row sum target vector and the column sum target vector less than the preset tolerance as the convergence criterion; According to the stop condition completion judgment, when the stop condition is met, the frame-level pose estimation is output and the updated soft assignment matrix is retained, when the stop condition is not met, the alternating update is continued, and when the upper limit of the iteration number is reached, the frame-level pose estimation is stopped and output.

5. The pose-driven controllable video generation method based on diffusion model according to claim 1, wherein, The initial pose sequence assembly and sequence generation based on the matching confidence specifically include: The frame-level pose estimation, the matching confidence, the visibility weight and the soft assignment matrix are adopted, the matching confidence and the visibility weight of each joint are weighted and summarized, and the frame confidence of each time frame is formed; A frame confidence threshold is set, time frames with frame confidence not lower than the threshold are retained, and a time index sequence is formed; A matching confidence threshold and a visibility weight threshold are set, for each joint of each time frame, when the matching confidence is not lower than the matching confidence threshold and the visibility weight is not lower than the visibility weight threshold, the joint is marked as valid, and a joint valid mask is generated; In each time frame, the joint parameters are extracted from the frame-level pose estimation according to the joint valid mask, for the unmarked joint, the previous valid frame and the next valid frame are searched in time sequence, linear interpolation is performed when both exist, the previous valid frame parameter is adopted when only the previous valid frame exists, and the next valid frame parameter is adopted when only the next valid frame exists, to obtain the time frame pose parameter; The time frame pose parameters are connected according to the time index sequence to generate the initial pose sequence; The visibility weights of each time frame are stacked according to the time index sequence to generate a visibility weight sequence; The soft assignment matrices of each time frame are stacked according to the time index sequence to generate a soft assignment matrix sequence.

6. The pose-driven controllable video generation method based on diffusion model according to claim 1, wherein, The construction of the cross-frame factor graph and the solving of the time-consistent pose sequence specifically include: Pose nodes are established for each time frame of the initial pose sequence on the time index sequence, the node variable is the joint parameter vector, and a node set is formed; A time sequence smoothing factor is set, a smoothing residual is constructed according to the difference between adjacent pose nodes, and is weighted by a weight matrix Q; A speed factor is set, a speed residual is constructed by first-order difference according to the time interval, and is weighted by a weight matrix R; At the current linearization point, the smoothing residual and the speed residual are combined into a weighted least squares problem, assembled into a block tri-diagonal normal equation arranged in time sequence, and the block diagonal items and the adjacent coupling items are recorded; Schur complement recursion and back substitution are performed on the block tri-diagonal normal equation; The pose increment vector is obtained by completing back substitution, the joint parameter vector is updated according to the time index sequence, and a time-consistent pose sequence is generated; The factor graph state is recorded, and the factor graph state includes the pose node update result, the weight matrix Q, the weight matrix R and the sparse connection relationship.

7. The pose-driven controllable video generation method based on diffusion model according to claim 1, wherein, The composition and standardized integration process of the video condition sequence specifically include: A time index alignment table is established, and the input is the time-consistent pose sequence, the factor graph state, the visibility weight sequence, the soft assignment matrix sequence and the observation tensor; The pose vector, the visibility weight vector, the soft assignment matrix and the observation tensor slice of each time frame are aligned according to the time index, and a frame-level data set is formed; The observation tensor slice is unfolded into a one-dimensional vector in a fixed channel order, the soft assignment matrix is unfolded into a one-dimensional vector in a column-first order, and the pose vector and the visibility weight vector are spliced in a predetermined order to form a frame-level original condition vector; Determine each dimension center value and scale value according to factor weight in factor graph state and statistic of time consistent pose sequence, record as normalization parameter; Perform decentering and scale adjustment on frame level original condition vector according to normalization parameter, obtain frame level normalized condition vector and reserve time index; Combine frame level normalized condition vector and time index to form video condition frame, connect as video condition sequence according to time index order.

8. The pose-driven controllable video generation method based on diffusion model according to claim 1, wherein, The matching temperature parameter in the diffusion denoising iteration is synchronously scheduled with the diffusion noise intensity and the pose consistent guidance generation specifically includes: Establish time step index and load video condition sequence and factor graph state initial value, set matching temperature parameter lower bound constant, matching temperature parameter upper bound constant, diffusion noise intensity upper bound constant and curve shape coefficient; Calculate diffusion noise intensity according to time step index, generate diffusion noise intensity time sequence; Calculate matching temperature parameter at each time step according to synchronous scheduling, generate matching temperature parameter time sequence; Input matching temperature parameter to temperature parameter interface unit and input factor graph state to factor graph interface unit at each time step; Establish frame to step mapping, extract video condition frame from video condition sequence according to time step index, and input matching temperature parameter and factor graph state into differentiable SoftPOSIT module together; Generate pose consistent guidance in differentiable SoftPOSIT module, the processing order is to extract time consistent pose vector, soft assignment matrix, visibility weight and observation tensor slice, perform skeleton projection according to time consistent pose vector and align with observation tensor slice, weight pixel difference and component difference according to soft assignment matrix and visibility weight to form residual field, superimpose into geometric residual set by adding bone length invariance constraint, joint amplitude constraint and motion chain consistency constraint, determine guidance direction according to matching temperature parameter and diffusion noise intensity, map guidance direction to pose consistent guidance component in denoising input space according to observation channel order, and generate factor graph state increment at the same time; Arrange pose consistent guidance component into pose consistent guidance sequence according to time step index, and align with matching temperature parameter time sequence and diffusion noise intensity time sequence.

Images