A method and system for generating videos of abnormally restless movements in infants with cerebral palsy

Abstract

This invention discloses a method and system for generating videos of abnormally restless movements in infants with cerebral palsy, belonging to the field of image processing technology. The method involves acquiring single-frame appearance images from different videos of infants with cerebral palsy's movements, and a sequence of original target poses composed of multiple consecutive single-frame appearance images. The single-frame appearance images and the original target pose sequence are then input into a pre-trained video generation network to generate a predicted video sequence. The video generation network includes a cascaded first generator and a second generator. The first generator takes the single-frame appearance images and the original target pose sequence as input and outputs a coarse-grained video sequence. The second generator takes the single-frame appearance images and the coarse-grained video sequence as input and generates the predicted video sequence. The video generation network constructed by this method can generate video sequences that possess both visual realism and faithfully preserve the pathological kinematic features of abnormal FM (cerebral palsy-related motor dysfunction).

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method and system for generating videos of abnormally restless movements in infants with cerebral palsy. Background Technology

[0002] Cerebral palsy (CP) is a persistent motor and postural disorder caused by brain injury or abnormal brain development during the developmental period. It is usually accompanied by activity limitations and can have long-term effects on the child's motor function and quality of life. Because early infancy presents a crucial window of neuroplasticity, early identification and timely intervention in infants at high risk of CP are of significant clinical importance. In current clinical practice, the Prechtl whole-body motor assessment is widely used for early screening of infant neurodevelopment. Among these assessments, restless movements (FM) are an important predictive indicator. Restless movements (FM) typically appear approximately 9 to 20 weeks after the infant's corrected gestational age. Normal FM is characterized by small, continuous, and varied spontaneous movements of the neck, trunk, and limbs; while infants at high risk of CP often exhibit abnormal characteristics such as absent, sparse, monotonous, or insufficient spatiotemporal variation in FM. Therefore, abnormal FM has become an important kinematic marker for the early identification of CP.

[0003] With the development of computer vision and artificial intelligence technologies, video-based computer-aided motor assessment (GMMA) has gradually become an important technical direction for infant neurodevelopmental evaluation. Compared with wearable sensor solutions, video acquisition has advantages such as being non-contact, low-interference, and not restricting the infant's natural spontaneous activities, making it more suitable for continuous observation and intelligent analysis of the infant's whole-body movements. However, video-based automated assessment methods typically rely on a large amount of high-quality, expert-annotated infant movement video data. Due to limitations such as medical ethics, privacy protection, complex clinical acquisition conditions, long follow-up periods, and high expert annotation costs, abnormal FM video samples in existing clinical data are particularly scarce. This leads to problems such as insufficient samples, imbalanced category distribution, and insufficient coverage of abnormal movement patterns during the training of related models, thereby limiting the generalization ability and clinical application effectiveness of the recognition models.

[0004] To alleviate the problem of insufficient abnormal video samples, existing technologies have proposed data augmentation methods such as generative adversarial networks and diffusion models. These methods can generate video sequences with a certain degree of visual realism by learning the image distribution and time-series variation patterns of video data.

[0005] However, existing general video generation methods are mainly geared towards natural video scenes, with optimization goals focusing on visual realism and overall smoothness. When directly applied to the generation of abnormal movement (FM) videos of infants with cerebral palsy, the inability to accurately model the pathological features of FM leads to problems such as the easy loss of pathological movement features and difficulty in balancing visual plausibility and pathological fidelity. The generated videos have low clinical recognizability and cannot effectively serve subsequent auxiliary diagnosis or model training. Summary of the Invention

[0006] To address the problems existing in the above-mentioned fields, this invention proposes a method and system for generating videos of abnormal restless movements in infants with cerebral palsy. The constructed video generation network can generate video sequences that are both visually realistic and faithfully preserve the pathological kinematic features of abnormal FM (muscle movement), providing reliable data support for early intelligent screening and auxiliary diagnostic model training for cerebral palsy.

[0007] To address the aforementioned technical problems, this invention discloses a method for generating videos of abnormally restless movements in infants with cerebral palsy, comprising the following steps: Acquire single-frame appearance images from movement videos of infants with cerebral palsy, and the original target pose sequence composed of multiple consecutive single-frame appearance images; A single-frame appearance image and the original target pose sequence are input into a pre-trained video generation network to generate a predicted video sequence. The video generation network includes a cascaded first generator and a second generator. The first generator is based on a U-Net structure, with its encoder part being a parallel dual-branch structure of conditional branch and pose branch, and a temporal enhancement module is introduced before the decoder. The second generator is based on a U-Net structure, with its encoder part being a parallel appearance branch and motion branch, and a 3D-SE attention module is introduced between the output of the appearance branch and the decoder. The conditional branch extracts shape features from a single frame of the appearance image; the pose branch extracts pose features from the original target pose sequence; the temporal enhancement module performs cross-frame dynamic information aggregation on the fused features based on the fusion of shape and pose features, outputs temporal enhancement features, and uses them as input to the decoder to generate a coarse-grained video sequence. Specifically, the appearance branch uses a pseudo-3D convolution with a temporal kernel size of 1 to extract pure spatial shape features from a single frame of the appearance image; the 3D-SE attention module performs channel recalibration on the pure spatial shape features to obtain enhanced shape features; the motion branch uses standard 3D convolution to extract motion features from the coarse-grained video sequence; the decoder fuses the enhanced shape features and motion features to generate a residual detail compensation image; and adds the residual detail compensation image to the coarse-grained video sequence pixel by pixel to generate a predicted video sequence.

[0008] Preferably, the attitude branch extracts attitude features from the original target attitude sequence, specifically including: The enhanced attitude representation is obtained by performing adjacent frame difference on the original target attitude sequence and then concatenating it with the original target attitude sequence channels. The posture branch extracts posture features that enhance the posture representation.

[0009] Preferably, the step of performing adjacent frame difference on the original target pose sequence and then concatenating it with the original target pose sequence channels to obtain the enhanced pose representation specifically includes: Based on the difference between the pose of the current video frame and the pose of the next video frame in the original target pose sequence, a pose difference sequence is obtained. The attitude difference sequence is concatenated with the original target attitude sequence in the channel dimension to obtain the enhanced attitude representation.

[0010] Preferably, the temporal enhancement module performs cross-frame dynamic information aggregation on the fused features based on shape and pose feature fusion, and outputs temporal enhancement features, specifically including: The timing enhancement module includes multiple cascaded 3D residual units, each with a different temporal receptive field. The local short-term dynamic information and the relative long-term dynamic information of multiple 3D residual units are aggregated and fused to obtain temporal enhancement features.

[0011] Preferably, the decoder fuses the enhanced shape features and motion features to generate a residual detail-compensated image, specifically including: The decoder of the second generator uses progressive upsampling and parallel convolutional units to decode the fused features, and uses long skip connections to introduce the multi-scale encoded features generated by the appearance branch in the process of extracting pure spatial shape features and the multi-scale encoded features generated by the motion branch in the process of extracting motion features into the decoder, so as to collaboratively recover multi-scale high-frequency information and generate residual detail compensation images for compensating for missing high-frequency textures and local deformation details in coarse-grained video sequences.

[0012] Preferably, the training process of the video generation network specifically includes: Acquire single-frame appearance images from movement videos of infants with cerebral palsy, a sequence of original target poses composed of multiple consecutive single-frame appearance images, and the labels corresponding to the original target pose sequences; A total loss function is constructed based on the difference between the predicted video sequence generated by the video generation network and the label. A training set is constructed based on single-frame appearance images from different videos of infants with cerebral palsy, a sequence of original target poses composed of multiple consecutive single-frame appearance images, and the labels corresponding to the original target pose sequences. By using the training set and the total loss function, the video generation network is optimized to obtain a pre-trained video generation network; The total loss function includes a spectral constraint loss, which is obtained by performing a global real-number fast Fourier transform on the predicted video sequence and the label along the time dimension to determine the difference in spectral energy distribution between the two in a preset frequency band. The second generator is constrained by the spectral constraint loss.

[0013] Preferably, the total loss function further includes adversarial loss and pixel-level reconstruction loss; The adversarial loss is output by a dual-head 3D PatchGAN discriminator; the dual-head 3D PatchGAN discriminator includes a shared 3D encoder, a spatial realism head, and a temporal consistency head; wherein, the shared 3D encoder extracts hierarchical spatiotemporal features of the predicted video sequence and the tags; the spatial realism head evaluates the realism of local textures based on the hierarchical spatiotemporal features; the temporal consistency head evaluates the rationality of dynamic changes based on the neighboring frame difference features of the hierarchical spatiotemporal features; The training process of the second generator is constrained by the adversarial loss; The pixel-level reconstruction loss is obtained by determining the L1 distance between the predicted video sequence and the label, and by weighting the L1 distance of the foreground region where the infant's body is located based on the human body mask obtained by thresholding the pose heatmap. By incorporating the pixel-level reconstruction loss into the total loss function, the network parameters of the first generator and the second generator are simultaneously optimized using the backpropagation algorithm.

[0014] Preferably, the total loss function further includes a consistency constraint based on frame difference; The frame difference-based consistency constraint is used to constrain the inter-frame difference between the predicted video sequence and the label at a preset sampling interval. The training processes of both the first and second generators are constrained by the frame difference-based consistency constraint.

[0015] Preferably, the total loss function further includes perceptual loss; The perceptual loss is obtained by inputting the predicted video sequence and the label into a pre-trained feature extraction network to obtain the corresponding high-level semantic feature representations; and is based on the difference between the high-level semantic feature representations of the predicted video sequence and the label. The training process of the second generator is constrained by the perceptual loss; The pre-trained feature extraction network includes the VGG-19 network.

[0016] Preferably, it also includes a system for generating videos of abnormally restless movements in infants with cerebral palsy, comprising: The data acquisition module is used to acquire single-frame appearance images and original target posture sequences composed of multiple consecutive single-frame appearance images from different videos of infants with cerebral palsy. The video generation module is used to input a single-frame appearance image and the original target pose sequence into a pre-trained video generation network to generate a predicted video sequence; A video generation network construction module is used to construct a video generation network, wherein the video generation network includes a cascaded first generator and a second generator; the first generator is based on a U-Net structure, its encoder part is a parallel dual-branch structure of conditional branch and pose branch, and a temporal enhancement module is introduced before the decoder; the second generator is based on a U-Net structure, its encoder part is a parallel appearance branch and motion branch, and a 3D-SE attention module is introduced between the output of the appearance branch and the decoder; The conditional branch extracts shape features from a single frame of the appearance image; the pose branch extracts pose features from the original target pose sequence; the temporal enhancement module performs cross-frame dynamic information aggregation on the fused features based on the fusion of shape and pose features, outputs temporal enhancement features, and uses them as input to the decoder to generate a coarse-grained video sequence. Specifically, the appearance branch extracts pure spatial shape features from a single frame of appearance image using a pseudo-3D convolution with a temporal kernel size of 1; the 3D-SE attention module performs channel recalibration on the pure spatial shape features to obtain enhanced shape features; the motion branch extracts motion features from the coarse-grained video sequence using standard 3D convolution; the decoder fuses the enhanced shape features and motion features to generate a residual detail-compensated image; and adds the residual detail-compensated image pixel-by-pixel to the coarse-grained video sequence to generate a predicted video sequence. The training process of the video generation network specifically includes: Acquire single-frame appearance images from movement videos of infants with cerebral palsy, a sequence of original target poses composed of multiple consecutive single-frame appearance images, and the labels corresponding to the original target pose sequences; A total loss function is constructed based on the difference between the predicted video sequence generated by the video generation network and the label. A training set is constructed based on single-frame appearance images from different videos of infants with cerebral palsy, a sequence of original target poses composed of multiple consecutive single-frame appearance images, and the labels corresponding to the original target pose sequences. By using the training set and the total loss function, the video generation network is optimized to obtain a pre-trained video generation network; The total loss function includes a spectral constraint loss, which is obtained by performing a global real-number fast Fourier transform on the predicted video sequence and the label along the time dimension to determine the difference in spectral energy distribution between the two in a preset frequency band. The second generator is constrained by the spectral constraint loss.

[0017] Compared with the prior art, the present invention has the following beneficial effects: This invention proposes a method for generating videos of abnormally restless movements in infants with cerebral palsy. The video generation network is constructed using a cascaded first and second generator, forming a two-stage progressive network. In the first stage, the focus is on generating coarse-grained video sequences with consistent appearance and reasonable geometric structure, providing a stable and reliable foundation for subsequent refinement. In the second stage, instead of relearning the global structure, the focus is on predicting residual-based detail-compensated images. This coarse-to-fine residual learning mechanism effectively avoids the problem of erasing subtle pathological dynamic features in single-stage generation in pursuit of overall visual smoothness, achieving a balance between visual plausibility and fidelity in pathological movement.

[0018] The constructed video generation network, by introducing a temporal enhancement module before the decoder of the first generator, fully integrates local and long-range temporal context information before restoring spatial resolution, significantly improving the motion coherence and temporal stability of coarse-grained video sequences. In the second generator, pure spatial shape features extracted through pseudo-3D convolution with a temporal kernel size of 1 avoid confusion with motion information. Simultaneously, a 3D-SE attention module is introduced at the output of the appearance branch encoder to recalibrate the channels of pure spatial shape features, adaptively enhancing the feature responses of human body boundaries, clothing textures, and local motion-sensitive areas. Combined with motion features extracted by standard 3D convolution for fusion decoding, the generated residual compensation image accurately compensates for the missing high-frequency textures and local deformation details in the coarse-grained video, significantly improving the clinical recognizability of the generated video. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method for generating videos of abnormally restless movements in infants with cerebral palsy proposed in this invention. Figure 2 The overall network architecture diagram of the video generation network constructed for this invention; Figure 3 A schematic diagram of enhanced attitude representation provided for an embodiment of the present invention; Figure 4 This is a network architecture diagram of the timing enhancement module provided in an embodiment of the present invention; Figure 5 This is a network architecture diagram of the decoder for the second generator provided in an embodiment of the present invention; Figure 6The method implementation steps provided in the embodiments of the present invention; Figure 7 Qualitative comparison results of the proposed method (Ours) of this invention and the above-mentioned representative methods on complex pose sequences, provided for embodiments of this invention; Figure 8 A comparison of the time-frequency spectra generated by the proposed method (Ours) in this embodiment of the invention with those generated by the aforementioned representative methods is provided, wherein... Figure 8 In the table, (a) is the time spectrum generated using the method of this invention, (b) is the time spectrum generated using Magicpose, (c) is the time spectrum generated using DreamPose, and (d) is the time spectrum generated using MinicMotion. Detailed Implementation

[0020] The following will refer to the appendices in the embodiments of the present invention. Figures 1-8 The technical solutions in the embodiments of the present invention will be clearly and completely described. It should be understood that the terminology used in the present invention is only for describing particular implementation methods and is not intended to limit the present invention.

[0021] Example like Figure 1 As shown, this invention provides a method for generating videos of abnormally restless movements in infants with cerebral palsy, comprising the following steps: S1: Obtain single-frame appearance images from movement videos of different infants with cerebral palsy. The original target pose sequence consists of multiple consecutive single-frame appearance images. ; S2: Transform single-frame appearance image and the original target attitude sequence The input is fed into a pre-trained video generation network to generate predicted video sequences; S3: Construct a video generation network, which includes a cascaded first generator. Second generator ,like Figure 2 As shown: First Generator The U-Net architecture is a symmetrical dual-branch network. Its encoder includes parallel conditional branches and pose branches, and a temporal enhancement module is introduced before the decoder. The conditional branch extracts a single-frame appearance image. Shape features; attitude branch based on the original target attitude sequence As input, corresponding pose features are extracted through multi-scale spatiotemporal coding; the temporal enhancement module performs cross-frame dynamic information aggregation on the fused features based on shape and pose feature fusion, outputting temporal enhancement features, which are then used as input to the decoder to generate coarse-grained video sequences. Its goal is to establish a basic video representation that is visually consistent, has controllable posture, and has a reasonable geometric structure.

[0022] Second generator This includes an asymmetric dual-branch U-Net architecture, where the encoder comprises an appearance branch and a motion branch, and a 3D-SE attention module is introduced between the output of the appearance branch and the decoder; the appearance branch extracts a single-frame appearance image using a pseudo-3D convolution with a temporal kernel size of 1. The pure spatial shape features are extracted; the 3D-SE attention module performs channel recalibration on the pure spatial shape features to obtain enhanced shape features; the motion branch uses standard 3D convolution to extract motion features from the coarse-grained video sequence; the decoder fuses the enhanced shape features and motion features to generate a residual detail-compensated image; and the residual detail-compensated image is added pixel-by-pixel to the coarse-grained video sequence to generate a predicted video sequence. .

[0023] The video generation network constructed in this invention employs a cascaded first generator. Second generator The progressive two-stage generation process emphasizes structural correctness in the first stage and emphasizes the authenticity of details and dynamic fidelity in the second stage, thereby achieving progressive optimization from appearance consistency to pathological motion fidelity.

[0024] Specifically, in step S1, this embodiment uses the MINI-RGBD dataset, which was legally and compliantly acquired. This dataset is considered a representative public dataset in the field of infant motion analysis. MINI-RGBD has been widely used in pose estimation, action recognition, and motion generation research, and is currently the only public dataset containing anomalous data.

[0025] This dataset contains 12,000 RGB images of 12 infants captured from a top-down viewpoint, with 1,000 frames per infant and a sampling frame rate of 25 FPS. Because the top-down viewpoint can preserve the infant's full-body motion information relatively completely, it is suitable for cross-individual pose modeling and motion generation tasks.

[0026] To improve the network's cross-individual generalization ability, this embodiment adopts a subject-independent data partitioning strategy, that is, 10 infants are selected for training and validation (3 abnormal and 7 normal), and the remaining 2 infants are used for testing (1 abnormal and 1 normal), and there is no overlap of subjects between the subsets.

[0027] In this embodiment, a cross-individual pairing method is adopted in the sample construction process: 20 consecutive appearance images of the same infant are used as the target pose sequence, and sliding sampling is performed with a step size of 10 frames. Each infant can generate 99 sequences, so a total of 198 samples are obtained in the testing phase.

[0028] During the training and validation phase, single-frame appearance images were sampled from motion videos of other subjects as reference images and paired with target pose sequences, ultimately resulting in 5,742 pairs of cross-individual samples.

[0029] Traditional pose-driven generation typically uses absolute pose heatmaps as input, requiring the network to implicitly learn both static pose structure and inter-frame motion changes, which can easily lead to timing jitter and pose jumps.

[0030] Furthermore, to explicitly enhance motion evolution modeling capabilities, this invention is based on the pose of the current video frame in the original target pose sequence. and the pose of the previous video frame The difference is used to obtain the pose difference sequence of the current video frame. for: ; Where T is the total number of frames in the video sequence, representing the total number of consecutive video frames contained in the original target pose sequence.

[0031] attitude difference sequence P=[ , ,..., ] and the original target attitude sequence P=[ , ,..., By stitching along the channel dimension, an enhanced pose representation is obtained. for: ; This design enables the network to directly perceive relative displacement trends while modeling absolute structure, thereby improving the coherence of motion between frames, such as... Figure 3 The diagram shown is a schematic representation of the enhanced attitude characterization provided in this embodiment. Figure 3 (a) in the diagram represents the original pose keypoint; Figure 3 In (b), the differential enhancement representation is shown, and the arrows in (b) represent the inter-frame keypoint displacement vectors.

[0032] Specifically, in step S2, the first generator is based on a U-Net structure, its encoder part is a parallel conditional branch and pose branch dual-branch structure, and a timing enhancement module is introduced before the decoder, wherein: The conditional branch is responsible for capturing a single frame of the appearance image. The appearance texture and identity features are analyzed to extract shape features; the pose branch is responsible for processing the original target pose sequence. The obtained enhanced pose representation Multi-scale spatiotemporal coding is performed to extract pose features. The two branches adopt a symmetrical design to maintain the semantic alignment of appearance and motion features in the latent space.

[0033] To further enhance cross-frame dependency modeling, the temporal enhancement module constructed in this invention includes multiple cascaded 3D residual units, each with a different temporal receptive field. The multiple 3D residual units aggregate and fuse the local short-term dynamic information and the relative long-term dynamic information of the features to obtain temporal enhancement features, which are then used as input to the decoder.

[0034] like Figure 4 The diagram shows the network architecture of the temporal enhancement module provided in this embodiment. The feature fusion part on the left concatenates pose features (FP) and shape features (FS) and maps them to a spatiotemporal feature tensor through a fully connected layer. The temporal modeling part in the middle uses two 3D residual blocks with kernel sizes of (5,3,3) and (3,3,3) to capture spatiotemporal dependencies. The right side shows the internal structure of the 3D residual block, which contains two 3D convolutional layers and skip connections.

[0035] The decoder, combined with long-hop connections, recovers spatial resolution step by step, ultimately generating a coarse-grained video sequence with visual consistency, structural rationality, and preliminary motion coherence. .

[0036] First generator The purpose of the video generation network in the first stage of the generation process is to establish a basic video representation with consistent appearance, controllable posture, and reasonable geometric structure. It is not to directly generate the final result, but to provide a stable and reliable structural basis for subsequent detailed pathological dynamics.

[0037] Second generator An asymmetric dual-branch U-Net architecture is adopted, in which the encoder includes an appearance branch and a motion branch, with the appearance branch based on conditional images. As input, pseudo-3D convolution with a temporal kernel size of 1 is used to extract pure spatial shape features, and its motion branch is based on coarse-grained video sequences. As input, standard 3D convolution is used to jointly model cross-frame spatiotemporal patterns, generating coarse-grained video sequences. The motion characteristics.

[0038] Unlike the first stage, the second stage no longer relearns the global structure, but focuses on enhancing identity textures, local deformations, and fine-grained motion representation.

[0039] To improve the characterization ability of key regions, this invention uses a second generator... A 3D squeeze-excited attention module (3D-SE attention module) is introduced between the output of the appearance branch and the decoder. It adaptively enhances the feature responses of human body boundaries, clothing textures and local motion-sensitive areas through a channel recalibration mechanism.

[0040] like Figure 5 The diagram shown is a network architecture diagram of the decoder of the second generator provided in this embodiment, including a conditional fusion block, decoding block 1, decoding block 2, and an output layer. The conditional fusion block concatenates enhanced shape features and motion features; decoding blocks 1 and 2 gradually restore spatiotemporal resolution through 3D convolution, upsampling, and skip connections; the output layer generates the final video sequence through tanh activation.

[0041] Second generator The decoder employs progressive upsampling and parallel convolutional units to decode the features after the fusion of enhanced shape and motion features. It also utilizes skip connections to introduce multi-scale encoded features generated by the appearance branch during the extraction of pure spatial shape features and multi-scale encoded features generated by the motion branch during the extraction of motion features into the decoder. This collaboratively recovers multi-scale high-frequency information and generates residual detail compensation images to compensate for missing high-frequency textures and local deformation details in coarse-grained video sequences, thereby improving local texture clarity and detail expression.

[0042] To make the refinement process more realistic not only at the single-frame level but also more in line with the laws of real pathological movement at the temporal level, this invention further designs a dual-head 3D PatchGAN discriminator, such as... Figure 2 As shown in "Second Stage: Discriminator (D)" in the diagram, the discriminator can constrain texture realism on a local patch basis, and its temporal extension in video tasks has been shown to help enhance cross-frame consistency. Based on this, the discriminator of this invention first extracts hierarchical spatiotemporal features of the predicted video sequence and labels through a shared 3D encoder, and then divides them into a spatial realism head and a temporal consistency head. The spatial realism head evaluates local texture realism based on hierarchical spatiotemporal features, while the temporal consistency head explicitly evaluates the reasonableness of dynamic changes based on the neighboring frame difference features of the hierarchical spatiotemporal features. The outputs of the spatial realism head and the temporal consistency head are weighted and fused to form the overall discriminant result, thus enabling the adversarial supervision in the second stage to simultaneously cover both the realism of local details and the reasonableness of temporal changes.

[0043] Network optimization To achieve progressive optimization from structural rationality to pathological realism, this invention constructs differentiated loss constraints around the functional differences between the two stages. Overall, the first stage focuses on structural reconstruction and basic temporal smoothing, while the second stage further emphasizes local texture realism, temporal consistency, and pathological dynamic fidelity.

[0044] First, pixel-level reconstruction loss is used in both stages to ensure that the generated result is consistent with the overall appearance of the real video.

[0045] The pixel-level reconstruction loss constructed in this invention To obtain the prediction loss, we determine the L1 distance between the predicted video sequence and the label, and then weight the L1 distance of the foreground region containing the infant's body based on the human mask obtained through pose heatmap thresholding. for; ; in, For tags, To predict video sequences, This is a human mask obtained by thresholding pose heatmaps. For Hadamard product; This is the foreground weighting coefficient, used to control the weight of the foreground region L1 loss in the total loss.

[0046] By incorporating pixel-level reconstruction loss into the total loss function, the backpropagation algorithm is used to simultaneously optimize the network parameters of the first and second generators.

[0047] Furthermore, to suppress inter-frame jitter and enhance cross-frame continuity, this invention introduces a consistency constraint based on frame difference: in, Indicates the sampling interval. The tag corresponding to frame t. For the t-th The tag corresponding to the frame, Let t be the predicted video sequence corresponding to the t-th frame. For the t-th The predicted video sequence corresponding to the frame.

[0048] This loss directly constrains the consistency of the predicted video sequence and the label in the inter-frame changes at a preset sampling interval, enabling the network to maintain a smooth and reasonable dynamic evolution while recovering details.

[0049] By incorporating a consistency constraint based on frame difference into the total loss function, the backpropagation algorithm is used to simultaneously optimize the network parameters of the first and second generators.

[0050] An adversarial loss output by the discriminator is constructed. By adding the adversarial loss to the total loss function, the network parameters of the second generator are optimized using the backpropagation algorithm to constrain the training process of the second generator, so that the generator is supervised at both the level of local texture realism and overall dynamic rationality.

[0051] Because abnormal movements in infants at high risk of cerebral palsy are not random perturbations but rather specific pathological kinematic patterns reflected in the temporal structure, existing research has shown that automated infant movement analysis can identify dynamic differences associated with cerebral palsy risk during the general movements (FM) stage. These differences are not only reflected in temporal movement patterns but also in frequency domain statistical features. Therefore, relying solely on pixel reconstruction and adversarial learning may still cause models to miss the most clinically significant subtle abnormal dynamics in the pursuit of visual smoothness.

[0052] Therefore, the spectrum-constrained loss constructed in this invention MoveNet-Thunder (a lightweight, high-precision human pose estimation model developed by Google, a variant of the MoveNet series) is used to extract predicted video keypoint sequences from predicted video sequences generated by the constructed video generation network. And the real video key point sequence of the tags. ,in n =14 indicates the number of key points selected for the limb joints, and 3 includes the two-dimensional coordinates x, y and the confidence score. b For batches, T The number of frames.

[0053] To focus on motion trajectories, this invention retains only the predicted video keypoint sequence. and real video key point sequences The two-dimensional coordinates x and y are used to reorganize and predict the video keypoint sequence. and real video key point sequences Converted into corresponding time-domain motion signals ,2 contains two-dimensional coordinates x and y.

[0054] Subsequently, the time-domain motion signal Perform global real number fast Fourier transform The energy of the spectrum is then normalized to obtain the amplitude-normalized spectrum. for: ; in, f It is the global real number fast Fourier transform. Frequency index, N This represents the total number of frequency components after the global real-number Fast Fourier Transform. To prevent constant terms with a denominator of zero, this embodiment takes... .

[0055] Furthermore, the spectral constraint loss for limb joints is defined as the normalized spectral amplitude of the predicted video sequence. Normalized spectral amplitude of the label The mean square error.

[0056] Obtain the spectral constraint loss of limb joints for: in, To predict the normalized spectral amplitude corresponding to a video sequence, The normalized spectral amplitude corresponding to the label. [.] represents the arithmetic mean operator, which means taking the arithmetic mean over all batches b, all key points n of all limb joints, and all coordinate components d, compressing the multidimensional error into a scalar loss value.

[0057] In this invention, the role of this loss is not simply to enhance motion amplitude, but rather to explicitly guide the generator to retain dynamic energy patterns consistent with real pathological motion during the refinement process. This elevates the optimization objective of the second stage from appearance refinement to aligning appearance refinement with the dynamic mechanism. Combined with the inherent bias in the frequency domain modeling of the generative model, this constraint constitutes a key design feature that distinguishes this invention from general human video generation methods.

[0058] Furthermore, to further improve the quality of visual perception, this invention introduces perceptual loss. and multi-scale structural similarity loss ,in: Perceived loss By inputting the predicted video sequence and the label into a pre-trained feature extraction network, the corresponding high-level semantic feature representations are obtained; and the high-level semantic feature representations of the predicted video sequence and the label are obtained based on the difference between them. The pre-trained feature extraction network includes the VGG-19 network.

[0059] By perceiving loss By incorporating a total loss function and utilizing backpropagation to optimize the network parameters of the second generator, the insufficient textural and structural representation capabilities of pixel-level loss are addressed. Multi-scale structural similarity loss is also employed. Measuring the generation quality from the perspective of multi-scale structural consistency is more in line with the human eye's perception of structural information.

[0060] Taking into account the above constraints, the total loss of the first generator in the network constructed in this invention is... Total loss of the second generator They are defined as follows: in, Combating losses , , , , , , and All are weighting coefficients.

[0061] Through this phased and goal-oriented optimization design, the first stage is mainly responsible for learning coarse-grained results with correct structure, smooth posture, and stable motion. The second stage further refines the pathological video to achieve realistic details, spectral alignment, and reliable dynamics. Thus, this invention, centered on the core goal of preserving the fidelity of pathological motion, establishes a step-by-step optimization from input representation, structure generation, refinement enhancement to spectral alignment.

[0062] like Figure 6 The diagram shows the implementation steps of the method provided in this embodiment, including input data construction, pose difference enhancement, coarse-grained structure generation and construction, pathological dynamic refinement generation and construction, dual discriminator joint optimization design, spectral constraint implementation, and evaluation of the predicted video sequence output by the network.

[0063] To verify the effectiveness of the proposed method, comparative experiments were conducted from both quantitative and qualitative perspectives. Representative methods for pose-guided human video generation were selected for comparison, including MiniCMotion, MagicPose, and DreamPose. To improve evaluation efficiency, the resolution of all videos was standardized to 256×256, and a consistent test set was used to comprehensively evaluate different metrics, thus ensuring fairness and reliability.

[0064] like Figure 7 As shown, the qualitative comparison results of the proposed method (Ours) and the aforementioned representative methods on complex pose sequences are presented, where GT represents the ground truth image. The comparison clearly shows that the aforementioned representative methods generally have significant shortcomings when handling large pose changes. For example, DreamPose shows obvious deficiencies when dealing with complex poses such as bent arms, failing to effectively capture detailed changes. The proposed method, however, demonstrates a stronger advantage in this regard, better preserving structural details, and exhibiting higher accuracy and stability, especially when handling complex pose changes.

[0065] Table 1. Quantitative evaluation results (test resolution 256×256) Model PSNR↑ tSSIM↑ FID↓ FVD↓ HE↓ Magicpose 24.95 0.910 128.49 65.80 4.40 MinicMotion 17.86 0.824 143.05 84.13 4.81 DreamPose 17.28 0.822 176.57 110.79 4.80 The method proposed in this invention (Our) 28.54 0.937 138.17 82.15 3.75 Table 1 shows the quantitative evaluation results of the proposed method and the above-mentioned representative methods on the MINI-RGBD dataset using different indicators in this embodiment.

[0066] Table 1 shows that the proposed method achieves good performance in various evaluation metrics, including Peak Signal-to-Noise Ratio (PSNR), Temporal Structural Similarity (tSSIM), Fréchet Inception Distance (FID), Fréchet Video Distance (FVD), and Spectral Error (HE). The HE metric represents the difference between the generated predicted video sequence and the label across the entire spectrum.

[0067] To further verify the quality of the predicted video sequence Figure 8 The proposed method (Ours) is compared with the time-frequency spectrum generated by the aforementioned representative methods. The analysis was conducted from the perspective of the low-frequency domain, namely the frequency domain of the main movements of infants and young children. The results show that the spectrum curve of the video generated by the present invention is closer to the spectrum curve of the real video, indicating that it has stronger fidelity in low-frequency information reconstruction.

[0068] ablation experiment To verify the contribution of each key module to the overall method, Table 2 shows the analysis results of the ablation experiment conducted at 96×128 resolution. The ablation results are reported by progressively introducing the temporal enhancement module and the 3D-SE attention module.

[0069] Table 2 Ablation Experiment Results (Test Resolution 96×128) Timing Enhancement Module 3D-SE Attention Module PSNR↑ tSSIM↑ FVD↓ HE↓ 22.81 0.863 46.47 0.47 √ 23.16 0.861 47.91 0.43 √ √ 23.22 0.867 34.60 0.37 The results show that the baseline model (referring to the simplified video generation network without the temporal enhancement module and the 3D-SE attention module) has a PSNR of 22.81 and an FVD of 46.47. After introducing the temporal enhancement module, the PSNR increased to 23.16, but the FVD increased to 47.91, indicating that pixel-level constraints may have a negative impact on temporal consistency. After adding 3D-SE, the FVD significantly decreased to 34.60 (a decrease of 27.8%), and the tSSIM increased to 0.867, verifying the key role of the proposed method in improving temporal consistency. The proposed method achieves a PSNR of 23.22, a tSSIM of 0.867, and an HE of 0.37, with all metrics reaching optimal levels.

[0070] The above ablation experiment results show that the temporal enhancement module and the 3D-SE attention module effectively complement each other, jointly optimizing the perception quality, and the synergistic effect of the two achieves a comprehensive performance improvement.

[0071] This invention also proposes a system for generating videos of abnormally restless movements in infants with cerebral palsy, comprising: The data acquisition module is used to acquire single-frame appearance images and original target posture sequences composed of multiple consecutive single-frame appearance images from different videos of infants with cerebral palsy. The video generation module is used to input a single-frame appearance image and the original target pose sequence into a pre-trained video generation network to generate a predicted video sequence; A video generation network building module is used to construct a video generation network, which includes a cascaded first generator and a second generator. The first generator is based on the U-Net structure, and its encoder part is a parallel dual-branch structure of conditional branch and pose branch, and a temporal enhancement module is introduced before the decoder. The second generator is based on the U-Net structure, and its encoder part is a parallel appearance branch and motion branch, and a 3D-SE attention module is introduced between the output of the appearance branch and the decoder. The conditional branch extracts the shape features of a single frame of appearance image; the pose branch extracts the pose features of the original target pose sequence; the temporal enhancement module performs cross-frame dynamic information aggregation on the fused features based on the fusion of shape features and pose features, outputs temporal enhancement features, and uses them as input to the decoder to generate a coarse-grained video sequence. The appearance branch extracts pure spatial shape features from a single frame of appearance image using a pseudo-3D convolution with a temporal kernel size of 1; the 3D-SE attention module performs channel recalibration on the pure spatial shape features to obtain enhanced shape features; the motion branch extracts motion features from the coarse-grained video sequence using standard 3D convolution; the decoder fuses the enhanced shape features and motion features to generate a residual detail compensation image; and then adds the residual detail compensation image to the coarse-grained video sequence pixel by pixel to generate a predicted video sequence. The training process of the video generation network specifically includes: Obtain single-frame appearance images from different movement videos of infants with cerebral palsy, and the labels corresponding to the original target pose sequence composed of multiple consecutive single-frame appearance images; A total loss function is constructed based on the difference between the predicted video sequences generated by the video generation network and the labels. A training set is constructed based on single-frame appearance images from different videos of infants with cerebral palsy and the labels corresponding to the original target pose sequence composed of multiple consecutive single-frame appearance images. By using the training set and the total loss function, the video generation network is optimized to obtain a pre-trained video generation network; The total loss function includes the spectral constraint loss, which is obtained by performing a global real-number fast Fourier transform on the predicted video sequence and the label along the time dimension to determine the difference in spectral energy distribution between the two in a preset frequency band. The second generator is constrained by spectral constraint loss.

[0072] The method proposed in this invention combines pose difference representation, two-stage progressive generation, temporal enhancement, and spectral constraints to effectively model the spatiotemporal characteristics of abnormal FM pathological movements in infants. This can improve the accuracy of the pose structure, temporal consistency, local detail recovery ability, and pathological fidelity of the generated video, reduce the problem of excessive smoothing of abnormal movements, and provide data support for early screening of cerebral palsy, training of auxiliary diagnostic models, and data augmentation in scenarios where abnormal samples are scarce.

[0073] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0074] Furthermore, unless otherwise stated, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. All references to this specification are incorporated by way of citation to disclose and describe methods relating to those references. In the event of any conflict with any incorporated reference, the content of this specification shall prevail.

Claims

1. A cerebral palsy infant abnormal fidgety movement video generation method characterized by, Includes the following steps: Acquire single-frame appearance images from movement videos of infants with cerebral palsy, and the original target pose sequence composed of multiple consecutive single-frame appearance images; A single-frame appearance image and the original target pose sequence are input into a pre-trained video generation network to generate a predicted video sequence. The video generation network includes a cascaded first generator and a second generator. The first generator is based on a U-Net structure, with its encoder part being a parallel dual-branch structure of conditional branch and pose branch, and a temporal enhancement module is introduced before the decoder. The second generator is based on a U-Net structure, with its encoder part being a parallel appearance branch and motion branch, and a 3D-SE attention module is introduced between the output of the appearance branch and the decoder. The conditional branch extracts shape features from a single frame of the appearance image; the pose branch extracts pose features from the original target pose sequence; the temporal enhancement module performs cross-frame dynamic information aggregation on the fused features based on the fusion of shape and pose features, outputs temporal enhancement features, and uses them as input to the decoder to generate a coarse-grained video sequence. Specifically, the appearance branch extracts pure spatial shape features from a single frame of appearance image using a pseudo-3D convolution with a temporal kernel size of 1; the 3D-SE attention module performs channel recalibration on the pure spatial shape features to obtain enhanced shape features; the motion branch extracts motion features from the coarse-grained video sequence using standard 3D convolution; the decoder fuses the enhanced shape features and motion features to generate a residual detail-compensated image; and adds the residual detail-compensated image pixel-by-pixel to the coarse-grained video sequence to generate a predicted video sequence. The training process of the video generation network specifically includes: Acquire single-frame appearance images from movement videos of infants with cerebral palsy, a sequence of original target poses composed of multiple consecutive single-frame appearance images, and the labels corresponding to the original target pose sequences; A total loss function is constructed based on the difference between the predicted video sequence generated by the video generation network and the label. A training set is constructed based on single-frame appearance images from different videos of infants with cerebral palsy, a sequence of original target poses composed of multiple consecutive single-frame appearance images, and the labels corresponding to the original target pose sequences. By using the training set and the total loss function, the video generation network is optimized to obtain a pre-trained video generation network; The total loss function includes a spectral constraint loss, which is obtained by performing a global real-number fast Fourier transform on the predicted video sequence and the label along the time dimension to determine the difference in spectral energy distribution between the two in a preset frequency band. The second generator is constrained by the spectral constraint loss.

2. The cerebral palsy infant abnormal fidgety movement video generation method of claim 1, wherein, The pose branch extracts pose features from the original target pose sequence, specifically including: The enhanced attitude representation is obtained by performing adjacent frame difference on the original target attitude sequence and then concatenating it with the original target attitude sequence channels. The posture branch extracts posture features that enhance the posture representation.

3. The cerebral palsy infant abnormal fidgety movement video generation method according to claim 2, characterized by, The step of performing adjacent-frame difference analysis on the original target attitude sequence and then concatenating it with the original target attitude sequence channels to obtain the enhanced attitude representation specifically includes: Based on the difference between the pose of the current video frame and the pose of the next video frame in the original target pose sequence, a pose difference sequence is obtained. The attitude difference sequence is concatenated with the original target attitude sequence in the channel dimension to obtain the enhanced attitude representation.

4. The method for generating videos of abnormally restless movements in infants with cerebral palsy according to claim 1, characterized in that, The temporal enhancement module performs cross-frame dynamic information aggregation on the fused features based on shape and pose feature fusion, and outputs temporal enhancement features, specifically including: The timing enhancement module includes multiple cascaded 3D residual units, each with a different temporal receptive field. The local short-term dynamic information and the relative long-term dynamic information of multiple 3D residual units are aggregated and fused to obtain temporal enhancement features.

5. The method for generating videos of abnormally restless movements in infants with cerebral palsy according to claim 1, characterized in that, The decoder fuses enhanced shape features and motion features, and then decodes and generates a residual detail-compensated image, specifically including: The decoder of the second generator uses progressive upsampling and parallel convolutional units to decode the fused features, and uses long skip connections to introduce the multi-scale encoded features generated by the appearance branch in the process of extracting pure spatial shape features and the multi-scale encoded features generated by the motion branch in the process of extracting motion features into the decoder, so as to collaboratively recover multi-scale high-frequency information and generate residual detail compensation images for compensating for missing high-frequency textures and local deformation details in coarse-grained video sequences.

6. The method for generating videos of abnormally restless movements in infants with cerebral palsy according to claim 1, characterized in that, The total loss function also includes adversarial loss and pixel-level reconstruction loss; The adversarial loss is output by a dual-head 3D PatchGAN discriminator; the dual-head 3D PatchGAN discriminator includes a shared 3D encoder, a spatial realism head, and a temporal consistency head; wherein, the shared 3D encoder extracts hierarchical spatiotemporal features of the predicted video sequence and the tags; the spatial realism head evaluates the realism of local textures based on the hierarchical spatiotemporal features; the temporal consistency head evaluates the rationality of dynamic changes based on the neighboring frame difference features of the hierarchical spatiotemporal features; By incorporating the adversarial loss into the total loss function, the backpropagation algorithm is used to optimize the network parameters of the second generator, thereby constraining the training process of the second generator; The pixel-level reconstruction loss is obtained by determining the L1 distance between the predicted video sequence and the label, and by weighting the L1 distance of the foreground region where the infant's body is located based on the human body mask obtained by thresholding the pose heatmap. By incorporating the pixel-level reconstruction loss into the total loss function, the network parameters of the first generator and the second generator are simultaneously optimized using the backpropagation algorithm.

7. The method for generating videos of abnormally restless movements in infants with cerebral palsy according to claim 1, characterized in that, The total loss function also includes a consistency constraint based on frame difference; The frame difference-based consistency constraint is used to constrain the inter-frame difference between the predicted video sequence and the label at a preset sampling interval. By incorporating the frame difference-based consistency constraint into the total loss function, the network parameters of the first generator and the second generator are simultaneously optimized using the backpropagation algorithm.

8. The method for generating videos of abnormally restless movements in infants with cerebral palsy according to claim 1, characterized in that, The total loss function also includes perceptual loss; The perceptual loss is obtained by inputting the predicted video sequence and the label into a pre-trained feature extraction network to obtain the corresponding high-level semantic feature representations; and is based on the difference between the high-level semantic feature representations of the predicted video sequence and the label. By incorporating the perceptual loss into the total loss function, the network parameters of the second generator are optimized using the backpropagation algorithm; The pre-trained feature extraction network includes the VGG-19 network.

9. A system for generating videos of abnormally restless movements in infants with cerebral palsy, characterized in that, include: The data acquisition module is used to acquire single-frame appearance images and original target posture sequences composed of multiple consecutive single-frame appearance images from different cerebral palsy infant movement videos. The video generation module is used to input a single-frame appearance image and the original target pose sequence into a pre-trained video generation network to generate a predicted video sequence; A video generation network construction module is used to construct a video generation network, wherein the video generation network includes a cascaded first generator and a second generator; the first generator is based on a U-Net structure, its encoder part is a parallel dual-branch structure of conditional branch and pose branch, and a temporal enhancement module is introduced before the decoder; the second generator is based on a U-Net structure, its encoder part is a parallel appearance branch and motion branch, and a 3D-SE attention module is introduced between the output of the appearance branch and the decoder; The conditional branch extracts shape features from a single frame of the appearance image; the pose branch extracts pose features from the original target pose sequence; the temporal enhancement module performs cross-frame dynamic information aggregation on the fused features based on the fusion of shape and pose features, outputs temporal enhancement features, and uses them as input to the decoder to generate a coarse-grained video sequence. Specifically, the appearance branch extracts pure spatial shape features from a single frame of appearance image using a pseudo-3D convolution with a temporal kernel size of 1; the 3D-SE attention module performs channel recalibration on the pure spatial shape features to obtain enhanced shape features; the motion branch extracts motion features from the coarse-grained video sequence using standard 3D convolution; the decoder fuses the enhanced shape features and motion features to generate a residual detail-compensated image; and adds the residual detail-compensated image pixel-by-pixel to the coarse-grained video sequence to generate a predicted video sequence. The training process of the video generation network specifically includes: Acquire single-frame appearance images from movement videos of infants with cerebral palsy, a sequence of original target poses composed of multiple consecutive single-frame appearance images, and the labels corresponding to the original target pose sequences; A total loss function is constructed based on the difference between the predicted video sequence generated by the video generation network and the label. A training set is constructed based on single-frame appearance images from different videos of infants with cerebral palsy, a sequence of original target poses composed of multiple consecutive single-frame appearance images, and the labels corresponding to the original target pose sequences. By using the training set and the total loss function, the video generation network is optimized to obtain a pre-trained video generation network; The total loss function includes a spectral constraint loss, which is obtained by performing a global real-number fast Fourier transform on the predicted video sequence and the label along the time dimension to determine the difference in spectral energy distribution between the two in a preset frequency band. The second generator is constrained by the spectral constraint loss.

Images