Content sequence generation method and device based on time alignment, equipment and medium

Abstract

The present application relates to the technical field of image detection, and discloses a content sequence generation method and device based on time sequence alignment, equipment and medium, comprising: acquiring reference object images, time sequence driving signals, target motion indication information and target action region contour information sequences, establishing a target sequence time axis and an initial noise latent variable sequence, extracting an initial object representation, local region features, time sequence driving features and latent space constraint mask sequences, and combining a spatio-temporal motion map input content generation network to obtain a generated content sequence. The present application can be applied to financial technology and medical health business scenarios, unifies the time correspondence relationship of various types of condition information and the initial noise latent variable sequence through the target sequence time axis, and inputs the appearance, local, driving, mask and motion conditions into the content generation network together, thereby reducing local region distortion and overall motion incoordination in the generated content.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and in particular to a method, apparatus, device and medium for generating content sequences based on temporal alignment. Background Technology

[0002] While existing diffusion-based media content generation technologies have advanced research in character animation, a core problem remains prevalent in content generation scenarios where temporal driving signals are the primary driving force: simultaneously achieving temporal consistency in driving forces and local structural stability in the generated content. This is particularly true in body-level content generation tasks, where the lack of a direct and stable correspondence between temporal driving signals and action changes easily leads to unnatural transitions between actions, distorted local morphological changes, and insufficient constraints on the main structure. These problems are common in existing technologies and directly impact the realism and usability of the generated content.

[0003] In the fintech sector, time-driven and signal-driven digital personas are commonly used for robo-advisor broadcasts, risk disclosure explanations, wealth management interactive displays, and digital human services for financial customer service. Existing technologies in these scenarios can often achieve basic voice-driven or simple facial expression changes. However, when the generated object involves upper body movements, gestures, and continuous explanations, issues arise such as unstable correspondence between mouth movements and speech rhythm, incomplete details in the hand and facial areas, and a lack of coordination between the overall movement amplitude and the broadcast rhythm. These problems ultimately affect the seriousness, credibility, and interactive coherence of the financial information delivery.

[0004] In the healthcare sector, time-driven and signal-driven digital persona content is commonly used for health education, rehabilitation guidance, remote consultation assistance, and medical service guidance. However, existing technologies also face significant limitations in these scenarios. When the driving source is primarily time-driven signals such as audio, the generated content struggles to maintain the structural integrity of key areas like the face and hands. It also fails to ensure that body movements remain consistent with the narration rhythm and semantic guidance, easily leading to distortion of continuous movements, abnormal expression in localized areas, and unnatural overall posture changes. This reduces the accuracy and standardization of content expression in healthcare settings. Summary of the Invention

[0005] The main objective of this invention is to provide a content sequence generation method, apparatus, device, and storage medium based on temporal alignment, aiming to solve the technical problem that existing technologies are unable to establish a stable correspondence between driving semantics, local action region contours, and overall motion indicators in a continuous time dimension, which leads to local region distortion and overall motion incoordination in the generated content sequence.

[0006] To achieve the above objectives, the present invention provides a content sequence generation method based on temporal alignment, comprising: Acquire a sequence of reference object image, timing drive signal, target motion indication information, and target action area contour information; determine the target sequence time axis based on the timing drive signal; and generate an initial noise latent variable sequence according to the target sequence time axis. Based on the reference object image, latent variable representation of appearance and posture structure feature encoding are extracted, and the latent variable representation of appearance and the posture structure feature encoding are fused to obtain an initial object representation; In the reference object image, the location of key points in a local region is determined, and a local region image is obtained by cropping based on the location of the key points in the local region. The local region features corresponding to the local region image are then extracted. Based on the time-driven signal, a semantic feature sequence is extracted, and the semantic feature sequence is time-aligned according to the time axis of the target sequence. The time-aligned semantic feature sequence is then converted into a time-driven feature corresponding to the initial noise latent variable sequence. The target action region contour information sequence is aligned according to the target sequence time axis, and a pixel space mask sequence is generated based on the aligned target action region contour information sequence. The pixel space mask sequence is then projected onto the latent space to obtain a latent space constraint mask sequence. The target motion indication information is converted into a spatiotemporal motion graph corresponding to the time axis of the target sequence. Based on the initial object representation, the local region features, the temporal driving features, the latent space constraint mask sequence, the spatiotemporal motion graph, and the initial noise latent variable sequence, a content generation network is used to perform conditional generation processing to obtain the generated content sequence.

[0007] Furthermore, to achieve the above objectives, the present invention provides a content sequence generation apparatus based on temporal alignment, comprising: The target sequence time axis generation module is used to acquire a reference object image, a timing drive signal, target motion indication information, and a sequence of target action area contour information, determine the target sequence time axis based on the timing drive signal, and generate an initial noise latent variable sequence according to the target sequence time axis. The initial object representation extraction module is used to extract appearance latent variable representation and posture structure feature encoding based on the reference object image, and fuse the appearance latent variable representation and the posture structure feature encoding to obtain the initial object representation; The local region feature extraction module is used to determine the location of key points in the local region in the reference object image, crop the local region image based on the location of the key points in the local region, and extract the local region features corresponding to the local region image. The time-driven feature generation module is used to extract semantic feature sequences based on the time-driven signal, perform time-series alignment of the semantic feature sequences according to the time axis of the target sequence, and convert the time-series aligned semantic feature sequences into time-driven features corresponding to the initial noise latent variable sequence. The latent space mask generation module is used to align the target action region contour information sequence according to the target sequence time axis, generate a pixel space mask sequence based on the aligned target action region contour information sequence, and project the pixel space mask sequence onto the latent space to obtain a latent space constraint mask sequence. The conditional content generation module is used to convert the target motion indication information into a spatiotemporal motion graph corresponding to the time axis of the target sequence, and to perform conditional generation processing through a content generation network based on the initial object representation, the local region features, the temporal driving features, the latent space constraint mask sequence, the spatiotemporal motion graph, and the initial noise latent variable sequence to obtain the generated content sequence.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a time-aligned content sequence generation program stored in the memory and executable on the processor, wherein when the time-aligned content sequence generation program is executed by the processor, it implements the steps of the time-aligned content sequence generation method as described above.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a time-aligned content sequence generation program, wherein the time-aligned content sequence generation program, when executed by a processor, implements the steps of the time-aligned content sequence generation method described above.

[0010] Beneficial Effects: This invention relates to the field of image detection technology and discloses a method, apparatus, device, and medium for generating content sequences based on temporal alignment. The method includes: acquiring a reference object image, a temporal driving signal, target motion indication information, and a target action region contour information sequence; establishing a target sequence time axis and an initial noise latent variable sequence; extracting an initial object representation, local region features, temporal driving features, and a latent space constraint mask sequence; and inputting these into a content generation network in conjunction with a spatiotemporal motion graph to obtain the generated content sequence. This invention can be applied to business scenarios such as fintech and healthcare. By unifying the temporal driving features, latent space constraint mask sequence, spatiotemporal motion graph, and initial noise latent variable sequence using the target sequence time axis, and by utilizing the initial object representation and local region features to provide appearance and local detail conditions, the content generation network receives driving semantics, local region, and overall motion constraints in the same generation process, thereby reducing local region distortion and overall motion inconsistency. Attached Figure Description

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a content sequence generation method based on temporal alignment according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the content sequence generation method based on temporal alignment according to the present invention. Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the content sequence generation device based on time alignment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] The content sequence generation method based on temporal alignment provided in this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can obtain a sequence of reference object images, temporal driving signals, target motion indication information, and target action region contour information from the client. It then establishes a target sequence time axis and an initial noise latent variable sequence, extracts the initial object representation, local region features, temporal driving features, and latent space constraint mask sequence, and combines this with the spatiotemporal motion graph input to the content generation network to obtain the generated content sequence. This invention can be applied to business scenarios such as fintech and healthcare. By unifying the temporal driving features, latent space constraint mask sequence, spatiotemporal motion graph, and initial noise latent variable sequence using the target sequence time axis, and by utilizing the initial object representation and local region features to provide appearance and local detail conditions, the content generation network receives driving semantics, local region, and overall motion constraints in the same generation process, thereby reducing local region distortion and overall motion incoordination. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0014] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the content sequence generation method based on temporal alignment provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0015] like Figure 2 As shown, the content sequence generation method based on temporal alignment proposed in this invention includes the following steps: S10, acquire the reference object image, the timing drive signal, the target motion indication information and the target action area contour information sequence, determine the target sequence time axis based on the timing drive signal, and generate the initial noise latent variable sequence according to the target sequence time axis; In this embodiment, the reference object image is used to provide information on the appearance boundary, subject proportion, and visible area of ​​the generated object. In implementation, the reference object image can be used for subject region localization, image size reading, visible boundary recording, and image quality marker generation. Subject region localization can be accomplished using an object segmentation network, a keypoint detection network, or a bounding box detection network; image size reading is used to determine the subsequent latent space scale; visible boundary recording is used to exclude areas that are severely occluded or exceed the image's range; image quality markers are generated based on sharpness, occlusion level, and subject integrity, and are used to determine whether the reference object image is suitable for subsequent generation.

[0016] Timing-driven signals provide information on driving changes over continuous time. These signals can be speech, text rhythm markers, action rhythm markers, or business script time markers. In implementation, effective driving intervals, pause intervals, driving unit change boundaries, and time sampling markers can be extracted from the timing-driven signal. Effective driving intervals determine the duration of the generated content sequence, driving unit change boundaries determine the change positions on the target sequence's timeline, and time sampling markers map different segments of the timing-driven signal to discrete time positions. In fintech applications, timing-driven signals can correspond to intelligent investment advisor broadcasts, risk disclosures, or financial customer service Q&A messages. Segments containing financial keywords such as return volatility, asset allocation, and redemption periods can form driving unit change boundaries.

[0017] Target motion indication information is used to provide constraints on the overall range and intensity of motion. This information may include trunk range of motion, head rotation range, gesture amplitude range, body sway range, and movement rhythm markers. In implementation, the target motion indication information can be saved as time-meaningful motion configuration data, recording the motion area, amplitude, direction, and intensity. At this stage, the target motion indication information is only received and processed as input; it is not expanded to generate a spatiotemporal motion graph to avoid confusion with subsequent processing.

[0018] The target action region contour information sequence provides information on the temporal contour changes of the local action region. The target action region can be a local area of ​​the mouth, hand, face, or other areas requiring fine constraints. In implementation, each contour unit can be recorded as a contour point set, contour sampling time, and contour confidence level. The contour point set represents the local boundary shape, the contour sampling time is used to establish a correspondence with the target sequence timeline, and the contour confidence level is used to indicate whether the contour is complete, whether there is occlusion, or positioning deviation. This stage only receives and organizes the target action region contour information sequence; it does not generate a pixel space mask sequence.

[0019] The target sequence timeline is formed based on the timing drive signal and serves to provide a unified time position for subsequent serialization. In implementation, the time range of the target sequence timeline can be determined based on the effective drive interval, the change nodes within the target sequence timeline can be determined based on the change boundaries of the drive units, and discrete time positions can be generated based on the output frame interval or target segment interval. The target sequence timeline can be represented using frame bit sequences, segment bit sequences, or latent variable position sequences. If the timing drive signal is speech, the target sequence timeline can be generated based on the speech duration and articulation change boundaries; if the timing drive signal is script timestamps, the target sequence timeline can be generated based on the statement duration, pause markers, and emphasis markers.

[0020] An initial noise latent variable sequence is generated according to the target sequence timeline and is used to form the initial serialized input in the latent space. In implementation, the number of latent variable frames is determined based on the target sequence timeline, the spatial size of each latent variable frame is determined based on the image size of the reference object image and the latent space compression scale, the feature dimensions are determined based on the latent space input requirements of the content generation network, and a noise latent variable frame is generated for each latent variable frame. Multiple noise latent variable frames are arranged according to the target sequence timeline to form the initial noise latent variable sequence. The temporal position of the initial noise latent variable sequence is consistent with the target sequence timeline, allowing subsequent reception of drive, contour, and motion inputs from the same time reference.

[0021] This can be implemented using audio-driven processing. The input receives a reference object image, a timing-driven signal in speech form, target motion indication information in the form of motion amplitude configuration, and a sequence of target motion region contour information in the form of mouth contours. The audio processing unit extracts the effective speech interval, silence interval, and articulation change boundary. Based on the effective speech interval, it determines the duration of the target sequence time axis and sets the change nodes in the target sequence time axis based on the articulation change boundary. The latent variable initialization unit reads the size information of the reference object image and a preset latent space compression scale, generates latent variable frame positions corresponding to the target sequence time axis, and generates noise latent variable frames at each latent variable frame position, arranging them according to their time positions to form an initial noise latent variable sequence. This implementation is suitable for risk disclosure broadcasting, intelligent investment advisor explanations, and wealth management customer service video generation in fintech businesses.

[0022] Alternatively, a script-driven rhythm approach can be used. The input receives a reference object image, a timing-driven signal consisting of text pause markers and accent markers, target motion indication information in the form of action templates, and a sequence of target action region contour information formed by local region contour templates. The timeline generation unit generates the target sequence timeline based on the duration, pause positions, and financial keyword positions of each statement segment in the script. The latent variable initialization unit generates an initial noise latent variable sequence based on the number of discrete time positions in the target sequence timeline. This implementation is suitable for financial product descriptions, compliance prompts in short videos, and digital human content for account services that require pre-arranged broadcast rhythms.

[0023] In the healthcare sector, when a health management platform generates digital content for rehabilitation guidance, it can receive illustrative images as reference images, rehabilitation guidance audio as a time-driven signal, movement amplitude limitation data as target movement indication information, and mouth or hand prompt region contour sequences as target movement region contour information sequences. The narration intervals, movement prompt intervals, and pause intervals in the audio are converted into the target sequence timeline. The initial noise latent variable sequence is generated according to the target sequence timeline, facilitating the maintenance of narration rhythm and temporal correspondence between local movement regions during subsequent content generation.

[0024] In the fintech business, when generating content explaining wealth management products, robo-advisor platforms can receive an image of an investment advisor as a reference image, product explanation audio as a time-driven signal, compliant broadcast action range as target motion indication information, and mouth contour sequence as a target action area contour information sequence. Segments in the audio related to risk level, return volatility, asset allocation, and redemption period are mapped to the target sequence timeline. The initial noise latent variable sequence is formed according to the target sequence timeline, ensuring that subsequent generated content maintains a stable time-driven relationship at the locations where financial keywords appear.

[0025] In this embodiment, the target sequence time axis is determined by a time-driven signal, and an initial noise latent variable sequence is generated according to the target sequence time axis. This allows the reference object image, target motion indication information, and target action region contour information sequence to be organized around the same time reference. The time position of the initial noise latent variable sequence corresponds to the target sequence time axis, and subsequent driving semantics, local contours, and motion constraints can participate in the generation at the same time position, thereby reducing time misalignment and local action discontinuity in the generated content sequence.

[0026] S20, based on the reference object image, extract the appearance latent variable representation and the posture structure feature encoding, and fuse the appearance latent variable representation and the posture structure feature encoding to obtain the initial object representation; In this embodiment, the reference object image carries information about the object's appearance, main outline, color texture, visible pose, and regional proportions. In practice, the image size, main bounding area, visible boundary, and sharpness marker of the reference object image can be read first. Then, the reference object image can be scaled, cropped, color normalized, and occluded area marked to extract appearance and structural information within the same image coordinate range. The reference object image can be a digital persona image, a business presentation image, a health education image, or a financial customer service image, or it can be a virtual object image with a clearly defined main outline.

[0027] Appearance latent variables are used to store information such as identity appearance, color distribution, texture details, and clothing appearance in a reference object image. In practice, the reference object image can be input into a variational autoencoder network or a multi-scale appearance encoder, and after passing through convolutional downsampling, normalization layers, attention aggregation layers, and latent space mapping layers, a low-dimensional latent space representation is obtained. If a variational autoencoder network is used, the appearance latent variable parameters can be output by the encoder, and then the appearance latent variable representation can be generated through reparameterization or deterministic mapping. If a multi-scale appearance encoder is used, appearance information can be extracted from shallow texture features and deep semantic features, and then mapped into an appearance latent variable representation with a unified latent space scale.

[0028] Pose structure feature encoding is used to store the structural positions, keypoint distributions, and part connections in a reference object image. In practice, pose keypoint coordinates can be detected from the reference object image, and a pose structure map can be generated based on these coordinates. The pose structure map can be represented by a keypoint heatmap, skeleton connection map, local part distribution map, or key region response map. After the pose structure map enters the pose encoder, it passes through a convolutional feature extraction layer, a structural relationship aggregation layer, and a latent space projection layer to obtain the pose structure feature encoding. For human images, pose structure feature encoding can express the positional relationships between the head, shoulders, hands, and torso; for non-human objects, pose structure feature encoding can express the spatial arrangement relationships between main body parts.

[0029] The appearance latent variable representation and the pose structure feature encoding need to remain fusionable in both spatial scale and feature dimension. In practice, the pose structure feature encoding can be mapped to a latent space resolution consistent with the appearance latent variable representation, and then the initial object representation can be obtained through channel concatenation, weighted fusion, gated fusion, or cross-attention fusion. When using channel concatenation, the appearance latent variable representation and the aligned pose structure feature encoding can be merged into a concatenated feature, which is then compressed to the target latent space dimension through a fusion mapping layer. When using gated fusion, structural weights can be generated based on the pose structure feature encoding, and these structural weights can be used to adjust different spatial positions in the appearance latent variable representation. The initial object representation contains both appearance preservation information and pose structure information, used to express the initial state of the object in the reference object image.

[0030] This can be achieved using a dual encoding approach with a variational autoencoder (VA) and a pose encoder. The reference object image, after being cropped and resized, is input into the VVA. The VVA's encoding layer consists of multiple convolutional blocks, normalization layers, and latent space mapping layers, outputting appearance latent variable parameters, which are then used to generate an appearance latent variable representation. The pose processing layer detects the coordinates of pose keypoints from the same reference object image, generating a pose structure map. This map is then input into the pose encoder to obtain pose structure feature encoding. The appearance latent variable representation and pose structure feature encoding are spatially aligned before entering a fusion mapping layer. This fusion mapping layer can use convolutional mapping or lightweight attention mapping to output an initial object representation. During training, the VVA can reconstruct the error-constrained appearance preservation capability using the reference object image, the pose encoder can reconstruct the error-constrained structure representation capability using keypoint heatmaps or pose structure maps, and the fusion mapping layer can reconstruct the combination of error-constrained appearance and pose using latent space mapping.

[0031] Alternatively, tokenization encoding can be used. The reference object image is divided into image patches, which are then processed by an appearance token encoder to generate appearance latent variable representations. The pose keypoint coordinates are converted into pose token sequences, which are then processed by a structure token encoder to generate pose structure feature codes. The appearance latent variable representations and pose structure feature codes exchange information through a cross-attention fusion layer. The cross-attention fusion layer adjusts the spatial response in the appearance latent variable representations based on the positional relationships in the pose structure feature codes, and then a linear mapping layer outputs the initial object representation. This implementation is suitable for generation scenarios with complex subject regions, numerous clothing textures, or high-resolution reference object images.

[0032] Alternatively, region-separate encoding can be used. The reference object image is divided into a main body region, a head region, a hand region, and a torso region. Each region is input into an appearance encoding branch to generate regional appearance features. The pose structure map is divided into structure response maps for the corresponding regions. Each structure response map is input into a structure encoding branch to generate regional structure features. The regional appearance features and regional structure features are fused separately and then summarized into an initial object representation. This implementation is suitable for scenarios such as financial customer service digital humans and health education digital humans that require maintaining facial stability, natural upper body posture, and consistency with clothing appearance.

[0033] In the healthcare sector, when health management platforms generate digital content for rehabilitation guidance, they can use a health narrator image as a reference object image. The facial features, clothing colors, and body contours in the image are encoded as latent appearance variables, while the positions of the head, shoulders, and hands are encoded as postural structural features. These two representations are then fused to obtain the initial object representation. During subsequent content generation, the digital narrator image can maintain a stable appearance and reasonable posture before any changes in the narration's actions.

[0034] In the fintech business, when generating content to explain wealth management products, robo-advisor platforms can use an image of the investment advisor as a reference image. The professional attire, facial features, and main outline of the image are encoded as latent appearance variables, while posture, hand positions, and upper body structure are encoded as posture structural features. These two are then fused to obtain the initial object representation. This process is suitable for generating financial content such as risk level explanations, asset allocation descriptions, and return volatility alerts, ensuring that the investment advisor's image maintains consistent appearance and structural stability in subsequent presentations and action generation.

[0035] This embodiment extracts latent appearance variable representations and pose structure feature codes from a reference object image, and fuses them in the latent space to generate an initial object representation. This allows appearance texture information and pose structure information to form a unified expression before entering subsequent generation processing. The latent appearance variable representation preserves the object's identity appearance and image texture, while the pose structure feature code preserves the object's part position and structural relationships. The fused initial object representation reduces the lack of pose information caused by a single appearance code and also reduces the loss of appearance details caused by a single structure code, thereby improving the consistency between the object's appearance and the initial structure in the generated content.

[0036] S30, determine the location of key points in the local region in the reference object image, crop the local region image based on the location of key points in the local region, and extract the local region features corresponding to the local region image. In this embodiment, the reference object image, after being sized, having its subject boundary located, and its visible area marked, is used to determine the local area requiring fine extraction. The reference object image can be a half-body image of a person, a virtual narrator image, a digital customer service avatar image, or a health guidance avatar image. During processing, the image detection network identifies the subject region from the image, and the keypoint detection network outputs the locations of key points in the local area within the subject region. These local keypoint locations can include facial boundary points, mouth contour points, hand joint points, shoulder reference points, or other local area location points. The keypoint output typically includes coordinate values, visibility, and confidence scores. The coordinate values ​​are used to determine the region's location, the visibility is used to determine occlusion, and the confidence score is used to exclude points with significant positioning deviations.

[0037] Keypoint locations within a local region are used to generate the local region cropping range. In implementation, a local region bounding box can be generated based on the minimum bounding radius of multiple keypoints within the same region. This bounding box is then expanded outwards according to a boundary expansion factor to prevent loss of edge details after cropping. Image boundaries are used to limit the cropping range and prevent local regions from exceeding their limits. For the mouth region, the local region bounding box can be formed based on the upper and lower lip contour points; for the hand region, it can be formed based on the palm, fingertip, and wrist points; and for the face region, it can be formed based on the boundary points of the eyes, nose, mouth, and cheeks.

[0038] The local region image is obtained through cropping transformation. During cropping, affine cropping, perspective cropping, or bounding box cropping can be used to maintain a uniform size and orientation of the local region image. If the local region is rotated or tilted, orientation angles can be generated based on the keypoint positions of the local region, and then rotation correction can be performed according to these orientation angles. The local region image can also be accompanied by region type markers, cropping scale markers, and quality markers. The region type marker is used to distinguish between identity-preserving regions and motion-changing regions; the cropping scale marker is used to record the proportion of the local region relative to the reference object image; and the quality marker is used to record the sharpness, occlusion ratio, and boundary integrity of the local region.

[0039] Local region features are obtained by locally encoding local region images. For identity-preserving regions, facial encoders, texture encoders, or identity embedding networks can be used to extract local region features, making the local region features include identity texture, basic facial expression shape, and local color distribution. For motion-changing regions, convolutional encoders, keypoint guided encoders, or local temporal prior encoders can be used to extract local region features, making the local region features include hand pose, mouth shape, and local motion tendency. During encoding, the local region image, region type label, and quality label can be input into the encoding network to obtain local region features with regional differences. If a local region is occluded, the encoding network can reduce the response of the occluded region based on the quality label, while preserving the effective representation of the visible region.

[0040] In one implementation, a reference object image is processed by a subject detection network to obtain a subject bounding box. A keypoint detection network outputs the positions of facial and hand keypoints within the subject bounding box. A processing unit generates a facial region bounding box based on the facial keypoint positions and a hand region bounding box based on the hand keypoint positions, then refines both bounding boxes according to the image boundaries. The facial region image is input to an identity preservation encoding branch, and the hand region image is input to an action prior encoding branch. The features output from these two encoding branches are then processed to form local region features. This implementation is suitable for intelligent investment advisor explanations and wealth management customer service videos in fintech businesses. The facial region is used to maintain a stable investment advisor image, while the hand region is used to convey gestures during risk warnings, return explanations, and asset allocation explanations.

[0041] In another implementation, the keypoint detection network outputs a local region keypoint heatmap. The processing unit extracts candidate keypoint locations from the local region keypoint heatmap and generates local region keypoint locations based on the spatial clustering of the candidate keypoints. The cropping unit generates a rotatable cropping box based on the local region keypoint locations and extracts the local region image from the reference object image according to the rotatable cropping box. The local region image enters a multi-branch encoding network, where different branches extract boundary morphology, texture details, and local pose respectively. The outputs of multiple branches are then synthesized into local region features. This implementation is suitable for rehabilitation guidance content in healthcare applications, where hand gestures, verbal explanations, and upper body poses can all be encoded separately by region.

[0042] Alternatively, quality-constrained cropping can be used. After the keypoint detection network outputs the keypoint locations of local regions, the processing unit determines the cropping priority based on the keypoint confidence and occlusion status. Local regions with high confidence are cropped using normal bounding boxes, while local regions with low confidence have their cropping range expanded and quality markers recorded. The local encoding network reads the local region image and quality markers to generate local region features. This implementation is suitable for scenarios where local regions of the reference object image are occluded by gestures, hair strands, collars, or edges of the image.

[0043] In the healthcare field, when health management platforms generate instructional content, they can detect key points of the mouth, hands, and shoulders in the health guidance images. The processing unit crops the mouth region image based on the mouth key points and the hand region image based on the hand key points, and then extracts local features of the mouth and hands respectively. The mouth region features are used to preserve the basic mouth morphology during instruction, while the hand region features are used to preserve the hand structure information in the rehabilitation movement prompts.

[0044] In the fintech business, when generating content to explain wealth management products, robo-advisor platforms can detect facial and hand key points in the advisor's image. The processing unit crops the facial region image based on the facial key points and the gesture region image based on the hand key points. The facial region image is input into the identity preservation encoding branch, and the gesture region image is input into the action prior encoding branch, resulting in local region features. These local region features are suitable for generating content with high requirements for facial stability and gesture clarity in scenarios such as risk level explanations, return volatility alerts, asset allocation explanations, and redemption period reminders.

[0045] This embodiment determines the location of key points in a local area in a reference object image and obtains a local area image by cropping based on the location of the key points. The boundary position and local shape of the local area can be extracted separately. After differential encoding, the local area image forms local area features, so that identity-related areas and action change areas can obtain clearer local expressions before subsequent generation, thereby reducing structural loss and detail distortion of areas such as face, mouth, and hands in the generated content.

[0046] S40, extract semantic feature sequences based on the time-driven signal, perform time-series alignment of the semantic feature sequences according to the time axis of the target sequence, and convert the time-aligned semantic feature sequences into time-driven features corresponding to the initial noise latent variable sequences; In this embodiment, the temporal driving signal is segmented, labeled, and encoded to form a semantic feature sequence. The temporal driving signal can be speech, rhythmically marked text, action cue rhythm, or business script time markers. Speech forms can undergo endpoint detection, silence interval removal, acoustic frame segmentation, and phoneme boundary recognition to obtain driving segments with time positions. Text rhythm forms can generate driving segments based on sentence pauses, accent marks, keyword positions, and playback duration. Each driving segment is configured with a time marker and then input into a temporal semantic encoder, which may include a spectral coding layer, a text embedding layer, a time position embedding layer, a context aggregation layer, and an output mapping layer. The semantic feature sequence stores the driving content, rhythmic variations, and segment boundaries, enabling speech content or script rhythm to be converted into continuously usable feature representations.

[0047] The target sequence timeline provides a temporal reference for the generated content sequence. The sampling interval of the semantic feature sequence itself is often not entirely consistent with the generated frame position, therefore temporal alignment based on the target sequence timeline is required. In implementation, a target time window can be generated based on the target sequence timeline, and the time markers in the semantic feature sequence can be matched with the target time window. One target time window can correspond to one semantic segment or the aggregation result of multiple semantic segments. If there are no directly matching semantic segments within a time window, interpolation or preservation can be performed based on the temporal distance between adjacent semantic segments. If multiple semantic segments fall into the same time window, a unified feature can be formed through attention weighting, average aggregation, or boundary-first aggregation. The temporally aligned semantic feature sequence corresponds to the temporal position in the target sequence timeline, ensuring consistency between semantic changes and the generated frame position during subsequent transformations.

[0048] The initial noise latent variable sequence provides the temporal location and feature dimension requirements in the latent space. The temporally aligned semantic feature sequence needs to be converted into temporally driven features corresponding to the initial noise latent variable sequence. In implementation, the temporal length, latent space resolution, and feature dimension of the initial noise latent variable sequence can be read. The temporally aligned semantic feature sequence is then input into the projection layer to ensure the output dimension matches the latent space feature dimension. Furthermore, positional embeddings are added based on the temporal location of the initial noise latent variable sequence, ensuring that the semantic feature at each temporal location corresponds to the latent variable frame at the same temporal location. For speech-driven scenarios, boundary weights can be generated based on pronunciation boundaries or rhythmic change boundaries to give lip movements, gesture cues, or pauses a clearer temporal distribution in the temporally driven features.

[0049] Temporally driven features are used to carry the driving expression after semantic encoding, temporal alignment, and latent space mapping. These features can be represented using sequence tensors, token sequences, or latent space conditional features. Sequence tensors are suitable for content generation networks with frame-level input, token sequences are suitable for generation through attention layers, and latent space conditional features are suitable for conditional fusion with the initial noisy latent variable sequence. In fintech businesses, temporally driven features can carry the broadcast position corresponding to financial keywords such as risk level, return volatility, asset allocation, and redemption period; in healthcare businesses, temporally driven features can carry the time position corresponding to action cues, breathing rhythms, rehabilitation reminders, and health education statements.

[0050] This can be implemented using speech coding. The input receives a time-driven signal in speech form. The speech processing unit extracts acoustic frames, silence intervals, and articulation change boundaries. The acoustic frames are input to the speech semantic encoder to obtain a semantic feature sequence. The time alignment unit generates a target time window based on the target sequence time axis and maps the time markers in the semantic feature sequence to the target time window. The latent space mapping unit reads the time length and feature dimensions of the initial noise latent variable sequence and maps the time-aligned semantic feature sequence to time-driven features. The speech semantic encoder can consist of an acoustic feature extraction layer, a time position embedding layer, a self-attention encoding layer, and a latent space projection layer. During training, audio segments can be paired with corresponding video frame time positions to form data. The time matching loss constrains the time position of the output features, and the reconstruction loss constrains the semantic content preservation.

[0051] Alternatively, script rhythm encoding can be used. The input receives a script rhythm signal with pause markers, accent markers, and keyword markers. The script encoding unit converts the text fragments into semantic embeddings and generates time markers based on the pause and accent markers. Time windows in the target sequence's timeline are matched with the script fragments to obtain a temporally aligned semantic feature sequence. The latent space mapping unit maps the semantic embedding of each time window to a temporally driven feature at the same time position based on the number of latent variable frames in the initial noise latent variable sequence. This implementation is suitable for scenarios requiring pre-arranged language rhythm, such as financial product descriptions, compliance reminders in short videos, and customer service broadcasts.

[0052] A hybrid approach can also be used. The voice signal provides the actual broadcast rhythm, while the script timestamps provide the positions of business keywords. These two signals are processed by a speech encoding layer and a text encoding layer, respectively, to form two sets of semantic feature sequences. The fusion layer aligns the two sets of semantic feature sequences to the same target time window based on the target sequence timeline, and then uses gating weights to form a temporally aligned semantic feature sequence. The latent space mapping layer generates temporally driven features based on the feature dimensions of the initial noise latent variable sequence. This implementation is suitable for generating fintech content where voice broadcasting and business scripts need to be strictly matched, and also suitable for scenarios in medical and health education where voice rhythm and professional terminology prompts need to be synchronized.

[0053] In the healthcare sector, when health management platforms generate rehabilitation guidance content, they can use the rehabilitation explanation audio as a time-driven signal. Action cues, pauses, and guidance statements in the audio are encoded into semantic feature sequences, which are then time-aligned according to the target sequence's timeline. The time-aligned semantic feature sequences are then converted into time-driven features based on the temporal position and feature dimensions of the initial noise latent variable sequence, enabling subsequent generated content to form a stable temporal correspondence around the action cues.

[0054] In the fintech business, when robo-advisory platforms generate content explaining wealth management products, they can use the rhythm markers of the product explanation voice or script as time-driven signals. Fragments containing financial keywords such as risk level, return volatility, asset allocation, redemption period, and compliance reminders form key time positions in the semantic feature sequence. After aligning these key time positions with the target sequence timeline, the latent space mapping layer generates time-driven features, ensuring that the broadcast rhythm of financial keywords corresponds to the time position of the generated content sequence.

[0055] This embodiment extracts semantic feature sequences from the temporal driving signal and performs temporal alignment based on the target sequence time axis. The semantic changes, pause positions, and boundary changes of the driving content can correspond to the time positions of the generated content sequence. By converting the temporally aligned semantic feature sequences into temporal driving features corresponding to the initial noise latent variable sequence, the driving features can match the subsequent generated input in terms of time length and latent space dimension, thereby reducing the temporal misalignment between semantic driving and generated frame bits.

[0056] S50, the target action region contour information sequence is aligned according to the target sequence time axis, and a pixel space mask sequence is generated based on the aligned target action region contour information sequence. The pixel space mask sequence is then projected onto the latent space to obtain a latent space constraint mask sequence. In this embodiment, the target action region contour information sequence is used to provide the boundary changes of the action region over continuous time. Contour information can originate from keypoint enclosure results, segmentation edge extraction results, polygon annotation results, or region boundary detection results. Common forms include a time-arranged set of contour points, boundary curves, or a set of vertices of a closed region. The target sequence timeline provides a unified time reference, ensuring that the contour information is consistent with the frame position of the generated content in the time dimension. During alignment, the sampling time corresponding to each frame's contour is typically read first, and then the sampling time is mapped to the target time position in the target sequence timeline. If there is a deviation between the contour sampling time and the target time position, time unification can be achieved through resampling, interpolation, translation of adjacent frames, or boundary smoothing. To facilitate subsequent region filling, the number of contour points can also be standardized to a uniform length, ensuring that the contours at adjacent times have a comparable point order and boundary orientation.

[0057] The aligned target action region contour information sequence is used to generate a pixel space mask sequence. Pixel space represents the image coordinate plane, and the mask represents the association of each pixel with the action region. During generation, the closed contour at each moment can be mapped onto the image plane, filling the inside of the contour with region values, setting background values ​​outside the contour, and introducing transition bands at the boundary positions to make the mask edges continuously varied. The pixel space mask sequence can be in binary form or continuous weight form; the binary form is suitable for clearly defining the range of the constrained region, while the continuous weight form is suitable for expressing boundary uncertainty and the importance of the region. The action region can be a mouth region, hand region, or torso region. The more stable the contour, the more concentrated the constraint of the pixel space mask sequence on the local region.

[0058] The latent space is used to hold the compressed spatial representation, and its resolution is typically lower than that of the image plane. When the pixel spatial mask sequence is projected into the latent space, the temporal position needs to be preserved, while the regional constraints of the image plane are transformed into constraint expressions consistent with the resolution of the latent space. In implementation, the pixel spatial mask sequence can be downsampled according to the dimensions of the latent space, or convolutional mapping, region pooling, or encoder mapping can be used to compress high-resolution mask information into the latent space grid. If jagged edges or breaks appear at the mask boundaries during projection, boundary smoothing and neighborhood consistency constraints can be added before and after projection to ensure that the latent space constraint mask sequence still maintains a continuous regional distribution. The latent space constraint mask sequence formed in this way retains the temporal correspondence, regional boundary information, and local weight distribution, which facilitates stable constraints on the action region at the compressed representation level in the subsequent generation process.

[0059] In one implementation, the target action region contour information sequence is stored as a set of contour points, with each time position corresponding to a set of sequentially arranged boundary points. A time alignment unit reads the contour sampling time and maps it to the target sequence time axis. For boundary points with missing times, interpolation between adjacent times is used to fill in the gaps. For contours with inconsistent point counts, uniform arc-length resampling is performed. A mask generation unit generates closed regions based on the resampled boundary points, fills the regions on the image plane, and performs a blurring transition on the boundaries to obtain a pixel space mask sequence. A projection unit performs block downsampling of the pixel space mask sequence according to the latent space resolution and combines it with local average weights to generate a latent space constraint mask sequence.

[0060] In another implementation, the target action region contour information sequence is stored in the form of segmented boundaries, with each time position corresponding to a boundary map. The temporal alignment unit performs temporal rearrangement of the boundary maps and fills in missing time points using optical flow estimation or boundary displacement estimation. The mask generation unit performs region closure and internal filling on the boundary maps to form a pixel-space mask sequence. The projection unit inputs the pixel-space mask sequence into a convolutional mapping network that matches the latent space resolution, outputting a latent space constraint mask sequence. This implementation is more suitable for scenarios with many boundary details and large deformations in the action region.

[0061] A hybrid implementation approach can also be used. The contour point set is used to preserve the boundary structure, while the segmentation boundary is used to supplement local shape changes; both are temporally aligned and participate in mask construction. The pixel-space mask sequence can generate hard boundary regions and soft transition regions separately, and then synthesize them into a unified mask representation. During projection, max pooling is used to preserve the coverage area of ​​hard boundary regions, while average pooling is used to preserve the boundary gradient of soft transition regions, ultimately resulting in a latent space constraint mask sequence that balances region range and edge continuity. This implementation is suitable for content generation scenarios where contour boundaries are clear but local actions change frequently.

[0062] In the healthcare field, when generating home rehabilitation guidance content, the target action area can be set as the mouth area or the hand area. The mouth contour information sequence can reflect the opening and closing changes during pronunciation training, while the hand contour information sequence can reflect the gesture range during rehabilitation action demonstration. After the aligned contours are filled and projected into latent space, stable local constraints can be formed, ensuring that the generated content maintains clear action boundaries in continuous time.

[0063] In the fintech field, when digital humans deliver financial explanations, risk warnings, or investment advisor interpretations, the target action areas can be set as the mouth and hand areas. The mouth contour information sequence is used to constrain lip movements during delivery, while the hand contour information sequence is used to constrain the gesture display area. After time alignment, mask generation, and latent space projection, the local action areas can maintain high consistency across consecutive frames, making the presentation of financial keyword explanations, return tips, and risk disclosures more stable.

[0064] This embodiment aligns the target action region contour information sequence to the target sequence time axis, ensuring that the positional changes of the action region in the time dimension remain consistent with the generated frame position. By converting the aligned contour into a pixel space mask sequence, the coverage and boundary transitions of the action region can be clearly expressed in the image plane. Through further projection, a latent space constraint mask sequence is obtained, allowing the region constraints to enter the compressed representation layer and maintain temporal correspondence, thereby improving the temporal consistency and boundary stability of the local action region.

[0065] S60, the target motion indication information is converted into a spatiotemporal motion graph corresponding to the time axis of the target sequence, and based on the initial object representation, the local region features, the temporal driving features, the latent space constraint mask sequence, the spatiotemporal motion graph, and the initial noise latent variable sequence, conditional generation processing is performed through a content generation network to obtain a generated content sequence.

[0066] In this embodiment, the target motion indication information carries motion change information over continuous time, which can be represented by a set of keypoint displacements, a set of skeleton connections, a set of regional motion parameters, or a dense motion field. When converting this information into a spatiotemporal motion graph, the motion elements at each time position are generally organized according to the target sequence timeline, and then the motion elements are encoded into motion representations with temporal dimensions and spatial structure. The spatiotemporal motion graph can be constructed using a temporally stacked heatmap, a spatiotemporal topology graph, or a motion feature tensor, internally preserving motion position, connection relationships, displacement amplitude, and direction of change, so that the motion state over continuous time can be directly read by the content generation network. The target sequence timeline here serves as a unified temporal reference; each motion segment in the target motion indication information is mapped to a specific time position, avoiding mismatches between motion conditions and generated frame positions.

[0067] The initial object representation provides a stable representation of the object's appearance and overall structure. Local region features supplement the detailed constraints of the face, mouth, hands, or other key areas. Temporal-driven features carry rhythmic and semantic change information consistent with the target sequence's timeline. The latent space constraint mask sequence provides local constraint regions in the latent space. The initial noise latent variable sequence provides the starting representation to be updated. These types of inputs play different roles in the content generation network, and therefore are usually fed into the network using a split-encoding approach. A common implementation is to set up conditional embedding units, spatiotemporal interaction units, region constraint units, and latent variable update units in the content generation network. The conditional embedding unit maps the initial object representation, local region features, temporal-driven features, latent space constraint mask sequence, and spatiotemporal motion graph to a unified feature space. The spatiotemporal interaction unit jointly models the time and space dimensions, enabling the object's preservation information, local detail information, rhythmic information, and motion information to be correlated within the same representation space. The region constraint unit uses the latent space constraint mask sequence to control the update range and intensity of local regions. The latent variable update unit generates content latent variables round by round around the initial noise latent variable sequence and restores the generated content latent variables to the generated content sequence.

[0068] Conditional generation processing manifests as a latent variable update process guided by multiple conditions. The initial object representation primarily constrains identity consistency and overall structural continuity; local region features primarily constrain local texture and action details; temporal-driven features primarily constrain temporal rhythm and semantic changes; latent spatial constraint mask sequences primarily constrain the influence range of local regions; and spatiotemporal motion graphs primarily constrain body movement trajectories and spatial displacement relationships. The content generation network can employ a diffusion-based denoising structure or a generation structure with spatiotemporal attention. When using a diffusion-based denoising structure, the network receives the current latent variable state and multi-condition information in each round of latent variable updates. It then writes the conditional information into the latent variable representation through cross-attention, gated modulation, or normalized modulation, and outputs a new latent variable state. When using a spatiotemporal attention structure, the network jointly inputs temporal slice features and spatial region features, maintaining inter-frame continuity through temporal attention and local structural stability through spatial attention, finally outputting a generated content sequence. The generated content sequence can be a video frame sequence or a character animation frame sequence.

[0069] In one implementation, the content generation network employs a spatiotemporal transformation structure. Target motion indication information is encoded into a spatiotemporal motion graph containing time, node, and connection dimensions, and then mapped to motion guidance features by a motion coding unit. The initial object representation enters the object preservation coding unit, local region features enter the local enhancement coding unit, temporal-driven features enter the rhythm semantic coding unit, and the latent space constraint mask sequence enters the region modulation unit. All conditional features are fed into a multi-layer spatiotemporal attention block under a unified dimension, interacting with and updating the initial noisy latent variable sequence. During the update process, the region modulation unit controls the feature writing intensity of local locations according to the latent space constraint mask sequence, the motion guidance features control the overall movement direction, the object preservation features stabilize the character's appearance, the local enhancement features correct facial and hand details, and the rhythm semantic features control the temporal variation density. After multiple rounds of updates, the content sequence is generated and output by the decoding unit. This implementation is suitable for scenarios with large motion amplitudes and high requirements for local region details.

[0070] In another implementation, the content generation network employs a dual-stream generation structure. One data stream carries the initial noise latent variable sequence and is responsible for generating the main content, while the other carries conditional information and is responsible for guiding modulation. The spatiotemporal motion map and temporally driven features are first combined into dynamic guiding features, the initial object representation and local region features are combined into object detail guiding features, and the latent space constraint mask sequence forms a region control feature separately. The dynamic guiding features are used to constrain the direction of motion and the intensity of action at each time point, the object detail guiding features are used to stabilize the appearance and local details, and the region control features are used to limit the injection location of local features. The two data streams interact at multiple latent variable update layers, with each layer outputting a new content latent variable representation. Finally, the generated content sequence is output through a temporal decoder. This implementation is suitable for scenarios that require both stable character identity and synchronized action time.

[0071] A phased generation approach can also be adopted. The first phase uses spatiotemporal motion maps and time-driven features to generate coarse-grained motion content, obtaining intermediate latent variables with an overall motion trend. The second phase incorporates the initial object representation, local region features, and latent space constraint mask sequence into the intermediate latent variable update process, refining local textures and region boundaries, and outputting a sequence of generated content. This phased approach is suitable for content generation tasks where both torso motion and local detail changes are significant.

[0072] In the field of healthcare, when generating digital humans for home rehabilitation guidance, the target movement instruction information can come from the upper limb movement trajectory or mouth training trajectory in the rehabilitation action template. Spatiotemporal motion maps are used to constrain limb swing and posture changes, local region features are used to enhance hand and mouth details, latent space constraint mask sequences are used to limit the update range of local regions, and the generated content sequence can be used for rehabilitation action demonstrations and voice-guided training output.

[0073] In the fintech business field, when digital employees broadcast financial product interpretations, risk warnings, or investment advisory explanations, the target motion indication information can come from the gesture trajectory and body displacement information in the explanation action template. The spatiotemporal motion map is used to constrain the overall movement trend during the explanation, the time-driven features are used to constrain the broadcast rhythm, and the local area features are used to preserve the details of the mouth and hands. The generated content sequence can be used for the output of financial product explanations, return descriptions, and risk disclosure screens.

[0074] This embodiment converts target motion indication information into a spatiotemporal motion graph, which is then input into the content generation network along with the initial object representation, local region features, temporally driven features, latent space constraint mask sequence, and initial noise latent variable sequence. The generation process simultaneously incorporates object preservation, local detail enhancement, temporal synchronization constraints, region range constraints, and motion trajectory constraints. These multiple conditions work synergistically within the same generation process, improving the consistency of identity, the integrity of local actions, inter-frame continuity, and the degree of action temporal matching in the generated content sequence.

[0075] In one embodiment, step S10 includes: S101, acquire the reference object image, timing drive signal, initial motion indication information and initial motion region contour information sequence; S102, extract the driving duration interval and driving unit change boundary based on the timing driving signal, and generate the target sequence time axis according to the driving duration interval and driving unit change boundary; S103, align the initial motion indication information with the target sequence time axis to obtain the target motion indication information; S104, perform contour time alignment on the initial action region contour information sequence according to the target sequence time axis to obtain the target action region contour information sequence. S105, determine the latent space feature dimension based on the reference object image, and generate a set of noisy latent variable frames according to the target sequence time axis and the latent space feature dimension; S106, organize the noise latent variable frame set according to the target sequence time axis to generate an initial noise latent variable sequence.

[0076] In this embodiment, the reference object image is read as an image tensor during the input phase, and the image size, visible subject area, cropping boundary, and quality marker are recorded simultaneously. The image size is used to determine the subsequent latent space feature dimensions, the visible subject area is used to eliminate interference from non-subject areas on the latent space scale determination, the cropping boundary is used to correct the latent space input range when there are missing edges in the image, and the quality marker can be generated from sharpness, occlusion ratio, and subject integrity. The reference object image does not participate in the timeline generation but participates in the determination of latent space feature dimensions; therefore, data related to the image size and subject area needs to be retained during the input phase.

[0077] After effective interval identification, the temporal driving signal forms a driving duration interval. The driving duration interval can be determined through endpoint detection, energy thresholding, semantic segment start and end markers, rhythm markers, or external timestamps. The driving duration interval provides the temporal coverage of the target sequence timeline, ensuring the generated sequence length corresponds to the duration of the driving content. Driving unit change boundaries are extracted from the temporal driving signal and are used to mark the locations where changes occur in the driving content. For speech signals, driving unit change boundaries can be formed by phoneme boundaries, syllable boundaries, pause boundaries, or stress boundaries; for script rhythm signals, driving unit change boundaries can be formed by sentence segmentation points, keyword time points, pause markers, and emphasis markers. The driving duration interval determines the timeline length, and the driving unit change boundaries determine the positions of change nodes within the timeline; the combination of these two elements generates the target sequence timeline.

[0078] The target sequence timeline is stored in discrete time position format, which can be a frame bit sequence, a segment bit sequence, or a latent variable position sequence. During generation, the start and end ranges are determined based on the driving duration interval, and the base time positions are generated based on the target output interval. Then, the driving unit change boundaries are mapped to adjacent time positions, so that the boundary positions have clear landing points in the target sequence timeline. If the driving unit change boundary falls between two time positions, the corresponding position can be determined using nearest neighbor mapping, linear interpolation, or boundary snapping. After the target sequence timeline is generated, subsequent motion time alignment, contour time alignment, and noise latent variable frame set organization are all referenced to this timeline.

[0079] Initial motion indication information records the overall motion information before alignment, and may include motion amplitude, direction, area, key node displacement, motion template number, and motion intensity range. During motion time alignment, the motion time markers or motion segment positions in the initial motion indication information are mapped to discrete time positions in the target sequence time axis. If some time positions in the target sequence time axis do not have directly corresponding motion information, supplementary motion information can be generated based on the displacement trends and intensity changes of adjacent motion segments. The aligned result serves as the target motion indication information, which has a time position consistent with the target sequence time axis and can be directly used in subsequent generation of spatiotemporal motion maps.

[0080] The initial action region contour information sequence records the boundaries of the local action region before alignment and may include contour point sets, contour sampling times, contour confidence levels, and region categories. During contour temporal alignment, the initial action region contour information sequence is temporally rearranged according to the target sequence time axis, mapping each contour sampling time to its corresponding position on the target sequence time axis. If contour information is missing at certain target time positions, supplementary contours can be generated based on the displacement direction, boundary deformation, and confidence level changes of adjacent contour point sets. The aligned result serves as the target action region contour information sequence, maintaining temporal consistency with the target sequence time axis, and can subsequently be used to generate pixel space mask sequences.

[0081] The latent space feature dimensions are determined based on the reference object image. In implementation, the image size and visible range of the subject in the reference object image can be read, and combined with the latent space compression scale, target output ratio, and latent space input format of the content generation network, the spatial scale and feature dimensions of each latent variable frame bit can be determined. The latent space feature dimensions can include latent space height, latent space width, and the number of feature channels, or they can be represented by the number of latent space tokens and the token feature length. If the reference object image has been cropped or scaled, the latent space feature dimensions are determined based on the effective image range after cropping or scaling, avoiding the expansion of invalid edge regions into the latent space input.

[0082] The set of noise latent variable frames is generated based on the target sequence's timeline and latent space feature dimensions. The target sequence's timeline determines the number and temporal positions of the noise latent variable frames, while the latent space feature dimensions determine the spatial shape and feature length of each frame. During generation, one noise latent variable frame can be generated for each time position on the target sequence's timeline. These frames can be generated using normal distribution sampling, pseudo-random seed generation, or controlled noise templates. To maintain controllable variations between different time positions, time-related perturbations can be set between adjacent noise latent variable frames, or each frame can be generated independently. After the set of noise latent variable frames is formed, they are arranged according to the temporal order of the target sequence's timeline to obtain the initial noise latent variable sequence. The initial noise latent variable sequence has the same temporal length as the target sequence's timeline and the same spatial scale as the latent space feature dimensions.

[0083] This embodiment generates a target sequence time axis by driving the continuous interval and the changing boundary of the driving unit. The generated sequence length and driving change position can maintain correspondence with the timing driving signal. The target sequence time axis is used to time-align the initial motion indication information and the initial action area contour information sequence, so that the target motion indication information and the target action area contour information sequence can share the same time reference. The latent space feature dimension is determined by the reference object image, and the set of noise latent variable frames is organized according to the target sequence time axis. The initial noise latent variable sequence can match the subsequent conditional input in terms of time position and latent space shape, thereby reducing the time misalignment and latent space size inconsistency in sequence generation.

[0084] In one embodiment, step S20 above includes: S201, Input the reference object image into a variational autoencoder network to generate appearance latent variable parameters; S202, Generate a representation of latent appearance variables based on the latent appearance variable parameters; S203, extract the coordinates of the attitude key points from the reference object image, and generate an attitude structure diagram based on the coordinates of the attitude key points; S204, Input the attitude structure diagram into the attitude coding network to generate attitude structure feature codes; S205, the attitude structure feature encoding is mapped to a latent space resolution consistent with the appearance latent variable representation to obtain the aligned attitude structure feature encoding; S206, The appearance latent variable representation and the alignment posture structural feature encoding are concatenated in the channel dimension to obtain the concatenated feature; S207, The spliced ​​features are input into the fusion mapping layer to generate an initial object representation.

[0085] In this embodiment, the reference image carries the identity, texture details, and overall shape of the object. After input, it undergoes size unification, color normalization, and main body region preservation, ensuring that the subsequent two branches extract information on the same image reference. A variational autoencoder network compresses appearance information from the pixel space into the latent space. The output appearance latent variable parameters can be stored as distribution parameters, such as mean parameters, fluctuation parameters, or equivalent distribution coefficients. This representation allows skin color, texture, clothing style, contour details, and identity appearance to be compressed into a compact representation, facilitating subsequent generation. When generating the appearance latent variable representation based on the appearance latent variable parameters, parameter sampling, reparameter transformation, or parameter mapping can be used to transform the appearance information from parameter form into a latent space feature tensor that can directly participate in fusion. This feature tensor preserves stable information about the object's appearance, making it suitable as a main representation for subsequent object preservation.

[0086] The coordinates of pose keypoints are extracted from the image of the reference object and are used to describe the spatial positions of key parts of the object. Keypoints can originate from facial keypoints, hand keypoints, upper limb keypoints, torso keypoints, or other location points related to structural morphology. Keypoint coordinates can be expressed using a two-dimensional coordinate set, and can also include confidence, visibility, and connectivity categories. After the pose keypoint coordinates are generated, they are not directly concatenated with the latent variable representation of appearance; instead, a pose structure diagram is first constructed. The pose structure diagram can be formed through keypoint drawing, skeleton connection, region topology filling, or multi-channel structural encoding, transforming discrete coordinates into an image-like representation with spatial distribution characteristics. This preserves the relative positional relationships between keypoints, limb connection relationships, and the overall pose contour, avoiding the inadequacy of structural relationships expressed using only coordinate sequences.

[0087] After the pose structure map is input into the pose encoding network, the network performs multi-layer feature extraction on the spatial layout and connection patterns in the structure map, outputting the pose structure feature code. This code highlights the object's pose, skeleton orientation, local opening and closing states, and relative positional relationships of parts, representing structural information. The appearance latent variable representation and the pose structure feature code are generated in different branches, and they typically differ in spatial resolution, channel size, and representation focus. Therefore, it is necessary to map the pose structure feature code to a latent spatial resolution consistent with the appearance latent variable representation. This mapping can be achieved through upsampling, downsampling, convolutional projection, linear mapping, or scaling transformation, aligning the two features in spatial dimensions and eliminating size inconsistencies in subsequent stitching. The aligned pose structure feature code, after alignment, can jointly express object information at the same latent spatial scale as the appearance latent variable representation.

[0088] Channel-dimensional concatenation is used to merge the latent variable representation of appearance and the aligned pose structural feature encoding into a unified feature. The concatenated feature contains both appearance and structural information, where appearance information reflects the object's identity and texture attributes, and structural information reflects the object's pose and morphological layout. However, the concatenated feature still suffers from uneven proportions of information in the two branches, local redundancy, and semantic dispersion. Therefore, a fusion mapping layer is introduced to further integrate the concatenated features. This fusion mapping layer can employ convolutional mapping, linear mapping, gating mapping, or attention mapping to redistribute the weights of appearance-related and structure-related components within a unified feature space, suppressing irrelevant components and strengthening the effective components that jointly participate in the generation. The resulting initial object representation retains both the object's appearance recognizability and contains initial structural constraint information, providing a stable and compact object conditional input for subsequent content generation.

[0089] This embodiment uses a variational autoencoder network to form latent variable parameters of appearance and generate latent variable representations of appearance, which can compress the identity and texture information in the reference object image into a stable latent space expression. By extracting the coordinates of pose key points, generating a pose structure map, and obtaining pose structure feature encoding through a pose encoding network, the structural layout and pose relationships of the object can be transformed into a learnable structural representation. Through latent space resolution alignment, channel dimension splicing, and fusion mapping layer integration, appearance information and structural information can be unified into the same object representation, thereby improving the object's identity preservation ability and structural consistency, and reducing appearance drift, pose distortion, and local structural inconsistency in the subsequent generation process.

[0090] In one embodiment, step S30 above includes: S301, Generate a local region key point heatmap in the reference object image, and extract candidate key point positions from the local region key point heatmap; S302, Generate a candidate region topology map based on the candidate key point locations, and filter the local region key point locations according to the topological completeness and spatial clustering of the candidate region topology map; S303, Generate a local region bounding box based on the location of the key points of the local region, and configure a region type label for the local region bounding box. The region type label includes identity preservation type and action prior type. S304, Generate cropping transformation parameters based on the local region bounding box and the image boundary of the reference object image; S305, based on the cropping transformation parameters, a local region image is obtained by cropping from the reference object image, and a region correspondence relationship is established between the local region image and the region type label; S306, Generate a region confidence weight corresponding to the region based on the visibility of the key point positions in the local region and the integrity of the local region bounding box; S307, based on the region correspondence and the region type label, input the local region image corresponding to the identity preservation type into the identity preservation coding branch to generate identity preservation local features, and input the local region image corresponding to the action prior type into the action prior coding branch to generate action prior local features; S308, modulate the identity preservation local features according to the regional credibility weight corresponding to the identity preservation type to obtain modulated identity preservation local features, and modulate the action prior local features according to the regional credibility weight corresponding to the action prior type to obtain modulated action prior local features. S309, the modulation identity preservation local features and the modulation action prior local features are fused to generate local region features.

[0091] In this embodiment, the local keypoint heatmap is formed by a keypoint detection network performing dense response prediction on a reference object image. Each channel in the heatmap corresponds to a type of local keypoint, and the pixel value within a channel represents the response intensity of that location belonging to the corresponding keypoint. By using a heatmap format instead of directly outputting discrete coordinates, occlusions, blurring, and missing boundaries in the local area can be preserved as continuous distribution information, facilitating subsequent filtering. Candidate keypoint locations can be obtained through heatmap peak detection, non-maximum suppression, or neighborhood response aggregation. Locations with higher response values ​​are extracted as candidate keypoint locations. If multiple high-response locations exist in the same area, effective candidate points can be retained by combining response intensity ranking and minimum interval constraints, reducing interference from adjacent false detection points on local area localization.

[0092] The candidate region topology map is constructed based on the geometric relationships between candidate key point locations. During construction, candidate key point locations can be used as graph nodes, and preset connection relationships, distance relationships, or relative direction relationships can be used as graph edges, thus forming a structural map reflecting the spatial organization of a local region. Topological completeness measures whether the key points and their connections are complete within a candidate region, and can be determined based on the retention ratio of key nodes, key connections, or the completeness ratio of closed structures. Spatial clustering measures whether candidate key point locations are concentrated around the same local region, and can be determined based on key point spacing, center deviation, enclosing area density, or distribution dispersion. By jointly filtering with topological completeness and spatial clustering, key point locations in local regions with more complete structures and more concentrated spatial distribution can be selected from multiple candidate regions, avoiding the problem of key points belonging to different parts or being mixed across regions when selecting points solely based on single-point response intensity.

[0093] Once the locations of key points in a local region are determined, a bounding box for that region can be generated based on the minimum bounding area of ​​the key points, a preset expansion ratio, and morphological constraints. The bounding box serves both to determine the cropping range and to distinguish between different region types. Region type labels are used to differentiate the functional emphasis of different local regions in subsequent feature extraction. Identity-preserving types are typically used for local regions carrying strong identity stability information, such as the face, the area near the hairline, or local regions with significant appearance recognition. Action-prior types are typically used for local regions carrying strong action change information, such as the hand, the area near limb joints, or local regions with more pronounced changes in the amplitude of movement. Directly configuring region type labels onto the local region bounding box ensures a clear correspondence between the subsequent cropping results and the encoding branches, preventing local region images from entering mismatched feature extraction branches.

[0094] The cropping transformation parameters are jointly determined by the local region bounding box and the image boundary of the reference object image. The local region bounding box provides the cropping center, width and height range, and local region scale information, while the image boundary constrains the cropped region to not exceed the effective image range. Cropping transformation parameters may include translation, scaling, boundary compensation, width and height adjustment coefficients, and coordinate normalization coefficients. When cropping regions near the boundary, if the local region bounding box exceeds the image boundary, mirroring, zero-value padding, or neighborhood interpolation padding can be used to supplement the cropping window, ensuring uniform image size in the local region. After local region image cropping is completed, a region correspondence between the local region image and region type labels needs to be established. This correspondence can be stored using index mapping, identifier binding, or a data table format, enabling each local region image to accurately trace its region type in subsequent processing.

[0095] The region confidence weight is jointly determined by the visibility of keypoints in the local region and the integrity of the local region's bounding box. Visibility reflects whether keypoints are occluded, whether they are within the effective field of view, and whether the response is stable. Integrity reflects the adequacy of the local region's bounding box in covering the target local region, such as whether key parts are truncated or whether too much background area is included. Region confidence weights can be formed through weighted combination, piecewise mapping, or normalized mapping. If the visibility of keypoints in a certain region is low or the bounding box integrity is insufficient, the region confidence weight is reduced accordingly to weaken the region's influence in subsequent feature fusion. Binding region confidence weights to region correspondences allows each type of local region to carry modulation information matching its own quality state.

[0096] The identity-preserving encoding branch and the action-prior encoding branch process different types of local region images respectively. When the local region image corresponding to the identity-preserving type is input into the identity-preserving encoding branch, the branch focuses on extracting texture distribution, color combinations, contour details, and local identity recognition information to generate identity-preserving local features. When the local region image corresponding to the action-prior type is input into the action-prior encoding branch, the branch focuses on extracting local morphological changes, joint opening and closing, regional pose trends, and action-related structural information to generate action-prior local features. The two encoding branches can use different convolutional layer configurations, receptive field settings, attention units, or mapping dimensions, allowing the local region images to extract more targeted representations according to their respective functional directions. This branch-separation approach can reduce information aliasing that occurs when different types of local regions are encoded uniformly.

[0097] Region credibility weights are used to modulate identity-preserving local features and action-prior local features. Modulation can be achieved through channel weight scaling, spatial location weighting, gated modulation, or feature recalibration. When the region credibility weights corresponding to the identity-preserving type are applied to the identity-preserving local features, modulated identity-preserving local features are obtained, giving higher weight to high-quality identity regions in the final local representation. When the region credibility weights corresponding to the action-prior type are applied to the action-prior local features, modulated action-prior local features are obtained, allowing local regions with clearer action information and more complete structures to participate more fully in subsequent fusion. After modulation, the fusion module integrates the modulated identity-preserving local features and modulated action-prior local features. Fusion can employ splicing mapping, weighted summation, interactive attention mapping, or gated fusion methods, enabling local region features to simultaneously carry identity stability information and action change information, forming a local representation suitable for subsequent generation processing in a unified feature space.

[0098] This embodiment improves the accuracy of local region localization by jointly filtering key point heatmaps, candidate region topology maps, topological completeness, and spatial clustering. Through region type labeling, region correspondence, and a dual-branch coding structure, identity-related information and action-related information can be extracted separately. Modulating the two types of local features with region confidence weights before fusion reduces interference from occlusion, incomplete cropping, and local false detections. The resulting local region features balance local identity recognition and action expression capabilities, resulting in more stable local details and clearer local action information. This reduces region drift, local distortion, and detail mixing during subsequent generation.

[0099] In one embodiment, step S40 above includes: S401, the timing drive signal is divided into a drive segment sequence, and a drive time marker is configured for the drive segment sequence; S402, input the driving segment sequence into the temporal semantic encoder to generate a semantic feature sequence; S403, extract the driving unit change boundary based on the semantic feature sequence and the driving time marker, and perform boundary segmentation on the semantic feature sequence according to the driving unit change boundary to generate a semantic segmentation feature sequence; S404, Generate a target time window sequence based on the target sequence time axis, and perform time window matching between the semantic segmentation feature sequence and the target time window sequence according to the driving time marker to obtain a temporally aligned semantic feature sequence; S405, extract the latent variable temporal position marker and latent variable feature dimension marker from the initial noise latent variable sequence; S406, The temporally aligned semantic feature sequence is embedded with the position based on the latent variable temporal position marker to obtain the position-embedded semantic feature sequence; S407, the position embedding semantic feature sequence is mapped into candidate temporal driving features based on the latent variable feature dimension label; S408, a boundary attention mask is generated based on the change boundary of the driving unit, the target time window sequence, and the latent variable temporal position marker, and the candidate temporal driving features are modulated according to the boundary attention mask to generate temporal driving features.

[0100] In this embodiment, the temporal-driven signal needs to be divided into a sequence of driving segments before entering semantic extraction. Segment boundaries can be formed based on time sampling points, energy change points, pause positions, semantic switching positions, or external timestamps. Each driving segment retains its start and end times, segment length, and segment order, and is configured with a driving time marker. The driving time marker is used to record the temporal position of the driving segment in the temporal-driven signal, enabling subsequent semantic feature sequences to retain the temporal source of the driving content. If the temporal-driven signal is speech, audio frames, phoneme intervals, or syllable intervals can be used as driving segments; if the temporal-driven signal is a rhythm script, sentence segments, pause segments, or emphasis segments can be used as driving segments.

[0101] After the driving segment sequence is input into the temporal semantic encoder, the encoder performs feature extraction and context aggregation on each driving segment to generate a semantic feature sequence. The temporal semantic encoder can include a segment embedding layer, a temporal position embedding layer, a sequence encoding layer, and an output mapping layer. The segment embedding layer converts the driving segments into vector representations, the temporal position embedding layer writes driving time markers into the segment vectors, the sequence encoding layer extracts the semantic continuity relationships between adjacent segments, and the output mapping layer organizes the encoded results into a semantic feature sequence of uniform dimensions. During training, time-labeled driving segments can be paired with target time positions to enable the temporal semantic encoder to learn the correspondence between driving segments and semantic change locations.

[0102] The driving unit change boundaries are extracted from the semantic feature sequence and driving time markers. These boundaries can be determined by semantic feature adjacency differences, pause intervals, phoneme changes, stress changes, or script markers. After extraction, the driving unit change boundaries are used to segment the semantic feature sequence, making the locations of significant semantic changes the segmentation criteria. The segmented semantic feature sequence retains the internal continuity of each semantic segment and the change relationships between segments, preventing different driving units from being mixed into boundless continuous features during subsequent time matching.

[0103] The target sequence timeline is converted into a target time window sequence. Each target time window corresponds to a time position or a set of adjacent time positions in the generated content. During time window matching, driving timestamps are used to determine the target time window into which each segment in the semantic segment feature sequence enters. If a target time window covers multiple semantic segments, it can be aggregated into a single time window feature using time proportion, boundary priority weight, or attention weight; if a semantic segment spans multiple target time windows, it can be split into multiple time windows according to the time coverage ratio. After matching, a temporally aligned semantic feature sequence is obtained, whose time positions are consistent with the target sequence timeline.

[0104] The initial noisy latent variable sequence provides the temporal location and latent spatial dimension of the subsequent generated input. Latent variable temporal location markers are extracted from the initial noisy latent variable sequence and are used to record the position of each latent variable frame position on the target sequence's timeline. Latent variable feature dimension markers are used to record the feature length, number of channels, or token dimension of each latent variable frame position. During location embedding, the latent variable temporal location markers are written into the temporally aligned semantic feature sequence, creating a one-to-one correspondence between semantic features and latent variable frame positions, resulting in a location-embedded semantic feature sequence.

[0105] The location-embedded semantic feature sequence needs to be mapped to candidate temporal-driven features. During mapping, latent variable feature dimension labels are used to determine the dimension of the output features, enabling the candidate temporal-driven features to match the initial noisy latent variable sequence in terms of feature dimension. The mapping layer can employ linear mapping, convolutional mapping, or multilayer perceptron mapping. If the initial noisy latent variable sequence is in tensor form, the candidate temporal-driven features can be organized into spatially broadcast features; if the initial noisy latent variable sequence is in token form, the candidate temporal-driven features can be organized into token features of the same length.

[0106] The boundary attention mask is generated based on the driving unit change boundary, the target time window sequence, and the latent variable temporal position markers. The driving unit change boundary provides the driving change location, the target time window sequence provides the target time range corresponding to the boundary, and the latent variable temporal position markers provide the landing point of the boundary in the latent variable sequence. The boundary attention mask can assign higher weights to candidate temporal driving features near the change boundary and lower weights to positions far from the change boundary. After being modulated by the boundary attention mask, the candidate temporal driving features form temporal driving features. This feature simultaneously contains semantic content, temporal location, latent space dimension, and boundary change information, and can be used as the driving condition input in subsequent content generation.

[0107] This embodiment preserves the temporal source of the temporal driving signal by using a driving segment sequence and driving time stamps. It generates a semantic feature sequence through a temporal semantic encoder and then uses the driving unit change boundary to form a semantic segmentation feature sequence, which can enhance the distinguishability of the driving content at the time boundary. By completing time window matching through the target time window sequence, the semantic features can be kept consistent with the time axis of the target sequence. By completing position embedding and dimension mapping through latent variable temporal position stamps and latent variable feature dimension stamps, the candidate temporal driving features can be matched with the initial noise latent variable sequence in terms of time position and feature dimension. By modulating the candidate temporal driving features through boundary attention mask, the driving expression near the change position of the driving unit can be strengthened, thereby reducing the misalignment between the driving semantics and the generated time position.

[0108] In one embodiment, step S50 above includes: S501, extract the contour sampling time marker, contour point set sequence and contour confidence marker from the contour information sequence of the target action area; S502, establish a contour time mapping relationship based on the target sequence time axis and the contour sampling time marker; S503, the contour point set sequence is rearranged in time according to the contour time mapping relationship to obtain a time rearranged contour point set sequence, the contour displacement trend between adjacent contour point sets in the time rearranged contour point set sequence is determined, and a complete contour point set is generated based on the contour displacement trend. S504, Based on the time-rearranged contour point set sequence and the completed contour point set, generate an aligned target action region contour information sequence. S505, the contour credibility mark is rearranged in time according to the contour time mapping relationship to obtain the time rearranged contour credibility mark, and the completed contour credibility mark is generated based on the completed contour point set and the time rearranged contour credibility mark. S506, Generate a contour filling map sequence and a boundary transition map sequence based on the aligned target action region contour information sequence; S507, generate a mask confidence map sequence based on the time rearrangement contour confidence mark and the completion contour confidence mark, and fuse the contour filling map sequence, the boundary transition map sequence and the mask confidence map sequence to generate a pixel space mask sequence; S508, Generate mask projection parameters based on the pixel space mask sequence and the latent space resolution corresponding to the initial noise latent variable sequence; S509, Project the pixel space mask sequence onto the latent space according to the mask projection parameters to obtain the latent space candidate mask sequence; S510, perform adjacent frame smoothing processing on the latent space candidate mask sequence according to the target sequence time axis to generate a latent space constraint mask sequence.

[0109] In this embodiment, the target action region contour information sequence includes temporal dimension information, spatial boundary information, and reliability information. After extracting the contour sampling time stamp, contour point set sequence, and contour reliability marker, the three dimensions of time, shape, and quality are separated, facilitating alignment and reconstruction separately. The contour sampling time stamp is used to indicate the sampling time of each frame contour, and can be a timestamp, frame number, or a position index calculated based on the sampling frequency. The contour point set sequence is used to represent the action region boundary and can be composed of key boundary points, equidistant sampling boundary points, or spline control points. The contour reliability marker is used to indicate the effectiveness of each frame contour and can be derived from the contour detection score, boundary closure degree, occlusion ratio, or point set completeness.

[0110] After establishing a contour time mapping relationship between the target sequence time axis and the contour sampling time markers, a corresponding basis is formed between the original contour sampling time and the target generation time. The contour time mapping relationship can be implemented using nearest-time matching, linear interpolation matching, or piecewise interpolation matching, allowing the original contour sequence to be rearranged according to the target sequence time axis. After time rearranging the contour point set sequence based on the contour time mapping relationship, a time-rearranged contour point set sequence is obtained, with each contour point set aligned with the target sequence time axis in terms of temporal position. For cases where missing positions or excessively large time intervals still exist after time rearrangement, it is necessary to determine the contour displacement trend between adjacent contour point sets. The contour displacement trend can be determined through the corresponding point displacement vector, the region centroid offset, the boundary scale change, or the boundary rotation change. When generating a complete contour point set based on the contour displacement trend, interpolation calculations can be performed on the boundary points at the missing time, or shape completion can be performed on local boundary segments to continuously extend the contour of the missing frame. After combining the time-rearranged contour point set sequence with the completed contour point set, an aligned target action region contour information sequence is obtained. This sequence already has the ability to express continuous boundaries facing the time axis of the target sequence.

[0111] The contour confidence marker needs to be consistent with the temporal alignment result of the contour point set. After temporally rearranging the contour confidence markers according to the contour temporal mapping relationship, a temporally rearranged contour confidence marker is obtained. The confidence level corresponding to the padded contour point set cannot directly use the original value; it needs to be generated by combining the padded source and the confidence level of adjacent frames. During generation, weighted interpolation of adjacent frame confidence levels can be used, or a new confidence value can be formed by superimposing the contour displacement amplitude and boundary continuity. In this way, the padded contour point set and the padded contour confidence marker correspond one-to-one, allowing the subsequent mask generation stage to distinguish the reliability differences between the original contour region and the padded contour region.

[0112] The aligned target action region contour information sequence is used to generate a contour fill map sequence and a boundary transition map sequence. The contour fill map sequence represents the effective coverage area within the action region and can be achieved through closed boundary filling, region rasterization, or polygon scan conversion. The boundary transition map sequence expresses the gradient relationship from the action region edge to the background region; it can generate attenuation regions based on boundary distance or form transition bands based on boundary normal expansion. The contour fill map sequence provides a clear foreground region range, while the boundary transition map sequence provides edge softening information; combining the two can reduce boundary breaks and mask jaggedness issues.

[0113] The temporal rearrangement contour confidence marker and the padded contour confidence marker are further used to generate a mask confidence map sequence. The mask confidence map sequence can assign higher weights to high-confidence regions and lower weights to regions with large padded areas or low integrity, allowing the mask to carry reliability differences in spatial distribution. After fusing the contour-filled map sequence, the boundary transition map sequence, and the mask confidence map sequence, a pixel-space mask sequence is obtained. The fusion method can employ pixel-wise weighted combination, partitioned weight modulation, or confidence-based boundary attenuation, enabling the pixel-space mask sequence to simultaneously express region location, boundary transition, and confidence distribution.

[0114] The pixel space mask sequence needs to be transformed into the latent space to maintain consistency with subsequent latent variable sequences. The latent space resolution corresponding to the initial noisy latent variable sequence is used to constrain the projected target size. After generating mask projection parameters based on the pixel space mask sequence and latent space resolution, the scaling ratio, spatial offset, sampling method, and pixel alignment rules can be clearly defined. The mask projection parameters determine both the spatial size transformation and the spatial position mapping relationship. After projecting the pixel space mask sequence into the latent space according to the mask projection parameters, a latent space candidate mask sequence is obtained. During projection, average pooling, bilinear scaling, or convolutional downsampling can be used to compress pixel-level region constraints into a region constraint representation at the latent space scale.

[0115] The latent space candidate mask sequence may still exhibit local jitter, boundary jumps, or accumulated padding errors in the temporal dimension. Therefore, it is necessary to perform adjacent frame smoothing based on the target sequence's timeline. Adjacent frame smoothing can employ sliding time window weighted averaging, temporal consistency filtering, or adjacent frame difference constraints to make the latent space mask changes more stable between adjacent time points. The smoothed latent space constraint mask sequence is more continuous in time, maintains the location and boundary range of the action region in space, and preserves the credibility differences in quality, thus serving as the region constraint input for subsequent generation stages.

[0116] This embodiment establishes a contour time mapping relationship with the target sequence time axis by using contour sampling time markers, which enables uniform alignment of the action region contour in the target time dimension. By time-rearranging the contour point set sequence, contour displacement trend, and filling in the contour point set, the continuity of contour information in missing moments and cross-frame change scenarios can be enhanced. By generating a mask confidence map sequence by time-rearranging contour confidence markers and filling in the contour confidence markers, the reliability of different regions can be preserved in the mask representation. By fusing the contour filling map sequence, boundary transition map sequence, and mask confidence map sequence to generate a pixel space mask sequence, and further projecting it into a latent space constraint mask sequence, the spatial constraints can be kept consistent with the latent space resolution. By smoothing adjacent frames, the jumps of the latent space mask in the time dimension can be reduced, thereby improving the continuity and stability of the action region constraints.

[0117] In one embodiment, step S60 above includes: S601, based on the target sequence time axis, the target motion indication information is time-expanded to generate a motion frame index sequence; S602, based on the motion frame index sequence, extract the motion key node sequence, motion connection sequence and motion intensity sequence from the target motion indication information, and generate a spatiotemporal motion map based on the motion key node sequence, the motion connection sequence and the motion intensity sequence; S603, generate an object preservation condition sequence based on the initial object representation, generate a local enhancement condition sequence based on the local region features, generate a driving semantic condition sequence based on the temporal driving features, generate a mask constraint condition sequence based on the latent space constraint mask sequence, and generate a motion guidance condition sequence based on the spatiotemporal motion graph. S604, based on the target sequence time axis, the object preservation condition sequence, the local enhancement condition sequence, the driving semantic condition sequence, the mask constraint condition sequence, and the motion guidance condition sequence are time-series arranged to generate a conditional temporal position correspondence, and condition type tags are configured for the object preservation condition sequence, the local enhancement condition sequence, the driving semantic condition sequence, the mask constraint condition sequence, and the motion guidance condition sequence, and a joint condition sequence is generated based on the conditional temporal position correspondence and the condition type tags; S605, input the joint conditional sequence and the initial noise latent variable sequence into the content generation network, and establish object preservation path, local enhancement path, temporal interaction path, mask constraint path and motion guidance path in the content generation network; S606, based on the condition type marker and the condition temporal position correspondence, separate the object preservation condition sequence, local enhancement condition sequence, driving semantic condition sequence, mask constraint condition sequence and motion guidance condition sequence from the joint condition sequence corresponding to each time position in the target sequence time axis; S607, based on the correspondence between the object preservation path and the conditional temporal position, the object preservation conditional sequence is injected into the initial noise latent variable sequence; based on the correspondence between the local enhancement path and the conditional temporal position, the local enhancement conditional sequence is injected into the initial noise latent variable sequence; based on the correspondence between the temporal interaction path and the conditional temporal position, the driving semantic conditional sequence is injected into the initial noise latent variable sequence; based on the correspondence between the mask constraint path and the conditional temporal position, the mask constraint conditional sequence is injected into the initial noise latent variable sequence; and based on the correspondence between the motion guidance path and the conditional temporal position, the motion guidance conditional sequence is injected into the initial noise latent variable sequence, thereby generating a stage content latent variable sequence; S608, Based on the latent variable sequence of the stage content, conditional generation processing is performed in the content generation network to obtain the generated content sequence.

[0118] In this embodiment, the target motion indication information carries motion control content that changes over time. After time unpacking, a motion frame index sequence is formed, giving each target time position a clear motion index. Time unpacking can be performed by resampling the target motion indication information according to the frame interval of the target sequence time axis, or by discretizing the continuous motion description according to a preset time granularity. After the motion frame index sequence is formed, the motion state in the target motion indication information is split into various time positions. This index sequence is used as the positioning basis when extracting the motion key node sequence, motion connection sequence, and motion intensity sequence. The motion key node sequence is used to express the positional changes of key parts of the moving subject at each time position, and can be composed of joint coordinates, displacement vectors, or local geometric centers. The motion connection sequence is used to express the structural connection relationship between key nodes, and can be represented by a set of node pairs, a set of skeleton edges, or a region correlation matrix. The motion intensity sequence is used to express the motion amplitude at different time positions, and can be determined by the node displacement amplitude, the change in connection stretching, or the region motion energy. After these three types of sequences are combined into a spatiotemporal motion diagram, the change information in the time dimension and the structural information in the spatial dimension are uniformly encoded, so that motion control no longer stays at the local description of a single moment, but forms a continuous motion expression for the entire sequence.

[0119] Since the initial object representation, local region features, temporally driven features, latent space constraint mask sequences, and spatiotemporal motion graphs have different functions, they are converted into object-preserving condition sequences, local enhancement condition sequences, driving semantic condition sequences, mask constraint condition sequences, and motion-guided condition sequences, respectively. Control information from different sources is then broken down into independently schedulable conditional expressions. The object-preserving condition sequence carries appearance consistency information of the reference object, typically including identity-related latent representations, pose-related basic representations, or a combination of both. The local enhancement condition sequence carries enhanced representations of the face, hands, or other local regions to improve the recognizability and motion details of these regions during generation. The driving semantic condition sequence carries rhythmic, semantic, or phonetic control quantities extracted from temporally driven signals to control the synchronization of actions over time. The mask constraint condition sequence carries information on the range of action regions and constraint boundaries to limit local changes during content generation. The motion-guided condition sequence carries the overall pose change trend in the spatiotemporal motion graph to control the continuity of the generated content at the global motion level. After the conditional sequences are formed, the control content that was originally mixed in different inputs is decomposed into conditional expressions oriented towards different generation goals, providing a basis for subsequent organization by time position and injection by type.

[0120] When arranging conditional sequences temporally, a one-to-one correspondence between conditional content and time position needs to be established based on the target sequence timeline. This temporal position correspondence indicates which time position or continuous time range each conditional information should apply to. The object-preserving conditional sequences are valid for all time periods in most scenarios, forming a cross-time period reuse correspondence. Local enhancement conditional sequences can be configured to corresponding time positions according to the activity level of local actions or the frequency of changes in local regions. Driving semantic conditional sequences naturally correspond to temporal driving signals and are usually directly matched according to the target sequence timeline. Mask constraint conditional sequences need to be consistent with the temporal changes in the action region contour. Motion-guided conditional sequences are associated with each time position based on the frame-level representation of the spatiotemporal motion graph. Condition type labels are used to distinguish the source category and control responsibility of each conditional sequence, enabling the subsequent content generation network to identify the injection direction of different conditions. After generating a joint condition sequence based on the correspondence between conditional temporal positions and conditional type labels, the previously scattered multiple types of conditions are integrated into a unified temporal input structure. The joint condition sequence retains both time position attributes and conditional type attributes, which facilitates subsequent classification and scheduling under a unified input state.

[0121] After receiving the joint conditional sequence and the initial noise latent variable sequence, the content generation network needs to form object preservation, local enhancement, temporal interaction, mask constraint, and motion guidance pathways within the network. The object preservation pathway applies the object preservation conditional sequence to the initial noise latent variable sequence, ensuring the generated content maintains the continuity of the reference object's appearance over the entire time span. The local enhancement pathway introduces relevant information about local region features, enhancing the expression of key local regions. The temporal interaction pathway introduces driving semantic conditional sequences, ensuring the driving rhythm corresponds to the temporal position of the latent variables. The mask constraint pathway applies mask constraint conditional sequences to the spatial range of the latent variables, controlling local changes within regional constraints. The motion guidance pathway introduces motion guidance conditional sequences into the latent variable evolution process, ensuring smooth temporal changes in the motion trajectory. The content generation network can employ a multi-layered latent variable transformation structure, with each layer containing a spatial transformation layer, a temporal interaction layer, and a conditional fusion layer. The conditional fusion layer can use cross-attention, gated modulation, or weighted mapping to introduce different conditional segments from the joint conditional sequence into the corresponding pathways.

[0122] Although the joint conditional sequence achieves unified organization, it still needs to be separated according to the correspondence between condition type labels and condition temporal positions when it is used in specific applications. The separated object-preserving conditional sequence, local enhancement conditional sequence, driving semantic conditional sequence, mask constraint conditional sequence, and motion-guided conditional sequence all carry temporal information corresponding to each time position in the target sequence's time axis. In this way, when the content generation network processes the initial noise latent variable sequence at each time position, it can extract the condition subset corresponding to the current time position from the joint conditional sequence, avoiding the mixing of conditions between different time positions. Separating actions also ensures that the differences of the same type of condition at different time positions are preserved, enabling the content generation network to maintain the overall stability of the object while reflecting fine-grained changes in actions and driving signals during continuous-time processing.

[0123] After the object-preserving conditional sequence, local enhancement conditional sequence, driving semantic conditional sequence, mask constraint conditional sequence, and motion-guided conditional sequence are injected into the initial noise latent variable sequence through their respective pathways, the initial noise latent variable sequence gradually transforms from an unstructured noise state into a stage content latent variable sequence controlled by multiple conditions. The injection of the object-preserving conditional sequence typically acts on the global semantic layer and appearance representation layer, maintaining identity consistency and overall contour stability in the generated result. The injection of the local enhancement conditional sequence typically acts on high-resolution latent variable regions or local attention regions, improving the quality of local details. The injection of the driving semantic conditional sequence typically acts on temporal interaction positions, keeping sequence changes synchronized with temporal driving signals. The injection of the mask constraint conditional sequence typically acts on the latent variable space region selection process, allowing adjustment in local regions while maintaining suppression in non-target regions. The injection of the motion-guided conditional sequence typically acts on the continuous temporal change process, controlling the pose and amplitude changes of the generated content. The stage content latent variable sequence formed after multi-path injection already contains five types of control information: object appearance, local details, temporal driving, region constraints, and overall motion.

[0124] As the latent variable sequence of the stage content continues to undergo conditional generation processing in the content generation network, the network updates the latent representation layer by layer based on the current latent variable state and the injected conditions, gradually converging to the target content expression that satisfies multiple conditional constraints. Conditional generation processing can manifest as multi-round iterative updates of latent variables or as multi-layer decoding transformation. During the update process, the temporal attention mechanism maintains continuous changes between different time positions, the spatial attention mechanism maintains the stability of details in local regions, and the conditional modulation mechanism maintains the continuous effect of various control information at different levels. The resulting generated content sequence exhibits motion continuity in the temporal dimension, object consistency and local detail fidelity in the spatial dimension, and simultaneously responds to the joint constraints of temporally driven features, latent spatial constraint mask sequences, and spatiotemporal motion graphs in the control dimension.

[0125] This embodiment organizes motion control information into a spatiotemporal motion graph by temporally unfolding the target motion indication information and extracting the motion key node sequence, motion connection sequence, and motion intensity sequence, thereby improving the global motion continuity of the entire generated content. By converting the initial object representation, local region features, temporal driving features, latent space constraint mask sequence, and spatiotemporal motion graph into corresponding condition sequences, and further generating a joint condition sequence, control information from different sources can be organized and scheduled within a unified time frame. By setting up object preservation paths, local enhancement paths, temporal interaction paths, mask constraint paths, and motion guidance paths in the content generation network, and implementing separation and injection based on the condition type label and the correspondence between condition temporal positions, multiple types of conditions can participate in latent variable updates at their respective suitable positions. The resulting stage content latent variable sequence can simultaneously carry object consistency, local details, driving synchronization, regional constraints, and overall motion information, thereby improving the performance of the generated content sequence in terms of temporal stability, local integrity, and motion controllability.

[0126] In one embodiment, a content sequence generation apparatus based on temporal alignment is provided, which corresponds one-to-one with the content sequence generation method based on temporal alignment described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the content sequence generation device based on time alignment of the present invention. The modules include a target sequence timeline generation module 10, an initial object representation extraction module 20, a local region feature extraction module 30, a time-driven feature generation module 40, a latent space mask generation module 50, and a conditional content generation module 60. Detailed descriptions of each functional module are as follows: The target sequence time axis generation module 10 is used to acquire a reference object image, a timing drive signal, target motion indication information, and a sequence of target action area contour information, determine the target sequence time axis based on the timing drive signal, and generate an initial noise latent variable sequence according to the target sequence time axis. The initial object representation extraction module 20 is used to extract appearance latent variable representation and posture structure feature encoding based on the reference object image, and fuse the appearance latent variable representation and the posture structure feature encoding to obtain the initial object representation; The local region feature extraction module 30 is used to determine the location of key points in the local region in the reference object image, crop the local region image based on the location of the key points in the local region, and extract the local region features corresponding to the local region image. The time-driven feature generation module 40 is used to extract semantic feature sequences based on the time-driven signal, perform time-series alignment of the semantic feature sequences according to the time axis of the target sequence, and convert the time-series aligned semantic feature sequences into time-driven features corresponding to the initial noise latent variable sequence. The latent space mask generation module 50 is used to align the target action region contour information sequence according to the target sequence time axis, generate a pixel space mask sequence based on the aligned target action region contour information sequence, and project the pixel space mask sequence onto the latent space to obtain a latent space constraint mask sequence. The conditional content generation module 60 is used to convert the target motion indication information into a spatiotemporal motion graph corresponding to the time axis of the target sequence, and to perform conditional generation processing through a content generation network based on the initial object representation, the local region features, the temporal driving features, the latent space constraint mask sequence, the spatiotemporal motion graph, and the initial noise latent variable sequence to obtain the generated content sequence.

[0127] In one embodiment, the target sequence timeline generation module 10 is specifically used for: Acquire the reference object image, timing drive signal, initial motion indication information, and initial motion region contour information sequence; Based on the time-series driving signal, the driving duration interval and the driving unit change boundary are extracted, and the target sequence time axis is generated according to the driving duration interval and the driving unit change boundary; The initial motion indication information is aligned with the target sequence time axis to obtain the target motion indication information. The initial action region contour information sequence is aligned with the target sequence time axis to obtain the target action region contour information sequence. The latent space feature dimension is determined based on the reference object image, and a set of noisy latent variable frames is generated according to the target sequence time axis and the latent space feature dimension. The noise latent variable frame set is organized according to the target sequence time axis to generate an initial noise latent variable sequence.

[0128] In one embodiment, the initial object representation extraction module 20 is specifically used for: The reference object image is input into a variational autoencoder network to generate latent appearance variable parameters. Generate a representation of latent appearance variables based on the aforementioned latent appearance variable parameters; Extract the coordinates of attitude key points from the reference object image, and generate an attitude structure diagram based on the coordinates of the attitude key points; The attitude structure diagram is input into the attitude coding network to generate attitude structure feature codes; The posture structure feature encoding is mapped to a latent space resolution consistent with the appearance latent variable representation to obtain the aligned posture structure feature encoding. The appearance latent variable representation and the alignment posture structural feature encoding are concatenated along the channel dimension to obtain the concatenated feature; The spliced ​​features are input into the fusion mapping layer to generate an initial object representation.

[0129] In one embodiment, the local region feature extraction module 30 is specifically used for: A local region key point heatmap is generated in the reference object image, and candidate key point locations are extracted from the local region key point heatmap. A candidate region topology map is generated based on the candidate key point locations, and the locations of local region key points are obtained by filtering based on the topological completeness and spatial clustering of the candidate region topology map. A local region bounding box is generated based on the location of key points in the local region, and a region type label is configured for the local region bounding box. The region type label includes identity preservation type and action prior type. Crop transformation parameters are generated based on the local region bounding box and the image boundary of the reference object image; Based on the cropping transformation parameters, a local region image is cropped from the reference object image to obtain a local region image, and a region correspondence relationship is established between the local region image and the region type label; Based on the visibility of the key point locations in the local region and the integrity of the local region's bounding box, a region confidence weight corresponding to the region's correspondence is generated. Based on the region correspondence and the region type label, the local region image corresponding to the identity preservation type is input into the identity preservation coding branch to generate identity preservation local features, and the local region image corresponding to the action prior type is input into the action prior coding branch to generate action prior local features. The identity preservation local features are modulated according to the regional credibility weights corresponding to the identity preservation type to obtain modulated identity preservation local features, and the action prior local features are modulated according to the regional credibility weights corresponding to the action prior type to obtain modulated action prior local features. By fusing the modulation identity-preserving local features and the modulation action prior local features, local region features are generated.

[0130] In one embodiment, the timing-driven feature generation module 40 is specifically used for: The timing drive signal is divided into a drive segment sequence, and a drive time marker is configured for the drive segment sequence; The driving segment sequence is input into a temporal semantic encoder to generate a semantic feature sequence; Based on the semantic feature sequence and the driving time marker, the driving unit change boundary is extracted, and the semantic feature sequence is segmented according to the driving unit change boundary to generate a semantic segmentation feature sequence; A target time window sequence is generated based on the target sequence time axis, and the semantic segmentation feature sequence is matched with the target time window sequence according to the driving time marker to obtain a temporally aligned semantic feature sequence. Extract the temporal position markers and feature dimension markers of the latent variables from the initial noise latent variable sequence; The temporally aligned semantic feature sequence is embedded with the latent variable temporal position markers to obtain the position-embedded semantic feature sequence. Based on the latent variable feature dimension labeling, the position embedding semantic feature sequence is mapped into candidate temporal driving features; Based on the change boundary of the driving unit, the target time window sequence, and the latent variable temporal position marker, a boundary attention mask is generated, and the candidate temporal driving features are modulated according to the boundary attention mask to generate temporal driving features.

[0131] In one embodiment, the latent space mask generation module 50 is specifically used for: Extract the contour sampling time marker, contour point set sequence, and contour confidence marker from the contour information sequence of the target action region; Establish a contour time mapping relationship based on the target sequence time axis and the contour sampling time marker; The contour point set sequence is rearranged in time according to the contour time mapping relationship to obtain a time rearranged contour point set sequence. The contour displacement trend between adjacent contour point sets in the time rearranged contour point set sequence is determined, and a complete contour point set is generated based on the contour displacement trend. Based on the time-rearranged contour point set sequence and the completed contour point set, an aligned target action region contour information sequence is generated. The contour credibility marker is rearranged in time according to the contour time mapping relationship to obtain the time rearranged contour credibility marker, and the completed contour credibility marker is generated based on the completed contour point set and the time rearranged contour credibility marker. Generate a contour filling map sequence and a boundary transition map sequence based on the aligned target action region contour information sequence; A mask confidence map sequence is generated based on the time rearrangement contour confidence marker and the completion contour confidence marker, and the contour filling map sequence, the boundary transition map sequence and the mask confidence map sequence are fused to generate a pixel space mask sequence; Based on the pixel space mask sequence and the latent space resolution corresponding to the initial noise latent variable sequence, mask projection parameters are generated; The pixel space mask sequence is projected onto the latent space according to the mask projection parameters to obtain the latent space candidate mask sequence; The latent space candidate mask sequence is smoothed by adjacent frames according to the target sequence time axis to generate a latent space constrained mask sequence.

[0132] In one embodiment, the conditional content generation module 60 is specifically used for: Based on the target sequence time axis, the target motion indication information is time-expanded to generate a motion frame index sequence; Based on the motion frame index sequence, the motion key node sequence, motion connection sequence, and motion intensity sequence are extracted from the target motion indication information, and a spatiotemporal motion graph is generated based on the motion key node sequence, the motion connection sequence, and the motion intensity sequence; Based on the initial object representation, an object preservation condition sequence is generated; based on the local region features, a local enhancement condition sequence is generated; based on the temporal driving features, a driving semantic condition sequence is generated; based on the latent space constraint mask sequence, a mask constraint condition sequence is generated; and based on the spatiotemporal motion graph, a motion guidance condition sequence is generated. Based on the target sequence timeline, the object preservation condition sequence, the local enhancement condition sequence, the driving semantic condition sequence, the mask constraint condition sequence, and the motion guidance condition sequence are time-series arranged to generate a conditional temporal position correspondence. Condition type tags are configured for the object preservation condition sequence, the local enhancement condition sequence, the driving semantic condition sequence, the mask constraint condition sequence, and the motion guidance condition sequence. A joint condition sequence is generated based on the conditional temporal position correspondence and the condition type tags. The joint conditional sequence and the initial noise latent variable sequence are input into the content generation network, and object preservation path, local enhancement path, temporal interaction path, mask constraint path and motion guidance path are established in the content generation network; Based on the condition type marker and the correspondence between the condition time sequence positions, the object preservation condition sequence, local enhancement condition sequence, driving semantic condition sequence, mask constraint condition sequence and motion guidance condition sequence corresponding to each time position in the target sequence time axis are separated from the joint condition sequence; Based on the correspondence between the object preservation path and the conditional temporal position, the object preservation conditional sequence is injected into the initial noise latent variable sequence; based on the correspondence between the local enhancement path and the conditional temporal position, the local enhancement conditional sequence is injected into the initial noise latent variable sequence; based on the correspondence between the temporal interaction path and the conditional temporal position, the driving semantic conditional sequence is injected into the initial noise latent variable sequence; based on the correspondence between the mask constraint path and the conditional temporal position, the mask constraint conditional sequence is injected into the initial noise latent variable sequence; and based on the correspondence between the motion guidance path and the conditional temporal position, the motion guidance conditional sequence is injected into the initial noise latent variable sequence, thereby generating a stage content latent variable sequence. Based on the latent variable sequence of the stage content, conditional generation processing is performed in the content generation network to obtain the generated content sequence.

[0133] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a time-aligned content sequence generation method on the server side.

[0134] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a time-aligned content sequence generation method.

[0135] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Acquire a sequence of reference object image, timing drive signal, target motion indication information, and target action area contour information; determine the target sequence time axis based on the timing drive signal; and generate an initial noise latent variable sequence according to the target sequence time axis. Based on the reference object image, latent variable representation of appearance and posture structure feature encoding are extracted, and the latent variable representation of appearance and the posture structure feature encoding are fused to obtain an initial object representation; In the reference object image, the location of key points in a local region is determined, and a local region image is obtained by cropping based on the location of the key points in the local region. The local region features corresponding to the local region image are then extracted. Based on the time-driven signal, a semantic feature sequence is extracted, and the semantic feature sequence is time-aligned according to the time axis of the target sequence. The time-aligned semantic feature sequence is then converted into a time-driven feature corresponding to the initial noise latent variable sequence. The target action region contour information sequence is aligned according to the target sequence time axis, and a pixel space mask sequence is generated based on the aligned target action region contour information sequence. The pixel space mask sequence is then projected onto the latent space to obtain a latent space constraint mask sequence. The target motion indication information is converted into a spatiotemporal motion graph corresponding to the time axis of the target sequence. Based on the initial object representation, the local region features, the temporal driving features, the latent space constraint mask sequence, the spatiotemporal motion graph, and the initial noise latent variable sequence, a content generation network is used to perform conditional generation processing to obtain the generated content sequence.

[0136] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, performs the following steps: Acquire a sequence of reference object image, timing drive signal, target motion indication information, and target action area contour information; determine the target sequence time axis based on the timing drive signal; and generate an initial noise latent variable sequence according to the target sequence time axis. Based on the reference object image, latent variable representation of appearance and posture structure feature encoding are extracted, and the latent variable representation of appearance and the posture structure feature encoding are fused to obtain an initial object representation; In the reference object image, the location of key points in a local region is determined, and a local region image is obtained by cropping based on the location of the key points in the local region. The local region features corresponding to the local region image are then extracted. Based on the time-driven signal, a semantic feature sequence is extracted, and the semantic feature sequence is time-aligned according to the time axis of the target sequence. The time-aligned semantic feature sequence is then converted into a time-driven feature corresponding to the initial noise latent variable sequence. The target action region contour information sequence is aligned according to the target sequence time axis, and a pixel space mask sequence is generated based on the aligned target action region contour information sequence. The pixel space mask sequence is then projected onto the latent space to obtain a latent space constraint mask sequence. The target motion indication information is converted into a spatiotemporal motion graph corresponding to the time axis of the target sequence. Based on the initial object representation, the local region features, the temporal driving features, the latent space constraint mask sequence, the spatiotemporal motion graph, and the initial noise latent variable sequence, a content generation network is used to perform conditional generation processing to obtain the generated content sequence.

[0137] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0138] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0139] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0140] It should be noted that any software tools or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0141] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A content sequence generation method based on temporal alignment, characterized in that, Includes the following steps: Acquire a sequence of reference object image, timing drive signal, target motion indication information, and target action area contour information; determine the target sequence time axis based on the timing drive signal; and generate an initial noise latent variable sequence according to the target sequence time axis. Based on the reference object image, latent variable representation of appearance and posture structure feature encoding are extracted, and the latent variable representation of appearance and the posture structure feature encoding are fused to obtain an initial object representation; In the reference object image, the location of key points in a local region is determined, and a local region image is obtained by cropping based on the location of the key points in the local region. The local region features corresponding to the local region image are then extracted. Based on the time-driven signal, a semantic feature sequence is extracted, and the semantic feature sequence is time-aligned according to the time axis of the target sequence. The time-aligned semantic feature sequence is then converted into a time-driven feature corresponding to the initial noise latent variable sequence. The target action region contour information sequence is aligned according to the target sequence time axis, and a pixel space mask sequence is generated based on the aligned target action region contour information sequence. The pixel space mask sequence is then projected onto the latent space to obtain a latent space constraint mask sequence. The target motion indication information is converted into a spatiotemporal motion graph corresponding to the time axis of the target sequence. Based on the initial object representation, the local region features, the temporal driving features, the latent space constraint mask sequence, the spatiotemporal motion graph, and the initial noise latent variable sequence, a content generation network is used to perform conditional generation processing to obtain the generated content sequence.

2. The content sequence generation method based on temporal alignment as described in claim 1, characterized in that, Acquire a sequence of reference object image, temporal driving signal, target motion indication information, and target action region contour information; determine the target sequence time axis based on the temporal driving signal; and generate an initial noise latent variable sequence according to the target sequence time axis, including: Acquire the reference object image, timing drive signal, initial motion indication information, and initial motion region contour information sequence; Based on the time-series driving signal, the driving duration interval and the driving unit change boundary are extracted, and the target sequence time axis is generated according to the driving duration interval and the driving unit change boundary; The initial motion indication information is aligned with the target sequence time axis to obtain the target motion indication information. The initial action region contour information sequence is aligned with the target sequence time axis to obtain the target action region contour information sequence. The latent space feature dimension is determined based on the reference object image, and a set of noisy latent variable frames is generated according to the target sequence time axis and the latent space feature dimension. The noise latent variable frame set is organized according to the target sequence time axis to generate an initial noise latent variable sequence.

3. The content sequence generation method based on temporal alignment as described in claim 1, characterized in that, Based on the reference object image, latent variable representations of appearance and pose structure feature encodings are extracted. The latent variable representations of appearance and the pose structure feature encodings are then fused to obtain an initial object representation, including: The reference object image is input into a variational autoencoder network to generate latent appearance variable parameters. Generate a representation of latent appearance variables based on the aforementioned latent appearance variable parameters; Extract the coordinates of attitude key points from the reference object image, and generate an attitude structure diagram based on the coordinates of the attitude key points; The attitude structure diagram is input into the attitude coding network to generate attitude structure feature codes; The posture structure feature encoding is mapped to a latent space resolution consistent with the appearance latent variable representation to obtain the aligned posture structure feature encoding. The appearance latent variable representation and the alignment posture structural feature encoding are concatenated along the channel dimension to obtain the concatenated feature; The spliced ​​features are input into the fusion mapping layer to generate an initial object representation.

4. The content sequence generation method based on temporal alignment as described in claim 1, characterized in that, Determine the locations of key points in a local region within the reference object image, crop the local region image based on these key point locations, and extract the local region features corresponding to the local region image, including: A local region key point heatmap is generated in the reference object image, and candidate key point locations are extracted from the local region key point heatmap. A candidate region topology map is generated based on the candidate key point locations, and the locations of local region key points are obtained by filtering based on the topological completeness and spatial clustering of the candidate region topology map. A local region bounding box is generated based on the location of key points in the local region, and a region type label is configured for the local region bounding box. The region type label includes identity preservation type and action prior type. Crop transformation parameters are generated based on the local region bounding box and the image boundary of the reference object image; Based on the cropping transformation parameters, a local region image is cropped from the reference object image to obtain a local region image, and a region correspondence relationship is established between the local region image and the region type label; Based on the visibility of the key point locations in the local region and the integrity of the local region's bounding box, a region confidence weight corresponding to the region's correspondence is generated. Based on the region correspondence and the region type label, the local region image corresponding to the identity preservation type is input into the identity preservation coding branch to generate identity preservation local features, and the local region image corresponding to the action prior type is input into the action prior coding branch to generate action prior local features. The identity preservation local features are modulated according to the regional credibility weights corresponding to the identity preservation type to obtain modulated identity preservation local features, and the action prior local features are modulated according to the regional credibility weights corresponding to the action prior type to obtain modulated action prior local features. By fusing the modulation identity-preserving local features and the modulation action prior local features, local region features are generated.

5. The content sequence generation method based on temporal alignment as described in claim 1, characterized in that, Based on the time-driven signal, a semantic feature sequence is extracted; the semantic feature sequence is time-aligned according to the time axis of the target sequence; and the time-aligned semantic feature sequence is converted into a time-driven feature corresponding to the initial noise latent variable sequence, including: The timing drive signal is divided into a drive segment sequence, and a drive time marker is configured for the drive segment sequence; The driving segment sequence is input into a temporal semantic encoder to generate a semantic feature sequence; Based on the semantic feature sequence and the driving time marker, the driving unit change boundary is extracted, and the semantic feature sequence is segmented according to the driving unit change boundary to generate a semantic segmentation feature sequence; A target time window sequence is generated based on the target sequence time axis, and the semantic segmentation feature sequence is matched with the target time window sequence according to the driving time marker to obtain a temporally aligned semantic feature sequence. Extract the temporal position markers and feature dimension markers of the latent variables from the initial noise latent variable sequence; The temporally aligned semantic feature sequence is embedded with the latent variable temporal position markers to obtain the position-embedded semantic feature sequence. Based on the latent variable feature dimension labeling, the position embedding semantic feature sequence is mapped into candidate temporal driving features; Based on the change boundary of the driving unit, the target time window sequence, and the latent variable temporal position marker, a boundary attention mask is generated, and the candidate temporal driving features are modulated according to the boundary attention mask to generate temporal driving features.

6. The content sequence generation method based on temporal alignment as described in claim 1, characterized in that, Align the target action region contour information sequence according to the target sequence time axis, generate a pixel space mask sequence based on the aligned target action region contour information sequence, and project the pixel space mask sequence onto the latent space to obtain a latent space constraint mask sequence, including: Extract the contour sampling time marker, contour point set sequence, and contour confidence marker from the contour information sequence of the target action region; Establish a contour time mapping relationship based on the target sequence time axis and the contour sampling time marker; The contour point set sequence is rearranged in time according to the contour time mapping relationship to obtain a time rearranged contour point set sequence. The contour displacement trend between adjacent contour point sets in the time rearranged contour point set sequence is determined, and a complete contour point set is generated based on the contour displacement trend. Based on the time-rearranged contour point set sequence and the completed contour point set, an aligned target action region contour information sequence is generated. The contour credibility marker is rearranged in time according to the contour time mapping relationship to obtain the time rearranged contour credibility marker, and the completed contour credibility marker is generated based on the completed contour point set and the time rearranged contour credibility marker. Generate a contour filling map sequence and a boundary transition map sequence based on the aligned target action region contour information sequence; A mask confidence map sequence is generated based on the time rearrangement contour confidence marker and the completion contour confidence marker, and the contour filling map sequence, the boundary transition map sequence and the mask confidence map sequence are fused to generate a pixel space mask sequence; Based on the pixel space mask sequence and the latent space resolution corresponding to the initial noise latent variable sequence, mask projection parameters are generated; The pixel space mask sequence is projected onto the latent space according to the mask projection parameters to obtain the latent space candidate mask sequence; The latent space candidate mask sequence is smoothed by adjacent frames according to the target sequence time axis to generate a latent space constrained mask sequence.

7. The content sequence generation method based on temporal alignment as described in claim 1, characterized in that, The target motion indication information is converted into a spatiotemporal motion graph corresponding to the time axis of the target sequence. Based on the initial object representation, the local region features, the temporal driving features, the latent space constraint mask sequence, the spatiotemporal motion graph, and the initial noise latent variable sequence, a content generation network is used for conditional generation processing to obtain a generated content sequence, including: Based on the target sequence time axis, the target motion indication information is time-expanded to generate a motion frame index sequence; Based on the motion frame index sequence, the motion key node sequence, motion connection sequence, and motion intensity sequence are extracted from the target motion indication information, and a spatiotemporal motion graph is generated based on the motion key node sequence, the motion connection sequence, and the motion intensity sequence; Based on the initial object representation, an object preservation condition sequence is generated; based on the local region features, a local enhancement condition sequence is generated; based on the temporal driving features, a driving semantic condition sequence is generated; based on the latent space constraint mask sequence, a mask constraint condition sequence is generated; and based on the spatiotemporal motion graph, a motion guidance condition sequence is generated. Based on the target sequence timeline, the object preservation condition sequence, the local enhancement condition sequence, the driving semantic condition sequence, the mask constraint condition sequence, and the motion guidance condition sequence are time-series arranged to generate a conditional temporal position correspondence. Condition type tags are configured for the object preservation condition sequence, the local enhancement condition sequence, the driving semantic condition sequence, the mask constraint condition sequence, and the motion guidance condition sequence. A joint condition sequence is generated based on the conditional temporal position correspondence and the condition type tags. The joint conditional sequence and the initial noise latent variable sequence are input into the content generation network, and object preservation path, local enhancement path, temporal interaction path, mask constraint path and motion guidance path are established in the content generation network; Based on the condition type marker and the correspondence between the condition time sequence positions, the object preservation condition sequence, local enhancement condition sequence, driving semantic condition sequence, mask constraint condition sequence and motion guidance condition sequence corresponding to each time position in the target sequence time axis are separated from the joint condition sequence; Based on the correspondence between the object preservation path and the conditional temporal position, the object preservation conditional sequence is injected into the initial noise latent variable sequence; based on the correspondence between the local enhancement path and the conditional temporal position, the local enhancement conditional sequence is injected into the initial noise latent variable sequence; based on the correspondence between the temporal interaction path and the conditional temporal position, the driving semantic conditional sequence is injected into the initial noise latent variable sequence; based on the correspondence between the mask constraint path and the conditional temporal position, the mask constraint conditional sequence is injected into the initial noise latent variable sequence; and based on the correspondence between the motion guidance path and the conditional temporal position, the motion guidance conditional sequence is injected into the initial noise latent variable sequence, thereby generating a stage content latent variable sequence. Based on the latent variable sequence of the stage content, conditional generation processing is performed in the content generation network to obtain the generated content sequence.

8. A content sequence generation device based on temporal alignment, characterized in that, The content sequence generation device based on temporal alignment includes: The target sequence time axis generation module is used to acquire a reference object image, a timing drive signal, target motion indication information, and a sequence of target action area contour information, determine the target sequence time axis based on the timing drive signal, and generate an initial noise latent variable sequence according to the target sequence time axis. The initial object representation extraction module is used to extract appearance latent variable representation and posture structure feature encoding based on the reference object image, and fuse the appearance latent variable representation and the posture structure feature encoding to obtain the initial object representation; The local region feature extraction module is used to determine the location of key points in the local region in the reference object image, crop the local region image based on the location of the key points in the local region, and extract the local region features corresponding to the local region image. The time-driven feature generation module is used to extract semantic feature sequences based on the time-driven signal, perform time-series alignment of the semantic feature sequences according to the time axis of the target sequence, and convert the time-series aligned semantic feature sequences into time-driven features corresponding to the initial noise latent variable sequence. The latent space mask generation module is used to align the target action region contour information sequence according to the target sequence time axis, generate a pixel space mask sequence based on the aligned target action region contour information sequence, and project the pixel space mask sequence onto the latent space to obtain a latent space constraint mask sequence. The conditional content generation module is used to convert the target motion indication information into a spatiotemporal motion graph corresponding to the time axis of the target sequence, and to perform conditional generation processing through a content generation network based on the initial object representation, the local region features, the temporal driving features, the latent space constraint mask sequence, the spatiotemporal motion graph, and the initial noise latent variable sequence to obtain the generated content sequence.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a time-aligned content sequence generation program stored in the memory and executable on the processor, wherein the time-aligned content sequence generation program, when executed by the processor, implements the steps of the time-aligned content sequence generation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a content sequence generation program based on time alignment, which, when executed by a processor, implements the steps of the content sequence generation method based on time alignment as described in any one of claims 1-7.

Images