Method for generating custom role video based on multi-reference feature fusion and related device

Abstract

This application relates to the field of video generation technology and provides a method and related equipment for generating custom character videos using multi-reference image feature fusion. The method includes: acquiring multiple reference images from different perspectives and performing identity recognition on the target character in the reference images to obtain global identity recognition data; acquiring a driving sequence for driving the target character's movement, and performing time-sequential encoding on the driving sequence to obtain motion information data and corresponding timestamps; inputting the global identity recognition data, motion information data, and timestamps into a motion adapter for risk avoidance and noise reduction to obtain a target video frame sequence and focus depth data; and reconstructing the focus of the target video frame sequence based on the focus depth data to output a custom character video. This invention solves the technical problem of identity drift when the character's posture changes in the generated video due to a single reference image carrying specific perspective, posture, and lighting deviation information.

Description

Technical Field

[0001] This invention relates to the field of video generation technology, and in particular to a method and related equipment for generating custom character videos by fusing multiple reference map features. Background Technology

[0002] Existing custom character video generation technologies based on reference images extract identity features from a single reference image. When the character's pose changes in the generated video, the lack of joint modeling of multi-view appearance features leads to identity feature drift. Furthermore, the subject's identity features and customized motion features are entangled within the network representation, and when the customized motion is transferred to different target characters, the generated results retain some of the original character's identity features. In existing technologies, visual tokens in long video sequence generation use a full self-attention mechanism to model temporal relationships, with computational complexity increasing quadratically with the number of frames. The reuse of timestamp information from the discrete prediction head of text tokens leads to biases in temporal boundary predictions. Moreover, the depth relationship of the scene is not coupled with the character's identity localization; depth estimation and blurring processing operate independently of the subject identity localization module, resulting in a mismatch between the focal plane and the character's actual depth position during character movement. Summary of the Invention

[0003] This invention provides a method and related equipment for generating custom character videos by fusing features from multiple reference images. This invention solves the technical problem of identity drift when the character's posture changes in the generated video due to the presence of specific viewpoint, posture, and lighting deviation information in a single reference image. It realizes relative motion risk prediction and active avoidance based on historical states, ensuring that the character's motion trajectory in the generated video conforms to scene constraints.

[0004] In a first aspect, embodiments of this application provide a method for generating custom character videos using multi-reference map feature fusion, including: Multiple reference images from different perspectives are acquired, and the target characters in the reference images are identified to obtain global identity recognition data of the target characters. Obtain the driving sequence used to drive the movement of the target character, encode the driving sequence in time order to obtain the encoded token sequence, and perform motion embedding and decoupling optimization on the encoded token sequence to obtain motion information data and the corresponding timestamp; The global identity recognition data, the motion information data, and the timestamp are input into the motion adapter for risk avoidance and noise reduction to obtain the target video frame sequence and focus depth data. Based on the focus depth data, the target video frame sequence is refocused and reconstructed to output a custom character video.

[0005] Optionally, in a first implementation of the first aspect of the present invention, the step of acquiring multiple reference images from different perspectives and performing identity recognition on the target character in the reference images to obtain global identity recognition data of the target character includes: Multiple reference images from different perspectives are acquired, and local identity features of the target characters in the reference images are extracted to obtain the first identity recognition data corresponding to each perspective. The preset identity anchoring vector and the first identity recognition data corresponding to each viewpoint are combined to form an input sequence. Multi-head self-attention calculation is performed on the input sequence to obtain the correlation weight between each first identity recognition data. The first identity recognition data corresponding to each viewpoint is weighted and fused according to the relevance weight, retaining the common identity feature components of each viewpoint and suppressing the non-common feature components caused by posture differences and lighting differences, to obtain the second identity recognition data corresponding to each viewpoint. Then, the second identity recognition data corresponding to each viewpoint is fused from multiple perspectives to obtain the global identity recognition data of the target character.

[0006] Optionally, in a second implementation of the first aspect of the present invention, the step of obtaining a driving sequence for driving the target character's movement, encoding the driving sequence in a time sequence to obtain an encoded token sequence, and performing motion embedding and decoupling optimization on the encoded token sequence to obtain motion information data and a corresponding timestamp includes: Obtain the driving sequence used to drive the movement of the target character, parse the driving sequence into a visual token sequence arranged in frames, and superimpose physical timestamp sine coding and sequence index sine coding on the visual token sequence to obtain the encoded token sequence. The encoded token sequence is subjected to Fourier transform along the time dimension and real-valued frequency domain features are extracted. The preset query matrix is ​​used as the query vector, and the real-valued frequency domain features are used as the key vector and value vector to perform cross-attention calculation to obtain the motion embedding summary vector. Construct an orthogonal constraint loss function between the motion embedding summary vector and the global identity recognition data. Decouple and optimize by minimizing the orthogonal constraint loss function to obtain motion information data, and regress the timestamp from the dedicated time identifier Token.

[0007] Optionally, in a third implementation of the first aspect of the present invention, the step of performing a Fourier transform on the encoded token sequence along the time dimension and extracting real-valued frequency domain features, using a preset query matrix as a query vector, and the real-valued frequency domain features as key vectors and value vectors to perform cross-attention calculation to obtain a motion embedding summary vector, includes: Along the time dimension, a one-dimensional real Fourier transform is performed on the encoded token sequence to obtain the original frequency domain features. The component with the lowest frequency index in the original frequency domain features is retained to obtain the low-frequency frequency domain features. Depth-separable convolution operations are performed on the first real part feature and the first imaginary part feature of the low-frequency frequency domain feature to obtain the second real part feature and the second imaginary part feature. The second real part feature and the second imaginary part feature are then concatenated and linearly projected in the channel dimension to obtain the real-valued frequency domain feature. Using a preset query matrix as the query vector and the real-valued frequency domain features as the key vector and value vector, cross-attention calculation is performed on the query vector, the key vector, and the value vector to obtain the global motion condition embedding. Then, mean pooling is performed on the global motion condition embedding in the token dimension to obtain the motion embedding summary vector.

[0008] Optionally, in a fourth implementation of the first aspect of the present invention, the step of constructing the orthogonal constraint loss function between the motion embedding summary vector and the global identity recognition data, performing decoupling optimization by minimizing the orthogonal constraint loss function to obtain motion information data, and regressing the timestamp from the dedicated time identifier Token, includes: Construct an orthogonal constraint loss function based on the motion embedding summary vector and the global identity recognition data; The parameters in the global motion condition embedding are updated by minimizing the orthogonal constraint loss function through backpropagation and then using the updated global motion condition embedding as motion information data after the orthogonal constraint loss function converges. The dedicated time identifier tokens in the encoded token sequence are subjected to depthwise separable convolution and cross-attention resampling to obtain time identifier resampling features. Regression calculation is then performed on the time identifier resampling features to obtain the timestamp.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, the step of inputting the global identity recognition data, the motion information data, and the timestamp into the motion adapter for risk avoidance and noise reduction to obtain the target video frame sequence and focus depth data includes: The global identity recognition data and the motion information data are respectively input into the video denoising generation network in the motion adapter. The first video frame sequence is obtained by injecting parameters and generating video frames through the identity cross-attention sublayer and motion cross-attention sublayer of each denoising module in the video denoising generation network. During the generation of each frame of the first video frame sequence, the velocity vector and acceleration vector are calculated and input into the risk discrimination network for correction, and a second video frame sequence is generated. The frames in the second video frame sequence are arranged according to the timestamp to generate the target video frame sequence. The local motion state vectors of each frame in the target video sequence are mapped to the character reference depth reference to obtain the character relative depth value of each frame, and the character relative depth values ​​of each frame are combined to form focus depth data.

[0010] Optionally, in a sixth implementation of the first aspect of the present invention, the step of calculating velocity vectors and acceleration vectors and inputting them into a risk discrimination network for correction during the generation of each frame of the first video frame sequence, generating a second video frame sequence, and arranging each frame of the second video frame sequence according to the timestamp to generate a target video frame sequence includes: During the process of generating each frame of the first video frame sequence, global average pooling is performed on the latent variable features of the current frame to obtain the local motion state vector of the current frame, and the local motion state vector of the current frame is stored in the history state buffer. The velocity vector and acceleration vector are calculated based on the local motion state vectors of adjacent frames in the historical state buffer, and the velocity vector and acceleration vector are input into the risk discrimination network to calculate the risk score. When the risk score exceeds a preset threshold, the gradient of the risk score relative to the predicted motion state is calculated, and correction is made along the negative direction of the gradient to generate a second video frame sequence. The frames in the second video frame sequence are then arranged according to the timestamp to generate a target video frame sequence.

[0011] Optionally, in a seventh implementation of the first aspect of the present invention, the step of reconstructing the focus of the target video frame sequence based on the focus depth data and outputting a custom character video includes: Calculate the relative depth distribution map of each frame in the target video frame sequence, and upsample the relative depth distribution map to the same spatial resolution as the corresponding frame to obtain the full resolution depth distribution map of each frame; Calculate the depth difference between the target depth value and the corresponding character relative depth value at each pixel position in the full-resolution depth distribution map of each frame, and calculate the blur circle diameter value at each pixel position based on the depth difference; In the preset Gaussian blur kernel library, blur kernels of adjacent size levels are selected and linear interpolation weighted between the two size levels according to the blur circle diameter value to obtain the adaptive blur kernel for each pixel position. The adaptive blur kernel is then used to reconstruct the focus of each pixel position in the target video frame sequence to output the custom character video.

[0012] Secondly, embodiments of this application provide a custom character video generation apparatus based on multi-reference map feature fusion, comprising: The character identification unit is used to acquire multiple reference images from different perspectives, identify the target character in the reference images, and obtain global identification data of the target character. The motion coding unit is used to acquire the driving sequence for driving the target character's motion, encode the driving sequence in time order to obtain the encoded token sequence, and perform motion embedding and decoupling optimization on the encoded token sequence to obtain motion information data and the corresponding timestamp. The risk avoidance and noise reduction unit is used to input the global identity recognition data, the motion information data and the timestamp into the motion adapter to perform risk avoidance and noise reduction, and obtain the target video frame sequence and focus depth data. The focus reconstruction unit is used to reconstruct the target video frame sequence based on the focus depth data and output a custom character video.

[0013] Optionally, in a first implementation of the second aspect of the present invention, the role identification unit is specifically used for: Multiple reference images from different perspectives are acquired, and local identity features of the target characters in the reference images are extracted to obtain the first identity recognition data corresponding to each perspective. The preset identity anchoring vector and the first identity recognition data corresponding to each viewpoint are combined to form an input sequence. Multi-head self-attention calculation is performed on the input sequence to obtain the correlation weight between each first identity recognition data. The first identity recognition data corresponding to each viewpoint is weighted and fused according to the relevance weight, retaining the common identity feature components of each viewpoint and suppressing the non-common feature components caused by posture differences and lighting differences, to obtain the second identity recognition data corresponding to each viewpoint. Then, the second identity recognition data corresponding to each viewpoint is fused from multiple perspectives to obtain the global identity recognition data of the target character.

[0014] Optionally, in a second implementation of the second aspect of the present invention, the motion coding unit is specifically used for: Obtain the driving sequence used to drive the movement of the target character, parse the driving sequence into a visual token sequence arranged in frames, and superimpose physical timestamp sine coding and sequence index sine coding on the visual token sequence to obtain the encoded token sequence. The encoded token sequence is subjected to Fourier transform along the time dimension and real-valued frequency domain features are extracted. The preset query matrix is ​​used as the query vector, and the real-valued frequency domain features are used as the key vector and value vector to perform cross-attention calculation to obtain the motion embedding summary vector. Construct an orthogonal constraint loss function between the motion embedding summary vector and the global identity recognition data. Decouple and optimize by minimizing the orthogonal constraint loss function to obtain motion information data, and regress the timestamp from the dedicated time identifier Token.

[0015] Optionally, in a third implementation of the second aspect of the present invention, the risk avoidance and noise reduction unit is specifically used for: The global identity recognition data and the motion information data are respectively input into the video denoising generation network in the motion adapter. The first video frame sequence is obtained by injecting parameters and generating video frames through the identity cross-attention sublayer and motion cross-attention sublayer of each denoising module in the video denoising generation network. During the generation of each frame of the first video frame sequence, the velocity vector and acceleration vector are calculated and input into the risk discrimination network for correction, and a second video frame sequence is generated. The frames in the second video frame sequence are arranged according to the timestamp to generate the target video frame sequence. The local motion state vectors of each frame in the target video sequence are mapped to the character reference depth reference to obtain the character relative depth value of each frame, and the character relative depth values ​​of each frame are combined to form focus depth data.

[0016] Optionally, in a fourth implementation of the second aspect of the present invention, the focus reconstruction unit is specifically used for: Calculate the relative depth distribution map of each frame in the target video frame sequence, and upsample the relative depth distribution map to the same spatial resolution as the corresponding frame to obtain the full resolution depth distribution map of each frame; Calculate the depth difference between the target depth value and the corresponding character relative depth value at each pixel position in the full-resolution depth distribution map of each frame, and calculate the blur circle diameter value at each pixel position based on the depth difference; In the preset Gaussian blur kernel library, blur kernels of adjacent size levels are selected and linear interpolation weighted between the two size levels according to the blur circle diameter value to obtain the adaptive blur kernel for each pixel position. The adaptive blur kernel is then used to reconstruct the focus of each pixel position in the target video frame sequence to output the custom character video.

[0017] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described custom character video generation method based on multi-reference graph feature fusion.

[0018] Fourthly, embodiments of this application provide a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned custom character video generation method involving multi-reference graph feature fusion.

[0019] Fifthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the above-described custom character video generation method based on multi-reference graph feature fusion.

[0020] In one of the solutions provided by the aforementioned method and related devices for generating custom character videos using multi-reference image feature fusion, a mechanism for extracting identity features from multiple view and multi-reference images and constructing global identity recognition data is used. After extracting local identity features from multiple reference images from different perspectives, the correlation between the features of each view is weighted through the self-attention mechanism in the cross-view fusion network. Common identity feature components such as facial structure proportions, relative positions of facial features, hairstyle outlines, and clothing styles are retained, while non-common components caused by differences in posture and lighting are suppressed. This makes the global identity recognition data independent of the character's posture state, achieving the separation of identity representation and posture representation from the source of feature extraction. This solves the technical problem of identity drift when the character's posture changes in the generated video due to a single reference image carrying specific viewpoint, posture, and lighting deviation information. This invention employs an absolute time-sequence joint encoding mechanism, concatenating the physical timestamp sine code and the sequence index sine code into a joint position code, which is then superimposed onto the tokens of each frame in the driving sequence. This allows the motion representation to simultaneously carry physical time and sequence order information. Through a Fourier-convolutional collaborative token compression mechanism, a one-dimensional real-valued Fourier transform is performed on the encoded token sequence while preserving low-frequency components. The real and imaginary parts of the low-frequency domain features are then subjected to depthwise separable convolutions and concatenated into real-valued frequency domain features. A fixed-length global motion condition embedding is obtained by cross-attention resampling with a preset query matrix and the real-valued frequency domain features. This addresses the technical problem that the computational complexity of the full self-attention mechanism in long video sequence generation increases quadratically with the number of frames. Furthermore, by resampling a dedicated time-identifier token and integrating it into a timestamp prediction head, continuous normalized timestamps are directly regressed, avoiding the reuse of the discrete classification prediction method for text tokens and resolving the time boundary prediction bias problem. This invention calculates cosine similarity between motion embedding summary vectors and global identity recognition data of target characters, and constructs an orthogonal constraint loss function. Orthogonal constraints are introduced during video generation training to reduce the directional correlation between motion embedding summary vectors and identity representations, helping to alleviate the coupling between motion features and identity features, and preserving action information in the driving sequence through the video generation task. Multiple reference images in the training samples correspond to the same character as the driving sequence, ensuring that identity constraints target the character identity information contained in the driving sequence. After training, the motion coding network is used for action transfer of different target characters. This invention utilizes a historical state motion adapter, maintaining a historical state buffer to store a preset number of frames of local motion state vectors during frame-by-frame generation. Velocity and acceleration vectors are calculated based on the difference between local motion state vectors between adjacent frames. These vectors are input into a risk discrimination network for risk score calculation. When the risk score exceeds a preset threshold, the motion condition representation is corrected along the negative direction of the risk score gradient and then re-participates in the denoising calculation. This achieves relative motion risk prediction and proactive avoidance based on historical states, ensuring that the character's motion trajectory in the generated video conforms to scene constraints.This invention employs a high-precision depth-of-field optimized focus reconstruction mechanism. It inputs a depth prediction head into each frame to obtain a full-resolution depth distribution map. The difference between the depth value of each pixel in the depth distribution map and the relative depth value of the character in the corresponding frame is calculated to obtain the blur circle diameter. Based on the blur circle diameter, a blur kernel of adjacent size levels is selected from a preset Gaussian blur kernel library and linearly interpolated and weighted to obtain an adaptive blur kernel for each pixel position. After performing adaptive blur kernel convolution processing on the neighborhood features of each pixel position, each frame is reconstructed, ensuring the focus plane remains continuously locked at the custom depth position of the character. The blurring degree of the background and foreground areas dynamically changes frame by frame according to their relative relationship with the actual depth position of the character. This solves the technical problem of a mismatch between the focus plane and the actual depth position of the character caused by depth estimation and blurring processing being independent of the subject's identity positioning. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of a custom character video generation system based on multi-reference image feature fusion in one embodiment of the present invention; Figure 2 This is a flowchart illustrating a method for generating custom character videos by fusing multiple reference image features according to an embodiment of the present invention. Figure 3 yes Figure 2 A schematic diagram of the implementation process of step S101; Figure 4 yes Figure 2 A schematic diagram of the implementation process of step S102; Figure 5 This is a schematic diagram of a structure for motion coding and decoupling optimization in one embodiment of the present invention; Figure 6 yes Figure 2 A schematic diagram of the implementation process of step S103; Figure 7 This is a schematic diagram of a motion adapter in one embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of a risk discrimination network in one embodiment of the present invention; Figure 9 yes Figure 2 A schematic diagram of the implementation process of step S104; Figure 10This is a schematic diagram of a custom character video generation device that integrates multiple reference image features according to an embodiment of the present invention; Figure 11 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. It should also be understood that, as used in this specification and the appended claims, the term "and / or" refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0026] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0027] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0028] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0029] To address the problems mentioned above in the background technology, this application proposes a method and related equipment for generating custom character videos using multi-reference map feature fusion. The method for generating custom character videos using multi-reference map feature fusion provided by this invention can be applied to applications such as... Figure 1 The custom character video generation system based on multi-reference map feature fusion shown includes: an image acquisition terminal 1 and an image processing device 2.

[0030] In one embodiment, such as Figure 2 As shown, a method for generating custom character videos by fusing features from multiple reference maps is provided, and this method is applied to... Figure 1 Taking a custom character video generation system based on multi-reference map feature fusion as an example, the following steps are included: S101: Acquire multiple reference images from different perspectives, and perform identity recognition on the target characters in the reference images to obtain global identity recognition data of the target characters; In this embodiment, through a multi-view, multi-reference image identity feature extraction and global identity recognition data construction mechanism, local identity features are extracted from multiple reference images from different perspectives. Then, the correlation between the features of each view is weighted by the self-attention mechanism in the cross-view fusion network. Common identity feature components such as facial structure proportions, relative positions of facial features, hairstyle outlines, and clothing patterns are retained, while non-common components caused by differences in posture and lighting are suppressed. This makes the global identity recognition data independent of the character's posture state, achieving the separation of identity representation and posture representation from the source of feature extraction. This solves the technical problem of identity drift when the character's posture changes in the generated video due to a single reference image carrying specific perspective, posture, and lighting deviation information.

[0031] S102: Obtain the driving sequence used to drive the target character's motion, encode the driving sequence in time order to obtain the encoded token sequence, and perform motion embedding and decoupling optimization on the encoded token sequence to obtain motion information data and the corresponding timestamp; In this embodiment, an absolute time-sequence joint encoding mechanism is used to concatenate the physical timestamp sine code and the sequence index sine code into a joint position code, which is then superimposed on the tokens of each frame in the driving sequence, so that the motion representation carries both physical time and sequence order information. A Fourier-convolutional collaborative token compression mechanism is used to perform a one-dimensional real Fourier transform on the encoded token sequence while retaining low-frequency components. The real and imaginary parts of the low-frequency domain features are then subjected to depthwise separable convolutions and concatenated into real-valued frequency domain features. A fixed-length global motion condition embedding is obtained by cross-attention resampling with the real-valued frequency domain features using a preset query matrix. This solves the technical problem that the computational complexity of the full self-attention mechanism in long video sequence generation increases quadratically with the number of frames. Finally, a dedicated time identifier token is resampled and then fed into a timestamp prediction head to directly regress a continuous normalized timestamp, avoiding the reuse of the discrete classification prediction method for text tokens and resolving the time boundary prediction bias problem. This invention calculates cosine similarity between motion embedding summary vectors and global identity recognition data of target characters, and constructs an orthogonal constraint loss function. Orthogonal constraints are introduced during video generation training to reduce the directional correlation between motion embedding summary vectors and identity representations, helping to alleviate the coupling between motion features and identity features, and preserving action information in the driving sequence through the video generation task. Multiple reference images in the training samples correspond to the same character as the driving sequence, ensuring that identity constraints target the character identity information contained in the driving sequence. After training, the motion coding network is used for action transfer of different target characters.

[0032] S103: Input global identity recognition data, motion information data and timestamps into the motion adapter for risk avoidance and noise reduction to obtain the target video frame sequence and focus depth data; In this embodiment, a historical state motion adapter is used to maintain a historical state buffer to store local motion state vectors of a preset number of frames during the frame-by-frame generation process. Velocity vectors and acceleration vectors are calculated based on the difference between local motion state vectors between adjacent frames. The velocity vectors and acceleration vectors are then input into a risk discrimination network to calculate risk scores. When the risk score exceeds a preset threshold, the motion condition representation is corrected along the negative direction of the risk score gradient and then re-participates in the denoising calculation. This achieves relative motion risk prediction and active avoidance based on historical states, ensuring that the character's motion trajectory in the generated video conforms to scene constraints.

[0033] S104: Reconstruct the focus of the target video sequence based on the focus depth data, and output a custom character video.

[0034] In this embodiment, a high-precision depth-of-field optimized focus reconstruction mechanism is used to input a depth prediction head into each frame to obtain a full-resolution depth distribution map. The difference between the depth value of each pixel position in the depth distribution map and the relative depth value of the character in the corresponding frame is calculated to obtain the blur circle diameter value. Based on the blur circle diameter value, a blur kernel of adjacent size level is selected from a preset Gaussian blur kernel library and linear interpolation weighting is performed to obtain an adaptive blur kernel for each pixel position. After performing adaptive blur kernel convolution processing on the neighborhood features of each pixel position, each frame is reconstructed, so that the focus plane is continuously locked at the depth position of the custom character. The blur degree of the background and foreground areas changes dynamically frame by frame according to their relative relationship with the actual depth position of the character. This solves the technical problem that the depth estimation and blur processing are independent of the subject identity positioning, resulting in a mismatch between the focus plane and the actual depth position of the character.

[0035] In one embodiment, such as Figure 3 As shown, step S101 specifically includes the following steps: S201: Collect multiple reference images from different perspectives, and extract local identity features of the target characters in the reference images to obtain the first identity recognition data corresponding to each perspective; S202: Combine the preset identity anchor vector with the first identity recognition data corresponding to each viewpoint to form an input sequence, perform multi-head self-attention calculation on the input sequence, and obtain the correlation weight between each first identity recognition data. S203: Based on the relevance weight, the first identity recognition data corresponding to each viewpoint is weighted and fused, retaining the common identity feature components of each viewpoint and suppressing the non-common feature components caused by pose differences and lighting differences, to obtain the second identity recognition data corresponding to each viewpoint. Then, the second identity recognition data corresponding to each viewpoint is fused from multiple perspectives to obtain the global identity recognition data of the target character.

[0036] In this embodiment, a unified multi-view reference image set is established for the target character. The number of reference images is set to N≥2, covering the front, left, and right sides of the target character, as well as perspectives with significant pose differences. This ensures that different images share stable identity information such as facial structure proportions, relative positions of facial features, hairstyle outlines, and clothing styles, while allowing for certain pose and lighting differences among the reference images. For each reference image, it is uniformly adjusted to a preset input resolution H×W and then input into a visual Transformer identity feature extraction subnetwork with shared parameters. The kernel size and stride of the convolutional block embedding layer are both set to 16, and the number of output channels is set to 768. This divides each reference image into multiple image blocks, and a learnable CLS Token is added before the image block sequence, along with a learnable absolute position encoding. The processed image patch sequence is input into an encoder consisting of 12 Transformer coding layers, each with 12 self-attention heads, each with a dimension of 64. The feedforward network has an intermediate dimension of 3072. GELU activation, LayerNorm normalization, and residual connections are used to enable the model to extract stable identity features from local textures, facial contours, clothing structure, and overall appearance relationships. Each reference image is computed using the same set of network parameters, and the 768-dimensional feature vector corresponding to the CLS Token in the final output sequence of each reference image is taken as the first identity recognition data for the current viewpoint. This yields first identity recognition data corresponding one-to-one with N reference viewpoints. A learnable identity anchor vector is pre-set, and the identity anchor vector and all first identity recognition data are combined in a uniform order to form an input sequence of length N+1. This input sequence is then fed into a cross-view fusion network. The cross-view fusion network can use a 4-layer Transformer encoder with a hidden feature dimension of 768, each layer having 8 multi-head self-attention heads, and the feedforward network having an intermediate dimension of 3072. During multi-head self-attention computation, a global association is established between the identity anchor vector and the first identity recognition data from different viewpoints. Simultaneously, pairwise feature associations are also performed between the first identity recognition data from different viewpoints. This allows the network to determine which features consistently appear across multiple viewpoints and which features are generated solely by a single viewpoint, head rotation, partial occlusion, or lighting changes. The attention responses obtained from different attention heads can measure the correlation between multiple viewpoints from different feature subspaces. Therefore, the corresponding attention responses can be used as correlation weights between the first identity recognition data. Furthermore, during the multi-layer self-attention iteration process, the response strength of shared identity features across viewpoints is gradually increased, while the response strength of viewpoint-related non-shared features is decreased.The first identity recognition data corresponding to each viewpoint is weighted and fused based on relevance weights. For example, the relative positions of facial features, overall facial proportions, main outline of hairstyle, and basic structure of clothing of the same character that can appear stably in frontal, left, and right images will obtain a high consistency response across multiple viewpoints, thus retaining a large feature contribution during fusion. However, unilateral contour changes caused by side angles, local shadows, exposure differences, and local deformations caused by specific poses are difficult to obtain stable correspondences in other viewpoints, and their relevance weights are relatively low, thus being suppressed during fusion. After relevance weighting, second identity recognition data corresponding to each viewpoint can be formed, which reduces the interference of viewpoint and lighting factors compared to the first identity recognition data, while still retaining the effective identity information provided by the current viewpoint. The second identity recognition data corresponding to each viewpoint is input into the same cross-view fusion network for further fusion, and the final hidden state after multi-layer self-attention calculation at the location of the preset identity anchor vector is read as the global identity recognition data of the target character, with its feature dimension maintained at 768. The resulting global identity recognition data no longer relies primarily on a single reference image, but is formed by multiple perspectives. Therefore, it can more stably represent the identity attributes of the target character and reduce the possibility of identity drift when the character turns around, looks up, looks down, or when lighting conditions change during video generation.

[0037] In one embodiment, such as Figure 4 As shown, step S102 specifically includes the following steps: S301: Obtain the driving sequence used to drive the target character's movement, parse the driving sequence into a visual token sequence arranged in frames, and superimpose physical timestamp sine coding and sequence index sine coding on the visual token sequence to obtain the encoded token sequence. S302: Perform Fourier transform on the encoded token sequence along the time dimension and extract real-valued frequency domain features. Use the preset query matrix as the query vector and the real-valued frequency domain features as the key vector and value vector to perform cross-attention calculation to obtain the motion embedding summary vector. S303: Construct an orthogonal constraint loss function between the motion embedding summary vector and the global identity recognition data. Decouple and optimize by minimizing the orthogonal constraint loss function to obtain motion information data, and obtain the timestamp from the dedicated time identifier Token.

[0038] In this embodiment, a driving sequence capable of representing the target's motion process is obtained. This driving sequence can be a user-specified motion trajectory sequence or a driving video sequence containing continuous human movements. The driving sequence is then parsed into a frame-by-frame visual token sequence of length T according to its temporal order. For each visual token at any given moment, its feature dimension is uniformly set to 768 dimensions using linear projection, ensuring that the motion representations of different frames reside in the same feature space. To simultaneously preserve both "real-time position" and "frame's position in the sequence," a physical timestamp sine code and a sequence index sine code are constructed for each visual token. The physical timestamp sine code occupies 384 dimensions. peacekeeping According to the respective and calculate, The value range is from 0 to 191. The physical timestamp of the current frame is represented by the sequence index sine code. The sequence index sine code uses the same encoding structure, but replaces the physical timestamp with the current frame's position within the entire driving sequence, also forming a 384-dimensional code. The 384-dimensional physical timestamp code and the 384-dimensional sequence index code are concatenated to form a 768-dimensional joint position code, which is then superimposed on the corresponding 768-dimensional visual token. This yields an encoded token sequence that includes both motion visual features and information on the real time scale and frame order. A one-dimensional real Fourier transform is performed channel-by-channel along the time dimension of the encoded token sequence, converting the continuous motion changes in the time domain into a frequency domain representation, while retaining the low-frequency components that reflect the overall motion trend of the character. For an encoded token sequence of length T, the K lowest frequency components are retained after the Fourier transform, where K is the integer obtained by rounding up the ratio of T to 4, thus compressing the number of tokens while preserving the main motion change trend. One-dimensional depthwise separable convolutions are performed on the real and imaginary parts of the low-frequency domain features, with a kernel size of 7, a group number of 768, and boundary padding of 3 to maintain consistent frequency lengths before and after processing and to fuse local relationships between adjacent frequency components. The processed real and imaginary parts are concatenated along the channel dimension and restored to 768-dimensional real-valued frequency domain features through linear projection. A query matrix consisting of 256 learnable query vectors is pre-defined. Using this pre-defined query matrix as the query vector and the aforementioned real-valued frequency domain features as both key and value vectors, a single-layer 8-head cross-attention computation is performed. This allows the 256 query vectors to actively extract representative global motion information from the compressed frequency domain motion features, thus forming a fixed-length global motion conditional embedding. Mean pooling is performed on the global motion conditional embedding along the token dimension to obtain a 768-dimensional motion embedding summary vector, which is used to summarize the main motion patterns in the entire driving sequence. The motion embedding summary vector and global identity recognition data are used to establish decoupling constraints between motion features and identity features. The correlation between the motion embedding summary vector and the global identity recognition data in the feature space is calculated, and an orthogonal constraint loss function is constructed accordingly. During training, the orthogonal constraint loss is continuously minimized through backpropagation, gradually weakening the feature components related to the target role's identity in the motion conditions, while retaining the feature components related to posture changes, limb movements, movement direction, and movement rhythm. When the orthogonal constraint loss tends to converge, the gradient-updated global motion condition embedding is determined as the motion information data, thereby enabling motion information to be transferred between different target roles while minimizing the carrying of the original driving role's identity information.Simultaneously, a dedicated time stamp token is set in the encoded token sequence, and this token undergoes depthwise separable convolution and cross-attention resampling to obtain time stamp resampled features. These resampled features are then input into the timestamp prediction head for continuous value regression. The timestamp prediction head employs a two-layer fully connected structure with dimensions set sequentially from 768 to 256 to 1. The intermediate layer uses the GELU activation function, and the output layer uses a linear output, thus obtaining a continuous normalized timestamp.

[0039] like Figure 5 As shown, the driving sequence is parsed to obtain a visual token sequence arranged in frames. After superimposing the physical timestamp sine coding and the sequence index sine coding, an encoded token sequence is formed. The global motion conditional embedding is obtained through Fourier-convolutional collaborative token compression and cross-attention resampling. The global motion conditional embedding is then averaged and pooled to obtain a motion embedding summary vector. At the same time, the orthogonal decoupling constraint between the motion embedding summary vector and the global identity recognition data is used to reduce the identity information in the motion features. The timestamp is obtained from the dedicated time identifier token through depthwise separable convolution, cross-attention resampling and timestamp prediction head.

[0040] In one embodiment, step S302 specifically includes the following steps: Along the time dimension, a one-dimensional real Fourier transform is performed on the encoded token sequence to obtain the original frequency domain features. The component with the lowest frequency index in the original frequency domain features is retained to obtain the low-frequency frequency domain features. Depth-separable convolution operations are performed on the first real part feature and the first imaginary part feature of the low-frequency domain feature to obtain the second real part feature and the second imaginary part feature. The second real part feature and the second imaginary part feature are then concatenated and linearly projected in the channel dimension to obtain the real-valued frequency domain feature. Using a preset query matrix as the query vector and real-valued frequency domain features as the key and value vectors, cross-attention calculation is performed on the query vector, key vector, and value vector to obtain the global motion condition embedding. Then, mean pooling is performed on the global motion condition embedding at the token dimension to obtain the motion embedding summary vector.

[0041] In this embodiment, when processing the encoded token sequence along the time dimension, an encoded token sequence of length T and feature dimension 768 is taken as input, and each of the 768 feature channels is treated as an independent processing object. A one-dimensional real Fourier transform is performed on each channel along the time dimension, converting the time-domain motion information originally arranged in frame order into a frequency-domain representation. Since the input is a real-valued sequence, the original frequency-domain features obtained after the transformation are in complex form, and their frequency dimension length is T. Extracting frequencies from lowest to highest, retaining only the first K frequency components with the lowest frequency indices, where K is... The low-frequency domain features are obtained by discarding the remaining higher-frequency components. The low-frequency portion is mainly used to express the relatively long-lasting and stable main motion trends in the entire driving sequence, such as overall body translation, turning, and continuous limb swinging. After truncation, the number of frequency domain tokens is also reduced accordingly, thus reducing the data scale of cross-attention processing while retaining the main motion change information. After obtaining the low-frequency domain features, since the frequency domain features are still composed of real and imaginary parts, the first real and first imaginary features corresponding to the low-frequency domain features are extracted separately, and one-dimensional depthwise separable convolutions are performed independently on both. The convolution kernel size is set to 7, the number of groups is set to 768, and the boundary padding is set to 3, so that the frequency dimension length after the convolution operation remains unchanged, while allowing each feature channel to focus on fusing local frequency domain information within the vicinity of the current frequency component, resulting in the second real and second imaginary features. In the channel dimension, the second real part feature and the second imaginary part feature are concatenated, so that each frequency position retains both the real and imaginary information after local fusion. The concatenated channel dimension is expanded from 768 to 1536. The 1536-dimensional features are remapped to 768-dimensional through a linear projection layer, resulting in real-valued frequency domain features that can be directly input into the attention network. In this embodiment, a query matrix consisting of 256 learnable query vectors is pre-set, with each query vector having a dimension of 768. The query matrix is ​​used as the Query, and the real-valued frequency domain features are used as both the Key and Value, performing a single-layer 8-head cross-attention calculation. During the calculation, instead of full self-attention among all low-frequency tokens, the 256 learnable query vectors actively retrieve and aggregate different types of motion information from the low-frequency frequency domain features. This allows different query vectors to focus on different frequency patterns, trends, and stages of motion within the entire motion, ultimately resulting in a global motion conditional embedding with a fixed length of 256 and a single token dimension of 768. Since the number of query tokens is fixed at 256, the output motion conditions maintain a uniform size even as the length T of the driving sequence increases. This is beneficial for using the same subsequent network structure for driving sequences of different lengths. Mean pooling is applied to the global motion condition embedding across the 256 token dimensions. The motion information obtained from aggregating all query tokens is then uniformly summarized into a 768-dimensional vector, which is used as the motion embedding summary vector. This provides a more compact feature representation of the overall motion pattern of the entire driving sequence and is used for decoupling motion features from target character identity features.

[0042] In one embodiment, step S303 specifically includes the following steps: Construct an orthogonal constraint loss function based on motion embedding summary vectors and global identity recognition data; The parameters in the global motion condition embedding are updated by minimizing the orthogonal constraint loss function through backpropagation and then by gradient update. Once the orthogonal constraint loss function converges, the updated global motion condition embedding is used as motion information data. The dedicated time identifier tokens in the encoded token sequence are subjected to depthwise separable convolution and cross-attention resampling to obtain time identifier resampled features. Regression calculation is then performed on the time identifier resampled features to obtain the timestamp.

[0043] In this embodiment, the global motion conditional embedding is averaged and pooled across 256 token dimensions to form a 768-dimensional motion embedding summary vector. This motion embedding summary vector and the 768-dimensional global identity recognition data corresponding to the target role are then placed in the same feature space for similarity calculation, thereby constructing an orthogonal constraint between motion features and identity features. The orthogonal constraint loss function can be expressed as: ,in, This represents the orthogonal constraint loss. Represents a motion embedding summary vector. This represents global identity recognition data. Since the closer the absolute value of the cosine similarity between two vectors is to 0, the more orthogonal the directions of the two features in the feature space tend to be in the same direction, the optimization objective during training is to reduce the orthogonal constraint loss. This minimizes the representation of identity information such as the target character's facial structure, relative positions of facial features, hairstyle outline, and clothing style in the global motion conditional embedding, while primarily retaining information related to the driving behavior, such as action posture, movement direction, movement rhythm, and continuous movement changes. The gradient corresponding to the orthogonal constraint loss is calculated through backpropagation, and the parameters in the global motion conditional embedding are continuously updated based on the gradient. In each training round, the motion embedding summary vector is re-obtained, and the orthogonal constraint loss is calculated, gradually decoupling motion features from identity features. After the video generation task and orthogonal constraints are trained together, the global motion conditional embedding output by the trained motion coding network is used as motion information data to drive the current target character. Orthogonal constraints are used to help reduce the directional correlation between motion representation and identity representation, and are not used alone as a criterion for determining whether the original driving character's identity information has been removed. In this embodiment, to obtain timestamps that describe the continuous temporal positions of each frame, a dedicated time identifier token is bound to each frame in the encoded token sequence, and these tokens are grouped into independent temporal processing branches. Depthwise separable convolution and cross-attention resampling maintain the one-to-one correspondence between the time identifier tokens and the original frame indices. Then, frame-by-frame regression is performed using a shared timestamp prediction head to obtain a continuous normalized timestamp sequence corresponding to each generated frame. The real timestamps used for training employ the same normalization rule as the predicted timestamps. Depthwise separable convolution is applied to the dedicated time identifier tokens to extract local features related to temporal changes. Then, a cross-attention mechanism is used to resample the processed time identifier tokens, enabling them to aggregate features related to the temporal position of the entire driving sequence from the motion information of the encoded sequence, thus obtaining the timestamp resampled features. The timestamp resampled features are input into the dedicated timestamp prediction head for continuous value regression. The timestamp prediction head uses a two-layer fully connected network with network dimensions set sequentially from 768 to 256 to 1. The intermediate layers use the GELU activation function, and the output layer uses linear output, resulting in continuous normalized timestamps corresponding to the driving sequence. During the training phase, timestamp prediction uses Smooth-L1 loss, i.e.: ,in, Indicates timestamp regression loss, This represents the continuously normalized timestamps obtained from the prediction. This represents the corresponding real timestamp. By predicting time information through continuous value regression instead of discrete classification, the time token can form a continuous time representation corresponding to the driving sequence motion process, and provide a time basis for the temporal organization of each frame in the generated video.

[0044] In one embodiment, such as Figure 6 As shown, step S103 specifically includes the following steps: S401: Input the global identity recognition data and motion information data into the video denoising generation network in the motion adapter respectively. Then, through the identity cross-attention sublayer and motion cross-attention sublayer of each denoising module in the video denoising generation network, parameter injection and video frame generation are performed to obtain the first video frame sequence. S402: During the generation of each frame of the first video frame sequence, the velocity vector and acceleration vector are calculated and input into the risk discrimination network for correction, and the second video frame sequence is generated. The frames in the second video frame sequence are arranged according to the timestamp to generate the target video frame sequence. S403: Map the local motion state vector of each frame in the target video frame sequence to the character reference depth reference to obtain the character relative depth value of each frame, and combine the character relative depth values ​​of each frame to form focus depth data.

[0045] In this embodiment, global identity recognition data is used as the character identity constraint, and motion information data is used as the character action driving condition. Both are input into the video denoising generation network in the motion adapter. The video denoising generation network can be constructed based on a pre-trained Transformer structure backbone denoising network. The backbone network consists of 28 structurally identical denoising modules connected in series, with the hidden feature dimension set to 1152. Each denoising module contains, in sequence, a self-attention sublayer, an identity cross-attention sublayer, a motion cross-attention sublayer, and a feedforward network sublayer. In implementation, on the one hand, in the identity cross-attention sublayer, video latent variable features are used as query vectors, and global identity recognition data are used as key and value vectors for cross-attention calculation. This ensures that identity features such as facial structure proportions, relative positions of facial features, hairstyle outlines, and clothing styles are continuously injected throughout the multi-step denoising process, thus guaranteeing the stability of the character's identity during generation. On the other hand, in the motion cross-attention sublayer, video latent variable features are used as query vectors, and motion information data are used as key and value vectors for action condition injection. Parameter injection is completed through a low-rank bypass in the motion adapter. The low-rank value can be set to 32, and the scaling factor can be set to 64. During training, the original projection parameters of the backbone network are frozen, and only the low-rank branch parameters are updated. This ensures that the target action features are stably introduced into each layer of the denoising process while preserving the pre-trained video generation capability as much as possible. As each denoising module iterates layer by layer, the identity information and action information in the latent variable features are synchronously fused, outputting the first video frame sequence generated according to the target character's identity and customized motion trajectory.

[0046] A historical state motion adaptation mechanism is introduced synchronously during the frame-by-frame generation of the first video sequence to promptly identify potential risks of boundary crossings, collisions, or occlusions. During the generation of frame t, global average pooling is performed on the latent variable features corresponding to that frame to extract the local motion state vector. A historical state buffer of length 8 is maintained to store the local motion states of several frames prior to the current frame. A velocity vector is calculated based on the difference between the local motion state vectors of adjacent frames, and an acceleration vector is calculated based on the difference between adjacent velocity vectors. The velocity and acceleration vectors of the most recent moment are input together into a trend prediction network to obtain the predicted motion state increment. This increment is then added to the local motion state vector of the current frame to obtain the predicted local motion state for the next frame. Simultaneously, constraint information such as the motion space boundaries and areas occupied by other objects in the scene is encoded into scene constraint vectors, concatenated with the prediction results, and input into a risk discrimination network. The risk discrimination network outputs the risk score corresponding to the current predicted motion trend. When the risk score exceeds a preset threshold of 0.5, the current action trend is considered to have a high risk. At this point, the gradient of the risk score on the predicted motion state is backpropagated to the current motion condition representation. Correction is performed along the negative direction of the gradient of the risk score on the motion condition representation, and the risk score is recalculated. The corrected result, which reduces risk, is then used in the denoising calculation of the current frame, resulting in a second video frame sequence adjusted for risk avoidance. After all frames are generated, the frames in the second video frame sequence are sequentially arranged according to the timestamps obtained from the regression of the preceding dedicated time identifier Token, ensuring that the order of the frames on the timeline matches the actual rhythm of the driving sequence, thus forming the target video frame sequence. In this embodiment, a depth representation is established for each frame's main character. The relative depth distribution map of each reference image is obtained in advance using a depth prediction head. Under a unified depth direction, scale, and zero point, the depth results of different reference images are aligned, and the median depth within the target character region is extracted. The median depth of each reference image is then averaged to obtain the character's reference depth baseline. The local motion state vector corresponding to each frame in the target video sequence is mapped to a character reference depth reference. The character reference depth reference represents the reference depth position of the target character, while the local motion state vector reflects the character's posture and spatial motion state in the current frame. In implementation, both can be input into a two-layer fully connected mapping network with an input dimension of 768 + 1 and an intermediate layer dimension of 256. The output layer obtains the relative depth value of the character in the current frame. When the character moves forward or backward, turns, or changes posture in the video, the relative depth value of the current frame can be dynamically updated instead of remaining fixed. After repeating the above mapping process for all frames, a sequence of character relative depth values ​​corresponding one-to-one with the target video sequence is obtained. This set of frame-by-frame depth values ​​is then used to form the focus depth data.

[0047] like Figure 7 As shown, the motion adapter receives global identity recognition data and motion information data respectively, and inputs them into the video denoising generation network. The identity features of the target character are continuously injected through the identity cross-attention sublayer, and the motion conditions of the character are injected through the motion cross-attention sublayer. At the same time, the risk discrimination network is combined to judge the motion state during the generation process and correct the motion conditions according to the risk judgment results, thereby outputting a video frame sequence that has undergone risk avoidance processing and the relative depth value of the character corresponding to each frame.

[0048] In one embodiment, step S402 specifically includes the following steps: During the generation of each frame of the first video frame sequence, global average pooling is performed on the latent variable features of the current frame to obtain the local motion state vector of the current frame, and the local motion state vector of the current frame is stored in the historical state buffer. The velocity and acceleration vectors are calculated based on the local motion state vectors of adjacent frames in the historical state buffer, and then the velocity and acceleration vectors are input into the risk discrimination network to calculate the risk score. When the risk score exceeds a preset threshold, the gradient of the risk score relative to the predicted motion state is calculated, and correction is made along the negative direction of the gradient to generate a second video frame sequence. The frames in the second video frame sequence are then arranged according to the timestamps to generate the target video frame sequence.

[0049] In this embodiment, when generating each frame of the first video frame sequence, during the process of generating the t-th frame frame by frame by the backbone denoising network, the latent variable features corresponding to the t-th frame are obtained from the current denoising module. Global average pooling is performed on these features to obtain a 1152-dimensional feature vector, which is then mapped to a 768-dimensional local motion state vector through linear projection to characterize the overall motion state of the character in the current frame. Subsequently, a historical state buffer is maintained, with a preset buffer length of 8 frames. When a new local motion state vector is generated, it is written into the historical state buffer. If the number of states in the buffer has reached 8, the local motion states corresponding to the most recent 8 frames are retained in chronological order, ensuring that risk assessment always revolves around the continuous motion changes of the character over a recent period, rather than solely based on a single frame state. This embodiment calculates the velocity vector based on the local motion state vectors of adjacent frames, that is, using the difference between local motion states at adjacent moments to represent the trend of motion state change, and calculating the acceleration vector based on the difference between adjacent velocity vectors, simultaneously obtaining first-order and second-order information about the character's motion changes. The velocity and acceleration vectors of the most recent moment are input into a two-layer fully connected motion trend prediction network. The input consists of two 768-dimensional vectors, with the intermediate layer having a dimension of 512 and the output having a dimension of 768. The GELU activation function is used to obtain the predicted motion state increment of the next frame relative to the current frame. The predicted motion state increment is added to the local motion state vector of the current frame to obtain the predicted motion state of the next frame. The local motion state vector contains information related to the character's current position, and its correspondence with the spatial reference used for scene constraints is established through training. At the same time, scene constraint information such as the spatial boundary coordinates of the character's allowed activities and the occupied areas of other objects in the scene are input into a two-layer fully connected scene constraint encoding sub-network. The encoding results are uniformly mapped to a 768-dimensional scene constraint vector. The predicted motion state and the scene constraint vector are then concatenated and input into a risk discrimination network. The risk discrimination network adopts a two-layer fully connected structure of 1536→256→1, and the output layer uses the Sigmoid activation function to obtain a risk score r, where r∈[0,1], which is used to characterize the risk level of boundary crossing, collision, or occlusion conflict in the predicted motion state of the next frame. In this embodiment, when the risk score exceeds a preset threshold of 0.5, it is considered that continuing to generate the current frame according to the original motion conditions carries a high motion risk. At this point, the gradient of the risk score relative to the predicted motion state is calculated, and this gradient is backpropagated along the differentiable computational path between the predicted motion state, the local motion state, and the motion condition representation to obtain the gradient of the risk score relative to the current motion condition representation. Then, the motion condition representation of the input motion cross-attention sublayer is corrected along its negative direction. After correction, the risk score is recalculated to confirm whether the correction has reduced the risk.When the recalculated risk score is lower than the original risk score, the corrected motion condition representation is used in the remaining denoising calculation of the current frame; otherwise, the correction result is not used, and the original motion condition representation is retained. If the original risk score does not exceed 0.5, the current motion conditions remain unchanged, and the generation of the corresponding frame continues, thus obtaining the second video frame sequence. After each frame has completed risk assessment and necessary correction, the continuous time information obtained from the previous timestamp regression is used to sort the frames in the second video frame sequence, so that the order of the generated frames is consistent with the actual time relationship of the driving sequence, resulting in the target video frame sequence after motion risk assessment and necessary correction.

[0050] like Figure 8 As shown, let v be the local motion state vector of the (i-1)th frame in the history state buffer. i-1 Let v be the local motion state vector of the current frame, i.e., the i-th frame. i Then according to v i With v i-1 The difference between the velocity vectors of two adjacent frames yields the velocity vector, and the difference between the velocity vectors of two adjacent frames yields the acceleration vector. The velocity and acceleration vectors are processed by a fully connected layer and GELU activation to form the predicted motion state increment, which is then added to the local motion state vector of the current frame to obtain the predicted motion state for the next frame. The scene constraint vector represents the boundaries of the character's movable space and the occupancy of other objects in the scene. It is processed by a fully connected layer and GELU activation to form a 768-dimensional scene constraint feature. After passing through two fully connected layers and an output sigmoid activation layer, a risk score ranging from 0 to 1 is obtained to represent the risk of boundary crossing, collision, or occlusion conflicts in the predicted motion state of the next frame.

[0051] In one embodiment, such as Figure 9 As shown, step S104 specifically includes the following steps: S501: Calculate the relative depth distribution map of each frame in the target video frame sequence, and upsample the relative depth distribution map to the same spatial resolution as the corresponding frame to obtain the full resolution depth distribution map of each frame; S502: Calculate the depth difference between the target depth value and the corresponding character relative depth value at each pixel position in the full-resolution depth distribution map of each frame, and calculate the blur circle diameter value at each pixel position based on the depth difference. S503: Select blur kernels of adjacent size levels from the preset Gaussian blur kernel library and perform linear interpolation weighting between the two size levels according to the blur circle diameter value to obtain the adaptive blur kernel for each pixel position. Then, use the adaptive blur kernel to reconstruct the focus of each pixel position in the target video frame sequence and output the custom character video.

[0052] In this embodiment, a pixel-level relative depth representation is established for each frame in the target video sequence. A feature map corresponding to the current frame is extracted from the latent variable features output by the last denoising module of the video denoising generation network. This feature map is then input into the depth prediction head, which employs a two-layer convolutional structure with a kernel size of 3×3. The number of channels is mapped sequentially from the backbone hidden dimension to 256, then back to 1, thus outputting a single-channel relative depth distribution map for the current frame. The depth prediction head maintains the same network structure as the depth prediction head used to determine the character reference depth benchmark and shares some weight initialization. Simultaneously, a unified-scale depth supervision is applied to pixel-level depth prediction and character relative depth mapping, ensuring that pixel depth, character relative depth, and character reference depth benchmark have consistent depth direction, scale, and zero point, enabling direct comparison. The character relative depth used for blur circle calculation is restricted to positive values, with the minimum effective value set to 0.001 under the unified depth scale. Since the relative depth distribution map obtained from convolutional prediction is still at a low resolution, it is upsampled to a spatial resolution completely consistent with the current video frame using bilinear interpolation, thus obtaining the full-resolution depth distribution map of the current frame. This process is repeated for all frames to obtain a set of full-resolution depth distribution maps corresponding to each frame of the target video sequence, ensuring that the foreground, main subject, and background regions in each frame have continuous relative depth representation. In this embodiment, the depth difference relationship required for depth modulation is calculated pixel-by-pixel within each frame. Let the pixel position in the full-resolution depth distribution map of the current frame be... The target depth value is The relative depth of the main character in this frame is [value]. Then, calculate the depth difference between the two values, and determine the diameter of the blur circle at the current pixel position based on this difference. The calculation relationship can be written as: ,in, This indicates the value of the blur circle diameter at the current pixel position. This represents a preset scaling factor that is positively correlated with the pre-set virtual camera aperture parameters. In implementation, the preset scaling factor is set to 1.2 pixels at the current output resolution, so that the calculated blur circle diameter is in pixels, used to adjust the approximate depth-of-field blur level between the foreground and background. When the depth value of a pixel is exactly the same as the depth value of the main character, the depth difference is 0, and the blur circle diameter is also 0, indicating that the pixel should be in the sharp focus area. However, as the pixel depth gradually deviates from the depth of the main character, the absolute value of the depth difference increases, and the corresponding blur circle diameter also increases. Therefore, a stronger blur effect should be applied to this pixel during reconstruction. This embodiment pre-establishes a hierarchical Gaussian blur kernel library and adaptively selects blur kernels according to the blur circle diameter for focus reconstruction. The number of size levels in the Gaussian blur kernel library can be set to 5, i.e., S=5, corresponding to kernel sizes of 1×1, 3×3, 5×5, 7×7, and 9×9, respectively. Simultaneously, the standard deviation of each kernel level increases sequentially to ensure that the smoothing effect within the pixel neighborhood gradually strengthens as the blur level increases. For any pixel in the current frame, when the blur circle diameter is no greater than 1 pixel, a 1×1 identity kernel is selected; when the blur circle diameter is no less than 9 pixels, a 9×9 Gaussian blur kernel is selected. When the blur circle diameter is between two adjacent size levels, the centers of the two blur kernels are aligned, and the smaller blur kernel is padded with zeros to the larger size. Then, linear interpolation weighting is performed according to the position of the blur circle diameter between the two size levels, and the weights of other levels are reset to zero to obtain the adaptive blur kernel for the current pixel position. This adaptive blur kernel is used to perform a weighted convolution operation on the image features in the neighborhood of the current pixel, and the reconstructed feature value of the pixel position is output. Since the blur circle diameter is very small near the focus depth of the main character, the corresponding adaptive blur kernel will approach the 1×1 identity kernel, so this part of the region can maintain the original clear texture; while for the background or foreground regions that are significantly away from the depth position of the main character, the blur circle diameter is larger, and the corresponding blur kernel will be closer to the larger Gaussian kernel, so these regions will present a stronger blur effect. After reconstructing all pixels in each frame in the above manner, an output image with continuous frame-by-frame focus and natural depth-of-field changes is obtained. Finally, the images are combined according to the original time sequence of the video frames to output a custom character video that maintains the consistency of the character's identity and actions while having a high-precision depth-of-field focus effect.

[0053] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0054] In one embodiment, a custom character video generation apparatus based on multi-reference image feature fusion is provided, which corresponds one-to-one with the custom character video generation method based on multi-reference image feature fusion in the above embodiments. For example... Figure 10 As shown, the custom character video generation device based on multi-reference map feature fusion includes: The character identification unit is used to acquire multiple reference images from different perspectives and identify the target character in the reference images to obtain global character identification data. The motion coding unit is used to acquire the driving sequence used to drive the motion of the target character, encode the driving sequence in time order to obtain the encoded token sequence, and perform motion embedding and decoupling optimization on the encoded token sequence to obtain motion information data and the corresponding timestamp. The risk avoidance and noise reduction unit is used to input global identity recognition data, motion information data and timestamps into the motion adapter for risk avoidance and noise reduction, and obtain the target video frame sequence and focus depth data. The focus reconstruction unit is used to reconstruct the focus of the target video frame sequence based on the focus depth data and output a custom character video.

[0055] Optionally, the role identification unit is specifically used for: Multiple reference images from different perspectives were collected, and local identity features of the target characters in the reference images were extracted to obtain the first identity recognition data corresponding to each perspective. The preset identity anchor vector and the first identity recognition data corresponding to each viewpoint are combined to form an input sequence. Multi-head self-attention calculation is performed on the input sequence to obtain the correlation weight between each first identity recognition data. The first identity recognition data corresponding to each viewpoint is weighted and fused according to the relevance weight. The common identity feature components of each viewpoint are retained and the non-common feature components caused by pose difference and illumination difference are suppressed to obtain the second identity recognition data corresponding to each viewpoint. The second identity recognition data corresponding to each viewpoint is then fused from multiple perspectives to obtain the global identity recognition data of the target character.

[0056] Optionally, the motion coding unit is specifically used for: Obtain the driving sequence used to drive the target character's movement, parse the driving sequence into a visual token sequence arranged in frames, and superimpose physical timestamp sine coding and sequence index sine coding on the visual token sequence to obtain the encoded token sequence. The encoded token sequence is subjected to Fourier transform along the time dimension and real-valued frequency domain features are extracted. The preset query matrix is ​​used as the query vector, and the real-valued frequency domain features are used as the key vector and value vector to perform cross-attention calculation to obtain the motion embedding summary vector. An orthogonal constraint loss function is constructed between the motion embedding summary vector and the global identity recognition data. The motion information data is obtained by decoupling and optimization by minimizing the orthogonal constraint loss function, and the timestamp is obtained by regression from the dedicated time identifier Token.

[0057] Optionally, the risk avoidance denoising unit is specifically used for: Global identity recognition data and motion information data are respectively input into the video denoising generation network in the motion adapter. The first video frame sequence is obtained by injecting parameters and generating video frames through the identity cross-attention sublayer and motion cross-attention sublayer of each denoising module in the video denoising generation network. During the generation of each frame of the first video frame sequence, the velocity vector and acceleration vector are calculated and input into the risk discrimination network for correction, and the second video frame sequence is generated. The frames in the second video frame sequence are arranged according to the timestamp to generate the target video frame sequence. The local motion state vectors of each frame in the target video sequence are mapped to the character reference depth benchmark to obtain the character relative depth value of each frame, and the character relative depth values ​​of each frame are combined to form focus depth data.

[0058] Optionally, the focus reconstruction unit is specifically used for: Calculate the relative depth distribution map of each frame in the target video frame sequence, and upsample the relative depth distribution map to the same spatial resolution as the corresponding frame to obtain the full resolution depth distribution map of each frame; Calculate the depth difference between the target depth value and the corresponding character relative depth value at each pixel location in the full-resolution depth distribution map of each frame, and calculate the blur circle diameter value at each pixel location based on the depth difference. Select blur kernels of adjacent size levels from the preset Gaussian blur kernel library and perform linear interpolation weighting between the two size levels according to the blur circle diameter value to obtain the adaptive blur kernel for each pixel position. Then, use the adaptive blur kernel to reconstruct the focus of each pixel position in the target video frame sequence and output the custom character video.

[0059] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0060] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0061] This application also provides a computer device, such as... Figure 11 As shown, the computer device includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, it implements the steps in any of the above method embodiments, or when the processor executes the computer program, it implements the functions of each module / unit in the above device embodiments.

[0062] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0063] Those skilled in the art will understand that Figure 11 The computer device described is merely an example and does not constitute a limitation on the computer device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0064] The aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0065] The memory can be an internal storage unit of the computer device, such as a hard drive or RAM. The memory can also be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units of the computer device.

[0066] This application also provides a readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0067] This application provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.

[0068] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0069] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0070] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0071] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0072] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0073] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for generating custom character videos through multi-reference image feature fusion, characterized in that, include: Multiple reference images from different perspectives are acquired, and the target characters in the reference images are identified to obtain global identity recognition data of the target characters. Obtain the driving sequence used to drive the movement of the target character, encode the driving sequence in time order to obtain the encoded token sequence, and perform motion embedding and decoupling optimization on the encoded token sequence to obtain motion information data and the corresponding timestamp; The global identity recognition data, the motion information data, and the timestamp are input into the motion adapter for risk avoidance and noise reduction to obtain the target video frame sequence and focus depth data. Based on the focus depth data, the target video frame sequence is refocused and reconstructed to output a custom character video.

2. The method for generating custom character videos by multi-reference map feature fusion as described in claim 1, characterized in that, The process involves acquiring multiple reference images from different perspectives and performing identity recognition on the target characters within these reference images to obtain global identity recognition data for the target characters, including: Multiple reference images from different perspectives are acquired, and local identity features of the target characters in the reference images are extracted to obtain the first identity recognition data corresponding to each perspective. The preset identity anchoring vector and the first identity recognition data corresponding to each viewpoint are combined to form an input sequence. Multi-head self-attention calculation is performed on the input sequence to obtain the correlation weight between each first identity recognition data. The first identity recognition data corresponding to each viewpoint is weighted and fused according to the relevance weight, retaining the common identity feature components of each viewpoint and suppressing the non-common feature components caused by posture differences and lighting differences, to obtain the second identity recognition data corresponding to each viewpoint. Then, the second identity recognition data corresponding to each viewpoint is fused from multiple perspectives to obtain the global identity recognition data of the target character.

3. The method for generating custom character videos by multi-reference map feature fusion as described in claim 2, characterized in that, The process of obtaining a driving sequence for driving the target character's movement, encoding the driving sequence in a time sequence to obtain an encoded token sequence, and performing motion embedding and decoupling optimization on the encoded token sequence to obtain motion information data and corresponding timestamps includes: Obtain the driving sequence used to drive the movement of the target character, parse the driving sequence into a visual token sequence arranged in frames, and superimpose physical timestamp sine coding and sequence index sine coding on the visual token sequence to obtain the encoded token sequence. The encoded token sequence is subjected to Fourier transform along the time dimension and real-valued frequency domain features are extracted. The preset query matrix is ​​used as the query vector, and the real-valued frequency domain features are used as the key vector and value vector to perform cross-attention calculation to obtain the motion embedding summary vector. Construct an orthogonal constraint loss function between the motion embedding summary vector and the global identity recognition data. Decouple and optimize by minimizing the orthogonal constraint loss function to obtain motion information data, and regress the timestamp from the dedicated time identifier Token.

4. The method for generating custom character videos by multi-reference map feature fusion as described in claim 3, characterized in that, The process involves performing a Fourier transform on the encoded token sequence along the time dimension and extracting real-valued frequency domain features. A preset query matrix is ​​used as the query vector, and the real-valued frequency domain features are used as the key vector. Cross-attention calculation is then performed between the key vector and the value vector to obtain a motion embedding summary vector, including: Along the time dimension, a one-dimensional real Fourier transform is performed on the encoded token sequence to obtain the original frequency domain features. The component with the lowest frequency index in the original frequency domain features is retained to obtain the low-frequency frequency domain features. Depth-separable convolution operations are performed on the first real part feature and the first imaginary part feature of the low-frequency frequency domain feature to obtain the second real part feature and the second imaginary part feature. The second real part feature and the second imaginary part feature are then concatenated and linearly projected in the channel dimension to obtain the real-valued frequency domain feature. Using a preset query matrix as the query vector and the real-valued frequency domain features as the key vector and value vector, cross-attention calculation is performed on the query vector, the key vector, and the value vector to obtain the global motion condition embedding. Then, mean pooling is performed on the global motion condition embedding in the token dimension to obtain the motion embedding summary vector.

5. The method for generating custom character videos by multi-reference map feature fusion as described in claim 4, characterized in that, The process of constructing the orthogonal constraint loss function between the motion embedding summary vector and the global identity recognition data, and decoupling and optimizing by minimizing the orthogonal constraint loss function to obtain motion information data, and regressing the timestamp from the dedicated time identifier token, includes: Construct an orthogonal constraint loss function based on the motion embedding summary vector and the global identity recognition data; The parameters in the global motion condition embedding are updated by minimizing the orthogonal constraint loss function through backpropagation and then using the updated global motion condition embedding as motion information data after the orthogonal constraint loss function converges. The dedicated time identifier tokens in the encoded token sequence are subjected to depthwise separable convolution and cross-attention resampling to obtain time identifier resampling features. Regression calculation is then performed on the time identifier resampling features to obtain the timestamp.

6. The method for generating custom character videos by multi-reference map feature fusion as described in claim 4, characterized in that, The step of inputting the global identity recognition data, the motion information data, and the timestamp into the motion adapter for risk avoidance and noise reduction to obtain the target video frame sequence and focus depth data includes: The global identity recognition data and the motion information data are respectively input into the video denoising generation network in the motion adapter. The first video frame sequence is obtained by injecting parameters and generating video frames through the identity cross-attention sublayer and motion cross-attention sublayer of each denoising module in the video denoising generation network. During the generation of each frame of the first video frame sequence, the velocity vector and acceleration vector are calculated and input into the risk discrimination network for correction, and a second video frame sequence is generated. The frames in the second video frame sequence are arranged according to the timestamp to generate the target video frame sequence. The local motion state vectors of each frame in the target video sequence are mapped to the character reference depth reference to obtain the character relative depth value of each frame, and the character relative depth values ​​of each frame are combined to form focus depth data.

7. The method for generating custom character videos by multi-reference map feature fusion as described in claim 6, characterized in that, In the process of generating each frame of the first video frame sequence, the velocity vector and acceleration vector are calculated and input into a risk discrimination network for correction to generate a second video frame sequence. The frames in the second video frame sequence are then arranged according to the timestamps to generate the target video frame sequence, including: During the process of generating each frame of the first video frame sequence, global average pooling is performed on the latent variable features of the current frame to obtain the local motion state vector of the current frame, and the local motion state vector of the current frame is stored in the history state buffer. The velocity vector and acceleration vector are calculated based on the local motion state vectors of adjacent frames in the historical state buffer, and the velocity vector and acceleration vector are input into the risk discrimination network to calculate the risk score. When the risk score exceeds a preset threshold, the gradient of the risk score relative to the predicted motion state is calculated, and correction is made along the negative direction of the gradient to generate a second video frame sequence. The frames in the second video frame sequence are then arranged according to the timestamp to generate a target video frame sequence.

8. The method for generating custom character videos by multi-reference map feature fusion as described in claim 7, characterized in that, The step of reconstructing the focus of the target video sequence based on the focus depth data and outputting a custom character video includes: Calculate the relative depth distribution map of each frame in the target video frame sequence, and upsample the relative depth distribution map to the same spatial resolution as the corresponding frame to obtain the full resolution depth distribution map of each frame; Calculate the depth difference between the target depth value and the corresponding character relative depth value at each pixel position in the full-resolution depth distribution map of each frame, and calculate the blur circle diameter value at each pixel position based on the depth difference; In the preset Gaussian blur kernel library, blur kernels of adjacent size levels are selected and linear interpolation weighted between the two size levels according to the blur circle diameter value to obtain the adaptive blur kernel for each pixel position. The adaptive blur kernel is then used to reconstruct the focus of each pixel position in the target video frame sequence to output the custom character video.

9. A custom character video generation device based on multi-reference image feature fusion, characterized in that, The steps for implementing the custom character video generation method of multi-reference map feature fusion as described in any one of claims 1 to 8 include: The character identification unit is used to acquire multiple reference images from different perspectives, identify the target character in the reference images, and obtain global identification data of the target character. The motion coding unit is used to acquire the driving sequence for driving the target character's motion, encode the driving sequence in time order to obtain the encoded token sequence, and perform motion embedding and decoupling optimization on the encoded token sequence to obtain motion information data and the corresponding timestamp. The risk avoidance and noise reduction unit is used to input the global identity recognition data, the motion information data and the timestamp into the motion adapter to perform risk avoidance and noise reduction, and obtain the target video frame sequence and focus depth data. The focus reconstruction unit is used to reconstruct the target video frame sequence based on the focus depth data and output a custom character video.

10. The custom character video generation apparatus for multi-reference map feature fusion as described in claim 9, characterized in that, The role identification unit is specifically used for: Multiple reference images from different perspectives are acquired, and local identity features of the target characters in the reference images are extracted to obtain the first identity recognition data corresponding to each perspective. The preset identity anchoring vector and the first identity recognition data corresponding to each viewpoint are combined to form an input sequence. Multi-head self-attention calculation is performed on the input sequence to obtain the correlation weight between each first identity recognition data. The first identity recognition data corresponding to each viewpoint is weighted and fused according to the relevance weight, retaining the common identity feature components of each viewpoint and suppressing the non-common feature components caused by posture differences and lighting differences, to obtain the second identity recognition data corresponding to each viewpoint. Then, the second identity recognition data corresponding to each viewpoint is fused from multiple perspectives to obtain the global identity recognition data of the target character.

11. The custom character video generation apparatus for multi-reference map feature fusion as described in claim 9, characterized in that, The motion coding unit is specifically used for: Obtain the driving sequence used to drive the movement of the target character, parse the driving sequence into a visual token sequence arranged in frames, and superimpose physical timestamp sine coding and sequence index sine coding on the visual token sequence to obtain the encoded token sequence. The encoded token sequence is subjected to Fourier transform along the time dimension and real-valued frequency domain features are extracted. The preset query matrix is ​​used as the query vector, and the real-valued frequency domain features are used as the key vector and value vector to perform cross-attention calculation to obtain the motion embedding summary vector. Construct an orthogonal constraint loss function between the motion embedding summary vector and the global identity recognition data. Decouple and optimize by minimizing the orthogonal constraint loss function to obtain motion information data, and regress the timestamp from the dedicated time identifier Token.

12. The custom character video generation apparatus for multi-reference image feature fusion as described in claim 9, characterized in that, The risk avoidance and noise reduction unit is specifically used for: The global identity recognition data and the motion information data are respectively input into the video denoising generation network in the motion adapter. The first video frame sequence is obtained by injecting parameters and generating video frames through the identity cross-attention sublayer and motion cross-attention sublayer of each denoising module in the video denoising generation network. During the generation of each frame of the first video frame sequence, the velocity vector and acceleration vector are calculated and input into the risk discrimination network for correction, and a second video frame sequence is generated. The frames in the second video frame sequence are arranged according to the timestamp to generate the target video frame sequence. The local motion state vectors of each frame in the target video sequence are mapped to the character reference depth reference to obtain the character relative depth value of each frame, and the character relative depth values ​​of each frame are combined to form focus depth data.

13. The custom character video generation apparatus for multi-reference image feature fusion as described in claim 9, characterized in that, The focus reconstruction unit is specifically used for: Calculate the relative depth distribution map of each frame in the target video frame sequence, and upsample the relative depth distribution map to the same spatial resolution as the corresponding frame to obtain the full resolution depth distribution map of each frame; Calculate the depth difference between the target depth value and the corresponding character relative depth value at each pixel position in the full-resolution depth distribution map of each frame, and calculate the blur circle diameter value at each pixel position based on the depth difference; In the preset Gaussian blur kernel library, blur kernels of adjacent size levels are selected and linear interpolation weighted between the two size levels according to the blur circle diameter value to obtain the adaptive blur kernel for each pixel position. The adaptive blur kernel is then used to reconstruct the focus of each pixel position in the target video frame sequence to output the custom character video.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the custom character video generation method of multi-reference map feature fusion as described in any one of claims 1 to 8.

15. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the custom character video generation method of multi-reference map feature fusion as described in any one of claims 1 to 8.

16. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the custom character video generation method based on multi-reference map feature fusion as described in any one of claims 1 to 8.

Images