A method and system for generating a head-tail frame backflow circulating video

Abstract

The application discloses a kind of head and tail frame backflow cyclic video generation method and system, method includes: generating forward video sequence by input image and extracting head and tail frames;Subject segmentation and global motion estimation are carried out to forward video sequence, determine associated visual area based on the directed time shift dependence of subject residual motion and candidate residual motion, construct area response kernel;According to area response kernel, form asynchronous condition sequence and inject video generation model, generate backflow video sequence;Splice forward video sequence and backflow video sequence, construct annular joint state trajectory, identify and fold overlapping state section;The cycle time index of reserved frame is configured and intermediate frame is generated, and cyclic video is output.The application realizes the cooperation of continuous subject motion, associated response and time sampling by associated visual state modeling, asynchronous backflow generation and annular state trajectory folding.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and system for generating looped video with first and last frame rewind. Background Technology

[0002] With the development of artificial intelligence video generation technology, dynamic videos can be generated from single images, and further loop videos that can be played continuously. This technology has been widely used in scenarios such as digital display, virtual characters, advertising materials, and short video production.

[0003] Existing methods typically first use a graph-generated video model to generate forward motion video, then extract the first and last frames of the video, use a video transition generation model to generate a reflow segment that returns from the last frame to the first frame, and finally splice the forward video and the reflow segment together, and improve playback smoothness through optical flow frame interpolation and other methods.

[0004] While this type of method can reduce abrupt changes in the beginning and end of the video, it still mainly relies on the similarity of endpoint images or pixel continuity as the basis for judgment, and easily ignores the dynamic continuity of the looped video on the circular time axis.

[0005] On the one hand, the last frame of the forward video and the starting frame of the reflow segment, as well as the ending frame of the reflow segment and the first frame of the forward video, may overlap or nearly overlap, causing the same motion state to be continuously maintained for multiple sampling moments, resulting in local deceleration, short pauses, or time dilation at the seams. Directly deleting the overlapping frames may cause a jump in motion phase. On the other hand, existing methods usually only constrain the position, posture, or appearance of the subject to return to the initial state, without considering the response lag or attenuation process of accompanying visual states such as projection, reflection, water ripples, smoke, clothing hems, hair, and particles relative to the subject's motion. This results in the subject having closed while the accompanying states still retain the motion memory of the previous cycle, leading to phenomena such as instantaneous shadow shifts, abrupt changes in reflection direction, or causeless reset of ripples.

[0006] Therefore, how to simultaneously ensure uniformity of the circular time sampling and the coordinated closure of the subject and the accompanying visual state remains a problem that needs to be solved in existing loop video generation technology. Summary of the Invention

[0007] To address the above problems, embodiments of the present invention provide a method for generating looped video with first and last frame rewinding, the method comprising:

[0008] Acquire an input image, generate a forward video sequence from the input image, and extract the first and last forward frames;

[0009] The forward video sequence is segmented to obtain the main body region and its corresponding mask, and global motion is estimated. The frame-by-frame motion field is calculated and the global motion components in the frame-by-frame motion field are removed to obtain the main body residual motion sequence and the candidate residual motion sequence. Based on the directed time-shift dependency between the main body residual motion sequence and the candidate residual motion sequence, the accompanying visual region and its mask are determined from the adjacent candidate regions of the main body, and the region response kernel is constructed by time offset and response attenuation.

[0010] Extract main state features and accompanying state features from the forward video sequence; perform a circular shift on the accompanying state features according to the region response kernel to form an asynchronous conditional sequence composed of the main state features and the shifted accompanying state features; using the forward tail frame and the forward first frame as boundaries, inject the asynchronous conditional sequence into the temporal layer of the video generation model according to the main region mask and the accompanying visual region mask to generate a reflow video sequence;

[0011] The forward video sequence and the reflow video sequence are spliced ​​together to form a circular candidate sequence; a circular joint state trajectory is constructed from the main state features and the accompanying state features aligned with the regional response kernel, and overlapping state segments with equivalent states and the same direction of change are identified and folded.

[0012] Configure the loop time index of the retained frames according to the folded circular joint state trajectory, generate intermediate frames according to the loop time index of adjacent retained frames, and output loop video.

[0013] Furthermore, the global motion estimation process includes:

[0014] Background feature points located outside the main body region are extracted in adjacent frames. Background feature points corresponding to the forward matching position and the reverse matching position are retained. The inter-frame global transformation is fitted based on the retained background feature points, and the global motion field is formed by the inter-frame global transformation.

[0015] The frame-by-frame motion field is subtracted from the global motion field, and the resulting difference motion fields are aggregated according to the main region mask to form the main residual motion sequence; the region outside the main region that is connected to the boundary of the main region and has continuous residual motion is divided into the main adjacent candidate region, and the resulting difference motion fields are aggregated according to the main adjacent candidate region to form the candidate residual motion sequence.

[0016] Furthermore, the process for determining the directed time-shift dependency includes:

[0017] Bidirectional time-shift prediction is performed using each sequence in the main residual motion sequence and the candidate residual motion sequence as a preceding sequence to obtain residual sequences corresponding to the two prediction directions.

[0018] The predicted residual in the forward direction of the subject is less than the reverse prediction residual, and the subject adjacent candidate regions that are temporally related after removing the zero-time-shift component are determined as the accompanying visual regions.

[0019] A delay term is generated based on the time offset corresponding to the minimum value of the predicted residual in the subject's forward direction. An attenuation term is generated based on the change of the candidate residual motion amplitude relative to the subject's residual motion amplitude after the time offset processing. The delay term and the attenuation term are organized according to the accompanying visual region to form the region response kernel.

[0020] Furthermore, the process of generating the reflow video sequence includes:

[0021] Encode the forward tail frame and the forward first frame to obtain the boundary frame latent space features;

[0022] According to the delay term in the regional response kernel, the associated state features are mapped to the ring index before the corresponding main state features, and the mapped associated state features are modulated according to the attenuation term to form the asynchronous condition sequence;

[0023] In the predetermined temporal layer of the video generation model, the subject state features are written into the current time index using the subject region mask, and the mapped accompanying state features are written into the corresponding circular index using the accompanying visual region mask. The asynchronous conditional sequence after writing is then input into the predetermined temporal layer in conjunction with the boundary frame latent space features.

[0024] Furthermore, the construction process of the circular joint state trajectory includes:

[0025] Extract the subject contour, subject pose, and subject residual motion from each frame of the circular candidate sequence, and update the subject state features;

[0026] Extract the region morphology, corresponding candidate residual motion, and spatial relationship with the main region from the associated visual region, and update the associated state features;

[0027] The associated state features are aligned to the time index corresponding to the main state features according to the regional response kernel;

[0028] The main state features of the same time index, the aligned accompanying state features, and the state direction quantities obtained by the difference between adjacent time indices are combined to form a joint state point, and the joint state points are connected in a circular order to form the circular joint state trajectory.

[0029] Furthermore, the process of identifying the overlapping state segments includes:

[0030] Select consecutive joint state point sequences from the connection end and the loop end of the circular joint state trajectory respectively, and monotonically pair the joint state points in the two sets of consecutive joint state point sequences in a circular order.

[0031] For each pair, determine whether the state distance between the joint state points meets the state equivalence condition, whether the corresponding state direction quantities are in the same direction, and whether the order of the corresponding associated state features on the corresponding index of the region response kernel is consistent.

[0032] Pairs that continuously meet the above criteria are extended into candidate overlapping state segments. The candidate overlapping state segments containing the most joint state points are selected at the connection end and the loop end, respectively, and the selected candidate overlapping state segments are combined to form the overlapping state segment.

[0033] Furthermore, the folding process of the overlapping state segment includes:

[0034] Connect the paired joint state points in the overlapping state segment into two paired trajectories, and calculate the state direction deviation between the entrance joint state point and the exit joint state point of the two paired trajectories and the adjacent non-overlapping joint state point.

[0035] Retain the paired trajectory with the smaller total state direction deviation, and delete the frame corresponding to the other paired trajectory;

[0036] Connect the preserved paired trajectory with the non-overlapping trajectories on both sides to obtain the folded circular joint state trajectory;

[0037] The state distance between adjacent joint state points is accumulated along the folded circular joint state trajectory, and the accumulated value is mapped to the cycle time index of the retained frame in a circular order.

[0038] Furthermore, the process of generating the intermediate frame includes:

[0039] Each pair of adjacent retained frames is encoded to obtain frame features, and the bidirectional motion field between adjacent retained frames is calculated;

[0040] The main interpolation index is determined based on the cyclic time index of two adjacent retained frames, and the main interpolation index is converted into the accompanying interpolation index corresponding to the accompanying visual region according to the delay term and attenuation term in the region response kernel.

[0041] Based on the main interpolation index and the accompanying interpolation index, bidirectional deformation is performed on the frame features within the range defined by the main region mask and the accompanying visual region mask, respectively.

[0042] An occlusion confidence map is generated based on the forward and reverse consistency of the bidirectional motion field. The frame features after bidirectional deformation are fused according to the occlusion confidence map, and the fused frame features are decoded into the intermediate frame.

[0043] Furthermore, a method for generating looped video with first and last frame rewinds further includes: dividing motion phase segments based on the direction changes, amplitude changes, and continuous low amplitude states of the residual motion sequence of the main body; configuring phase identifiers for the motion phase segments; constructing corresponding phase response sub-cores and forming a phase state chain in a circular time sequence; calling the phase response sub-cores according to the main body state direction quantity and the phase identifier of the previous time index; constructing the asynchronous condition sequence; limiting the pairing of overlapping state segments; and determining the accompanying interpolation index.

[0044] On the other hand, this application also provides a system for generating looped video with first and last frame rewinding, the system comprising:

[0045] Forward video generation module: acquires input image, generates forward video sequence from input image, and extracts forward first frame and forward last frame;

[0046] The accompanying response modeling module performs subject segmentation on the forward video sequence to obtain the subject region and its corresponding mask, and estimates the global motion; it calculates the frame-by-frame motion field and removes the global motion components from the frame-by-frame motion field to obtain the subject residual motion sequence and candidate residual motion sequence; based on the directed time-shift dependency between the subject residual motion sequence and the candidate residual motion sequence, it determines the accompanying visual region and its mask from the adjacent candidate regions of the subject, and constructs the region response kernel based on the time offset and response attenuation;

[0047] Asynchronous reflow generation module: Extracts main state features and accompanying state features from the forward video sequence; performs a circular shift on the accompanying state features according to the region response kernel to form an asynchronous conditional sequence composed of the main state features and the shifted accompanying state features; using the forward tail frame and the forward first frame as boundaries, injects the asynchronous conditional sequence into the temporal layer of the video generation model according to the main region mask and the accompanying visual region mask to generate a reflow video sequence;

[0048] Overlapping State Folding Module: The forward video sequence and the reflow video sequence are spliced ​​together to form a circular candidate sequence; a circular joint state trajectory is constructed from the main state features and the accompanying state features aligned with the regional response kernel; overlapping state segments with equivalent states and the same direction of change are identified and folded.

[0049] Looping Timing Interpolation Module: Configures the looping time index of the retained frames according to the folded circular joint state trajectory, generates intermediate frames according to the looping time index of adjacent retained frames, and outputs looping video.

[0050] The technical solution provided by this invention has at least the following technical effects and advantages:

[0051] This invention achieves coordinated continuity of subject motion, accompanying response, and time sampling through accompanying visual state modeling, asynchronous reflow generation, and circular state trajectory folding. By removing global motion components and determining the accompanying visual region based on the directed time-shift dependency between the subject's residual motion and candidate residual motion, the accompanying state determined by the subject's adjacent candidate region can establish a correspondence with the subject's motion. The accompanying state features are circularly shifted and amplitude modulated using delay and attenuation terms in the region response kernel, ensuring that the subject's state and accompanying state in the reflow video change continuously according to their respective response processes. By constructing a circular joint state trajectory, overlapping state segments with equivalent states and unidirectional changes are identified and folded, reducing repetitive occupation at seam positions, local pauses, and motion phase jumps. The subject interpolation index and accompanying interpolation index are determined separately using a cyclic time index, and intermediate frames are generated by combining a bidirectional motion field and occlusion confidence map, ensuring continuity between frame motion and region boundaries in the looped video. Attached Figure Description

[0052] Figure 1 This is a flowchart of a method for generating looped video with first and last frame rewinding, as described in Example 1.

[0053] Figure 2 This is a flowchart of the phase state chain driven loop video generation method in Embodiment 2;

[0054] Figure 3 This is a schematic diagram of the connection of a first and last frame reflow loop video generation system in Embodiment 3. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Example 1:

[0057] Please see Figure 1 As shown, an embodiment of the present invention provides a method for generating looped video with first and last frame rewinding, the method comprising:

[0058] S1. Obtain the input image, generate a forward video sequence from the input image, and extract the first and last forward frames;

[0059] S2. Perform subject segmentation on the forward video sequence to obtain the subject region and its corresponding mask, and estimate the global motion; calculate the frame-by-frame motion field and remove the global motion components in the frame-by-frame motion field to obtain the subject residual motion sequence and candidate residual motion sequence; based on the directed time-shift dependency between the subject residual motion sequence and the candidate residual motion sequence, determine the accompanying visual region and its mask from the adjacent candidate regions of the subject, and construct the region response kernel based on the time offset and response attenuation;

[0060] S3. Extract the main state features and accompanying state features from the forward video sequence; perform a circular shift on the accompanying state features according to the region response kernel to form an asynchronous conditional sequence composed of the main state features and the shifted accompanying state features; using the forward tail frame and the forward first frame as boundaries, inject the asynchronous conditional sequence into the temporal layer of the video generation model according to the main region mask and the accompanying visual region mask to generate the reflow video sequence;

[0061] S4. The forward video sequence and the return video sequence are spliced ​​together to form a circular candidate sequence; a circular joint state trajectory is constructed from the main state features and the accompanying state features aligned with the regional response kernel, and overlapping state segments with equivalent states and the same direction of change are identified and folded.

[0062] S5. Configure the loop time index of the retained frames according to the folded circular joint state trajectory, generate intermediate frames according to the loop time index of adjacent retained frames, and output loop video.

[0063] In this embodiment, after generating the forward video sequence and segmenting the main body, global motion estimation is performed sequentially on adjacent frames in the forward video sequence. The main body region mask is used to mark the main body pixels in the corresponding frame. The pixels outside the main body region mask constitute the extraction range of background feature points. Background feature points are image positions in the background where the local brightness changes significantly and can be stably located between adjacent frames. They are used to characterize the positional changes of the background as the overall screen moves.

[0064] For temporally adjacent current and next frames, background feature points are first extracted outside the main body region mask of the current frame. Then, the background feature points are tracked from the current frame to the next frame to obtain the forward matching position. Subsequently, the forward matching position is used as the starting point to track back to the current frame to obtain the reverse matching position. When the reverse matching position and the position of the corresponding background feature point satisfy the bidirectional consistency condition, the background feature point and its forward matching position are retained. When the two do not satisfy the bidirectional consistency condition, the corresponding matching relationship is deleted. The bidirectional consistency condition is used to exclude unreliable point pairs caused by occlusion, local deformation, feature disappearance, and incorrect matching.

[0065] Based on the retained background feature points and their forward matching positions, the inter-frame global transformation is fitted. In this embodiment, affine transformation is used to describe the overall translation, rotation and scale changes between adjacent frames. During the fitting process, the reprojection deviation between the predicted position of the background feature point after affine transformation and the forward matching position is used as the screening criterion. Outliers that do not conform to the motion trend of the background subject are iteratively eliminated. The retained point pairs are used to determine the inter-frame global transformation. The pixel position in the current frame is substituted into the inter-frame global transformation to calculate the predicted position of the pixel in the next frame. The displacement vector between the predicted position and the original position constitutes the global motion field. The global motion field represents the common motion components formed by camera movement, image translation and overall scaling between adjacent frames.

[0066] Furthermore, the frame-by-frame motion field from the current frame to the next frame is calculated. The frame-by-frame motion field is a set of two-dimensional displacement vectors indexed by pixel position, which is used to represent the corresponding movement direction and movement magnitude of the current frame pixel in the next frame. For the same pixel position, the displacement vector in the frame-by-frame motion field is subtracted from the displacement vector in the global motion field to obtain the difference motion field. The difference motion field represents the local relative motion retained after removing the overall motion of the image.

[0067] Within the range defined by the main region mask, the displacement vectors in the difference motion field are weighted and aggregated according to the motion estimation confidence to obtain the subject residual motion representation corresponding to the current frame. The subject residual motion representations are arranged in frame order to form the subject residual motion sequence. The subject residual motion sequence is used to record the motion direction, motion amplitude and its changes over time of the subject relative to the overall picture.

[0068] Outside the main region mask, pixels connected to the boundary of the main region are used as the starting point for region growth. Pixels that are spatially continuous and whose direction and amplitude changes of adjacent displacement vectors meet the continuity condition are classified into the same main adjacent candidate region. Boundary connectivity means that the candidate region can be connected to the boundary of the main region through continuous pixel paths. Residual motion continuity means that the difference motion vector on the continuous pixel path does not undergo abrupt changes that are inconsistent with the trend of neighboring motions. The displacement vectors in the difference motion field are aggregated for the formed main adjacent candidate regions to obtain candidate residual motion representations, which are then arranged in frame order to form a candidate residual motion sequence.

[0069] For example: When the forward video sequence contains a person moving laterally and the hem of clothing moving with the person, the inter-frame global transformation obtained by fitting background feature points is used to characterize the overall lateral movement of the image. After subtracting the global motion field, the person and the hem of clothing retain residual motion relative to the background. The hem area that is connected to the outline of the person and has continuous residual motion is divided into candidate areas of the subject's adjacency. The candidate residual motion sequence is used for subsequent directed time-shift dependency analysis.

[0070] In this implementation, step S2 involves determining the directed time-shift dependency and constructing the regional response kernel, which follows the formation process of the main residual motion sequence and the candidate residual motion sequence.

[0071] For a candidate region adjacent to a subject, the residual motion sequence of the subject is taken as the subject sequence, and the candidate residual motion sequence corresponding to the region is taken as the candidate sequence. First, the two sequences are time-aligned and amplitude-normalized. Then, the subsequent motion representation of the candidate sequence is predicted using the historical fragments of the subject sequence to obtain the predicted value of the subject's preceding direction. Next, the subsequent motion representation of the subject sequence is predicted using the historical fragments of the candidate sequence to obtain the reverse predicted value. The difference between the predicted value and the actual motion representation of the predicted sequence is arranged in chronological order to form the residual sequence of the corresponding direction. The residual is used to represent the motion components that the preceding sequence could not explain. The smaller the residual amplitude, the more stable the temporal dependency corresponding to the predicted direction.

[0072] To reduce the interference of camera residual motion and synchronous deformation on the judgment results, candidate motion representations are fitted using the subject motion representation under the same time index, and this fitted component is subtracted from the candidate sequence to form a candidate sequence with the zero time shift component removed. For the th For each subject adjacent to a candidate region, the relative time position of the subject's residual motion sequence and the candidate residual motion sequence is changed within the candidate time offset range, and the amplitude of the subject's residual motion sequence is adjusted. The delay term and attenuation term are jointly determined by the following formula:

[0073] ;

[0074] In the formula, All the calculation terms within together constitute the first... The joint response cost of adjacent candidate regions of each entity; among which... This indicates the region number of the candidate adjacent regions of the subject; when there are multiple candidate adjacent regions of the subject... When defining a region, assign a region number based on its spatial location within the image.

[0075] This represents the time offset obtained when the joint response cost is minimized, and is used as the first... The delay term of the accompanying visual region; This represents the response amplitude coefficient obtained when the joint response cost is minimized, and is used as the attenuation term for the associated visual region;

[0076] This indicates that within a defined candidate range, the combination of variables that minimizes the joint response cost within the curly braces is obtained; its output is a variable pair consisting of time offset and response amplitude coefficient, rather than the numerical value of the joint response cost.

[0077] This indicates the candidate time offset during the traversal process, used to change the relative time position of the main residual motion sequence and the candidate residual motion sequence; This indicates the time position at which the main motion and the candidate region motion are completely synchronized;

[0078] Indicates the first The candidate time offset set corresponding to each subject's adjacent candidate region is determined based on the continuous visible time period of the subject's adjacent candidate region in the forward video sequence, and the time range in which the subject's residual motion sequence and the candidate residual motion sequence can form an effective overlap.

[0079] Represents the amplitude coefficient of the candidate response during the traversal process; This indicates that the response amplitude of the candidate region is not greater than the amplitude of the corresponding subject's motion, so that the calculation results are consistent with the meaning of the data in the "attenuation term";

[0080] This represents a discrete-time index in the forward video sequence, corresponding to a video frame position. Indicates that at the candidate time offset is At that time, in the residual motion sequence of the main body Candidate residual motion sequences and candidate residual motion sequences located at time index Candidate residual motion characterization at the location All of them contain valid data and are time index sets that can be calculated for the corresponding difference terms;

[0081] Indicates time index The residual motion representation of the main body is obtained by aggregating the difference motion field within the mask range of the main body region. It is used to record the motion direction, motion amplitude and motion distribution of the main body within the region after removing the global motion components.

[0082] This indicates that the residual motion sequence of the main body is shifted according to the candidate time. After the move, with time index The main residual motion representation corresponding to the candidate residual motion representation;

[0083] Indicates the first Each subject's adjacent candidate region in the time index The candidate residual motion characterization at the location is obtained by aggregating the difference motion fields within the candidate regions adjacent to the corresponding subject;

[0084] Indicates the first The motion component weight matrix corresponds to the adjacent candidate regions of each subject. The motion component weight matrix is ​​formed based on the motion estimation confidence within the region, so that vector components with stable motion correspondences have larger computational weights, while occluded positions and positions with unstable motion estimations have smaller computational weights.

[0085] The first term in the formula is the response residual in the subject-ahead direction, where, This represents the residual motion of the main body after time shift and amplitude decay. This indicates the difference between the actual movement of the candidate region and the predicted movement of the main body. The smaller this value is, the more stable the temporal relationship is, where the main body moves first and the candidate region moves later.

[0086] The second term is the asymmetric constraint term for prediction direction. The first weighted residual represents the residual generated by predicting the candidate residual motion sequence using the main residual motion sequence, and the second weighted residual represents the residual generated by predicting the main residual motion sequence in reverse using the candidate residual motion sequence.

[0087] This represents a non-negative adjustment coefficient for the asymmetric constraint term in the prediction direction, used to adjust the weight of the positive and negative prediction relationships in the joint response cost.

[0088] The third term is the zero-time-shift synchronization constraint term, used to calculate the normalized correlation between the main residual motion sequence and the candidate residual motion sequence when no time shift occurs.

[0089] Representation of the residual motion of the main body transpose; superscript The transpose operation represents a matrix or vector; This represents the inner product of the main residual motion representation and the candidate residual motion representation at the same time index, used to reflect the degree of synchronization between their motion direction and motion amplitude.

[0090] This represents the Euclidean norm, used to calculate the magnitude of the motion representation vector; The third term represents the non-negative adjustment coefficient of the zero-time-shift synchronization constraint. The higher the degree of synchronization correlation between the subject motion and the candidate region motion at zero time shift, the larger the third term becomes, thereby reducing the possibility that camera residual motion, overall translation, and synchronous background motion are identified as accompanying responses; the fourth term is the motion change direction constraint term. Representation of the residual motion of the main body The difference between the residual motion representation of the subject and its previous time index is used to represent the changes in the direction and amplitude of the subject's motion between adjacent time points. Representation of candidate residual motion characteristics The difference between the candidate residual motion representation and its previous time index is used to represent the changes in the direction and amplitude of motion of the candidate region between adjacent time points. This represents the inner product between two motion change vectors after time alignment.

[0091] The fourth fraction represents the degree of consistency in the directions of the two motion vectors; the closer their directions are, the closer the fraction is to one, and the smaller the value inside the parentheses; the greater the difference in their directions, the larger the value inside the parentheses.

[0092] This represents the non-negative adjustment coefficient of the motion change direction constraint term, used to adjust the weight of motion change direction consistency in the joint response cost. These are preset positive numbers, added to the denominators of the third and fourth terms respectively, to avoid division by zero when the motion amplitude is zero. These preset positive numbers are only used to maintain numerical calculation stability and do not participate in determining the meaning of the delay and attenuation terms.

[0093] Traversal The candidate time offset and the candidate response amplitude coefficient within the specified range will be used to determine the response with the minimum joint response cost. If it is determined to be a delayed item, the corresponding Once identified as an attenuation term, the delay term, attenuation term, and corresponding accompanying visual region mask are associated to form a response record in the region response kernel.

[0094] In the subsequent step S3, using Perform a circular shift on the associated state features, utilizing Amplitude modulation is performed on the associated state characteristics of the completed ring shift; in step S4, the amplitude is modulated using... and The associated state characteristics are time-aligned and amplitude-normalized; in step S5, the associated state characteristics are used... and Convert the main interpolation index to the companion interpolation index.

[0095] Based on the joint solution results, candidate regions adjacent to the subject whose forward prediction residual is less than the backward prediction residual and whose delay term is positive are identified as associated visual regions. Their mask identifiers, delay terms, and attenuation terms are then associated to form a region response kernel. For example, when a person moves and their clothing hem swings with a delay, the corresponding region is identified as an associated visual region, and its delay and attenuation terms are written into the region response kernel for use in subsequent asynchronous conditional sequence construction.

[0096] The process of generating the reflow video sequence in step S3 follows the process of constructing the regional response kernel.

[0097] First, the forward tail frame and the forward first frame are input into the image encoder that is paired with the video generation model. The image encoder converts the two boundary frames from pixel data into feature tensors of the same size according to the same spatial compression rules. It records the main structure, background layout and texture distribution in the boundary frames respectively, and obtains the latent space features of the boundary frames. The latent space features are multi-channel numerical representations formed after the image is encoded, which serve as the start and end boundaries of the reflow video sequence.

[0098] According to the same spatial scale as the latent space features of the boundary frame, the main body region mask and the accompanying visual region mask are size-mapped so that the mask position corresponds to the latent space feature position. The pose, contour and residual motion of the main body region in consecutive frames are extracted based on the forward video sequence to form the main body state features; the accompanying visual region morphology and candidate residual motion are used to form the accompanying state features.

[0099] The time positions of the video sequence to be generated are connected end to end to form a circular index. For the current subject state feature, the delay term of the corresponding accompanying visual region is read from the regional response kernel, and the corresponding time position is searched forward along the circular index. The state feature of the accompanying visual region is written into the time position. When the index crosses the beginning position of the sequence, it continues to count from the end of the sequence. Thus, the subject state feature and the accompanying state feature at the same generation time use different time indices.

[0100] Furthermore, the attenuation term is read from the regional response kernel, and the amplitude modulation is applied to the associated state features that have completed the circular shift. The amplitude modulation is to weight the motion components in the associated state features according to the attenuation term so that they maintain the change relationship corresponding to the aforementioned candidate residual motion sequence. The main state features arranged according to the current time index and the associated state features that have undergone circular shift and amplitude modulation are combined to form an asynchronous condition sequence.

[0101] The predetermined temporal layer is a network layer in the video generation model that associates continuous frame features along the time dimension. When processing the current time index, the predetermined temporal layer uses a subject region mask to select the spatial location corresponding to the subject and writes the subject state features into the spatial location. It also uses a companion visual region mask to select the spatial location corresponding to the companion visual region and writes the mapped companion state features into the spatial location of the corresponding circular index. The locations outside the mask retain the original frame features of the temporal layer.

[0102] After writing is complete, the asynchronous conditional sequence and the boundary frame latent space features are input into the predetermined temporal layer. The video generation model uses the boundary frame latent space features corresponding to the forward tail frame as the sequence starting point and the boundary frame latent space features corresponding to the forward first frame as the sequence ending point, and gradually updates the reflow frame latent space features between the two. During the update process, the main body region calls the main body state features according to the current time index, and the accompanying visual region calls the accompanying state features according to the ring index determined by the region response kernel. The updated reflow frame latent space features are input into the decoder and converted into a reflow video sequence arranged in chronological order.

[0103] The circular joint state trajectory construction process in step S4 includes splicing the forward video sequence and the return video sequence in chronological order, and establishing an adjacency relationship between the end of the sequence and the beginning of the sequence to form a circular candidate sequence.

[0104] For a frame in the circular candidate sequence, the boundary of the main region is determined according to the main region mask. The boundary pixels are arranged according to the contour extension direction and transformed to relative coordinates with the center of the main region as the origin to form the main contour. The main contour is used to represent the shape of the main body and its relative scale. Furthermore, pose points that can characterize the position of the main body structure are extracted from the main region. The position of the pose points relative to the center of the main region and the connection direction between adjacent pose points are used to form the main body pose. For the main body without a clear limb structure, the pose points are replaced by structural feature points that are stably distributed in the main region.

[0105] Read the residual motion representation of the subject corresponding to the frame, and combine its motion direction, motion amplitude and motion distribution within the region with the subject outline and subject posture in a fixed field order to form the subject state features. Thus, the subject state features of different frames have the same data arrangement, and can be compared between corresponding fields.

[0106] For the identified associated visual regions, the region boundaries, center, and extension direction are extracted based on the associated visual region mask to form the region morphology; the candidate residual motion representations corresponding to the region are read to obtain the direction and amplitude of their relative motion to the overall image; the position of the center of the associated visual region relative to the center of the main region, the interval between the boundaries of the two regions, and the direction of boundary connection are calculated to form the spatial relationship relative to the main region; the region morphology, candidate residual motion representations, and spatial relationship are combined to form associated state features.

[0107] Because the accompanying visual region has a response delay relative to the main region, the accompanying state features of the current frame are not directly paired with the main state features of the current frame. Based on the delay term in the region response kernel, the accompanying state features are moved along the circular candidate sequence to the time index corresponding to the main state that triggered the accompanying state. Based on the attenuation term in the region response kernel, the motion amplitude in the accompanying state features is reverse-normalized so that the aligned accompanying state features represent the response generated by the corresponding main state.

[0108] The main state features and the aligned accompanying state features under the same time index are sequentially connected to form joint state points. The corresponding fields of temporally adjacent joint state points are differentially analyzed to obtain state direction quantities that represent the direction and magnitude of state change. The state direction quantities are written into the corresponding joint state points. Finally, the joint state points are connected according to the frame order of the circular candidate sequence, and the end joint state point is connected to the beginning joint state point to form a circular joint state trajectory for subsequent recognition of overlapping state segments.

[0109] In step S4, after the construction of the circular joint state trajectory is completed, the positions where the forward video sequence and the return video sequence meet, and the positions where the end of the return video sequence and the beginning of the forward video sequence meet, are determined respectively. The former position is determined as the connection end and the latter position is determined as the loop end. Based on the connection end and the loop end, continuous joint state points are selected along the two extension directions of the circular joint state trajectory to form two sets of continuous joint state point sequences.

[0110] The pairing relationship between the joint state points is established according to the original time order of the two sets of continuous joint state point sequences. When pairing, starting from the joint state point closest to the corresponding end, the joint state points with similar states are searched sequentially in the other set of continuous joint state point sequences. The time index corresponding to the next pairing is limited to be after the time index corresponding to the previous pairing, so that the formed pairing relationship maintains the consistency of time order.

[0111] State distance is used to characterize the overall state difference between two joint state points. When calculating the state distance, the main body contour, main body pose, main body residual motion, aligned accompanying state features, and spatial relationship between the accompanying visual region and the main body region are obtained from the two joint state points respectively. The difference is calculated for the same fields, and the difference of different fields is converted to a unified comparison scale and then combined. The state equivalence condition is determined based on the state distance of non-end adjacent joint state points in the circular joint state trajectory. When the state distance between paired joint state points meets the state equivalence condition, the paired joint state points are determined to be state equivalent.

[0112] Further compare the state direction quantities corresponding to the paired joint state points, and determine the change direction of the main state feature and the aligned companion state feature in the corresponding fields. When the change order and change direction of the two at the pairing position are consistent, it is determined that the corresponding state direction quantities are in the same direction. Then, based on the delay term in the regional response kernel, determine the original time index of the companion state feature in the circular candidate sequence, and determine whether the companion state features in the continuous pairing maintain the same order as the main state feature. When the order is reversed, stop the extension of the current pairing relationship.

[0113] Paired joint state points that continuously satisfy state equivalence, state direction, and consistent arrangement order are formed into candidate overlapping state segments. The candidate overlapping state segments formed by the connection end and the loop end are compared respectively. The candidate overlapping state segment containing the most consecutive joint state points is selected. The selected candidate overlapping state segments are combined to form an overlapping state segment for subsequent folding processing.

[0114] In step S4, after identifying the overlapping state segments, the paired joint state points located on the forward video sequence side are connected sequentially according to the pairing relationship to form the first paired trajectory, and the paired joint state points located on the reflow video sequence side are connected sequentially to form the second paired trajectory. In each paired trajectory, the joint state point closer to the previous non-overlapping trajectory is the entry joint state point, and the joint state point closer to the next non-overlapping trajectory is the exit joint state point; the trajectory portion located outside the overlapping state segments and adjacent to the entry joint state point and the exit joint state point is the non-overlapping trajectory.

[0115] The joint state point at the entrance, the joint state point at the exit, and the adjacent non-overlapping joint state points of the first and second paired trajectories are obtained respectively. Based on the difference between the main state characteristics of the adjacent joint state points and the accompanying state characteristics after alignment, the state direction before entering the paired trajectory, the state direction at the entrance of the paired trajectory, the state direction at the exit of the paired trajectory, and the state direction after leaving the paired trajectory are determined.

[0116] State direction deviation is used to represent the difference in state change between the paired trajectory and the adjacent non-overlapping trajectory. The state direction at the entrance of the paired trajectory is compared with the state direction before entering the paired trajectory to obtain the entrance state direction deviation. The state direction at the exit of the paired trajectory is compared with the state direction after leaving the paired trajectory to obtain the exit state direction deviation. The entrance state direction deviation and the exit state direction deviation are combined to obtain the sum of the state direction deviations of the paired trajectory.

[0117] Compare the sum of the state direction deviations of the first and second paired trajectories, retain the paired trajectory with the smaller value and its corresponding frame, and delete the frame corresponding to the other paired trajectory from the circular candidate sequence. Connect the entry of the retained paired trajectory to the previous non-overlapping trajectory, and connect the exit of the retained paired trajectory to the next non-overlapping trajectory. At the same time, retain the original circular head-tail connection relationship to obtain the folded circular joint state trajectory. Folding means reducing two paired trajectories representing the same state change process to one paired trajectory without changing the original arrangement order of the joint state points in the retained trajectory.

[0118] Along the folded circular joint state trajectory, starting from the predetermined retained frame, the state distance between adjacent joint state points is calculated sequentially, and the state distances are accumulated to obtain the cumulative state distance corresponding to the retained frame. The sum of the cumulative state distances of the folded circular joint state trajectory is mapped to a cycle, and the corresponding cycle time index is determined according to the position of the cumulative state distance of the retained frame in the sum of cumulative state distances. The next time position of the last retained frame is connected to the cycle time index of the starting retained frame to form a continuous circular time sequence.

[0119] In step S5, after configuring the cyclic time index of the retained frames, adjacent retained frames are selected in a circular order, and the two frames are converted into frame features with consistent size and channel arrangement using an encoder adapted to the video generation model. The frame features are used to record the spatial structure, texture information and region boundaries in the corresponding retained frames. Based on the pixel correspondence between the two adjacent retained frames, the forward motion field from the previous retained frame to the next retained frame and the reverse motion field from the next retained frame to the previous retained frame are calculated respectively, forming a bidirectional motion field.

[0120] For an intermediate frame to be generated between two adjacent retained frames, the main interpolation index is determined based on the relative position of the intermediate frame's cycle time index between the cycle time indices of the two adjacent retained frames. The main interpolation index is used to indicate the degree to which the main state progresses from the previous retained frame to the next retained frame.

[0121] For the accompanying visual region, the corresponding delay term and attenuation term are read from the region response kernel. The main interpolation index is shifted in a circular manner according to the delay term, and the advance amount of the shifted index relative to the starting index is adjusted according to the attenuation term to obtain the accompanying interpolation index corresponding to the accompanying visual region. When the shifted index crosses the end of the current cycle, the calculation continues from the beginning of the current cycle.

[0122] Based on the subject interpolation index, the motion vectors within the subject region mask range in the bidirectional motion field are temporally scaled, and the scaled forward motion vectors are used to sample subject features from the frame features of the previous retained frame, and the scaled backward motion vectors are used to sample subject features from the frame features of the next retained frame. Based on the accompanying interpolation index, accompanying features are sampled in the accompanying visual region mask range in the same way. Bidirectional deformation refers to mapping the target temporal position backward to the feature position in the adjacent retained frame according to the motion vector, and obtaining the frame features of that feature position.

[0123] Furthermore, the position pointed to by the forward motion vector is added to the corresponding reverse motion vector to obtain the forward and reverse mapping deviation. The forward confidence is determined according to the continuity between the mapping deviation and the neighboring motion vectors. The same processing is performed in the opposite direction to obtain the reverse confidence. The forward confidence and the reverse confidence form an occlusion confidence map. The occlusion confidence map is used to record the confidence of the corresponding spatial position in the intermediate frame using the features of the previous and next preserved frames.

[0124] The main features and associated features obtained from bidirectional deformation are weighted and fused according to the occlusion confidence map, and the frame features outside the mask range are fused according to the main interpolation index to obtain the intermediate frame features. Finally, the intermediate frame features are input into the decoder corresponding to the encoder to restore the pixel data and obtain the intermediate frame located between two adjacent retained frames.

[0125] Example 2:

[0126] like Figure 2 As shown, this embodiment further improves the design based on embodiment one. The difference is that in the actual operation of embodiment one, it was found that when the subject's motion reverses direction, pauses and resumes motion, or the motion amplitude changes abruptly within the cycle, the same subject posture will repeat in different motion stages. The accompanying visual regions are in different states of continuing to swing, decaying back swing, and reverse response, respectively. This causes the single region response kernel to correspond the same subject state to the same delay term and decay term, failing to distinguish the motion direction and preceding accompanying state when the subject enters the posture. Based on this, a method for generating looped video with first and last frame rewinds further includes: dividing the motion phase segment according to the subject's residual motion sequence, constructing a phase response sub-kernel for the motion phase segment, and establishing a phase state chain according to the circular time sequence. During the generation of the rewind video sequence, the identification of overlapping state segments, and the generation of intermediate frames, the phase response sub-kernel is called according to the subject state direction quantity and the phase identifier corresponding to the previous time index.

[0127] Specifically, the direction changes, amplitude changes, and continuous low amplitude states of adjacent motion representations in the residual motion sequence of the main body are calculated. When the motion direction changes in the opposite direction, the direction turning point is determined. When the amplitude of the residual motion of the main body changes from the motion state to the continuous low amplitude state and then increases again from the continuous low amplitude state, the pause entry point and pause exit point are determined respectively. When the amplitude change of adjacent motion representations deviates from the amplitude change trend of their respective continuous motion segments, the amplitude turning point is determined. The residual motion sequence of the main body is segmented according to the above positions to obtain motion phase segments, and phase markers are configured for the motion phase segments.

[0128] For a motion phase segment, the main residual motion sequence of the segment and the candidate residual motion sequence corresponding to the accompanying visual region are extracted. The delay term and attenuation term corresponding to the segment are determined according to the bidirectional time-shift prediction process in Example 1. The delay term, attenuation term, phase identifier, segment entry state direction quantity and segment exit state direction quantity are associated to form a phase response sub-core. The phase response sub-cores are connected according to the order of the motion phase segment in the circular candidate sequence to form a phase state chain. The accompanying state characteristics at the end of the previous motion phase segment are recorded between adjacent phase response sub-cores as the end response state of the next motion phase segment.

[0129] In step S3, the current phase response sub-core is determined in the phase state chain based on the state direction quantity corresponding to the current main state feature and the phase identifier of the previous time index. When the phase identifier is switched, the end response state corresponding to the previous phase response sub-core is read, and the end response state is used as the initial state of the accompanying state feature in the next phase response sub-core. Then, an asynchronous condition sequence is constructed according to the delay term and attenuation term in the next phase response sub-core.

[0130] In step S4, the phase identifier is written into the corresponding joint state point. When identifying overlapping state segments, only joint state points with corresponding order in the phase state chain and the same state direction are paired. The same subject posture in opposite motion directions is not included in the same overlapping state segment. In step S5, the phase response sub-core is called according to the phase identifier corresponding to the intermediate frame, and the accompanying interpolation index is determined by using the end response state recorded between adjacent phase response sub-cores.

[0131] Example 3:

[0132] like Figure 3 As shown, based on the same inventive concept as the first and last frame rewind loop video generation method in the foregoing embodiments, this application provides a first and last frame rewind loop video generation system. The system and method embodiments in this application are based on the same inventive concept. The system includes:

[0133] Forward video generation module: acquires the input image, generates a forward video sequence from the input image, and extracts the first and last forward frames;

[0134] The accompanying response modeling module performs subject segmentation on the forward video sequence to obtain the subject region and its corresponding mask, and estimates the global motion; it calculates the frame-by-frame motion field and removes the global motion components from the frame-by-frame motion field to obtain the subject residual motion sequence and candidate residual motion sequence; based on the directed time-shift dependency between the subject residual motion sequence and the candidate residual motion sequence, it determines the accompanying visual region and its mask from the adjacent candidate regions of the subject, and constructs the region response kernel based on the time offset and response attenuation;

[0135] Asynchronous reflow generation module: Extracts main state features and accompanying state features from the forward video sequence; performs a circular shift on the accompanying state features based on the region response kernel to form an asynchronous conditional sequence composed of the main state features and the shifted accompanying state features; uses the forward tail frame and the forward first frame as boundaries, and injects the asynchronous conditional sequence into the temporal layer of the video generation model according to the main region mask and the accompanying visual region mask to generate the reflow video sequence;

[0136] Overlapping State Folding Module: The forward video sequence and the reflow video sequence are spliced ​​together to form a circular candidate sequence; a circular joint state trajectory is constructed from the main state features and the accompanying state features aligned with the regional response kernel, and overlapping state segments with equivalent states and the same direction of change are identified and folded.

[0137] Looping Timing Interpolation Module: Configures the looping time index of the retained frames according to the folded circular joint state trajectory, generates intermediate frames according to the looping time index of adjacent retained frames, and outputs looping video.

[0138] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0139] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present application, based on the technical solution and concept of the present application, should be covered within the scope of protection of the present application.

Claims

1. A method for generating looped video with first and last frame rewinding, characterized in that, The methods include: Acquire an input image, generate a forward video sequence from the input image, and extract the first and last forward frames; The forward video sequence is segmented to obtain the main body region and its corresponding mask, and global motion is estimated. The frame-by-frame motion field is calculated and the global motion components in the frame-by-frame motion field are removed to obtain the main body residual motion sequence and the candidate residual motion sequence. Based on the directed time-shift dependency between the main body residual motion sequence and the candidate residual motion sequence, the accompanying visual region and its mask are determined from the adjacent candidate regions of the main body, and the region response kernel is constructed by time offset and response attenuation. Extract main state features and accompanying state features from the forward video sequence; perform a circular shift on the accompanying state features according to the region response kernel to form an asynchronous conditional sequence composed of the main state features and the shifted accompanying state features; using the forward tail frame and the forward first frame as boundaries, inject the asynchronous conditional sequence into the temporal layer of the video generation model according to the main region mask and the accompanying visual region mask to generate a reflow video sequence; The forward video sequence and the reflow video sequence are spliced ​​together to form a circular candidate sequence; a circular joint state trajectory is constructed from the main state features and the accompanying state features aligned with the regional response kernel, and overlapping state segments with equivalent states and the same direction of change are identified and folded. Configure the loop time index of the retained frames according to the folded circular joint state trajectory, generate intermediate frames according to the loop time index of adjacent retained frames, and output loop video.

2. The method for generating looped video with first and last frame rewinding according to claim 1, characterized in that, The global motion estimation process includes: Background feature points located outside the main body region are extracted in adjacent frames. Background feature points corresponding to the forward matching position and the reverse matching position are retained. The inter-frame global transformation is fitted based on the retained background feature points, and the global motion field is formed by the inter-frame global transformation. The frame-by-frame motion field is subtracted from the global motion field, and the resulting difference motion fields are aggregated according to the main region mask to form the main residual motion sequence; the region outside the main region that is connected to the boundary of the main region and has continuous residual motion is divided into the main adjacent candidate region, and the resulting difference motion fields are aggregated according to the main adjacent candidate region to form the candidate residual motion sequence.

3. The method for generating looped video with first and last frame rewinding according to claim 2, characterized in that, The process for determining the directed time-shift dependency includes: Bidirectional time-shift prediction is performed using each sequence in the main residual motion sequence and the candidate residual motion sequence as a preceding sequence to obtain residual sequences corresponding to the two prediction directions. The predicted residual in the forward direction of the subject is less than the reverse prediction residual, and the subject adjacent candidate regions that are temporally related after removing the zero-time-shift component are determined as the accompanying visual regions. A delay term is generated based on the time offset corresponding to the minimum value of the predicted residual in the subject's forward direction. An attenuation term is generated based on the change of the candidate residual motion amplitude relative to the subject's residual motion amplitude after the time offset processing. The delay term and the attenuation term are organized according to the accompanying visual region to form the region response kernel.

4. The method for generating looped video with first and last frame rewinding according to claim 3, characterized in that, The process of generating the reflow video sequence includes: Encode the forward tail frame and the forward first frame to obtain the boundary frame latent space features; According to the delay term in the regional response kernel, the associated state features are mapped to the ring index before the corresponding main state features, and the mapped associated state features are modulated according to the attenuation term to form the asynchronous condition sequence; In the predetermined temporal layer of the video generation model, the subject state features are written into the current time index using the subject region mask, and the mapped accompanying state features are written into the corresponding circular index using the accompanying visual region mask. The asynchronous conditional sequence after writing is then input into the predetermined temporal layer in conjunction with the boundary frame latent space features.

5. The method for generating looped video with first and last frame rewinding according to claim 4, characterized in that, The process of constructing the circular joint state trajectory includes: Extract the subject contour, subject pose, and subject residual motion from each frame of the circular candidate sequence, and update the subject state features; Extract the region morphology, corresponding candidate residual motion, and spatial relationship with the main region from the associated visual region, and update the associated state features; The associated state features are aligned to the time index corresponding to the main state features according to the regional response kernel; The main state features of the same time index, the aligned accompanying state features, and the state direction quantities obtained by the difference between adjacent time indices are combined to form a joint state point, and the joint state points are connected in a circular order to form the circular joint state trajectory.

6. The method for generating looped video with first and last frame rewinding according to claim 5, characterized in that, The process of identifying the overlapping state segments includes: Select consecutive joint state point sequences from the connection end and the loop end of the circular joint state trajectory respectively, and monotonically pair the joint state points in the two sets of consecutive joint state point sequences in a circular order. For each pair, determine whether the state distance between the joint state points meets the state equivalence condition, whether the corresponding state direction quantities are in the same direction, and whether the order of the corresponding associated state features on the corresponding index of the region response kernel is consistent. Pairs that continuously meet the above criteria are extended into candidate overlapping state segments. The candidate overlapping state segments containing the most joint state points are selected at the connection end and the loop end, respectively, and the selected candidate overlapping state segments are combined to form the overlapping state segment.

7. The method for generating looped video with first and last frame rewinding according to claim 6, characterized in that, The folding process of the overlapping state segment includes: Connect the paired joint state points in the overlapping state segment into two paired trajectories, and calculate the state direction deviation between the entrance joint state point and the exit joint state point of the two paired trajectories and the adjacent non-overlapping joint state point. Retain the paired trajectory with the smaller total state direction deviation, and delete the frame corresponding to the other paired trajectory; Connect the preserved paired trajectory with the non-overlapping trajectories on both sides to obtain the folded circular joint state trajectory; The state distance between adjacent joint state points is accumulated along the folded circular joint state trajectory, and the accumulated value is mapped to the cycle time index of the retained frame in a circular order.

8. The method for generating looped video with first and last frame rewinding according to claim 7, characterized in that, The process of generating the intermediate frame includes: Each pair of adjacent retained frames is encoded to obtain frame features, and the bidirectional motion field between adjacent retained frames is calculated; The main interpolation index is determined based on the cyclic time index of two adjacent retained frames, and the main interpolation index is converted into the accompanying interpolation index corresponding to the accompanying visual region according to the delay term and attenuation term in the region response kernel. Based on the main interpolation index and the accompanying interpolation index, bidirectional deformation is performed on the frame features within the range defined by the main region mask and the accompanying visual region mask, respectively. An occlusion confidence map is generated based on the forward and reverse consistency of the bidirectional motion field. The frame features after bidirectional deformation are fused according to the occlusion confidence map, and the fused frame features are decoded into the intermediate frame.

9. A method for generating looped video with first and last frame rewinding according to claim 8, characterized in that, It also includes: dividing the motion phase segment according to the direction change, amplitude change and continuous low amplitude state of the residual motion sequence of the main body, configuring phase identifiers for the motion phase segment, constructing corresponding phase response sub-cores and forming a phase state chain in a circular time sequence; calling the phase response sub-core according to the main body state direction quantity and the phase identifier of the previous time index, constructing the asynchronous condition sequence, limiting the pairing of overlapping state segments and determining the accompanying interpolation index.

10. A system for generating looped video with first and last frame rewinding, characterized in that, The system includes: Forward video generation module: acquires input image, generates forward video sequence from input image, and extracts forward first frame and forward last frame; The accompanying response modeling module performs subject segmentation on the forward video sequence to obtain the subject region and its corresponding mask, and estimates the global motion; it calculates the frame-by-frame motion field and removes the global motion components from the frame-by-frame motion field to obtain the subject residual motion sequence and candidate residual motion sequence; based on the directed time-shift dependency between the subject residual motion sequence and the candidate residual motion sequence, it determines the accompanying visual region and its mask from the adjacent candidate regions of the subject, and constructs the region response kernel based on the time offset and response attenuation; Asynchronous reflow generation module: Extracts main state features and accompanying state features from the forward video sequence; performs a circular shift on the accompanying state features according to the region response kernel to form an asynchronous conditional sequence composed of the main state features and the shifted accompanying state features; using the forward tail frame and the forward first frame as boundaries, injects the asynchronous conditional sequence into the temporal layer of the video generation model according to the main region mask and the accompanying visual region mask to generate a reflow video sequence; Overlapping State Folding Module: The forward video sequence and the reflow video sequence are spliced ​​together to form a circular candidate sequence; a circular joint state trajectory is constructed from the main state features and the accompanying state features aligned with the regional response kernel; overlapping state segments with equivalent states and the same direction of change are identified and folded. Looping Timing Interpolation Module: Configures the looping time index of the retained frames according to the folded circular joint state trajectory, generates intermediate frames according to the looping time index of adjacent retained frames, and outputs looping video.

Images