A method for generating consistent intermediate video frames under keyframe constraints
By establishing bidirectional local states and errors during video generation and updating the comparison benchmark to correct intermediate frames, the problem of discontinuous states in intermediate frames of video is solved, and the temporal order consistency of intermediate frames is achieved.
Patent Information
- Application Number
- CN202611600893.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-16
- Publication Date
- 2026-11-17
AI Technical Summary
In video generation, when multiple intermediate frames are generated between consecutive keyframes, the state changes of the same image object may be discontinuous, leading to problems such as local reverse motion, posture swing, or discontinuous structural changes.
By establishing bidirectional local states and corresponding errors in the same image object in consecutive keyframes, the relative change process is determined, and the comparison benchmark is updated based on the backoff tolerance to correct intermediate frames and maintain state continuity.
This reduces the occurrence of state sequence reversal or local reverse changes in intermediate frames during continuous playback, maintaining the continuity and consistency of multiple intermediate frames in time sequence.
Smart Images

Figure CN123123644A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video generation and video intermediate frame processing technology, specifically to a method for generating consistent video intermediate frames with keyframe constraints. Background Technology
[0002] Video generation and frame interpolation techniques typically require the creation of one or more intermediate frames between known video frames to fill in missing images on the video timeline and obtain a continuous sequence of video frames. In keyframe-constrained video generation scenarios, preceding and following keyframes are usually used to define the start and end points of object state changes within a video time interval. Intermediate frames are then generated based on the image information, motion information, or target time position at both ends. For example, CN115065796A discloses a method for generating intermediate video frames, which constructs an image pyramid using target video frame pairs and combines bidirectional optical flow estimation and pixel synthesis to form intermediate frames layer by layer. CN117793480A discloses a long video generation method based on text input, which inputs consecutive video keyframes into a local frame filling and diffusion stage to generate intermediate frames between adjacent keyframes. These techniques demonstrate that using information from both ends of the video frame or keyframes to form intermediate video frames is an important processing method in the fields of video generation and frame interpolation.
[0003] However, when multiple target intermediate timeframes need to be continuously generated between the preceding and following keyframes, the two keyframes mainly define the start and end range of object state changes, while each target intermediate timeframe mainly defines the position of the intermediate frame on the video timeline. When the same image object has multiple intermediate states from the preceding keyframe state to the following keyframe state that can satisfy the constraints of the two end images, if each intermediate timeframe position is mainly generated based on the reference information from the two end frames, or if only general inter-frame motion estimation, image fusion, or local temporal correlation processing is used, then a reasonable correspondence between a single intermediate frame and the two keyframes does not necessarily mean that the same object state in multiple intermediate frames maintains the same continuous change process according to the video time sequence.
[0004] In the aforementioned scenarios, the object state corresponding to a later intermediate time point may reappear within the state range already traversed by the previous time point, or the object state changes at different intermediate time points may exhibit reversed or repeated order. Therefore, while each intermediate frame may still satisfy the image constraints of the preceding and following keyframes when viewed individually, when played continuously in chronological order, it may manifest as localized reverse motion, pose reversals, or discontinuous changes in contours or structures. Thus, given that the preceding and following keyframes are already determined and multiple intermediate frames need to be formed in between, how to maintain the constraints of the two keyframes while ensuring that the state changes of the same image object at multiple intermediate time points are consistent with the video's temporal order remains a technical problem that needs further resolution in the generation and processing of such intermediate frames. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention provides a method for generating consistent intermediate video frames with keyframe constraints, thereby resolving the problems mentioned in the background section.
[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for generating video intermediate frame consistency with keyframe constraints, comprising: S1. Obtain the previous keyframe, the next keyframe, and the intermediate times of multiple targets in between. Determine the bidirectional corresponding local state and corresponding error in the same image object between the two keyframes. S2. Form initial intermediate frames for the intermediate time of each target, obtain the corresponding local states, determine the relative change process based on the difference between the states corresponding to the previous and subsequent keyframes, and take the starting process corresponding to the previous keyframe as the initial reference. S3. Determine the backoff tolerance based on the corresponding error, and judge each intermediate state in chronological order; update the baseline when the backoff of the current state in the direction of the forward key frame does not exceed the tolerance, and determine the failure location and local failure state when it exceeds the tolerance for the first time. S4. Take the nearest intermediate frame used to update the baseline before the failure position to replace the previous key frame as the front reference, and use the subsequent key frame as the back reference. Re-determine the relative target time of the failure position according to the new front and back reference time and form a corrected intermediate frame. S5. The process of re-determining the correction state is based on the corresponding states of the previous and subsequent keyframes. If the process of reverting to the correction state does not exceed the revert tolerance, the process updates the comparison benchmark of the next target intermediate time and continues to process, forming the final intermediate frame sequence.
[0007] Furthermore, candidate local regions are formed within the same image object in both the preceding and following keyframes; After performing forward matching on the candidate local regions of the previous keyframe, a reverse search is performed. Then, starting from the matched local regions in the subsequent keyframe, matching and return searches are performed in the opposite direction. When the centers of the returned local regions in both directions fall within the coverage area of the corresponding starting local regions, a bidirectional correspondence is determined. The larger value of the local state difference between the returned local regions in both directions and their respective starting local regions is taken as the correspondence error.
[0008] Furthermore, determining the relative change process includes: calculating the difference between the intermediate state and the corresponding state of the previous keyframe, and the difference between the intermediate state and the corresponding state of the subsequent keyframe, respectively. The relative change process is obtained by dividing the difference between the intermediate state and the corresponding state of the previous keyframe by the sum of the differences between the intermediate state and the subsequent keyframe.
[0009] Furthermore, the backoff tolerance is obtained by dividing the corresponding error by the state difference between the corresponding state of the previous keyframe and the corresponding state of the subsequent keyframe. When the state difference is 0 or less than or equal to the corresponding error, the local states corresponding to both sides do not participate in the process rollback judgment.
[0010] Furthermore, when judging according to the intermediate time sequence of the target, for the same bidirectional corresponding local state, the relative change process of the corresponding intermediate state most recently judged by sequence is used as the comparison benchmark. When the current relative change process is lower than the benchmark, the difference between the two is used as the backoff amount. If the backoff amount does not exceed the corresponding backoff tolerance, the current intermediate state is confirmed, and the benchmark is replaced by the current relative change process.
[0011] Furthermore, for each target intermediate time, based on the proportion of the target intermediate time in the time interval of the preceding and following keyframes, the coordinates of the four sides of the circumscribed rectangle of the same image object in the preceding and following keyframes are converted to form an object reference rectangle. Then, a local window side length is extended outward around the object reference rectangle to form a candidate local region, thus forming the object search region. For each bidirectional corresponding local state in the preceding and following keyframes, the center position of the candidate local region is determined according to the fixed arrangement order within the object reference rectangle. The local image content corresponding to each candidate local region is obtained from the object search region. The sum of the local state differences between each candidate local region and the corresponding states in the preceding and following keyframes is calculated. The candidate local region with the smallest sum of differences is taken as the corresponding local state of the target at the intermediate time.
[0012] Furthermore, the re-determination of the relative target time includes: calculating the time difference between the failure location and the preceding reference time, and the total time difference between the following reference time and the preceding reference time, and using the ratio of the former to the latter as the relative target time of the failure location in the new preceding and following reference time intervals.
[0013] Furthermore, after the intermediate frame is formed, the center position of the candidate local region is determined in a fixed order within the object reference rectangle corresponding to the failure position, according to the same candidate local region formation and corresponding local state determination rules as the initial intermediate frame. The local image content corresponding to each candidate local region is obtained from the corresponding object search area, and all corresponding local states participating in the current intermediate frame order judgment are re-obtained. The relative change process is re-determined with reference to the corresponding states of the previous and subsequent keyframes. When the process rollback for all corresponding local states does not exceed its respective rollback tolerance, the intermediate frame is confirmed to be corrected, and the corresponding comparison benchmark is updated according to the relative change process of each corresponding local state.
[0014] Furthermore, at the same failure location, only one front reference replacement and intermediate frame correction are performed; If any reacquired local state still experiences a process rollback exceeding its rollback tolerance, the intermediate frame is not confirmed, the corresponding comparison benchmark is not updated based on the relative changes in the corresponding local states in the intermediate frame, and the intermediate frame sequence confirmation process between the current previous keyframe and the next keyframe is terminated.
[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. By establishing bidirectional corresponding local states and corresponding errors in the same image object in the previous and subsequent keyframes, and updating the comparison benchmark in chronological order according to the relative change process of the intermediate state with respect to the corresponding states of the previous and subsequent keyframes and the backoff tolerance formed by the corresponding errors, it is possible to identify the failure position where the intermediate state has over-tolerance backoff to the state range that has already been passed, thereby reducing the situation where the state order backoff or local reverse change occurs when multiple intermediate frames satisfy the constraints at both ends but are played continuously.
[0016] 2. After determining the failure location, the most recent intermediate frame used to update the comparison benchmark before the failure location is used as the new preceding reference, and the subsequent keyframe is kept as the following reference. The relative target time is re-determined according to the new preceding and following reference times to form the corrected intermediate frame. The corrected state is still re-determined based on the corresponding state of the preceding and following keyframes as a unified reference. This ensures that while the correction generation conditions are adjusted, the state process before and after the correction has a unified comparison caliber, which is beneficial for restoring the continuous connection between the failure location and the confirmed preceding state.
[0017] 3. After the intermediate frame is formed, the relative change process of all corresponding local states involved in the current intermediate frame order judgment is re-acquired and checked against the existing comparison benchmark and backoff tolerance respectively. The intermediate frame is confirmed and the comparison benchmark is updated only when no backoff exceeding the tolerance occurs in all corresponding local states. If there is still a backoff exceeding the tolerance, the subsequent comparison benchmark is not updated and the confirmation process of the current key frame interval is terminated, thereby avoiding the correction state that fails the check from continuing to affect the state order judgment of subsequent intermediate frames. Attached Figure Description
[0018] Figure 1 This is a flowchart illustrating a method for generating consistent intermediate video frames under keyframe constraints. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] Example: Combined with Appendix Figure 1 This embodiment provides a method for generating consistent intermediate video frames with keyframe constraints, including: S1. Obtain the previous keyframe, the next keyframe, and the intermediate times of multiple targets in between. Determine the bidirectional corresponding local state and corresponding error in the same image object between the two keyframes. S2. Form initial intermediate frames for the intermediate time of each target, obtain the corresponding local states, determine the relative change process based on the difference between the states corresponding to the previous and subsequent keyframes, and take the starting process corresponding to the previous keyframe as the initial reference. S3. Determine the backoff tolerance based on the corresponding error, and judge each intermediate state in chronological order; update the baseline when the backoff of the current state in the direction of the forward key frame does not exceed the tolerance, and determine the failure location and local failure state when it exceeds the tolerance for the first time. S4. Take the nearest intermediate frame used to update the baseline before the failure position to replace the previous key frame as the front reference, and use the subsequent key frame as the back reference. Re-determine the relative target time of the failure position according to the new front and back reference time and form a corrected intermediate frame. S5. The process of re-determining the correction state is based on the corresponding states of the previous and subsequent keyframes. If the process of reverting to the correction state does not exceed the revert tolerance, the process updates the comparison benchmark of the next target intermediate time and continues to process, forming the final intermediate frame sequence.
[0021] This implementation uses a video editing workstation or a video generation server as the implementation platform. The computing device can read keyframe images and timeline information from a video project, establish cross-frame correspondences for the same object in an image, and generate a video frame corresponding to the target time position given two reference frames and the target time position. The processor, memory, graphics processing resources, and file read / write interface of the computing device can use general-purpose computing resources capable of performing video image reading, local image comparison, and video frame generation. Power supply, network communication, file encapsulation, video playback, and display are not involved in the state order judgment and correction processing in this method; they are only considered as normal operating conditions.
[0022] Both the preceding and following keyframes are image frames that are already determined within the current video time interval and remain unchanged during this intermediate frame processing. The preceding keyframe is located at the beginning of this time interval, and the following keyframe is located at the end of this time interval. Multiple target intermediate times lie between these two and have a unique chronological order according to the video timeline. Internal processing preferentially uses frame numbers to represent time positions; when the video project uses millisecond timestamps, the timestamps are first converted to the corresponding frame numbers based on the project's frame rate, or the timestamps are directly retained for relative time conversion. Both representations maintain the same standard within the same keyframe interval.
[0023] In this embodiment, the same image object refers to image regions in the previous and subsequent keyframes that are considered to belong to the same persistent object, such as the same person, vehicle, mechanical part, or other subject that persists between two keyframes. Object association itself can be obtained from existing object trajectories in the video project, object identifiers determined during manual editing, or conventional image object matching processing; object association is only used to limit the object regions to be compared and does not participate in the calculation of subsequent state relative progress and state order invalidation. Image regions without established same object associations are not used as state comparison objects for the current object.
[0024] To ensure that the local states participating in the comparison have repeatable technical meaning, multiple candidate local regions are formed within the same image object in the previous and subsequent keyframes. The main implementation method uses overlapping local windows to form candidate local regions: 10% of the shorter side of the object's bounding rectangle is used as the representative side length of the local window. The center-to-center interval between adjacent candidate local regions, both horizontally and vertically, is half the window's side length. Starting from the upper left corner of the object's bounding rectangle, local windows are moved sequentially from left to right within the same row. After completing one row, they move down to the next row, continuing from left to right until the area within the object's bounding rectangle that can completely accommodate a local window is covered. Only local windows that completely fall within the object's bounding rectangle are retained. Each candidate local region records its center position, local width, local height, and corresponding local image block. The center position and local size are normalized according to the width and height of the current object's bounding rectangle, and the local image block is resampled to the same pixel size.
[0025] For a candidate local region in the previous keyframe, the local state difference is calculated among the candidate local regions of the same image object in the subsequent keyframe, and the candidate local region with the smallest local state difference is selected as the forward matching region. Starting from the forward matching region, a reverse search is performed in the candidate local regions of the same image object in the previous keyframe according to the same rules to form the first returned local region. When the center position of the first returned local region falls within the image range covered by the forward matching starting local region, the return relationship on the previous keyframe side is determined to be valid.
[0026] Using the forward matching region as the starting local region on the later keyframe side, matching is performed in the opposite direction from the later keyframe to the previous keyframe and then back to the later keyframe, forming a second returning local region. When the center of the second returning local region falls within the image area covered by the starting local region on the later keyframe side, the returning relationship on the later keyframe side is determined to be valid. When the returning relationships on both the previous and later keyframe sides are valid, the starting local regions on both sides are determined as bidirectional corresponding object state units.
[0027] "Return" indicates that the matching result still falls within the local spatial range corresponding to the initial local state, and does not require the returned local region to have the exact same center position, size, and image content as the initial candidate local region. Therefore, the returned local region can form a non-zero local state difference with the initial local region, and this difference is used to form the corresponding error of the object state unit in the subsequent process.
[0028] In addition to the aforementioned overlapping local window method, local feature localization processing that can output local center position, local scale, and local image block can also be used to form candidate local regions. When other candidate region formation methods are used, the subsequent bidirectional correspondence, local state difference, correspondence error, and state relative process are still executed according to the same processing caliber.
[0029] This implementation standardizes the direction and process uniformly: the direction of change is from the previous keyframe state to the next keyframe state. The relative change process of the corresponding state in the previous keyframe is recorded as the starting process, and the relative change process of the corresponding state in the next keyframe is recorded as the ending process. Changes from intermediate states towards the next keyframe are considered as progress increases, and reverting to the previous keyframe state are considered as progress regresses. This directional standard remains unchanged throughout the entire processing interval defined by the current previous and next keyframes. Even if the previous generation reference is changed during correction in an intermediate frame, the corrected state still returns to the unified reference relationship composed of the original previous and next keyframes for verification.
[0030] After acquiring a pair of determined previous and subsequent keyframes, the computing device establishes a stable local state correspondence within the same image object in both frames and uses the deviation generated by forward and reverse matching to determine the degree of uncertainty in the correspondence of this local state. Initial intermediate frames are formed for each target intermediate time, and the local states are placed in a unified state reference between the previous and subsequent keyframes to determine the relative change process. The computing device confirms the intermediate states frame by frame in video time sequence. The current state is only used to update subsequent comparison benchmarks if no over-tolerance backoff occurs to the already confirmed process. When the first over-tolerance backoff occurs, the nearest confirmed intermediate frame before the failure position is used to tighten the preceding generation reference for that position, while the subsequent keyframe remains as the following reference. The relative target time of the failure position is re-determined with the new reference interval. Corrected intermediate frames are returned to the original unified keyframe reference for verification. If the verification passes, subsequent comparison benchmarks continue to be updated; if the verification still fails, the confirmation process for the current keyframe interval ends.
[0031] The computing device reads the current preceding keyframe, the current following keyframe, and all target intermediate times in between from the video project's timeline. The image data of the preceding and following keyframes retains their original pixel dimensions, color space, and time signatures. To avoid differences in state caused by different input specifications leading to variations in computational accuracy, images compared within the current keyframe interval are standardized to the same pixel dimensions and color representation; this standardization process only changes the data representation, not the image content or time position.
[0032] After locating the same image object in the previous and subsequent keyframes, a local coordinate system is established using the object's bounding rectangle. The top-left corner of the bounding rectangle serves as the starting point for the local coordinates. The horizontal and vertical directions are normalized using the width and height of the bounding rectangle, respectively. Therefore, even if the object as a whole is translated within the frame, the relative positions within the object can still be compared. The object state unit stores the normalized x-coordinate and y-coordinate of its local center, the normalized width of the local region relative to the object's width, the normalized height relative to the object's height, and the resampled local image patch. The local image patch is used to represent contour and texture variations and does not require the object as a whole to remain rigid.
[0033] During forward matching, for a candidate local region in the previous keyframe, the local state difference is calculated for all candidate local regions of the same image object in the subsequent keyframe, and the candidate local region with the smallest local state difference is selected as the forward matching region. The local state difference is formed by the difference in normalized center position, the difference in normalized size, and the difference in resampled local image patches.
[0034] Starting from the forward matching region in the subsequent keyframe, a reverse search is performed in the candidate local regions of the same image object in the previous keyframe according to the same rules to obtain the first returned local region. When the center position of the first returned local region is within the image range covered by the original candidate local region of the previous keyframe, the forward-reverse return relationship is confirmed to be valid. In the opposite direction, starting from the corresponding candidate local region in the subsequent keyframe, backward-forward return matching is performed, and when the center position of the obtained second returned local region is still within the range covered by the original candidate local region of the subsequent keyframe, the return relationship in the other direction is confirmed to be valid. When the return relationships in both directions are valid, the candidate local regions of the previous and subsequent keyframes are determined to be bidirectional corresponding object state units.
[0035] The corresponding error is formed by the return deviations in two directions. For the previous keyframe side, the local state difference between the original starting local region of the previous keyframe and the first returned local region is calculated according to a unified calculation method for local state differences, resulting in the first return deviation. For the subsequent keyframe side, the local state difference between the original starting local region of the subsequent keyframe and the second returned local region is calculated, resulting in the second return deviation. The larger of the first and second return deviations is taken as the corresponding error of the current object state unit.
[0036] Since the returned local region only requires it to fall within the local state range represented by the initial local region, and does not require it to completely overlap with the initial candidate local region, the return deviation can reflect the actual return error of local position, local scale, and image content during the bidirectional matching process. The subsequent backoff tolerance is formed based on the difference between this corresponding error and the actual state of the corresponding states in the preceding and following keyframes.
[0037] When a candidate local region cannot satisfy the return relationships in both directions—that is, the center of the first returned local region does not fall within the coverage area of the original preceding keyframe's starting local region, or the center of the second returned local region does not fall within the coverage area of the original following keyframe's starting local region—this candidate local region will not be included in the state order determination of the current keyframe interval. This process is only used to ensure that the object state units participating in the state order determination have definite preceding and following state references, and does not change other object state units that have already formed stable bidirectional correspondences. When an image object has multiple stable corresponding local regions, each object state unit establishes its corresponding error and forms its subsequent state relative process separately, without pre-combining multiple local states into a single object score.
[0038] For local regions where the state remains largely unchanged between previous and subsequent keyframes, if the state difference between the corresponding local states before and after is 0, or less than or equal to the corresponding error of the current object state unit, then that object state unit will not participate in the process rollback judgment, but its corresponding image content will still be retained in the video frame. When some object state units in the same image object do not meet the conditions for participating in the process rollback judgment, the remaining object state units that meet the conditions continue to form a state-relative process and participate in the time sequence judgment. When no object state units in the same image object meet the participation conditions, that image object will not be used as the object for the state sequence judgment of the current keyframe interval. When there is at least one object state unit in the current keyframe interval that can form a stable bidirectional correspondence and whose states before and after are distinguishable, the subsequent state sequence confirmation is performed on the object state units that meet the conditions.
[0039] The initial intermediate frames corresponding to each target intermediate time are formed by the same video generation and processing interface. This interface receives at least the preceding reference frame, the following reference frame, and the relative position of the target time between the two reference frames, and outputs an intermediate image corresponding to the target time. In the initial processing, the preceding reference frame is the preceding keyframe, and the following reference frame is the following keyframe. The internal implementation of the intermediate frame generation process can adopt existing video interpolation or video generation processes that can form target time images based on the two reference images. Its internal network structure or motion estimation method does not determine the state order judgment result of this method. This implementation fixes its input and output interfaces and the calling method within the same keyframe interval, so that different target intermediate times obtain corresponding initial intermediate frames.
[0040] In a representative implementation, the video frame rate is 25 frames per second, the first keyframe is at 0ms, the second keyframe is at 200ms, and four intermediate frames are needed at 40ms, 80ms, 120ms, and 160ms. Initially, the relative positions of each target time with respect to the interval between the first and second keyframes are 0.2, 0.4, 0.6, and 0.8, respectively. These values are only for illustrating the continuous operation process; the actual target intermediate times are determined by the video project's frame rate and the positions of the frames to be supplemented.
[0041] To obtain the local state corresponding to the same image object in the initial intermediate frame and the corrected intermediate frame, a corresponding object search region is pre-formed for each target intermediate time. The main implementation method uses the bounding rectangle of the image object in the previous keyframe and the bounding rectangle of the same image object in the subsequent keyframe as the two object regions at both ends. The proportion of the time difference between the current target intermediate time and the previous keyframe time to the total time difference between the previous and subsequent keyframes is calculated. For the left, upper, right, and lower boundaries of the object's bounding rectangle, the difference between the corresponding boundary coordinates in the subsequent keyframe and the corresponding boundary coordinates in the previous keyframe is calculated. This difference is multiplied by the time proportion corresponding to the current target intermediate time and then added to the corresponding boundary coordinates in the previous keyframe to obtain the left, upper, right, and lower boundaries corresponding to the current target intermediate time. These four boundaries form the object reference rectangle for the current target intermediate time.
[0042] For the current target intermediate time, 10% of the shorter side of the object reference rectangle is used as the window side length of the candidate local region for the current intermediate frame. The center-to-center interval between adjacent candidate local regions horizontally and vertically is half the window side length. One window side length is extended outwards from each of the four sides of the object reference rectangle, and any portion exceeding the current video frame boundary is cropped within the image boundary, forming the object search region corresponding to the current target intermediate time. The center position of the candidate local regions is determined according to a fixed arrangement order within the object reference rectangle. The image content corresponding to the local window is obtained from the object search region. The object search region ensures that candidate local regions located near the edge of the object reference rectangle can obtain complete local image patches without changing the position normalization reference of the candidate local regions.
[0043] After the initial intermediate frame is formed, the center position of the candidate local region is determined according to the fixed arrangement order within the reference rectangle of the target intermediate time. The image content corresponding to the local window is obtained from the object search area, and then the local state difference is compared with the corresponding object state unit of the previous keyframe and the next keyframe.
[0044] When the intermediate frame at the initial state sequence failure location is re-formed, although the corrected generation uses the confirmed intermediate frame closest to the failure location as the new preceding generation reference, the object search area in the corrected intermediate frame is still formed according to the original preceding keyframe, the original following keyframe, and the original target time of the failure location. Therefore, the local states in the initial intermediate frame and the corrected intermediate frame maintain the same object space caliber, and the corrected state can still be compared against the unified state reference composed of the original preceding and following keyframes.
[0045] After an initial intermediate frame is formed, the computing device determines the center position of candidate local regions according to a fixed arrangement order within the reference rectangle corresponding to the current target intermediate time, and obtains the local image content of each candidate local region from the corresponding object search area. For each candidate local region, the difference between it and the corresponding state of the previous keyframe and the corresponding state of the next keyframe is calculated according to the unified local state difference caliber, and the two differences are added together; the candidate local region with the smallest sum of differences is taken as the current intermediate state. When the sum of differences of multiple candidate local regions is the same, the distance from the center of each candidate local region to the line connecting the centers of the corresponding states of the previous and next keyframes is compared, and the one with the smallest distance is taken as the current intermediate state. When the sum of differences of multiple candidate local regions is the same, and the distance from the center of each candidate local region to the line connecting the centers of the corresponding states of the previous and next keyframes is still the same, the candidate local region at the front of the line is selected as the current intermediate state according to the fixed arrangement order of moving downwards after completing a row. The fixed arrangement order is only used to eliminate the selection uncertainty under completely identical comparison results and does not participate in the state relative process or backtracking judgment. The determined intermediate state saves the local center position, local size, and local image patch required for comparison with the two end states.
[0046] The relative state progression is formed by the differences between the two endpoints. For an intermediate state, the preceding state difference between the intermediate state and the corresponding state of the previous keyframe is first calculated according to the unified local state difference rules. Then, the following state difference between the intermediate state and the corresponding state of the subsequent keyframe is calculated. Dividing the preceding state difference by the sum of the preceding and following state differences yields a relative state progression between 0 and 1. The closer the current intermediate state is to the previous keyframe state, the smaller the preceding state difference, and the closer the relative state progression is to 0; the closer the current intermediate state is to the subsequent keyframe state, the smaller the following state difference, and the closer the relative state progression is to 1. This progression only represents the relative position of the current local state between the determined endpoints; it does not imply that the motion must be uniform, nor does it require the relative progression to be equal to the target time ratio.
[0047] When the sum of the differences between the current state and the subsequent state is 0, it indicates that the intermediate state and the states at both ends are indistinguishable under the current state description. Since object state units where the differences between the preceding and following states are 0 or close to the corresponding error are no longer involved in process rollback, their state relative to the process will not be used as the basis for order determination in the normal processing chain. If the sum of the differences between the two sides is still 0 during numerical calculation, the object state unit is marked as being at its current time position and will not participate in order determination; the remaining object state units continue processing.
[0048] The state relative process of an object state unit always uses the original previous keyframe state and the original subsequent keyframe state as its two endpoints for reference. If the generation reference of the failed frame is changed due to the failure of the initial state sequence, this change only affects the generation input of the corrected intermediate frame and does not change the reference of the state relative process. Therefore, the state relative processes corresponding to the initial intermediate frame and the corrected intermediate frame are both located within the same 0 to 1 process scope, and can be directly compared sequentially.
[0049] When an image object has multiple object state units, each object state unit obtains its own relative state progress. Whether the current intermediate frame can update the subsequent comparison benchmark is based on the sequential judgment of each participating object state unit. As soon as an object state unit experiences a process rollback exceeding its own rollback tolerance for the first time, it can be determined that there is a state sequence failure in the intermediate time of the target, and the object state unit that experienced the rollback is recorded. The progress of different object state units is not directly compared with each other because the change magnitude and corresponding error of each local structure from the previous keyframe to the subsequent keyframe can be different.
[0050] The backoff tolerance for each object state unit is calculated from its bidirectional correspondence error to the state relative progress caliber. The computing device has already obtained the state differences between the preceding and following keyframes and the bidirectional correspondence error for this object state unit. The progress uncertainty ratio of this object state unit is obtained by dividing the bidirectional correspondence error by the state differences between the preceding and following keyframes; when this ratio is greater than 1, it is taken as 1. The progress uncertainty ratio is directly used as the backoff tolerance for the current object state unit. Since the backoff tolerance originates from the forward and reverse matching residuals of the same state unit itself, the tolerance is smaller when the local structure positioning is stable, and the tolerance increases accordingly when the local structure correspondence is uncertain.
[0051] For example, if the state difference between the preceding and following keyframes of a certain local state unit is 0.50, and the largest return difference obtained by corresponding values in both directions is 0.03, then the backoff tolerance is 0.06. This 0.06 is not an empirical threshold manually selected for the current video, but rather is formed by the corresponding error of the local state unit itself and the actual change amplitude at both ends. If the state difference between the two ends of another state unit is 0.25 and the return difference is 0.025, then its backoff tolerance is 0.10. Different state units therefore have their own judgment boundaries.
[0052] The state sequence judgment is performed sequentially from the target intermediate time to the next. The state relative progress of the state corresponding to the previous keyframe is 0, which is used to establish the initial comparison caliber. The state relative progress of the first target intermediate time is compared with 0; since the state relative progress itself is within the range of 0 to 1, the first intermediate state does not constitute a negative backtracking in the direction of the previous keyframe under normal circumstances. After the first intermediate state passes the judgment, its state relative progress becomes the comparison benchmark for the next target intermediate time.
[0053] When processing the intermediate time of the next target, the relative progress of the most recently determined state for the same object state unit is read, and the difference between the benchmark and the current state relative progress is calculated. When the current state relative progress is greater than or equal to the benchmark, the current state does not backtrack in the direction of the previous keyframe and is directly determined through sequential judgment; when the current state relative progress is less than the benchmark, the difference represents the actual backtracking amount. When the actual backtracking amount is less than or equal to the backtracking tolerance of the current object state unit, the change is considered to be within the uncertainty corresponding to the local state, the current intermediate state is still determined through sequential judgment, and the original benchmark is replaced by the current state relative progress. Only when the actual backtracking amount is greater than the backtracking tolerance is the current object state unit determined to have experienced a state sequence failure.
[0054] The processing rule of updating the comparison benchmark after sequentially judging the current intermediate state ensures that the comparison benchmark always corresponds to the intermediate state that is closest to the current target intermediate time in the video time sequence and has been confirmed not to have experienced excessive backtracking, rather than always comparing it with previous keyframes. This allows for the identification of situations where a single frame is close to both keyframes but backtracks to a previously visited state in a continuous sequence. This judgment does not require multiple states to be strictly arithmetic progressions, nor does it require the current state to be equal to the target time ratio relative to the progress; as long as the progress has not backtracked to a confirmed state beyond its corresponding error tolerance range, the actual nonlinear change of the current state is maintained.
[0055] When an intermediate frame contains multiple object state units participating in the judgment, the computing device reads their respective comparison benchmarks and backoff tolerances according to a fixed correspondence between the object state units. If none of the object state units participating in the judgment have experienced over-tolerance backoff, the intermediate frame is considered a confirmed intermediate frame, and the current state of each object state unit relative to the process replaces its corresponding comparison benchmark. If at least one object state unit experiences over-tolerance backoff, the target intermediate time is determined as the state sequence failure position, and all object state units that experienced over-tolerance backoff are recorded; the intermediate frame as a whole is not confirmed, and no object state unit in the current intermediate frame is used to update the comparison benchmark. Each object state unit continues to maintain the comparison benchmark corresponding to the most recently confirmed intermediate frame before its failure position.
[0056] The processing proceeds from front to back along the timeline. Therefore, the target intermediate time that first meets the over-tolerance backoff condition naturally becomes the first state sequence failure position. The state relative to the progress of the first target intermediate time is in the range of 0 to 1, and its comparison benchmark is the starting process of the previous keyframe (0). Therefore, as long as the object's state unit participates in the sequence judgment, the first intermediate state will not produce a negative backoff relative to the starting process in the direction of the previous keyframe. After the first intermediate frame passes the judgment, it can form the first confirmed intermediate state. When the over-tolerance backoff occurs for the first time subsequently, there has already been at least one intermediate frame that has passed the sequence judgment. Among them, the one whose time position is closest to the failure position is the confirmed intermediate frame closest to the failure position. Its image data, time position, and the confirmed state relative to the progress are used together to redefine the generation conditions of the failure position.
[0057] In practical applications, the number and interval of target intermediate times are determined by the video generation task. State order determination only uses the existing target intermediate time order, without creating additional detection time points or resampling the timeline. Updates to the comparison benchmark are only stored in the runtime logs required during the current keyframe interval processing. These runtime logs include at least the object state unit identifier, the most recently confirmed state's relative progress, and the corresponding intermediate frame time position; no separate database or cross-project state system is required.
[0058] After the initial state sequence failure location is determined, the computing device reads the nearest confirmed intermediate frame preceding that location and uses this intermediate frame as the preceding reference frame for correction generation. The following reference frame still uses the original following keyframe within the current keyframe interval. This process does not change the status of the preceding and following keyframes as unified references for the state relative process; it only changes the generation input range when the image corresponding to the failure location is re-formed.
[0059] The basis for the preceding reference replacement comes from the already completed sequence confirmation. While ordinary recursive generation can directly use any already generated intermediate frame as subsequent input, this implementation only uses the intermediate frame closest to the failure location that has already passed the over-tolerance backoff check. This confirmed intermediate frame is later than the original preceding keyframe, therefore the reference time interval formed from this confirmed intermediate frame to the original following keyframe is closer to the failure location, and the preceding reference already retains the most recently confirmed local state of the object in the video time sequence.
[0060] The relative target time is recalculated based on the new reference times on both sides. Let the actual time of the confirmed intermediate frame be the front reference time, and the actual time of the original subsequent keyframe be the back reference time. The actual time of the initial state sequence failure position remains unchanged. The computing device first calculates the time difference between the failure position and the front reference time, then calculates the total time difference between the back reference time and the front reference time. The former is divided by the latter, and the ratio between 0 and 1 is taken as the relative target time of the failure position in the new reference interval. This conversion only changes the relative time coordinates used by the generation interface; it does not change the actual time of the failure position on the original video timeline.
[0061] Typically, the previous keyframe is at 0ms, the next keyframe is at 200ms, the intermediate frame at 80ms has already been determined by sequence, and the state sequence failure first occurs at 120ms. When regenerating the 120ms position, the confirmed intermediate frame at 80ms is used as the previous reference, and the next keyframe at 200ms is used as the next reference. The distance from 120ms to 80ms is 40ms, and the total interval from 80ms to 200ms is 120ms. Therefore, the new relative target time is the ratio of 40ms to 120ms, approximately 0.333. The original 0.6 relative to the 0ms to 200ms interval is no longer used as the target time parameter for this modified generation interface.
[0062] The new preceding reference frame, the following reference frame, and the redefined relative target time constitute the complete input for generating the corrected intermediate frame. Failure state units are used to indicate the local state of objects requiring focused inspection, but do not require the intermediate frame generation process to use a specific local patching algorithm. In the main implementation path, the same video frame generation interface used when the initial intermediate frame was formed can be directly used to reconstruct the complete intermediate frame corresponding to the target time using the new preceding reference, the original following keyframe, and the new relative target time. Using the same generation interface avoids introducing new generation algorithms into the core processing chain.
[0063] When generating resources allows and it is desirable to minimize changes in other correctly defined areas, the image region corresponding to the failed state unit can be used as the primary area for reconstruction while maintaining the original image content of other regions. This approach further limits the scope of correction without altering the core relationship of the new front reference, the fixed rear reference, and the target time reset. Regardless of whether full-frame reconstruction or limited local region reconstruction is used, the corrected state after generation must be returned to the state reference composed of the original front keyframe and the original rear keyframe to recalculate the relative state process.
[0064] After the corrected intermediate frame is formed, the computing device determines the center position of the candidate local regions in a fixed order within the object reference rectangle corresponding to the failure location, according to the same candidate local region formation and local state acquisition rules as the initial intermediate frame. It then obtains the local image content of each candidate local region from the corresponding object search area and re-acquires the local states corresponding to all object state units involved in the current intermediate frame's order determination. The local state acquisition rules are consistent with those in the initial intermediate frame, using the same object reference rectangle, object search area, local state difference caliber, and dual-end state reference. The failed object state unit is used to verify whether the original state order failure has been eliminated. The remaining object state units involved in the determination are simultaneously re-acquired to confirm that no new over-tolerance rollback has occurred in the entire corrected intermediate frame due to regeneration.
[0065] The relative progress of the corrected state is still formed by the state differences between the corrected state and the corresponding states of the original previous keyframe and the original subsequent keyframe. The confirmed intermediate frame used during the correction generation is only a new forward generation reference and is not used as the starting point of the relative progress of the state. Therefore, the relative progress of the state before and after the correction is in the same state reference at both ends, and can be directly compared with the comparison benchmark of the most recent confirmed intermediate state before the failure position.
[0066] The computing device reads the relative progress of the state of each participating object state unit in the most recently confirmed intermediate frame before the failure location, and verifies and corrects the corresponding state in the intermediate frame according to the same directional caliber as the initial judgment. If the corrected relative progress of the state is greater than or equal to the corresponding comparison benchmark, it directly passes the sequential verification; if the corrected relative progress of the state is less than the comparison benchmark, the difference between the comparison benchmark and the corrected relative progress of the state is calculated and compared with the existing backoff tolerance of the object state unit. If the backoff amount does not exceed the backoff tolerance, it passes the sequential verification. When all participating object state units pass the verification, the corrected intermediate frame passes the sequential verification.
[0067] After passing the sequential check, the corrected intermediate frame is confirmed as a valid frame for the current target intermediate time. The relative state progress of each participating state unit in the corrected intermediate frame replaces the corresponding comparison reference used before the failure, and is used to determine the relative state progress of the next target intermediate time. The original failed intermediate frame no longer participates in subsequent state sequence checks. Intermediate frames that have passed the sequence check and are located before the failure position retain their original confirmation results and their relative state progress is not recalculated.
[0068] If any object state unit involved in the judgment in the corrected intermediate frame still experiences a process rollback exceeding the corresponding rollback tolerance, the corrected intermediate frame is not confirmed. The state of each object state unit relative to the process is not used to update the comparison benchmark for subsequent target intermediate times, and the intermediate frame sequence confirmation process between the current previous keyframe and the next keyframe is terminated. The current keyframe interval does not form a final confirmed complete intermediate frame sequence. A first-time state sequence failure location only performs one previous reference replacement and correction generation; if the corrected intermediate frame passes the verification, processing continues; if the corrected intermediate frame fails the verification, the confirmation process for the current keyframe interval ends.
[0069] When the corrected state passes verification and processing of subsequent target intermediate times continues, the subsequent intermediate frames still use their original initial generation results for state order determination, and the comparison benchmark has been updated to reflect the corrected state relative to the process. If a later target intermediate time experiences an over-tolerance rollback, it constitutes a new first unprocessed failure location relative to the updated comparison benchmark. The processing procedure continues to verify each intermediate frame in chronological order and performs the same preceding reference replacement and correction processing at the new failure location; for any case where the correction result still fails, the current keyframe interval is terminated according to the failure termination relationship.
[0070] The final intermediate frame sequence is formed by arranging the previous keyframe, all initial intermediate frames that have passed the state order judgment, corrected intermediate frames that have passed the correction check, and the subsequent keyframe in the original video time order. Whether the intermediate frames have been corrected or not does not change their original target time. Video encoding or file encapsulation only performs normal output on the already formed frame sequence, without changing the object's state relative to the process, comparison benchmark, or state order judgment results.
[0071] The following video clip illustrates a complete process using a person's right arm changing from a drooping to an upward position. This implementation clip demonstrates how the described rules are executed continuously and does not imply that this method is only applicable to human motion. The video frame rate is 25 frames per second, with the first keyframe at 0ms and the second keyframe at 200ms. Intermediate frames at 40ms, 80ms, 120ms, and 160ms need to be generated. The same person object can be located in both the first and second keyframes. After the candidate local regions near the outline of the person's right forearm complete the return matching in both directions, the center positions of the two returned local regions are still within the coverage area of the corresponding starting local regions. Therefore, the local regions of the first and second keyframes are determined as bidirectional corresponding object state units and are retained for state order determination.
[0072] The local state difference of this object state unit between the previous and subsequent keyframes is 0.50. After forward matching of the starting local region of the right forearm in the previous keyframe to the corresponding local region in the subsequent keyframe, a reverse search is performed. The center of the first returned local region is still within the coverage area of the original starting local region of the right forearm in the previous keyframe. The return deviation between the original starting local region and the first returned local region is calculated to be 0.02 according to the unified local state difference caliber. Then, a reverse return matching is performed starting from the corresponding local region of the subsequent keyframe. The center of the second returned local region is still within the coverage area of the original local region of the subsequent keyframe, and its return deviation is 0.03. Since the return relationships in both directions are valid, the larger value of 0.03 is taken as the corresponding error of this right forearm object state unit. Dividing the corresponding error of 0.03 by the previous and subsequent state difference of 0.50 yields a backtracking tolerance of 0.06. This backtracking tolerance is only used for this right forearm object state unit.
[0073] The computing device uses the previous keyframe, the next keyframe, and four target intermediate times to form initial intermediate frames. The local state corresponding to the right forearm is extracted from the four initial intermediate frames, and the local state differences between the right forearm state in the previous keyframe and the right forearm state in the next keyframe are calculated. The relative state progression is formed according to the proportion of the difference in the previous state to the sum of the differences in the states on both sides, resulting in a ratio of 0.18 for the 40ms position, 0.42 for the 80ms position, 0.31 for the 120ms position, and 0.82 for the 160ms position.
[0074] At the 40ms position, the state relative to the process 0.18 has not rolled back from the starting process 0 of the previous keyframe, and 0.18 becomes the subsequent comparison benchmark. At the 80ms position, the state relative to the process 0.42 is greater than 0.18, and the comparison benchmark is updated to 0.42 through sequential judgment. At the 120ms position, the state relative to the process 0.31 is less than 0.42, and the actual rollback amount is 0.11. Since 0.11 is greater than the rollback tolerance of 0.06 for this object's state unit, the 120ms position is determined as the initial state sequence failure position, and the right forearm object's state unit is determined as the local state that has failed.
[0075] The nearest intermediate frame to the failure position that has already passed the sequence check is the intermediate frame at the 80ms position. The computing device reads this intermediate frame as the preceding reference for the correction generation, and continues to use the following keyframe at the 200ms position as the following reference. The relative target time of the 120ms position in the new reference interval is determined to be approximately 0.333 based on the ratio of 40ms to 120ms. Using the confirmed 80ms intermediate frame, the following 200ms keyframe, and the relative target time of 0.333, the same video frame generation interface is called again to obtain the corrected intermediate frame at the 120ms position.
[0076] The local state of the right forearm in the corrected intermediate frame is compared with the right forearm states in the keyframe 0ms ago and 200ms later, yielding a relative progress of 0.57. Since 0.57 is greater than the baseline of 0.42 before failure, there is no progress rollback in the direction of the previous keyframe. The corrected intermediate frame is checked sequentially, and the baseline is updated to 0.57. The relative progress of the right forearm state in the original initial intermediate frame at the 160ms position is 0.82, which is greater than 0.57. The 160ms position is determined sequentially. The current keyframe interval ultimately uses the 40ms initial intermediate frame, the 80ms initial intermediate frame, the 120ms corrected intermediate frame, and the 160ms initial intermediate frame.
[0077] Within the same implementation segment, if the relative progress of the right forearm state corresponding to the newly formed 120ms corrected intermediate frame is 0.34, then the backoff amount relative to the 0.42 comparison benchmark is 0.08, which is still greater than the 0.06 backoff tolerance. This corrected intermediate frame is not confirmed, 0.34 does not replace the 0.42 comparison benchmark, and the 160ms position is no longer included in the final confirmation process of the current keyframe interval. The intermediate frame sequence confirmation ends for the current 0ms to 200ms keyframe interval. This failed branch does not increase the number of second corrections, nor does it generate a new temporary judgment state.
[0078] In the above implementation segment, 0.18, 0.42, 0.31, 0.82, and 0.57 are representative values used to illustrate the process relationship. The actual state relative to the process is calculated from the local state differences in the current video image; the 0.06 backoff tolerance is also formed by the bidirectional correspondence error of the current object state unit and the difference between the previous and subsequent states, rather than using 0.06 fixedly in different videos.
[0079] The target intermediate time is directly obtained from the video timeline and the frame positions that need to be supplemented. When the current video uses a fixed frame rate, the frame number can be used as the time position; when using a variable frame rate or editing the timeline, millisecond or higher precision timestamps can be used. The relative target time is always formed by the actual ratio between the target time and the reference times generated on both sides of the current time, so there is no need to set a separate time interpolation threshold.
[0080] The local region size of the object state unit is obtained based on the pixel size of the position and size normalized reference rectangle of the corresponding video frame. In the main implementation path, the candidate local region adopts a square local window; in the previous keyframe and the next keyframe, the window side length is 10% of the shorter side of the object's bounding rectangle of the current image object; in the initial intermediate frame and the corrected intermediate frame, the window side length is 10% of the shorter side of the object's reference rectangle corresponding to the current target intermediate time; the center interval between adjacent candidate local regions in both the horizontal and vertical directions is 1 / 2 of the corresponding window side length, so that adjacent candidate regions remain overlapping. This value is used to give the main implementation method that can be directly reproduced; while ensuring that the candidate local regions can represent the same local structure, the window side length can also be adjusted within the range of 5% to 15% of the shorter side of the corresponding position and size normalized reference rectangle. Once the window size ratio and center spacing ratio are determined within the current keyframe interval, the previous keyframe, the next keyframe, the initial intermediate frame, and the revised intermediate frame all maintain the same window size ratio and center spacing ratio; the center position of the candidate local region and the acquisition of the local image content in the initial intermediate frame and the revised intermediate frame are still performed according to the aforementioned relationship between the object reference rectangle and the object search region.
[0081] The main implementation caliber for local state differences is the arithmetic mean of normalized position difference, normalized local size difference, and resampled image patch difference. All three components are converted to the range of 0 to 1 before averaging, ensuring that data of different dimensions can be compared within the same difference value. If the local state of a certain implementation object mainly manifests as contour changes, the contour pixel distribution can be used instead of the original brightness distribution in the local image patch difference; as long as the previous keyframe, subsequent keyframe, initial intermediate frame, and corrected intermediate frame all maintain the same state difference caliber, the relative progress of the state and the backtracking judgment relationship remain unchanged.
[0082] The bidirectional correspondence error is directly obtained from the difference between the forward and reverse return values, without using an independent empirical threshold. The backoff tolerance is formed by the ratio of the bidirectional correspondence error to the actual state difference between the preceding and following keyframes. Therefore, it is simultaneously related to the matching stability of the local state and the actual change magnitude of that state within the current keyframe interval. When the state changes at both ends are large and the matching is stable, the backoff tolerance is small; when the local state correspondence has a large uncertainty, the backoff tolerance increases to avoid directly judging the matching noise as a state sequence failure.
[0083] When the bidirectional correspondence error is equal to or greater than the difference between the states of the preceding and following keyframes, the object state unit does not enter the process backtracking judgment according to the aforementioned participation conditions. Therefore, all object state units that actually participate in the state sequence judgment have a backtracking tolerance of less than 1. Other stable state units continue to participate in the processing of the current image object, and this processing does not require additional independent manual scoring or extra empirical thresholds.
[0084] This method can process multiple state units within a single image object simultaneously, or multiple persistent objects within the same keyframe interval. Each object state unit maintains an independent bidirectional correspondence, correspondence error, backoff tolerance, and relative progress of the most recently confirmed state. If any state unit involved in the judgment at a certain intermediate time position experiences an excessive backoff, that time position is designated as the failure position of the state sequence to be processed, and the specific information of the failed state unit is retained. When multiple object state units are all normal, the intermediate frame as a whole is confirmed and proceeds to the judgment of subsequent time positions.
[0085] For different object states such as vehicle translation, mechanical component rotation, and changes in human posture, as long as a local bidirectional correspondence can be established in the previous and subsequent keyframes, and the same state difference criterion can be used to compare the front-end, intermediate, and back-end states, the process can be implemented according to the same relative state progress and backtracking judgment relationship. For example, a vehicle's local state unit can correspond to a local area such as the headlights, wheel hubs, or body edge; a mechanical component can correspond to a local structure with a stable texture or contour; and a human object can correspond to areas that can be bidirectionally matched, such as the head, arms, and clothing edges. The above object differences only affect the specific image content of the local state unit and do not change the processing order.
[0086] Both the initial intermediate frame and the corrected intermediate frame are generated using the same type of video frame generation interface. This interface can utilize existing motion compensation, image generation, or other video frame generation techniques, provided it can receive reference frames from both sides and a relative target time, and output an image at the target time. Since the order determination, reference replacement generation, and correction verification in this method all occur outside the generation interface, the specific algorithm within the interface does not alter the core data handling. Actual video systems can utilize existing frame generation processing, only needing to maintain consistency in the input-output relationship between the reference frames from both sides, the relative target time, and the output intermediate image.
[0087] When a video generation task begins processing a keyframe interval, the computing device first locks the preceding keyframe, the following keyframe, and all intermediate timestamps to be generated within that interval, ensuring that the keyframes at both ends and the timeline positions remain unchanged during the processing of the current interval. After the same image object in the preceding and following keyframes is associated, a local state unit that can return consistent results in both directions is obtained from within the object. For each state unit, the local images at both ends, the normalized position within the object, the local size, and the bidirectional correspondence error are saved. If the difference between the states at both ends is insufficient to form a distinguishable process local state, it will not be included in the subsequent process judgment.
[0088] Multiple initial intermediate frames can be formed either once or sequentially according to the target time, but the state order judgment is strictly performed from front to back according to the video time. The corresponding local state in each initial intermediate frame is calculated using the original previous keyframe and the original subsequent keyframe to obtain the relative state progress within the same reference caliber. After converting the bidirectional correspondence error of the object state unit with the state difference at both ends to obtain the backoff tolerance, this backoff tolerance remains corresponding to the object state unit within the current keyframe interval and does not change again due to a correction in a certain intermediate frame.
[0089] During time-series processing, the most recently determined intermediate frame continuously provides a comparison reference for the next time position. If a new intermediate state continues to evolve towards the next keyframe, or if the reverse change does not exceed its own backoff tolerance, the intermediate state is confirmed and immediately replaces the comparison reference for that object's state unit. When a reverse change exceeding the backoff tolerance occurs for the first time, the current time position and the local state where the backoff occurred are fixed, and the comparison reference remains the state relative to the most recently confirmed intermediate state before the failure position.
[0090] The actual action triggered by the initial state sequence failure is not to modify the comparison threshold, but to change the input reference interval when regenerating the target time position. After the most recently confirmed intermediate frame replaces the original previous keyframe, the previous image state seen by the generation interface has advanced from the start of the keyframe interval to the most recently confirmed position; the subsequent keyframe continues to maintain its original termination state. The target time is recalculated within this new reference interval, thus correcting the generation to both retain the absolute position of the original target time on the video timeline and use the confirmed previous image state that is closer to the current failure position.
[0091] After the corrected intermediate frame is formed, it is not directly written into the final sequence. The corresponding local state is re-compared with the original previous and subsequent keyframes to form a state relative process with the same caliber as all initial intermediate states. After the corrected state is judged through over-tolerance backoff, the corrected intermediate frame replaces the original failed intermediate frame, and its state relative process is used for subsequent target intermediate times. If the correction result still fails, the confirmation process of the current keyframe interval is immediately terminated, and the unconfirmed corrected state is not propagated to subsequent intermediate times.
[0092] Through the aforementioned continuous operational relationship, any intermediate frame that can participate in subsequent comparisons or regenerate conditionally defined frames must first undergo state sequence confirmation. The comparison benchmark is derived only from the initial state of the previous keyframe or an intermediate state that has already passed the sequence judgment; the forward reference generated by correction is derived only from the most recent confirmed intermediate frame before the failure position; the correction result can only change the subsequent comparison benchmark after being re-judged through the same process. The source, judgment conditions, and subsequent uses of each intermediate result remain consistent within the same keyframe interval.
[0093] Once all target intermediate times within the current keyframe interval have been confirmed through initial judgment or after one correction, the computing device arranges the previous keyframe, confirmed intermediate frames, and subsequent keyframes in the original timeline order to obtain the final video frame sequence for that interval. This frame sequence is used for regular video encoding, previewing, or subsequent video editing. If any corrected intermediate frame still experiences an over-tolerance backoff, the current keyframe interval will not output a final confirmed complete intermediate frame sequence, and the processing within that keyframe interval by this method will end.
[0094] To ensure that local state differences are calculated using the same caliber across the previous keyframe, subsequent keyframe, initial intermediate frame, and revised intermediate frame, candidate local regions in the previous and subsequent keyframes are normalized for position and size using the bounding rectangle of their respective image objects. Candidate local regions in the initial and revised intermediate frames are normalized for position and size using the object reference rectangle formed at the corresponding intermediate time of the target. The position value (ranging from 0 to 1) is obtained by subtracting the left boundary of the corresponding normalized reference rectangle from the x-coordinate of the local region's center and dividing by the width of the reference rectangle; and by subtracting the upper boundary of the reference rectangle from the y-coordinate of the center and dividing by the height of the reference rectangle. The width and height of the local region are then divided by the width and height of the corresponding normalized reference rectangle, respectively, to obtain the size value (ranging from 0 to 1).
[0095] The comparison of local image content is performed at the same resampling size. The main implementation path scales the candidate local region to 32×32 pixels and normalizes each color channel using the minimum and maximum pixel values of that local image patch: each pixel value is subtracted from the minimum pixel value of that channel, and then divided by the difference between the maximum and minimum pixel values, so that the normalization result is in the range of 0 to 1; when the maximum and minimum pixel values are the same, the normalized pixel value of that channel is uniformly set to 0. The absolute difference between two normalized local image patches is taken pixel by pixel and averaged, and the resulting average value is directly used as the image content difference in the range of 0 to 1.
[0096] Positional differences are obtained by averaging the absolute values of the differences between the normalized center x-coordinates and y-coordinates of the two local regions; size differences are obtained by averaging the absolute values of the differences between the normalized width and height; image content differences are formed according to the aforementioned resampled image blocks. The main implementation path takes the arithmetic mean of positional differences, size differences, and image content differences to obtain the local state differences between the current two local regions. Since all three components have been normalized to the range of 0 to 1, no empirical weights need to be added when using the arithmetic mean. If a certain state of the actual video object does not have comparable texture, the image content difference can be replaced by the distribution difference of foreground contour pixels within the same local region. However, once the local state difference caliber is determined for the same keyframe interval, the preceding and following keyframes and all intermediate frames use this caliber.
[0097] The candidate search region for forward matching is limited to the bounding rectangle of the same image object in the subsequent keyframe. For a candidate local region in the previous keyframe, the local state difference between it and each candidate local region in the subsequent keyframe is calculated, and the candidate local region with the smallest local state difference is selected as the forward matching region; when there are multiple candidate regions with the same local state difference, the candidate region that appears first is selected according to the fixed arrangement order when the candidate local regions were formed.
[0098] Starting from the forward matching region in the subsequent keyframe, a reverse search is performed within the candidate local regions of the same image object in the previous keyframe to obtain the first returned local region. When the center of the first returned local region falls within the coverage area of the candidate local region of the previous keyframe used at the start of the forward matching, the return relationship on the previous keyframe side is confirmed. The forward matching region in the subsequent keyframe is then used as the starting local region on the subsequent keyframe side. Matching from the subsequent keyframe to the previous keyframe and back to the subsequent keyframe is performed in the opposite direction to obtain the second returned local region. When the center of the second returned local region falls within the coverage area of the starting local region on the subsequent keyframe side, the return relationship on the subsequent keyframe side is confirmed. When the return relationships in both directions are valid, the candidate local regions of the previous and subsequent keyframes form a bidirectional corresponding object state unit.
[0099] The bidirectional correspondence error is represented by the difference in local states after the return. For the previous keyframe side, the local state difference between the original starting local region of the previous keyframe and the first returned local region is calculated using a unified standard of positional difference, size difference, and image content difference, resulting in the first return deviation. For the subsequent keyframe side, the local state difference between the original starting local region of the subsequent keyframe and the second returned local region is calculated using the same standard, resulting in the second return deviation. The larger of the first and second return deviations is taken as the correspondence error of the current object state unit. The returned local region only needs to remain within the local spatial range represented by the starting local region and is not required to have the exact same center position, size, and image content as the starting local region. Therefore, a non-zero return deviation reflecting the degree of uncertainty in the local state correspondence can be formed.
[0100] After an object state unit enters an intermediate frame, it does not re-establish another set of object local definitions. The computing device uses the corresponding local regions in the previous and subsequent keyframes as templates at both ends. It determines the center position of candidate local regions according to the fixed arrangement order within the reference rectangle corresponding to the current target intermediate time, and obtains the local image content of each candidate local region from the corresponding object search area. For each candidate local region, it calculates the local state difference to the front and back templates, adds the two differences, and selects the candidate region with the smallest sum as the current intermediate state. If the sum of differences for multiple candidate regions is the same, it calculates the shortest distance from the normalized center of the candidate region to the normalized centers of the front and back templates, and selects the candidate region with the smallest shortest distance. When the shortest distances are still the same, it selects the candidate local region that appears first according to the fixed arrangement order when the candidate local regions were formed as the current intermediate state. Positional relationships are only used to eliminate matching ambiguities under the same difference conditions and do not participate in the calculation of the relative state process.
[0101] When a single image object has multiple bidirectional corresponding object state units, each object state unit stores the corresponding error, the difference between the two ends of the state, the backoff tolerance, and the relative progress of the most recently confirmed state. The sequential confirmation of an intermediate frame does not average these values before making a single judgment, because a significant state backoff in one local structure may be offset by the increased progress of other normal local structures. The computing device reads the relative progress of the state units participating in the judgment in the current intermediate frame one by one and compares each with its most recently confirmed relative progress.
[0102] For example, a single character object may have three stable bidirectional corresponding state units: right forearm, left shoulder, and head. At a certain intermediate time, the relative progress of the state units is 0.46, 0.51, and 0.48, respectively, and the most recently confirmed progress is 0.42, 0.49, and 0.45, with respective backoff tolerances of 0.06, 0.05, and 0.04. Since none of the three state units have backed up in the direction of the previous keyframe, this intermediate frame is confirmed, and the comparison benchmarks of the three state units are updated with 0.46, 0.51, and 0.48, respectively. If, at a subsequent intermediate time, the right forearm progress becomes 0.37, while the left shoulder and head still increase, and the right forearm backs up by 0.09 relative to 0.46, exceeding the 0.06 backoff tolerance, the current intermediate time is still determined as the state sequence failure position, and the failed state unit is recorded as the local state corresponding to the right forearm.
[0103] When multiple failure state units appear in the same intermediate frame, the initial state sequence failure location is still determined by the target time of that intermediate frame. The local failure state includes all object state units that have undergone over-tolerance rollback. The new preceding reference still uniformly uses the nearest confirmed intermediate frame before the failure location, without selecting different preceding reference times for different failure state units. This uniform processing ensures that the corrected intermediate frame still corresponds to a specific video time position and a specific pair of generation reference frames.
[0104] After an intermediate frame is sequentially evaluated, its image data, along with the relative progress of the states of each participating object state unit, serves as the confirmation result for the current time position. Subsequent target times only read the latest comparison benchmark corresponding to the required object state unit; old comparison benchmarks that have been replaced by earlier intermediate frames are not compared alongside subsequent intermediate states. This forms a hierarchical replacement relationship: the initial process of the previous keyframe provides the initial comparison benchmark, the first intermediate frame that passes evaluation replaces the initial process, and the next intermediate frame that passes evaluation replaces the previous comparison benchmark, until the first over-tolerance backoff occurs.
[0105] After the corrected intermediate frames are checked sequentially, they are also entered into the relationship in a step-by-step replacement manner. The most recently confirmed intermediate frame before failure is still retained in the final frame sequence, but its state relative to the process is no longer used as the latest comparison benchmark after the correction position; the corrected state relative to the process becomes the new comparison benchmark. Subsequent initial intermediate frames are backtracked based on the corrected state relative to the process, so the correction result will actually change the subsequent judgment conditions.
[0106] If a subsequent initial intermediate frame experiences an over-tolerance backtracking, the most recently confirmed intermediate frame before the new failure location is likely the intermediate frame that was previously corrected and verified. This intermediate frame can be used again as a reference for new front-end generation because it has already been verified through state order within the unified reference of the original preceding and following keyframes. Each failure handling uses the most recent confirmation result in the current time sequence, without needing to backtrack to all earlier intermediate frames for reassessment.
[0107] The aforementioned step-by-step replacement relationship does not alter the state reference status of the original preceding and following keyframes. The comparison benchmark can be updated step-by-step as confirmed intermediate states are established, and the preceding reference during generation can be replaced by the most recently confirmed intermediate frame, replacing the original preceding keyframe. However, the relative change process between any initial and corrected states is still calculated using the same caliber from the state of the original preceding keyframe to the state of the original following keyframe. Therefore, the process values between different time points have uniform comparability, avoiding the loss of a common reference for process values due to changes in local generation intervals.
[0108] Another representative implementation segment uses the example of a vehicle turning between two keyframes, accompanied by a change in vehicle position. The first keyframe is located at frame number 100, and the second keyframe is located at frame number 106. Frame numbers 101 to 105 represent the midpoint of five targets. The same vehicle is already associated in the first and second keyframes; bidirectional correspondence can be achieved in the local areas near the left front headlight, the local area of the front wheel hub, and the local area of the door edge. The following describes the state sequence processing for the local area of the front wheel hub.
[0109] The local state difference between the front and rear keyframes of the front wheel hub local area is 0.40. After performing forward matching and reverse return on the front wheel hub local area, the center of the first returned local area is still within the coverage of the starting local area of the front keyframe, and the local state difference between the original starting local area and the first returned local area is 0.018. After performing reverse return matching on the rear keyframe wheel hub local area, the center of the second returned local area is still within the coverage of the starting local area of the rear keyframe, and the corresponding return deviation is 0.024. Since the return relationship in both directions is valid, 0.024 is taken as the corresponding error of the state unit of this front wheel hub object.
[0110] Frame number 101 corresponds to process 0.12, frame number 102 corresponds to process 0.29, and frame number 103 corresponds to process 0.47. These three are determined sequentially based on their state order. Frame number 104 corresponds to process 0.43. Relative to the most recent comparison benchmark 0.47, it has a backslid of 0.04, which does not exceed the 0.06 backsliding tolerance. Therefore, it is still considered to be within the uncertainty range corresponding to the current local state. Frame number 104 is confirmed, and the comparison benchmark is updated to 0.43. This process demonstrates that the state relative to the process does not require strict mathematical monotonicity, and small reverse changes within the matching uncertainty range of the object's state unit will not trigger corrections.
[0111] Frame number 105 corresponds to process time 0.78, which is greater than 0.43. Continuing with the sequential check, all intermediate frames in the current keyframe interval have been confirmed. If the state of frame number 104 relative to the process time is 0.36 instead of 0.43 in the same task, then the backoff amount relative to 0.47 is 0.11, exceeding 0.06. Frame number 104 becomes the first position where the state sequence fails. The nearest confirmed intermediate frame to this position is frame number 103, and the subsequent reference is still the keyframe after frame number 106. Frame number 104 is located at 1 / 3 of the new interval from 103 to 106. The video frame generation interface uses the image of frame number 103, the image of frame number 106, and 1 / 3 of the relative target time to reconstruct frame number 104.
[0112] The corrected hub state in frame number 104 is still compared with the corresponding states in the keyframes before frame number 100 and after frame number 106. If the relative progress of the corrected state is 0.58, there is no backsliding relative to the comparison benchmark of 0.47, the corrected frame is confirmed, and 0.58 becomes the comparison benchmark for frame number 105; the original progress of frame number 105, 0.78, continues to pass the judgment. If the relative progress of the corrected state is 0.39, there is still a backsliding of 0.08 relative to 0.47 and exceeds 0.06, the corrected frame is not confirmed, and the confirmation process of the intermediate frame sequence between keyframes 100 and 106 ends.
[0113] Vehicle and human action segments use the exact same bidirectional correspondence, correspondence error, relative state progress, backtracking tolerance, temporal sequence comparison, and preceding reference replacement relationship. The only difference lies in the specific image content carried by the object state unit. Therefore, this method does not process specific human joints or vehicle components, but rather local image regions of objects that can form a stable bidirectional correspondence between preceding and following keyframes and exhibit distinguishable state changes.
[0114] In actual deployment, the runtime record for a keyframe interval only needs to maintain the information necessary for the current processing: the time and image of the previous and next keyframes, the target intermediate time list, the bidirectional corresponding state units for the same image object, the corresponding errors and backoff tolerances of each state unit, the images of each initial intermediate frame, the relative progress of the most recently confirmed state, and the time position of the most recently confirmed intermediate frame. All of the above information is directly generated from the input image and processing results of the current keyframe interval, without needing to read historical data from other video projects.
[0115] When the computing device begins processing a new keyframe interval, the most recent comparison reference from the previous keyframe interval is not directly carried over to the new keyframe interval. The new previous and subsequent keyframes re-establish the local state correspondence, correspondence error, and state differences for the same object, and the backoff tolerance is also re-formed according to the new two endpoints. This approach ensures that the state relative to the process always corresponds to the two fixed endpoints of the current keyframe interval. If the same keyframe serves as both the subsequent keyframe of the previous interval and the preceding keyframe of the subsequent interval, its image can participate in the processing of both intervals separately, but each interval uses its own endpoint state differences and correspondence errors.
[0116] After the intermediate frame generation interface generates images, the image files or frame buffers only serve as carriers for local state extraction and final sequence output. State relative to process, backoff tolerance, and comparison benchmarks are stored in memory as runtime data associated with the object's state unit. After the current keyframe interval is confirmed, the final frame sequence is written to the video project in its original chronological order; in case of failure and termination, unconfirmed correction frames are not considered the final confirmation result for the current interval. Regular file saving and cache clearing do not participate in state order determination.
[0117] It should be noted that the methods, systems, devices, or equipment described in the above embodiments can be implemented in software, hardware, or a combination of both, depending on the actual application scenario. When implemented in software, the corresponding program instructions can be stored in a computer-readable storage medium and executed by a processor to achieve the processing steps or functions described in the above embodiments. For the specific processing procedures of each functional module in the system, device, or equipment, refer to the corresponding processing procedures in the foregoing method embodiments.
[0118] The above description is merely a specific embodiment of the present invention, used to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Without departing from the technical concept of the present invention, any equivalent substitutions, reasonable modifications, or combinations made by those skilled in the art based on the disclosure of this specification should be determined according to the claims to determine whether they fall within the scope of protection of the present invention.
Claims
1. A method for generating consistent intermediate video frames with keyframe constraints, characterized in that, include: S1. Obtain the previous keyframe, the next keyframe, and the intermediate times of multiple targets in between. Determine the bidirectional corresponding local state and corresponding error in the same image object between the two keyframes. S2. Form initial intermediate frames for the intermediate time of each target, obtain the corresponding local states, determine the relative change process based on the difference between the states corresponding to the previous and subsequent keyframes, and take the starting process corresponding to the previous keyframe as the initial reference. S3. Determine the backoff tolerance based on the corresponding error, and judge each intermediate state in chronological order; update the baseline when the backoff of the current state in the direction of the forward key frame does not exceed the tolerance, and determine the failure location and local failure state when it exceeds the tolerance for the first time. S4. Take the nearest intermediate frame used to update the baseline before the failure position to replace the previous key frame as the front reference, and use the subsequent key frame as the back reference. Re-determine the relative target time of the failure position according to the new front and back reference time and form a corrected intermediate frame. S5. The process of re-determining the correction state is based on the corresponding states of the previous and subsequent keyframes. If the process of reverting to the correction state does not exceed the revert tolerance, the process updates the comparison benchmark of the next target intermediate time and continues to process, forming the final intermediate frame sequence.
2. The method for generating video intermediate frame consistency with keyframe constraints according to claim 1, characterized in that, Candidate local regions are formed within the same image object in the preceding and following keyframes; After performing forward matching on the candidate local regions of the previous keyframe, a reverse search is performed. Then, starting from the matched local regions in the subsequent keyframe, matching and return searches are performed in the opposite direction. When the centers of the returned local regions in both directions fall within the coverage area of the corresponding starting local regions, a bidirectional correspondence is determined. The larger value of the local state difference between the returned local regions in both directions and their respective starting local regions is taken as the correspondence error.
3. The method for generating video intermediate frame consistency with keyframe constraints according to claim 1, characterized in that, Determining the relative change process includes: calculating the difference between the intermediate state and the corresponding state of the previous keyframe, and the difference between the intermediate state and the corresponding state of the subsequent keyframe, respectively. The relative change process is obtained by dividing the difference between the intermediate state and the sum of the differences between the intermediate state and the subsequent keyframe.
4. The method for generating video intermediate frame consistency with keyframe constraints according to claim 3, characterized in that, The backoff tolerance is obtained by dividing the corresponding error by the state difference between the corresponding state of the previous keyframe and the corresponding state of the subsequent keyframe. When the state difference is 0 or less than or equal to the corresponding error, the local states corresponding to both sides do not participate in the process rollback judgment.
5. The method for generating video intermediate frame consistency with keyframe constraints according to claim 4, characterized in that, When judging according to the intermediate time sequence of the target, for the same bidirectional corresponding local state, the relative change process of the corresponding intermediate state that was most recently judged by sequence is used as the comparison benchmark. When the current relative change process is lower than the benchmark, the difference between the two is used as the backoff amount. If the backoff amount does not exceed the corresponding backoff tolerance, the current intermediate state is confirmed, and the benchmark is replaced by the current relative change process.
6. The method for generating video intermediate frame consistency with keyframe constraints according to claim 1, characterized in that, For each target intermediate time, based on the proportion of the target intermediate time in the time interval of the preceding and following keyframes, the coordinates of the four sides of the outer rectangle of the same image object in the preceding and following keyframes are converted to form an object reference rectangle. Then, a local window side length is extended outward around the object reference rectangle to form a candidate local region, thus forming the object search region. For each bidirectional corresponding local state in the preceding and following keyframes, the center position of the candidate local region is determined according to the fixed arrangement order within the object reference rectangle. The local image content corresponding to each candidate local region is obtained from the object search region. The sum of the local state differences between each candidate local region and the corresponding states in the preceding and following keyframes is calculated. The candidate local region with the smallest sum of differences is taken as the corresponding local state of the target at the intermediate time.
7. The method for generating video intermediate frame consistency with keyframe constraints according to claim 1, characterized in that, The re-determination of the relative target time includes: calculating the time difference between the failure location and the preceding reference time, and the total time difference between the following reference time and the preceding reference time. The ratio of the former to the latter is used as the relative target time of the failure location in the new preceding and following reference time intervals.
8. The method for generating video intermediate frame consistency with keyframe constraints according to claim 6, characterized in that, After the intermediate frame is corrected, the center position of the candidate local region is determined in a fixed order within the object reference rectangle corresponding to the failure position, according to the same candidate local region formation and corresponding local state determination rules as the initial intermediate frame. The local image content corresponding to each candidate local region is obtained from the corresponding object search area. All corresponding local states that participate in the current intermediate frame order judgment are re-obtained, and the relative change process is re-determined with reference to the corresponding states of the previous and subsequent keyframes. When the process rollback for all corresponding local states does not exceed its respective rollback tolerance, the intermediate frame is confirmed to be corrected, and the corresponding comparison benchmark is updated according to the relative change process of each corresponding local state.
9. The method for generating video intermediate frame consistency with keyframe constraints according to claim 8, characterized in that, Only one front reference replacement and intermediate frame correction are performed at the same failure location; If any reacquired local state still experiences a process rollback exceeding its rollback tolerance, the intermediate frame is not confirmed, the corresponding comparison benchmark is not updated based on the relative changes in the corresponding local states in the intermediate frame, and the intermediate frame sequence confirmation process between the current previous keyframe and the next keyframe is terminated.
Citation Information
Patent Citations
Method and device for generating video intermediate frame
CN115065796A
Long video generation method and device based on text input
CN117793480A
