Method for constructing action consistency pairwise dataset based on unstructured video

Abstract

The application discloses a motion consistency paired dataset construction method based on unstructured videos. The method comprises the following steps: performing scene detection and subject screening on original video data to be processed, and extracting a basic motion video segment containing a target role; decoupling and editing the features of the role in the basic motion video by using an image generation model to generate target role images with diversified appearances and postures; establishing a random matching mechanism to pair the target role images with the basic motion video, and driving a video generation model to synthesize a new video with corresponding motions; and constructing the synthesized video and the corresponding basic motion video into a paired video dataset. The application can effectively utilize extensive video resources, construct paired video data with high diversity and high motion consistency through automatic cleaning and generation technology, and solve the problems of difficult training data acquisition and single samples.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and relates to computer vision and big data processing technology, specifically to a method for constructing action consistency pairwise datasets based on unstructured videos. Background Technology

[0002] With the rapid development of deep learning technology, artificial intelligence has made significant breakthroughs in video understanding and video generation. High-quality, large-scale video datasets are fundamental for training high-performance action recognition, robot imitation learning, and video generation models. In advanced tasks such as action transfer, video restoration, and video generation, models typically need to learn the fine motion laws of the physical world, which places extremely high demands on the quality, coherence, and structure of the training data. However, the construction and application of existing video datasets still face many technical challenges, limiting further improvements in the generalization ability and generation quality of related models. The main problems are reflected in the following two aspects:

[0003] First, existing dataset construction methods are insufficient for processing unstructured long videos, resulting in a large amount of motion noise in the datasets. Early video datasets mainly relied on extensive scraping of publicly available videos from the internet. Since these unstructured videos (such as movies, TV series, and documentaries) typically contain complex shot transitions and varied scenes, without high-precision shot boundary detection technology, the extracted video segments often contain multiple unrelated shots, causing breaks in action semantics along the timeline. Existing shot segmentation techniques mostly rely on single pixel histogram differencing or simple optical flow detection, which is prone to false positives or false negatives when faced with complex lighting changes or dynamic backgrounds, making it difficult to guarantee the physical continuity and semantic purity of the basic video segments. In addition, the lack of multi-dimensional quality filtering mechanisms targeting image composition, lighting aesthetics, and subject motion amplitude results in a large number of low-quality or static redundant samples in the datasets.

[0004] Second, there is a severe shortage of action-consistent paired data for specific generation tasks. In video-driven generation or action transfer tasks, the ideal training data format is for input and target videos to appear in pairs, meaning they maintain a high degree of consistency in action logic and trajectory, differing only in character appearance or background texture (i.e., same action, different appearance). However, most current mainstream video datasets are video-text pairs, lacking such pixel-level or feature-level aligned paired video samples. Directly collecting fully synchronized but different real-world paired videos in nature is extremely costly and difficult to achieve, making it difficult for existing models to use high-fidelity real videos as supervision signals to decouple action and appearance features. This severely restricts the model's ability to accurately control complex actions and the realism of the generated results.

[0005] Therefore, there is an urgent need for a method that can effectively utilize the widely available unstructured video resources, perform high-precision cleaning and structuring through automated means, and construct pairwise video datasets with deep semantic consistency, in order to solve the current problems of data scarcity and low quality. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies in unstructured video data processing, such as inaccurate shot segmentation, inconsistent data quality, and lack of action-consistent paired samples. This invention provides a method for constructing action-consistent paired datasets based on unstructured videos. This method can automatically extract high-quality segments from complex unstructured video sources and construct paired training data with deep semantic alignment, effectively improving the training performance of video generation models.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] Step (1) Construct a general basic video pool, as follows:

[0009] (1-1) Read the unstructured raw video stream in chronological order to obtain the video frame sequence; for each frame image in the video stream, convert it from the original RGB color space to the HSV color space and LUV color space respectively through a color space conversion algorithm;

[0010] (1-2) In the HSV space, calculate the average absolute pixel difference of the H, S, and V channels of two adjacent frames. In LUV space, calculate the average absolute pixel difference of the L, U, and V components of two adjacent frames. ;

[0011] (1-3) Cut the long video stream into physically continuous, independent video segments without camera jumps;

[0012] (1-4) Eliminate interference from labels and subtitles;

[0013] (1-5) Input the independent video clips after removing the interference of labels and subtitles into the image-text matching model or a dedicated aesthetic scoring network, output a score representing visual quality, and remove independent video clips with scores lower than the set aesthetic threshold.

[0014] (1-6) Trim the black borders;

[0015] (1-7) Calculate the optical flow field between adjacent frames of an independent video segment using the dense optical flow algorithm, and solve for the mean value of the optical flow vector amplitude of the entire segment; remove independent video segments with a mean value lower than a preset static threshold, and finally retain the independent video segments. Import into the general basic video pool;

[0016] (1-8) Input each independent video segment in the general basic video pool into the video automatic description generation model. The model outputs a detailed natural language description P for each independent video segment and binds the natural language description as metadata to the corresponding independent video segment for storage.

[0017] Step (2) Filter single subject for each independent video segment in the general basic video pool; first, uniformly sample the video at fixed time intervals to obtain sample frames that can represent the overall content of the independent video segment; then input each sample frame into the pose detection model and output the corresponding detection box and confidence score; count the number of detection boxes with confidence scores higher than the set threshold in each sample frame, retain the videos in which the number of detection boxes in all sample frames is always 1, and remove the independent video segments corresponding to other videos from the general basic video pool.

[0018] Step (3) Generate a counterfactual character reference diagram; first, randomly select a frame from the selected independent video clips as the reference base frame. Pre-set prompt word library containing various counterfactual text commands The instructions in the library are used to describe states that are inconsistent with the original video actions; a prompt word is randomly selected from the prompt word library. The reference base frame will be used. With prompt words A common input instruction-driven image editing model outputs a counterfactual role reference diagram. ; The facial features, hairstyles, and clothing textures of the characters are similar to those in the game. Consistency, body posture or orientation should be in accordance with Forced change, serving as the character's identity anchor; reference diagram for all counterfactual characters. Character reference image library.

[0019] Step (4) Input all frames of the independent video clip of a single subject into the full-body pose estimation model, and infer the continuous motion state of the character on the timeline frame by frame; extract the complete set of human skeleton key point data for each frame to form skeleton sequence data; map the extracted key point coordinate sequence onto a pure black background canvas, connect adjacent key points with lines, and render to generate skeleton video. .

[0020] Step (5) Randomly sample an image from the character reference image library as the heterogeneous reference image. , The identity of the person in the middle is completely different from that of the person in the target video; use a different reference image. Skeleton video A video generation model that supports skeleton control and uses natural language description P as input generates motion videos. .

[0021] Step (6) constructs standard sample units from the dataset. Thus, action consistency pairwise datasets are constructed.

[0022] This invention effectively solves the problem of false positives and false negatives in traditional single-feature detection under complex lighting or dynamic backgrounds by introducing HSV-LUV dual color space judgment logic. It ensures that the basic segments segmented from unstructured long videos have strict physical continuity, significantly reducing semantic noise in the temporal dimension. This invention employs a multi-dimensional cascaded quality filtering mechanism, combining logo / subtitle detection, aesthetic scoring, black border cropping, and optical flow detection. This enables automated, high-throughput cleaning of high-quality video segments with excellent composition and effective motion information, greatly improving the utilization rate of unstructured data. This invention proposes an innovative pairwise data construction strategy. By using heterogeneous driven synthetic videos as input and real original videos as ground truth, this construction method enables the model to not only learn precise motion control during training but also utilize high-frequency details of real videos as strong supervision signals, thereby significantly improving the image quality and realism of the generated videos. By generating counterfactual character reference maps, identity features and action features are forcibly decoupled. During training, the model is forced to extract identity information from the reference map and action information from the skeleton or input video, effectively avoiding the problem of feature confusion or direct copying of reference map actions during training, and enhancing the model's consistent control over character actions. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;

[0024] Figure 2 This is a detailed logical flowchart illustrating the generation of the counterfactual role reference diagram and the skeleton-guided heterogeneous video generation in the embodiment. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0026] like Figure 1 and 2 As shown, a method for constructing action consistency pairwise datasets based on unstructured videos is as follows:

[0027] Step (1) Construct a general basic video pool;

[0028] (1-1) Reading the Video Stream and Color Space Conversion: Read the unstructured raw video stream (such as movie, TV series, or documentary files) in chronological order to obtain the video frame sequence. For each frame in the video stream, convert it from the original RGB color space to the HSV color space and then to the LUV color space using a color space conversion algorithm. The HSV color space is used to capture significant changes in hue, saturation, and brightness, and is suitable for detecting obvious scene transitions; the LUV color space has perceptual uniformity and is more sensitive to small changes in brightness and chromaticity, making it suitable for detecting sudden changes in lighting or soft transitions.

[0029] (1-2) Calculate the difference between adjacent frames: For the t-th and t+1-th frames, calculate the difference between the two frames in the two color spaces. In the HSV space, calculate the average absolute pixel difference of the H, S, and V channels of the two frames. In LUV space, calculate the mean absolute pixel difference of the L, U, and V components of two frames. The mean absolute difference is used to quantify the degree of change between adjacent frames in different visual dimensions.

[0030] (1-3) Dual threshold determination: To address the issue of single features being easily interfered with, three thresholds are preset: a high threshold for colorimetric detection of hard shearing. (e.g., 30.0), a low chromaticity threshold used to detect potential changes. (e.g., 10.0), Threshold for perceptual micro-changes used to assist in confirmation (e.g., 5.0) The decision logic is: if If a strong scene change occurs, the current frame is marked as the segmentation point; if Although the chromaticity change did not reach the extreme threshold, it simultaneously met the following requirements. When a significant change in brightness or chromaticity is detected in the LUV space, indicating a dramatic change in lighting or a smooth scene transition, it is also marked as a segmentation point. Through this step, the long video stream is cut into physically continuous, independent video segments without camera jumps.

[0031] (1-4) Removal of logos and subtitle interference: Using object detection models (such as the YOLOv8 model) or text detection models (such as the CRAFT model), identify the logo, watermark, and subtitle areas in the first, last, and middle frames of each independent video segment to generate an interference mask. Based on this mask, use a coordinate discretization grid enumeration method to search for the largest inscribed rectangle in the frame free of any interference. Calculate the length and width of this rectangle. If the length or width of the rectangle is less than a preset threshold (e.g., 512), the independent video segment is determined to have excessive occlusion or too small an effective frame, and is marked as an invalid independent video segment and discarded.

[0032] (1-5) Input the independent video clips, after removing labels and captions, into a text-image matching model (such as the CLIP model) or a dedicated aesthetic scoring network (such as the improved-aesthetic-predictor model). The model outputs a score representing visual quality (e.g., 0-10). Set the aesthetic threshold to 8.5 points. Independent video clips with scores below this threshold are considered to have quality problems such as blurriness, overexposure, underexposure, or cluttered composition, and are removed to ensure that the dataset has excellent visual appeal.

[0033] (1-6) Black Border Cropping: For the top, bottom, left, and right black borders generated by the Movie Mailbox mode, an edge detection operator is used to scan the gradient changes in the edge regions of the video frames. By identifying the scan lines where pixel gradient changes abruptly, the black border boundary lines are located. Based on the boundary lines, independent video segments are cropped without loss, retaining only the effective image area and unifying the aspect ratio characteristics of the data.

[0034] (1-7) Dynamic Filtering: The dense optical flow algorithm is used to calculate the optical flow field between adjacent frames of an independent video segment, and the mean value of the optical flow vector amplitude of the entire segment is calculated. If the mean value is lower than a preset static threshold (e.g., 1.5 pixels / frame), the segment is determined to be a still image or slideshow content, lacking the motion information required for training, and is therefore discarded. The final retained independent video segments... Import into the general basic video pool.

[0035] (1-8) Generate dense text descriptions; each independent video segment from the general basic video pool is input into the video auto-description generation model (such as the cogvlm2-llama3-caption model). The video auto-description generation model outputs a detailed natural language description P for each independent video segment, and binds and stores the natural language description as metadata with the corresponding independent video segment. The video auto-description generation model can perform deep semantic analysis of the main features, action details, scene environment lighting and shadows, and object interaction logic in the video frame.

[0036] Step (2) involves filtering each independent video segment in the general basic video pool for a single subject; specifically as follows:

[0037] First, video sampling is performed. To improve processing efficiency and cover the entire video content, the system extracts video from a general basic video pool and samples it evenly at fixed time intervals (e.g., every 24 frames) to obtain sampled frames that can represent the overall content of an independent video segment.

[0038] Then, human detection is performed; each sampled frame is input into the pose detection model (such as the YOLOv8-pose model), and the corresponding detection box and confidence score are output; the pose detection model is specifically designed to detect human faces or human targets in the image.

[0039] Detection box statistics and filtering: Count the number of detection boxes with a confidence level higher than a set threshold in each sampled frame. Only retain videos where the number of detection boxes in all sampled frames is always 1, ensuring that each independent video segment contains only a single main character. If the number of detection boxes in any frame is 0 (no one) or greater than 1 (multiple people), it is determined that the independent video segment has a missing subject or feature confusion, and the corresponding independent video segment is removed from the general basic video pool.

[0040] Step (3) Generate a counterfactual role reference diagram; details are as follows:

[0041] First, a reference base frame is extracted; from the selected independent video clips, a frame is randomly extracted as the reference base frame. , Includes the complete appearance features of the target character.

[0042] Build a prompt word library; pre-configure a prompt word library containing various counterfactual text commands. The library contains commands used to describe states that are inconsistent with the original video actions, including descriptive words for actions or perspectives such as raising a hand, turning around, looking to the side, and sitting down.

[0043] Randomly select a prompt word from the prompt word library. The reference base frame will be used. With prompt words Common input instruction-driven image editing models (such as the Z-Image model). These models execute editing operations and output a counterfactual role reference image. . The facial features, hairstyles, and clothing textures of the characters are similar to those in the game. Consistency, body posture or orientation should be in accordance with Forced change, serving as the character's identity anchor. (See all counterfactual character reference images.) Character reference image library.

[0044] Step (4) Extract the full-body skeleton motion; input all frames of the independent video clip of a single subject into the full-body pose estimation model (such as the DWPose model), and infer frame by frame for all frames of the independent video clip of a single subject to capture the continuous motion state of the character on the timeline. Extract a complete set of human skeleton keypoint data for each frame, including body torso keypoints, fine-grained hand keypoints, and facial contour keypoints (such as 17 keypoints in the COCO dataset annotation format) to form skeleton sequence data. Map the extracted keypoint coordinate sequence onto a pure black background canvas, connect adjacent keypoints with lines, and render to generate a skeleton video. This video removes all background and texture information from the original video, recording only the character's movement trajectory.

[0045] Step (5) generates a skeleton-guided motion video; details are as follows:

[0046] Randomly sample an image from the character reference image library as the heterogeneous reference image. ,make sure The identity of the person in the video is completely different from that of the person in the target video, creating a discrepancy in identity.

[0047] heterogeneous reference diagram (For appearance reference) Skeleton video Inputting a video generation model (such as the Wan-Animate model) that supports skeleton control with natural language description P (as conditional input) and a prompt word P (as a cue word) as input, the model generates motion videos. . The physical characteristics of Chinese characters are derived from The decision, body movements, and trajectory of movement are strictly followed. The video content conforms to the original natural language description P, and is completely consistent with the original video action logic (same action) but has a completely different visual appearance (different appearance).

[0048] Step (6) constructs standard sample units from the dataset. Thus, action consistency pairwise datasets are constructed.

[0049] When training downstream video generation models (such as video redrawing and action transfer models) using this dataset, and As input to the model, the model is required to reconstruct the output. Provides action structure but with incorrect identity (heterogeneous source). Providing the correct identity but with an incorrect pose (counterfactual), forcing the model to... Identity characteristics "migrated" to The action structure. Simultaneously, with realistic... As a truth value calculation loss function As a high-quality video shot in real life, its physical lighting and texture details are superior to those of synthetic videos, and it can be used as a strong supervisory signal to improve the realism of the model's generated results.

[0050] To facilitate subsequent training of dedicated video generation models, the constructed action consistency pairwise dataset can be standardized and encapsulated. Standardization includes: unifying the transcoding parameters of videos within data units, adjusting the synthetic and original real videos to preset resolutions (e.g., 512×512, 1024×1024 pixels) and frame rates (e.g., 24 frames / second, 30 frames / second), and storing them in a standard video encoding format; standardizing and normalizing the counterfactual role reference graphs to ensure consistent input sizes; structurally annotating the metadata of data units, writing the paths of synthetic videos, reference graphs, skeleton videos, and ground truth videos into an index file; and optionally, serializing and storing the cleaned and aligned massive data units in an efficient binary record format to improve data reading efficiency and input / output throughput during large-scale model training.

[0051] Those skilled in the art will understand that the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical disk, optical storage, portable hard disk, USB flash drive, cloud computing platform storage, solid-state drive, etc.) containing computer-usable program code.

[0052] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments. It should be understood that each flow, block, and combination of flowchart illustrations and / or block diagrams can be implemented by computer program instructions. These instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to generate machine instructions that cause the processor of the computer or other programmable data processing device to execute, thereby implementing the flow. Figure 1 Process or multiple processes, box Figure 1 The function specified by the box or multiple boxes.

[0053] These computer program instructions may also be stored in a computer-readable storage medium that directs a computer or other programmable data processing device to operate in a particular manner, causing the instructions stored in the storage medium to generate an article of manufacture containing the instructions, which implements the process. Figure 1 Process or multiple processes, box Figure 1 The function specified by the box or multiple boxes.

[0054] These computer program instructions can also be loaded into a computer or other programmable data processing device, causing the computer or other programmable device to perform a series of operational steps, generating computer-implemented processing, thereby providing the implementation flow of the instructions executed by the computer or other programmable device. Figure 1 Process or multiple processes, box Figure 1 The boxes or multiple boxes specify the steps for the function.

[0055] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some technical features (such as replacing the skeleton-guided generation model with other generation networks with attitude control capabilities, or adjusting the dual color space threshold parameters, etc.). These modifications or substitutions do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a pairwise dataset of action consistency based on unstructured video, characterized by: Step (1) Construct a general basic video pool, as follows: (1-1) Read the unstructured raw video stream in chronological order to obtain the video frame sequence; for each frame image in the video stream, convert it from the original RGB color space to the HSV color space and LUV color space respectively through a color space conversion algorithm; (1-2) In the HSV space, calculate the average absolute pixel difference of the H, S, and V channels of two adjacent frames. ; In the LUV space, calculate the average absolute pixel difference of the L, U, and V components of two adjacent frames. ; (1-3) Cut the long video stream into physically continuous, independent video segments without camera jumps; (1-4) Eliminate interference from labels and subtitles; (1-5) Input the independent video clips after removing the interference of labels and subtitles into the image-text matching model or a dedicated aesthetic scoring network, output a score representing visual quality, and remove independent video clips with scores lower than the set aesthetic threshold. (1-6) Trim the black borders; (1-7) Calculate the optical flow field between adjacent frames of an independent video segment using the dense optical flow algorithm, and solve for the mean value of the optical flow vector amplitude of the entire segment; remove independent video segments with a mean value lower than a preset static threshold, and finally retain the independent video segments. Import into the general-purpose basic video pool; (1-8) Input each independent video segment in the general basic video pool into the video automatic description generation model. The model outputs a detailed natural language description P for each independent video segment and binds and stores the natural language description as metadata with the corresponding independent video segment. Step (2) Filter single subject for each independent video segment in the general basic video pool; first, uniformly sample the video at fixed time intervals to obtain sample frames that can represent the overall content of the independent video segment; then input each sample frame into the pose detection model and output the corresponding detection box and confidence score; count the number of detection boxes with confidence scores higher than the set threshold in each sample frame, retain the videos in which the number of detection boxes in all sample frames is always 1, and remove the independent video segments corresponding to other videos from the general basic video pool; Step (3) Generate a counterfactual character reference diagram; first, randomly select a frame from the selected independent video clips as the reference base frame. Pre-set prompt word library containing various counterfactual text commands The instructions in the library are used to describe states that are inconsistent with the original video actions; Randomly select a prompt word from the prompt word library. The reference base frame will be used. With prompt words A common input instruction-driven image editing model outputs a counterfactual role reference diagram. ; The facial features, hairstyles, and clothing textures of the characters are similar to those in the game. Consistency, body posture or orientation should be in accordance with Forced change, serving as the character's identity anchor; Reference image of all counterfactual characters Character reference image library; Step (4) Input all frames of the independent video clip of a single subject into the full-body pose estimation model, and infer the continuous motion state of the character on the timeline frame by frame; extract the complete set of human skeleton key point data for each frame to form skeleton sequence data; map the extracted key point coordinate sequence onto a pure black background canvas, connect adjacent key points with lines, and render to generate skeleton video. ; Step (5) Randomly sample an image from the character reference image library as the heterogeneous reference image. , The identity of the person in the middle is completely different from that of the person in the target video; use a different reference image. Skeleton video A video generation model that supports skeleton control and uses natural language description P as input generates motion videos. ; Step (6) constructs standard sample units from the dataset. Thus, action consistency pairwise datasets are constructed.

2. The method for constructing action consistency pairwise datasets based on unstructured video as described in claim 1, characterized in that: Steps (1-3) involve setting three thresholds, which are the high thresholds for colorimetric detection used to detect hard cutting. Low threshold colorimetric values ​​used to detect potential changes Threshold for perceptual micro-changes used to assist in confirmation , The decision logic is: if If a strong scene change occurs, the current frame is marked as the segmentation point; if At the same time satisfy When a sudden change in lighting or a smooth scene transition occurs, the current frame is marked as the segmentation point.

3. The method for constructing action consistency pairwise datasets based on unstructured video as described in claim 1, characterized in that: Steps (1-4) use object detection models or text detection models to identify the logo, watermark, and subtitle areas in the first, last, and middle frames of each independent video segment, and generate interference masks. Based on the mask, a coordinate discretization grid enumeration method is used to search for the largest inscribed rectangle in the image without any interference. The length and width of the rectangle are calculated. If the length or width of the rectangle is less than a preset threshold, the independent video segment is determined to be too obscured or the effective image is too small. It is then marked as an invalid independent video segment and discarded.

4. The method for constructing a pairwise dataset of action consistency based on unstructured video as described in claim 1, characterized in that: In steps (1-6), for the top and bottom or left and right black borders generated by the movie mailbox mode, the edge detection operator is used to scan the gradient changes in the edge region of the video frame. By identifying the scanning line position of the pixel gradient change, the black border boundary line is located. Based on the boundary line, the independent video segments are cropped without loss, retaining only the effective screen area and unifying the aspect ratio features of the data.

Images