A background anti-warping method and system applied to video human body shape processing

Abstract

This application relates to a background anti-distortion method and system for video body shaping. The method includes: after acquiring the original video, performing transition detection, human detection, cross-frame identity association, and reference frame determination on the video; generating a human body mask sequence based on the reference frame; extracting the foreground of the human image according to the human body mask sequence; applying a preset deformation simulation to the original video and the human body mask sequence; determining local background removal areas based on the difference between the masks before and after deformation; performing background completion to obtain a completed background video; then performing deformation on the foreground of the human image according to the user's body shaping parameters; and combining the deformed foreground of the human image with the completed background video in a partitioned manner with the target mask sequence to output the target video with anti-distortion background. This invention can improve the temporal stability and boundary accuracy of human body masking, reduce the background completion range and computational overhead, reduce background stretching distortion and compositing traces, and improve the naturalness and processing efficiency of video body shaping effects.

Description

Technical Field

[0001] This application relates to the field of video image processing technology, and in particular to a background anti-distortion method and system for processing human body figures in videos. Background Technology

[0002] With the development of short video and live streaming applications, users' demand for portrait retouching functions such as slimming, leg lengthening, and arm slimming is constantly increasing. Existing video body retouching solutions typically perform deformation processing on the portrait and background simultaneously, which easily causes distortions such as background stretching and warping, affecting the final visual effect. To achieve a body retouching effect with a natural and distortion-free background, it is usually necessary to complete three steps in sequence: human body segmentation, background removal, and body compositing. Human body segmentation is used to separate the portrait from the background, background removal is used to obtain a clean background, and body compositing is used to merge the deformed portrait with the background for output. In existing technologies, human body segmentation mostly adopts a frame-by-frame independent processing method, and background removal mostly adopts a general global completion method. There is a lack of collaborative optimization for video body retouching scenarios, resulting in problems such as insufficient stability of masking timing, background susceptibility to deformation interference, and low processing efficiency, making it difficult to simultaneously achieve both retouching effect and running performance. Summary of the Invention

[0003] To at least partially overcome the lack of collaborative optimization for video body shaping scenarios in related technologies, which leads to insufficient stability of masking timing, susceptibility of background to deformation interference, and low processing efficiency, this application provides a background anti-distortion method and system for video human body shaping.

[0004] The proposed solution is as follows:

[0005] According to a first aspect of the embodiments of this application, a background anti-distortion method applied to video human body processing is provided, comprising: Obtain the original video; The original video is subjected to frame-by-frame transition detection and human detection, and the detected human bodies are assigned identity tags and cross-frame identity associations are performed. Calculate a quality score for the detected human body, and construct a human body instance based on the human body's quality score, identity identifier, and the occurrence of the same human body in consecutive non-transition frames; Based on the quality score of each human body instance, video segments are divided, and reference frames for each video segment are determined; Human body segmentation is performed based on each reference frame, and the human body mask corresponding to the reference frame is synchronized to other video frames in the video segment to obtain the human body mask sequence corresponding to the original video. Based on the human body masking sequence, the foreground of the human image is extracted from the original video, and a preset deformation simulation is applied to the original video and the human body masking sequence to obtain a simulated deformation video and a simulated deformation masking sequence. The local background removal region is determined based on the difference region between the human body masking sequence and the simulated deformation masking sequence, and background completion is performed only on the local background removal region to obtain a background-completed video. The user deforms the foreground of the portrait according to the user's body shaping parameters, and simultaneously obtains the target mask sequence corresponding to the deformed foreground of the portrait; Based on the completed background video, the original video, and the target mask sequence, the background is partitioned and mixed, and the distorted foreground image is pasted back into the mixed background to output the target video with anti-distortion background.

[0006] Preferably, the transition detection includes: Perform size normalization on the current video frame and scale the current video frame to a preset threshold; Convert the color image to a grayscale image, perform corner detection on the grayscale image, extract significant feature points, and set an upper limit on the number of feature points and a minimum spacing between feature points; Calculate the corresponding positions of feature points in the previous video frame in the current video frame, and record the tracking status and tracking error of each feature point; The proportion of effective tracking points and the average tracking error of the current video frame are statistically analyzed. When the average tracking error exceeds the first preset threshold or the proportion of effective tracking points is lower than the second preset threshold, the current video frame is determined to be a transition frame. When the current video frame is a transition frame, the feature point tracking records of the previous video frame are cleared, and the feature points of the current video frame are re-extracted as the tracking reference for subsequent video frames; when the current video frame is not a transition frame, the tracking status and tracking error of each feature point of the current video frame are used as the reference for calculation of the next video frame.

[0007] Preferably, human detection and cross-frame identity association include: A pre-trained human detection model is applied to the current video frame, and a list of detected human bodies is output. The list of human bodies includes human body information for each detected human body. The human body information includes: bounding box coordinates, key point coordinates, and detection confidence. Filter out human information with a detection confidence level lower than a preset threshold; When the current video frame is a non-transition frame and there are human detection results from the previous video frame, a pose distance matrix is ​​constructed based on the pose distance between each human body in the previous video frame and the current video frame, and cross-frame identity matching is performed based on the pose distance matrix. Pass the identity identifier of the human body in the previous video frame that was successfully matched to the corresponding human body in the current video frame. Assign a new identity identifier to any human body in the current video frame that does not match an existing identity identifier. When the current video frame is a transition frame, clear the identity information and tracking records of all human bodies in the previous video frame, and reassign identity identifiers to each human body detected in the current video frame. The pose distance between each human body in the previous video frame and the current video frame is calculated from the coordinates of the key points.

[0008] Preferably, the process of calculating a quality score for the detected human body and constructing a human body instance includes: The minimum bounding rectangle of the human body is generated by fitting the coordinates of multiple key points of each human body in the current video frame, and the geometric center of the minimum bounding rectangle is taken as the center position of the human body. The coordinates of the center position of the human body are normalized based on the coordinate range of the frame size. A position score is determined based on the distance between the center of the human body and the center of the image; a size score is determined based on the ratio of the area of ​​the minimum bounding rectangle to the frame area; and a comprehensive score is obtained based on the position score and the size score. The comprehensive score is then mapped to multiple discrete levels. The human detection results and comprehensive scoring results of all frames of the video are traversed, and human instance records are constructed according to identity identifiers; the human instance records include at least the first occurrence frame index, the last occurrence frame index, the frame-by-frame scoring sequence, and the corresponding frame key point information.

[0009] Preferably, the video segments are divided based on the quality score of each human body instance, and reference frames for each video segment are determined, including: The frame-by-frame scoring sequence of each human body instance is subjected to maximum value filtering and smoothing. Using the turning point of the rating trend as the instance segmentation point, the corresponding human instance is split into multiple sub-instances; the turning point is the turning point where the rating trend changes from continuous decline to continuous rise, or from continuous rise to continuous decline. For each sub-instance, find the highest score value from its frame-by-frame scoring sequence, and identify the continuous frame interval that reaches the highest score value as the best scoring interval; when there are multiple best scoring intervals, select the interval with the longest duration as the best scoring interval for the corresponding sub-instance. The sub-instances are sorted in descending order of frame length. For the first sub-instance, a video segment is created based on the frame range in which it appears, and the start frame, end frame, and best score range of the video segment are recorded. For each subsequent sub-instance, the intersection of the frame indices between the best score range of the current sub-instance and the best score ranges of each created video segment is calculated. When the intersection of the frame indices is not empty, the current sub-instance is merged into the corresponding video segment, and the start frame, end frame, and best score range of the video segment are updated; when the intersection of the frame indices is empty, a new video segment is created based on the occurrence frame range of the current sub-instance, and the start frame, end frame, and best score range of the new video segment are recorded. For each merged video segment, the midpoint frame of the best scoring interval is selected as the reference frame; Based on the reference frames of each video segment, human body segmentation is performed by combining human body key point constraints and human body bounding box constraints to obtain the human body mask corresponding to each reference frame. The human body mask corresponding to each reference frame is synchronized to other video frames within the video segment through optical flow mapping, and the boundary of the synchronized human body mask is adjusted in combination with the human body detection results of each video frame to maintain the temporal consistency of the human body mask within the same video segment. The human body masks corresponding to the video frames in each video segment are integrated to generate a human body mask sequence corresponding to the original video; wherein, the mask value of the human body region is 1, and the mask value of the background region is 0.

[0010] Preferably, background completion is performed on the local background removal area to obtain a background-completed video, including: For each video frame, a synthesis strategy is used to construct the input frame of the elimination model, which consists of the original background, the gray-filled elimination area, and the deformed foreground image. The synthesis strategy includes: retaining the background pixels in the original video that are outside the local background elimination area and outside the deformed foreground image, filling the local background elimination area with a uniform gray image, and retaining the deformed foreground image pixels. The input frame sequence of the elimination model and the corresponding local background elimination region masking sequence are divided into multiple consecutive frame segments according to the preset frame window length; when the number of frames in the last consecutive frame segment is less than the preset frame window length, the frame is supplemented by repeated frame supplementation. Each consecutive frame segment and the corresponding local background removal region masking sequence are sequentially input into the pre-trained removal model. The removal model is used to simultaneously extract single-frame spatial features and inter-frame temporal correlation features, and to perform background completion for the local background removal region of each video frame. The background segments are stitched together, and duplicate frames are removed to obtain the background-complete video.

[0011] Preferably, the training process for eliminating the model includes: Collect a portrait video dataset, a portrait matting video dataset corresponding to the portrait video dataset, and a pure background video dataset; Randomly extract a preset window length of frames of portrait videos and their corresponding masks from the portrait video dataset, apply maximum deformation to obtain the deformed video and the deformed mask, and determine the local background removal area based on the difference between the masks before and after deformation; Randomly select a preset window length of frames from the pure background video dataset as the real background labels; The training input frame is constructed according to the synthesis strategy, and a training sample is formed consisting of the training input frame, the local background elimination region mask, and the real background label. Data augmentation is performed on the training samples to increase the number of training samples; The elimination model is trained using a multi-objective loss function based on the data-augmented training samples. The multi-objective loss function includes at least pixel-level consistency loss, texture structure loss, and adversarial loss. An optimizer and learning rate decay strategy are set during training until the loss converges and stabilizes. The weight coefficients of each loss term in the multi-objective loss function are adjusted according to the actual scenario.

[0012] Preferably, user-defined deformation is performed on the foreground of the portrait based on user-defined body shaping parameters, and a target mask sequence corresponding to the deformed foreground is obtained simultaneously, including: The original video, human body masking sequence, background video completion, and user body shaping parameters are input synchronously, and frame alignment and format standardization are completed; wherein, the user body shaping parameters include at least deformation type, deformation intensity, and deformation range; Based on the human body masking sequence, the foreground of the human figure is extracted from the original video, and a user deformation operation formed by the user's body shaping parameters is applied to the foreground of the human figure to obtain the deformed foreground of the human figure. The same deformation processing as the user deformation operation is applied to the human body mask sequence synchronously to obtain the target mask sequence.

[0013] Preferably, background partitioning and mixing are performed based on the completed background video, the original video, and the target mask sequence, and the distorted foreground image is pasted back into the mixed background to output a background-anti-distortion target video, including: In the non-human portrait area, the background pixels in the original video are retained, and the background pixels in the original human portrait area are used to complete the background in the complete background video. At the junction of the original human portrait area and the non-human portrait area, a pixel-weighted gradient method is used to make a smooth transition to obtain a mixed background. The deformed foreground of the portrait is pasted back into the blended background, and edge feathering, color consistency correction and shadow reconstruction are performed based on the target mask sequence; The synthesized video frame sequence is processed with frame timing smoothing, contrast fine-tuning, and noise suppression to output a target video that meets playback requirements.

[0014] According to a second aspect of the embodiments of this application, a background anti-distortion system for video human body processing is provided, comprising: Processor and memory; The processor and memory are connected via a communication bus: The processor is used to call and execute the program stored in the memory; The memory is used to store a program, which is at least used to execute a background anti-distortion method for video human body processing as described in any of the above.

[0015] The technical solution provided in this application may include the following beneficial effects: This application uses the original video as the processing object. It first generates a human mask sequence corresponding to the original video through transition detection, human detection, cross-frame identity association, reference frame determination, and human segmentation and mask synchronization based on the reference frame. Since this mask sequence is not obtained independently frame by frame, but rather generated by combining video segments, reference frames, and cross-frame synchronization relationships, it provides more stable human body region boundaries for subsequent foreground extraction and background processing, reducing problems such as human body edge jitter, mask misalignment, and inaccurate foreground-background separation caused by frame-by-frame fluctuations. Furthermore, this application does not directly perform uniform deformation on the entire video frame. Instead, it first applies a preset deformation simulation to the original video and the human mask sequence, then determines local background removal areas based on the differences between the masks before and after deformation, and performs background completion only on these local areas. Therefore, it can limit the background processing scope to the local areas newly exposed due to human image deformation, avoiding redundant processing of irrelevant backgrounds, thereby reducing the difficulty and computational cost of background completion. Subsequently, the foreground of the portrait is independently deformed according to the user's body shaping parameters. The deformed foreground of the portrait is then mixed and pasted back into the composite with the completed background video in separate areas. This separates the foreground deformation from the background processing, effectively avoiding the problem of the background being pulled, stretched and distorted synchronously with the portrait in the existing technology. This allows the output target video to maintain the body shaping effect while having better background stability, naturalness of the picture and overall visual consistency.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] Figure 1 This is a schematic flowchart of a background anti-distortion method for video human body processing provided in one embodiment of this application; Figure 2 This is a schematic diagram of the structure of a background anti-distortion system for video human body processing provided in another embodiment of this application.

[0019] Reference numerals: Processor-21; Memory-22. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0021] Example 1 Figure 1 This is a flowchart illustrating a background anti-distortion method for video human body processing, provided in one embodiment of this technical solution. (Refer to...) Figure 1 A background anti-distortion method for video human body processing includes: S11. Obtain the original video; S12. Perform frame-by-frame transition detection and human detection on the original video, assign identity tags to the detected human bodies and perform cross-frame identity association. S13. Calculate the quality score for the detected human body, and construct a human body instance based on the human body's quality score, identity identifier, and the occurrence of the same human body in consecutive non-transition frames. S14. Divide the video segments according to the quality score of each human body instance, and determine the reference frame for each video segment; S15. Perform human body segmentation based on each reference frame, and synchronize the human body mask corresponding to the reference frame to other video frames in the video segment to obtain the human body mask sequence corresponding to the original video. S16. Extract the human foreground from the original video based on the human body masking sequence, and apply a preset deformation simulation to the original video and the human body masking sequence to obtain the simulated deformation video and the simulated deformation masking sequence. S17. Determine the local background removal region based on the difference region between the human body masking sequence and the simulated deformation masking sequence, and perform background completion only on the local background removal region to obtain the background-completed video. S18. Perform user deformation on the foreground of the portrait according to the user's body shaping parameters, and simultaneously obtain the target mask sequence corresponding to the deformed foreground of the portrait. S19. Based on the completed background video, the original video and the target mask sequence, perform background partitioning and mixing, and paste the deformed foreground image back into the mixed background to output the target video with anti-distortion background.

[0022] For ease of understanding, the following explains some key terms in this embodiment: Raw video refers to video data that is captured directly from a camera device without any processing. This video data contains a series of consecutive image frames, recording dynamic information about the scene.

[0023] Transition detection refers to the process of identifying scene or shot transitions in a video. By analyzing visual changes between consecutive frames, it determines whether there are changes in image content or composition, thereby distinguishing different video segments.

[0024] Human detection refers to the process of identifying and locating human targets in video frames. This process typically outputs information such as the bounding box and key points of the human target for subsequent human analysis and processing.

[0025] An identification identifier is a unique identifier assigned to each human body detected in a video. This identifier is used to track the movement and state of the same human body across different video frames, ensuring consistency across frames.

[0026] Cross-frame identity association refers to the process of matching and connecting the detection results of the same human body in consecutive video frames. Through this association, the trajectory of each human body on the video timeline can be established.

[0027] The quality score is a numerical evaluation of the human targets detected in a video frame. This score comprehensively considers factors such as the human's position, size, and sharpness within the frame to measure the reliability and importance of the human detection results.

[0028] A human instance refers to a collection of consecutive human detection results and related information (such as quality scores and keypoint information) with the same identity identifier across the entire timeline of a video. This instance represents a complete human lifecycle in the video.

[0029] A video segment is a sequence of sub-videos that are divided from an original video based on specific criteria (such as changes in the quality score of human instances) and possess a certain degree of continuity and integrity. Each video segment typically revolves around one or more human instances.

[0030] A reference frame is a video frame selected as a baseline within each video segment. This frame typically has high human detection quality or representative poses and is used for human segmentation and mask synchronization within that video segment.

[0031] Human body segmentation refers to the process of separating the human body region from the background in a video frame. This process outputs a human body mask, in which the pixels of the human body region are labeled and the pixels of the background region are distinguished.

[0032] A human body masking sequence refers to a series of human body masking images corresponding to each frame of the original video. This sequence indicates the spatial position and shape of each human target in the video at each moment.

[0033] The foreground of the human figure refers to the portion of the human image extracted from the original video based on a human body masking sequence. This portion contains only human body pixels, while background pixels are removed or made transparent.

[0034] Preset deformation simulation refers to applying a pre-defined, non-user-defined deformation operation to the foreground of a person in the original video. The degree of deformation in the preset deformation simulation is usually greater than the maximum deformation that the user can apply. This simulated deformation is used to generate differences before and after the deformation to help determine the background removal area.

[0035] Local background removal areas refer to the regions where differences appear between the original human body mask and the simulated deformed human body mask after foreground distortion. These areas are usually parts of the background that are exposed or obscured due to human body deformation, requiring background restoration.

[0036] Background completion refers to the process of filling in localized areas of background removal with pixels to make them consistent with the surrounding background content. This process aims to eliminate background holes or unnatural areas caused by human body deformation.

[0037] Background-complete videos refer to video sequences that have undergone background completion processing. In these videos, areas where background removal was performed have been appropriately filled, resulting in a complete and continuous presentation of the background content.

[0038] User-defined body shaping parameters refer to parameters set by the user to adjust the foreground deformation of a portrait according to their own needs. These parameters may include deformation type, deformation intensity, and deformation range.

[0039] User-defined deformation refers to the deformation operation applied to the foreground of a portrait based on the user's body shaping parameters. This operation directly changes the visual shape of the foreground of the portrait.

[0040] A target masking sequence refers to a series of human body masking images corresponding to the foreground of a human figure after user deformation. This sequence reflects the outline and position of the deformed human body.

[0041] Background blending refers to the process of fusing the completed background video, the original video, and the target mask sequence to construct the final background. This process typically uses different background sources in different areas and performs a smooth transition.

[0042] The target video refers to the final output video after background distortion removal processing, including the distorted foreground portrait and the blended background. This video presents a natural, distortion-free visual effect.

[0043] This embodiment provides a method for preventing background distortion when processing human body figures in videos.

[0044] First, the raw video is acquired. This raw video can be acquired in several ways. For example, the video data can be read from a local storage device or downloaded from a remote server via a network interface. As an alternative implementation, the video data can also be received directly from a camera or other video source via a real-time streaming protocol.

[0045] Furthermore, frame-by-frame transition detection and human detection are performed on the acquired raw video, and detected human bodies are assigned identities and cross-frame identity associations are established. Transition detection can be accomplished by comparing pixel differences or histogram similarity between adjacent frames; when the difference exceeds a certain fixed threshold, it is determined to be a transition. Human body detection can employ traditional methods based on image feature matching, such as identifying human contours in the image through pre-defined template matching. After detecting a human body, a digital identifier can be assigned to each human body as an identity identifier. Cross-frame identity association can be accomplished by comparing the spatial position or color features of human bodies in adjacent frames, treating human bodies with similar positions or colors as the same target.

[0046] Subsequently, a quality score is calculated for each detected human body. Human instances are then constructed based on the quality score, identity identifier, and the occurrence count of the same human body in consecutive non-transition frames. After human detection is complete, a quality score can be calculated for each human body. For example, this score can be based on the size of the human body's bounding box; a larger area results in a higher score. Then, based on this quality score, the assigned identity identifier, and the occurrence count of the human body in consecutive non-transition frames, the detection results of human bodies belonging to the same identity identifier are aggregated to form a corresponding human instance.

[0047] Based on this, video segments are divided according to the quality scores of each human instance, and reference frames for each video segment are determined. After constructing the human instances, video segments can be divided based on the quality score sequence of each instance. For example, a score threshold can be set; when the score of a human instance is higher than the threshold for multiple consecutive frames, these frames are divided into a video segment. For each segmented video segment, the starting frame or an intermediate frame of the segment can be selected as the reference frame.

[0048] Next, human body segmentation is performed based on each reference frame, and the human body mask corresponding to the reference frame is synchronized to other video frames within the same video segment to obtain a human body mask sequence corresponding to the original video. After determining the reference frames, human body segmentation can be performed on each reference frame. For example, color- or texture-based image segmentation algorithms can be used to manually or semi-automatically outline the human body contour and generate a human body mask. Then, the human body mask of the reference frame is directly copied or synchronized to other video frames within the same video segment through a translation operation. Finally, the masks from all video segments are stitched together to form a complete human body mask sequence.

[0049] Therefore, based on the human body masking sequence, the foreground of the human figure is extracted from the original video, and a preset deformation simulation is applied to the original video and the human body masking sequence to obtain a simulated deformation video and a simulated deformation masking sequence. Using the generated human body masking sequence, the foreground of the human figure can be cropped from the original video. Subsequently, a preset deformation simulation can be applied to the human figure region in the original video, for example, by scaling or stretching to simulate human body deformation. Simultaneously, the same deformation operation is applied to the human body masking sequence to obtain the simulated deformation video and the simulated deformation masking sequence.

[0050] Furthermore, local background removal regions are determined based on the differences between the human body masking sequence and the simulated deformation masking sequence, and background completion is performed only on these local background removal regions to obtain a completed background video. By comparing the original human body masking sequence and the simulated deformation masking sequence, regions with different pixel values ​​can be identified, and these regions are determined as local background removal regions. Then, background completion is performed only on these local background removal regions. For example, color filling can be used, filling these regions with the average color of the surrounding background or a preset solid color to obtain a completed background video.

[0051] Subsequently, user-input shaping is performed on the foreground of the portrait based on the user's body shaping parameters, and a target mask sequence corresponding to the shaped foreground is simultaneously obtained. The system receives user-inputted body shaping parameters; for example, the user can manually adjust a slider to control the degree of slimming of the portrait. Based on these parameters, a shaping operation is performed on the previously extracted foreground, for example, manually adjusted using the liquefy tool in image processing software. Simultaneously, the same manual shaping operation is applied to the original human body mask sequence to generate a target mask sequence corresponding to the shaped foreground.

[0052] Finally, based on the completed background video, the original video, and the target mask sequence, background partitioning and blending are performed. The deformed foreground image is then pasted back into the blended background, outputting the target video with anti-distortion background. During the background partitioning and blending stage, the background can be divided into portrait and non-portrait regions according to the target mask sequence. In the non-portrait regions, the background pixels of the original video are directly retained. In the original portrait regions, pixels from the completed background video are used. At the boundary between the two, linear interpolation can be used for blending. Finally, the user-deformed foreground image is directly superimposed onto the blended background to form the final video frame, which is then output as the target video.

[0053] This method improves the temporal consistency of human segmentation results by introducing frame-by-frame transition detection, human detection, and cross-frame identity association, combined with quality scoring to construct human instances and segment video segments, thus resolving boundary jumps and visual artifacts. By completing local background removal areas, it avoids the high computational cost and incomplete human image pixels associated with global removal. Therefore, this method can achieve background-free human body processing in videos, outputting visually natural-looking target videos with improved running efficiency.

[0054] Example 2 It should be noted that the transition inspection includes: Perform size normalization on the current video frame and scale the current video frame to a preset threshold; Convert the color image to a grayscale image, perform corner detection on the grayscale image, extract significant feature points, and set an upper limit on the number of feature points and a minimum spacing between feature points; Calculate the corresponding positions of feature points in the previous video frame in the current video frame, and record the tracking status and tracking error of each feature point; The proportion of effective tracking points and the average tracking error of the current video frame are statistically analyzed. When the average tracking error exceeds the first preset threshold or the proportion of effective tracking points is lower than the second preset threshold, the current video frame is determined to be a transition frame. When the current video frame is a transition frame, the feature point tracking records of the previous video frame are cleared, and the feature points of the current video frame are re-extracted as the tracking reference for subsequent video frames; when the current video frame is not a transition frame, the tracking status and tracking error of each feature point of the current video frame are used as the reference for calculation of the next video frame.

[0055] Preferably, after locking the aspect ratio, the longest side of the frame is scaled to a preset threshold to ensure the efficiency of subsequent processing; It should be noted that setting an upper limit on the number of feature points and a minimum spacing between feature points is to ensure that feature points are evenly distributed within the frame and improve tracking stability. Preferably, in this embodiment, the pyramid optical flow algorithm is used to construct an image pyramid hierarchy to calculate the corresponding position of feature points in the previous video frame in the current video frame, and at the same time record the tracking status (success / failure) and tracking error (pixel deviation value) of each feature point. In this embodiment, setting a dual threshold judgment rule can more accurately judge the transition frame; where, the effective tracking point ratio = the number of successfully tracked feature points / the total number of extracted feature points.

[0056] Through the aforementioned transition detection technology, this solution can effectively identify scene transition points in videos. This refined transition detection mechanism significantly improves the accuracy of video content analysis, providing reliable frame boundary information for subsequent human detection, identity association, and background distortion prevention processing. This avoids background distortion or foreground processing errors caused by misjudged transitions, ensuring the visual quality and processing effect of the final output video.

[0057] Example 3 It should be noted that human detection and cross-frame identity association include: Apply a pre-trained human detection model to the current video frame and output a list of detected human bodies. The list includes human body information for each detected human body, including bounding box coordinates, key point coordinates, and detection confidence. Filter out human information with a detection confidence level lower than a preset threshold; When the current video frame is a non-transition frame and there are human detection results from the previous video frame, a pose distance matrix is ​​constructed based on the pose distance between each human body in the previous video frame and the current video frame, and cross-frame identity matching is performed based on the pose distance matrix. Pass the identity identifier of the human body in the previous video frame that was successfully matched to the corresponding human body in the current video frame. Assign a new identity identifier to any human body in the current video frame that does not match an existing identity identifier. When the current video frame is a transition frame, clear the identity information and tracking records of all human bodies in the previous video frame, and reassign identity identifiers to each human body detected in the current video frame. The pose distance between each human body in the previous video frame and the current video frame is calculated from the coordinates of key points.

[0058] In this embodiment, a pre-trained human detection model is applied to the current video frame, and a list of detected human bodies is output. Each human body information includes bounding box coordinates (x1, y1, x2, y2), coordinates of multiple key points (head, torso, limb key points, etc.), and detection confidence. Human targets with confidence scores lower than a preset threshold are filtered out.

[0059] When there is a human detection result in the previous frame and the current frame is a non-transition frame, cross-frame identity association is performed: First, the pose distance between each human in the previous frame and each human in the current frame (based on the normalized value of Euclidean distance of key point coordinates) is calculated, and a pose distance matrix is ​​constructed; a greedy strategy is used for matching, and each time the human body corresponding to the minimum value in the distance matrix is ​​selected for pairing. If the distance is less than the matching threshold, the identity identifier of the human body in the previous frame is passed to the human body in the current frame, and the pairing is removed from the distance matrix; the process is repeated until there is no matching pairing. For human bodies in the current frame that have not been matched with an existing identity, a new unique identity identifier is assigned.

[0060] When the current frame is determined to be a transition frame, the identity information and tracking records of all human bodies in the previous frame are cleared, and a new identity identifier is assigned to each human body detected in the current frame to avoid incorrect association of human body identities across different shot scenes.

[0061] Example 4 It should be noted that the process of calculating quality scores and constructing human instances for detected human bodies includes: The minimum bounding rectangle of the human body is generated by fitting the coordinates of multiple key points of each human body in the current video frame, and the geometric center of the minimum bounding rectangle is taken as the center position of the human body. The coordinates of the center position of the human body are normalized based on the coordinate range of the frame size. The position score is determined based on the distance between the center of the human body and the center of the image, the size score is determined based on the ratio of the area of ​​the smallest bounding rectangle to the area of ​​the frame, and a comprehensive score is obtained based on the position score and the size score. The comprehensive score is then mapped to multiple discrete levels. The human detection results and comprehensive scoring results of all frames in the video are traversed, and human instance records are constructed according to identity identifiers. The human instance record includes at least the first occurrence frame index, the last occurrence frame index, the frame-by-frame scoring sequence, and the corresponding frame key point information.

[0062] Calculate the center position of the human body: Based on the coordinates of 17 key points of each human body in the current frame, fit and generate the smallest bounding rectangle of the human body, and take the geometric center of the rectangle ((x1+x2) / 2, (y1+y2) / 2) as the center position of the human body; normalize the center coordinates to the range of 0-1 (relative to the frame size) to eliminate the influence of different frame sizes on the score.

[0063] Calculate position score: Set the center of the screen as the optimal position (normalized coordinates (0.5, 0.5)), calculate the Euclidean distance between the center of the human body and the center of the screen, and use an inverse proportional function to map the distance to a position score in the range of 0-1. The closer the distance, the higher the score. When the distance exceeds the preset threshold, the position score drops to below 0.2, excluding the possibility of human bodies at the edge position as reference frames.

[0064] Calculate the size score: Calculate the ratio of the area of ​​the smallest bounding rectangle of the human body to the frame area, and set an appropriate proportion range. When the proportion is within this range, the size score is the highest (1.0). When the proportion is lower than 0.2 (the human body is too small) or higher than 0.6 (the human body is too large and easy to occlude), the score is reduced by a linear ratio to ensure that the human body size in the reference frame is appropriate and the features are complete.

[0065] Calculate the overall score: Multiply the position score and the size score to obtain the overall score value (range 0-1), and map the overall score to 5 discrete levels (level 1-5, level 5 is the best). Simultaneously record the score result and corresponding human identity of the current frame, and update it to the human instance list.

[0066] Constructing human instances: Traverse all frames of the video for human detection and scoring results, classify them by identity, and construct an independent instance record for each human. The record includes the frame index of the first appearance of the human, the frame index of the last appearance, the frame-by-frame scoring sequence, and the key point information of the corresponding frame.

[0067] Furthermore, based on the quality score of each human body instance, video segments are divided, and reference frames for each video segment are determined, including: The frame-by-frame scoring sequence of each human body instance is subjected to maximum value filtering and smoothing. Using the inflection point of the rating trend as the instance splitting point, the corresponding human instance is split into multiple sub-instances; the inflection point is the point where the rating trend changes from continuous decline to continuous rise, or from continuous rise to continuous decline. For each sub-instance, find the highest score value from its frame-by-frame scoring sequence, and identify the continuous frame interval that reaches the highest score value as the best scoring interval; when there are multiple best scoring intervals, select the interval with the longest duration as the best scoring interval for the corresponding sub-instance. The sub-instances are sorted in descending order of frame length. For the first sub-instance, a video segment is created based on the frame range in which it appears, and the start frame, end frame, and best score range of the video segment are recorded. For each subsequent sub-instance, the intersection of the frame indices between the best score range of the current sub-instance and the best score ranges of each created video segment is calculated. When the frame index intersection is not empty, the current sub-instance is merged into the corresponding video segment, and the start frame, end frame and best score range of the video segment are updated; when the frame index intersection is empty, a new video segment is created based on the occurrence frame range of the current sub-instance, and the start frame, end frame and best score range of the new video segment are recorded. For each merged video segment, the midpoint frame of the best scoring interval is selected as the reference frame; Based on the reference frames of each video segment, human body segmentation is performed by combining human body key point constraints and human body bounding box constraints to obtain the human body mask corresponding to each reference frame. The human body mask corresponding to each reference frame is synchronized to other video frames within the video segment through optical flow mapping, and the boundary of the synchronized human body mask is adjusted in combination with the human body detection results of each video frame to maintain the temporal consistency of the human body mask within the same video segment. The human body masks corresponding to the video frames in each video segment are integrated to generate a human body mask sequence corresponding to the original video; wherein, the mask value of the human body region is 1, and the mask value of the background region is 0.

[0068] It should be noted that maximum value filtering (with a window size of 3 frames) is applied to the frame-by-frame scoring sequence of each human instance to eliminate brief scoring jitters caused by slight changes in posture, making the scoring trend more stable and avoiding misjudgment of segmentation points.

[0069] The original human body instance is split into multiple sub-instances based on the segmentation points. Each sub-instance corresponds to a human body appearance cycle with a stable scoring trend. The continuous frame interval that reaches the highest score value in the sub-instance is the frame segment with the optimal human body pose and the best segmentation quality, and this is determined as the optimal scoring interval.

[0070] For the first sub-instance, a new video segment is created directly based on the frame range in which it appears, and the start frame, end frame, and best score range of the segment are recorded. For subsequent sub-instances, the intersection of the frame index of its best score range and the best score range of the existing segment is calculated. If there is a non-empty intersection (i.e., the frames overlap or are adjacent), the sub-instance is merged into the corresponding segment, and the start frame, end frame, and best score range of the segment are updated. If there is no intersection, a new video segment is created.

[0071] The reference frame is located in the middle of the frame segment with the optimal human pose and the most stable score, providing a reliable benchmark for the segmentation of other frames within the segment.

[0072] Example 5 It should be noted that background completion is performed on the local background removal area to obtain a video with a completed background, including: For each video frame, a synthesis strategy is used to construct the input frame of the elimination model, which consists of the original background, the gray-filled elimination area, and the deformed foreground image. The synthesis strategy includes: retaining the background pixels in the original video that are outside the local background elimination area and outside the deformed foreground image, filling the local background elimination area with a uniform gray image, and retaining the deformed foreground image pixels. The input frame sequence of the elimination model and the corresponding local background elimination region masking sequence are divided into multiple consecutive frame segments according to the preset frame window length; when the number of frames in the last consecutive frame segment is less than the preset frame window length, the frame is supplemented by repeated frame supplementation. Each consecutive frame segment and the corresponding local background removal region mask sequence are sequentially input into the pre-trained removal model. The removal model is used to simultaneously extract single-frame spatial features and inter-frame temporal correlation features, and to perform background completion for the local background removal region of each video frame. The background segments are stitched together, and duplicate frames are removed to obtain the background-complete video.

[0073] Preferably, before completing the background video, this embodiment first performs frame synchronization processing on the original video and the human body mask sequence to ensure that each frame of video matches the corresponding mask one by one, avoiding timing misalignment errors.

[0074] Then, a human body masking sequence is used to mask and extract the foreground region of the human figure frame by frame from the original video. After extraction, the foreground region retains the original image pixels, while the non-foreground region is temporarily set to transparent in preparation for subsequent compositing.

[0075] Based on common body modification scenarios in business applications (slimming, leg lengthening, arm slimming, etc.), a set of preset deformation operations is defined, and each deformation parameter is set to the maximum value allowed by the system to simulate the user's extreme modification effect. This deformation simulation is applied to the original video and the human body masking sequence to obtain the simulated deformation video and the simulated deformation masking sequence.

[0076] Calculate the elimination region, which is defined as the region of difference between the masking before and after deformation.

[0077] After completing the above steps, perform background completion for the local background removal area in this embodiment.

[0078] Preferably, in the synthesis strategy, a uniform gray image with a grayscale value of 127 is used to fill the local background removal area.

[0079] Preferably, each segment is spliced ​​into a frame sequence, the duplicate frames used for the last segment of completion are removed, and the complete background video sequence is restored to ensure that all pixels in the elimination area are replaced with content that is consistent with the surrounding background texture and lighting, without obvious completion traces or inter-frame splicing gaps.

[0080] The training process for the elimination model includes: Collect portrait video datasets, portrait matting video datasets corresponding to the portrait video datasets, and pure background video datasets; Randomly extract a number of frames of portrait videos with a preset window length and their corresponding masks from the portrait video dataset, apply maximum deformation to obtain the deformed video and the deformed mask, and determine the local background removal area based on the difference between the masks before and after deformation; Randomly select a preset window length of frames from the pure background video dataset as the real background labels; Training input frames are constructed according to the synthesis strategy, and training samples are formed consisting of training input frames, local background elimination region masks, and real background labels. Perform data augmentation on the training samples to increase the number of training samples; A multi-objective loss function is used to train the elimination model based on the data-augmented training samples. The multi-objective loss function includes at least pixel-level consistency loss, texture structure loss, and adversarial loss. An optimizer and learning rate decay strategy are set during training until the loss converges and stabilizes. The weight coefficients of each loss term in the multi-objective loss function are adjusted according to the actual scenario.

[0081] Preferably, the sample size is expanded by data augmentation techniques such as random flipping, brightness adjustment, and slight rotation to improve the model's generalization ability.

[0082] Example 6 It should be noted that user-defined body shaping is performed on the foreground of the portrait based on the user's body shaping parameters, and a target mask sequence corresponding to the shaped foreground is obtained simultaneously, including: The original video, human body masking sequence, background video completion, and user body shaping parameters are input simultaneously, and frame alignment and format standardization are completed; among them, user body shaping parameters include at least deformation type, deformation intensity, and deformation range; Based on the human body masking sequence, the foreground of the human figure is extracted from the original video, and a user deformation operation formed by the user's body shaping parameters is applied to the foreground of the human figure to obtain the deformed foreground of the human figure. Simultaneously apply the same deformation processing as the user's deformation operation to the human body mask sequence to obtain the target mask sequence.

[0083] In this embodiment, the user's body shaping parameters generally include: deformation type, intensity level 0-100, and range, forming a transformation function.

[0084] The foreground of the human figure is extracted from the original video based on the human body masking sequence, and the non-foreground areas are set to transparent.

[0085] Simultaneously apply the same deformation processing as the user's deformation operation to the human body mask sequence to obtain the target mask sequence, ensuring that the human body mask sequence matches the boundary of the deformed human image.

[0086] Furthermore, based on the completed background video, the original video, and the target mask sequence, background partitioning and blending are performed, and the distorted foreground image is pasted back into the blended background to output the background-anti-distortion target video, including: In the non-human area, the background pixels in the original video are retained, and in the original human area, the background pixels in the original video are filled in. At the boundary between the original human area and the non-human area, a pixel-weighted gradient method is used to make a smooth transition, resulting in a mixed background. The deformed foreground of the portrait is pasted back into the blended background, and edge feathering, color consistency correction and shadow reconstruction are performed based on the target mask sequence; The synthesized video frame sequence is processed with frame timing smoothing, contrast fine-tuning, and noise suppression to output a target video that meets playback requirements.

[0087] The hybrid logic in this embodiment is as follows: non-human areas retain the original pixels of the original video to preserve high-definition details; the original human areas use the supplemented background video to supplement pixels; and the mask edges are smoothly transitioned through pixel-weighted gradients to avoid abrupt changes.

[0088] Edge feathering includes: applying a 2-3 pixel Gaussian blur to the edges to generate a gradient alpha channel; color consistency correction includes: fine-tuning the foreground hue to match the background lighting; shadow reconstruction includes: generating soft shadows that match the human body contours to enhance realism.

[0089] In summary, this technical solution uses the original video as the processing object. It first generates a human mask sequence corresponding to the original video through transition detection, human detection, cross-frame identity association, reference frame determination, and human segmentation and mask synchronization based on the reference frame. Since this mask sequence is not obtained independently frame by frame, but rather generated by combining video segments, reference frames, and cross-frame synchronization relationships, it provides more stable human body region boundaries for subsequent foreground extraction and background processing, reducing problems such as human body edge jitter, mask misalignment, and inaccurate foreground-background separation caused by frame-by-frame fluctuations. Furthermore, this technical solution does not directly apply uniform deformation to the entire video frame. Instead, it first applies a preset deformation simulation to the original video and the human mask sequence, then determines local background removal areas based on the differences between the masks before and after deformation, and performs background completion only on these local areas. Therefore, it limits the background processing scope to the local areas newly exposed due to human image deformation, avoiding redundant processing of irrelevant backgrounds, thereby reducing the difficulty and computational cost of background completion. Subsequently, the foreground of the portrait is independently deformed according to the user's body shaping parameters. The deformed foreground of the portrait is then mixed and pasted back into the composite with the completed background video in separate areas. This separates the foreground deformation from the background processing, effectively avoiding the problem of the background being pulled, stretched and distorted synchronously with the portrait in the existing technology. This allows the output target video to maintain the body shaping effect while having better background stability, naturalness of the picture and overall visual consistency.

[0090] The following example will provide a more detailed explanation of the above technical solution: Suppose we have an original video in which user A performs a dance in an outdoor setting. The video includes multiple camera transitions, changes in user A's posture, and dynamic elements in the background. This video needs to undergo body reshaping to optimize user A's proportions while avoiding background distortion or visual artifacts.

[0091] First, the system acquires the original video. Then, the system processes the original video frame by frame. During processing, the system performs transition detection. Specifically, the system normalizes the size of each video frame, scaling it to a preset threshold, and converts the color image to a grayscale image. Next, corner detection is performed on the grayscale image to extract salient feature points, setting an upper limit on the number of feature points and a minimum distance between them. The system calculates the corresponding positions of feature points from the previous video frame in the current video frame and records the tracking status and tracking error of each feature point. By statistically analyzing the proportion of effective tracking points and the average tracking error of the current video frame, when the average tracking error exceeds a first preset threshold, or the proportion of effective tracking points is lower than a second preset threshold, the system determines the current video frame as a transition frame. For example, when the video switches from one scene to another, the tracking error of feature points increases, and the proportion of effective tracking points decreases; at this time, the system identifies the transition frame. When the current video frame is a transition frame, the system clears the feature point tracking records of the previous video frame and re-extracts the feature points of the current video frame as the tracking reference for subsequent video frames, thereby avoiding the problem of cross-scene incorrect association in existing technologies. When the current video frame is a non-transition frame, the system uses the tracking status and tracking error of each feature point in the current video frame as the benchmark for calculating the next video frame.

[0092] Simultaneously, the system performs human detection on video frames. The system applies a pre-trained human detection model to the current video frame, outputting a list of detected human bodies, including their information such as bounding box coordinates, keypoint coordinates, and detection confidence. The system filters out human information with a detection confidence below a preset threshold. When the current video frame is a non-transition frame and human detection results from a previous video frame exist, the system constructs a pose distance matrix based on the pose distances between each human body in the previous and current video frames, and performs cross-frame identity matching based on this matrix. For example, by comparing the keypoint coordinates of user A in consecutive frames, the system can transfer user A's identity from the previous frame to the current frame, maintaining identity consistency even when user A experiences brief occlusion or pose changes, solving the problem of maintaining cross-frame consistency of human identity in multi-person dynamic scenes in existing technologies. For human bodies in the current video frame that do not match an existing identity, the system assigns a new identity. When the current video frame is a transition frame, the system clears the identity information and tracking records of all human bodies in the previous video frame, and reassigns identity identifiers to each human body detected in the current video frame, ensuring the accuracy of identity recognition after the transition.

[0093] After completing human detection and identity association, the system calculates a quality score for the detected human bodies and constructs human instances. The system generates a minimum bounding rectangle for each human body based on the coordinates of multiple key points in the current video frame, taking its geometric center as the human body's center position. The coordinates of the human body's center position are normalized based on the frame size's coordinate range. The system determines a position score based on the distance between the human body's center position and the frame center, a size score based on the ratio of the minimum bounding rectangle's area to the frame area, and a comprehensive score based on both the position and size scores. This comprehensive score is then mapped to multiple discrete levels. For example, when user A is located in the center of the frame and occupies a large proportion, their comprehensive score is higher. The system iterates through all frames of the video for human detection results and comprehensive score results, constructing human instance records by identity identifier. These records include the first-appearance frame index, the last-appearance frame index, the frame-by-frame score sequence, and the corresponding frame key point information.

[0094] Next, the system divides video segments based on the quality scores of each human instance and determines the reference frames for each video segment. The system performs maximum value filtering smoothing on the frame-by-frame score sequence of each human instance to eliminate score fluctuations. For example, for user A's score sequence, the system smooths out short-term score drops or rises. The system uses the inflection point of the score change trend as the instance splitting point, dividing the corresponding human instance into multiple sub-instances. For example, when user A's score changes from a continuous decline to a continuous rise, this point is identified as the inflection point, and user A's instance is split. For each sub-instance, the system finds the highest score value from its frame-by-frame score sequence and identifies the continuous frame interval reaching the highest score value as the optimal score interval. When multiple optimal score intervals exist, the system selects the longest interval as the optimal score interval for the corresponding sub-instance. The system arranges each sub-instance in descending order of frame length. For the first sub-instance, a video segment is created based on the frame range in which it appears, and the start frame, end frame, and optimal score interval of the video segment are recorded. For each subsequent sub-instance, the system calculates the intersection of the frame indices between the best scoring range of the current sub-instance and the best scoring ranges of all previously created video segments. When the frame index intersection is not empty, the system merges the current sub-instance into the corresponding video segment and updates the start frame, end frame, and best scoring range of that video segment. When the frame index intersection is empty, the system creates a new video segment based on the frame range in which the current sub-instance appears and records the start frame, end frame, and best scoring range of the new video segment. In this way, the system can effectively divide the frame segments of user A in different performance stages or different scenarios.

[0095] For each merged video segment, the system selects the midpoint frame of the best scoring interval as the reference frame. For example, in a dance performance by user A, the system selects the frame with clear posture and a high score as the reference frame. Based on the reference frames of each video segment, the system performs human body segmentation by combining human keypoint constraints and human body bounding box constraints, obtaining the human body mask corresponding to each reference frame. Unlike the existing frame-by-frame independent segmentation method, this scheme synchronizes the human body mask corresponding to each reference frame to other video frames within the same video segment through optical flow mapping, and adjusts the boundaries of the synchronized human body mask based on the human detection results of each video frame to maintain the temporal consistency of the human body mask within the same video segment, effectively solving problems such as poor temporal consistency, boundary jumps, and flickering in the inter-frame segmentation results. Finally, the system integrates the human body masks corresponding to the video frames within each video segment to generate a human body mask sequence corresponding to the original video, where the mask value for the human body region is 1 and the mask value for the background region is 0.

[0096] Next, the system extracts the foreground of the human figure from the original video based on the human figure masking sequence, and applies a preset deformation simulation to the original video and the human figure masking sequence to obtain a simulated deformation video and a simulated deformation masking sequence. For example, the system simulates the deformation of user A's body becoming thinner, obtaining the simulated deformed foreground of the human figure and its mask.

[0097] Then, the system determines the local background removal region based on the difference between the human body masking sequence and the simulated deformation masking sequence. For example, when user A's waist is simulated to shrink, the difference between the original waist area and the deformed waist area is the local background removal region. The system performs background completion only on this local background removal region to obtain a completed background video. Unlike existing solutions that use a general video removal algorithm to globally remove and complete the entire human body masking region, this solution reduces the difficulty and computational cost of completion by only completing the local background removal region, and avoids the problem of incomplete human image pixels easily remaining when shrinking the removal region. Specifically, for each video frame, the system constructs the removal model input frame using a synthesis strategy of the original background, gray-filled removal region, and deformed human image foreground. This synthesis strategy includes: retaining background pixels in the original video that are outside the local background removal region and outside the deformed human image foreground, filling the local background removal region with a uniform gray image, and retaining the deformed human image foreground pixels. The system divides the input frame sequence of the elimination model and the corresponding local background elimination region masking sequence into multiple consecutive frame segments according to a preset frame window length. When the number of frames in the last consecutive frame segment is less than the preset frame window length, the system uses repeated frame padding to fill in the gaps. The system sequentially inputs each consecutive frame segment and the corresponding local background elimination region masking sequence into the pre-trained elimination model, which is used to simultaneously extract single-frame spatial features and inter-frame temporal correlation features, and performs background padding for the local background elimination regions of each video frame. Finally, the system splices the padded background segments and removes the repeated padding parts to obtain the background-painted video.

[0098] Subsequently, the system performs user-defined deformation on the foreground of the portrait based on the user's body shaping parameters, and simultaneously obtains a target mask sequence corresponding to the deformed foreground. The system synchronously inputs the original video, the human body mask sequence, the completed background video, and the user's body shaping parameters, and completes frame alignment and format standardization. The user's body shaping parameters include at least the deformation type, deformation intensity, and deformation range. Based on the human body mask sequence, the system extracts the foreground of the portrait from the original video and applies the user-defined deformation operation formed by the user's body shaping parameters to the foreground to obtain the deformed foreground. For example, if user A selects the "slim waist" parameter, the system will perform a waist-shrinking deformation on user A's foreground. Simultaneously, the system applies the same deformation processing as the user's deformation operation to the human body mask sequence to obtain the target mask sequence, ensuring a precise correspondence between the deformed portrait and the mask.

[0099] Finally, the system performs background partitioning and blending based on the completed background video, the original video, and the target mask sequence, and then pastes the deformed foreground image back into the blended background, outputting a target video with anti-distortion background. In non-human areas, the system retains the background pixels from the original video; in the original human areas, the system uses the completed background pixels from the completed background video; and a pixel-weighted gradient is used at the boundary between the original human and non-human areas for a smooth transition, resulting in a blended background. This partitioning and blending strategy effectively avoids the problem of background distortion due to human body deformation, resulting in a natural visual effect. The system pastes the deformed foreground image back into the blended background and performs edge feathering, color consistency correction, and shadow reconstruction based on the target mask sequence, further enhancing the realism of the composite. For example, the system adjusts the skin tone of user A after deformation and the hue of the new background, and reconstructs its shadow on the completed background. Finally, the system performs frame timing smoothing, contrast fine-tuning, and noise suppression on the synthesized video frame sequence to output a target video that meets playback requirements. This solves the problems commonly found in existing video body shaping technologies, such as background distortion due to human body shape changes, unnatural visual effects, and low operating efficiency.

[0100] Example 7 Figure 2 This is a schematic diagram of a background anti-distortion system for video human body processing, provided in another embodiment of this technical solution. (Refer to...) Figure 2 A background anti-distortion system for video human body shape processing includes: Processor 21 and memory 22; Processor 21 and memory 22 are connected via a communication bus: The processor 21 is used to call and execute the program stored in the memory 22; The memory 22 is used to store a program, which is at least used to execute one of the background anti-distortion methods applied to video human body processing in the above embodiments.

[0101] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0102] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means at least two.

[0103] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0104] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0105] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0106] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0107] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0108] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0109] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for preventing background distortion in video human body processing, characterized in that, include: Obtain the original video; The original video is subjected to frame-by-frame transition detection and human detection, and the detected human bodies are assigned identity tags and cross-frame identity associations are performed. Calculate a quality score for the detected human body, and construct a human body instance based on the human body's quality score, identity identifier, and the occurrence of the same human body in consecutive non-transition frames; Based on the quality score of each human body instance, video segments are divided, and reference frames for each video segment are determined; Human body segmentation is performed based on each reference frame, and the human body mask corresponding to the reference frame is synchronized to other video frames in the video segment to obtain the human body mask sequence corresponding to the original video. Based on the human body masking sequence, the foreground of the human image is extracted from the original video, and a preset deformation simulation is applied to the original video and the human body masking sequence to obtain a simulated deformation video and a simulated deformation masking sequence. The local background removal region is determined based on the difference region between the human body masking sequence and the simulated deformation masking sequence, and background completion is performed only on the local background removal region to obtain a background-completed video. The user deforms the foreground of the portrait according to the user's body shaping parameters, and simultaneously obtains the target mask sequence corresponding to the deformed foreground of the portrait; Based on the completed background video, the original video, and the target mask sequence, the background is partitioned and mixed, and the distorted foreground image is pasted back into the mixed background to output the target video with anti-distortion background.

2. The method according to claim 1, characterized in that, The transition detection includes: Perform size normalization on the current video frame and scale the current video frame to a preset threshold; Convert the color image to a grayscale image, perform corner detection on the grayscale image, extract significant feature points, and set an upper limit on the number of feature points and a minimum spacing between feature points; Calculate the corresponding positions of feature points in the previous video frame in the current video frame, and record the tracking status and tracking error of each feature point; The proportion of effective tracking points and the average tracking error of the current video frame are statistically analyzed. When the average tracking error exceeds the first preset threshold or the proportion of effective tracking points is lower than the second preset threshold, the current video frame is determined to be a transition frame. When the current video frame is a transition frame, the feature point tracking records of the previous video frame are cleared, and the feature points of the current video frame are re-extracted as the tracking reference for subsequent video frames; when the current video frame is not a transition frame, the tracking status and tracking error of each feature point of the current video frame are used as the reference for calculation of the next video frame.

3. The method according to claim 1, characterized in that, Human detection and cross-frame identity association include: A pre-trained human detection model is applied to the current video frame, and a list of detected human bodies is output. The list of human bodies includes human body information for each detected human body. The human body information includes: bounding box coordinates, key point coordinates, and detection confidence. Filter out human information with a detection confidence level lower than a preset threshold; When the current video frame is a non-transition frame and there are human detection results from the previous video frame, a pose distance matrix is ​​constructed based on the pose distance between each human body in the previous video frame and the current video frame, and cross-frame identity matching is performed based on the pose distance matrix. Pass the identity identifier of the human body in the previous video frame that was successfully matched to the corresponding human body in the current video frame. Assign a new identity identifier to any human body in the current video frame that does not match an existing identity identifier. When the current video frame is a transition frame, clear the identity information and tracking records of all human bodies in the previous video frame, and reassign identity identifiers to each human body detected in the current video frame. The pose distance between each human body in the previous video frame and the current video frame is calculated from the coordinates of the key points.

4. The method according to claim 3, characterized in that, Calculate quality scores for the detected human bodies and construct human instances, including: The minimum bounding rectangle of the human body is generated by fitting the coordinates of multiple key points of each human body in the current video frame, and the geometric center of the minimum bounding rectangle is taken as the center position of the human body. The coordinates of the center position of the human body are normalized based on the coordinate range of the frame size. A position score is determined based on the distance between the center of the human body and the center of the image; a size score is determined based on the ratio of the area of ​​the minimum bounding rectangle to the frame area; and a comprehensive score is obtained based on the position score and the size score. The comprehensive score is then mapped to multiple discrete levels. The human detection results and comprehensive scoring results of all frames of the video are traversed, and human instance records are constructed according to identity identifiers; the human instance records include at least the first occurrence frame index, the last occurrence frame index, the frame-by-frame scoring sequence, and the corresponding frame key point information.

5. The method according to claim 4, characterized in that, Based on the quality score of each human body instance, video segments are divided, and reference frames for each video segment are determined, including: The frame-by-frame scoring sequence of each human body instance is subjected to maximum value filtering and smoothing. Using the turning point of the rating trend as the instance segmentation point, the corresponding human instance is split into multiple sub-instances; the turning point is the turning point where the rating trend changes from continuous decline to continuous rise, or from continuous rise to continuous decline. For each sub-instance, find the highest score value from its frame-by-frame scoring sequence, and identify the continuous frame interval that reaches the highest score value as the best scoring interval; when there are multiple best scoring intervals, select the interval with the longest duration as the best scoring interval for the corresponding sub-instance. The sub-instances are sorted in descending order of frame length. For the first sub-instance, a video segment is created based on the frame range in which it appears, and the start frame, end frame, and best score range of the video segment are recorded. For each subsequent sub-instance, the intersection of the frame indices between the best score range of the current sub-instance and the best score ranges of each created video segment is calculated. When the intersection of the frame indices is not empty, the current sub-instance is merged into the corresponding video segment, and the start frame, end frame, and best score range of the video segment are updated; when the intersection of the frame indices is empty, a new video segment is created based on the occurrence frame range of the current sub-instance, and the start frame, end frame, and best score range of the new video segment are recorded. For each merged video segment, the midpoint frame of the best scoring interval is selected as the reference frame; Based on the reference frames of each video segment, human body segmentation is performed by combining human body key point constraints and human body bounding box constraints to obtain the human body mask corresponding to each reference frame. The human body mask corresponding to each reference frame is synchronized to other video frames within the video segment through optical flow mapping, and the boundary of the synchronized human body mask is adjusted in combination with the human body detection results of each video frame to maintain the temporal consistency of the human body mask within the same video segment. The human body masks corresponding to the video frames in each video segment are integrated to generate a human body mask sequence corresponding to the original video; wherein, the mask value of the human body region is 1, and the mask value of the background region is 0.

6. The method according to claim 1, characterized in that, Background completion is performed on the local background removal area to obtain a background-completed video, including: For each video frame, a synthesis strategy is used to construct the input frame of the elimination model, which consists of the original background, the gray-filled elimination area, and the deformed foreground image. The synthesis strategy includes: retaining the background pixels in the original video that are outside the local background elimination area and outside the deformed foreground image, filling the local background elimination area with a uniform gray image, and retaining the deformed foreground image pixels. The input frame sequence of the elimination model and the corresponding local background elimination region masking sequence are divided into multiple consecutive frame segments according to the preset frame window length; when the number of frames in the last consecutive frame segment is less than the preset frame window length, the frame is supplemented by repeated frame supplementation. Each consecutive frame segment and the corresponding local background removal region masking sequence are sequentially input into the pre-trained removal model. The removal model is used to simultaneously extract single-frame spatial features and inter-frame temporal correlation features, and to perform background completion for the local background removal region of each video frame. The background segments are stitched together, and duplicate frames are removed to obtain the background-complete video.

7. The method according to claim 6, characterized in that, The training process for the elimination model includes: Collect a portrait video dataset, a portrait matting video dataset corresponding to the portrait video dataset, and a pure background video dataset; Randomly extract a preset window length of frames of portrait videos and their corresponding masks from the portrait video dataset, apply maximum deformation to obtain the deformed video and the deformed mask, and determine the local background removal area based on the difference between the masks before and after deformation; Randomly select a preset window length of frames from the pure background video dataset as the real background labels; The training input frame is constructed according to the synthesis strategy, and a training sample is formed consisting of the training input frame, the local background elimination region mask, and the real background label. Data augmentation is performed on the training samples to increase the number of training samples; The elimination model is trained using a multi-objective loss function based on the data-augmented training samples. The multi-objective loss function includes at least pixel-level consistency loss, texture structure loss, and adversarial loss. An optimizer and learning rate decay strategy are set during training until the loss converges and stabilizes. The weight coefficients of each loss term in the multi-objective loss function are adjusted according to the actual scenario.

8. The method according to claim 1, characterized in that, User-defined body shaping parameters are applied to the foreground of the portrait, and a target mask sequence corresponding to the shaped foreground is obtained simultaneously, including: The original video, human body masking sequence, background video completion, and user body shaping parameters are input synchronously, and frame alignment and format standardization are completed; wherein, the user body shaping parameters include at least deformation type, deformation intensity, and deformation range; Based on the human body masking sequence, the foreground of the human figure is extracted from the original video, and a user deformation operation formed by the user's body shaping parameters is applied to the foreground of the human figure to obtain the deformed foreground of the human figure. The same deformation processing as the user deformation operation is applied to the human body mask sequence synchronously to obtain the target mask sequence.

9. The method according to claim 1, characterized in that, Based on the completed background video, the original video, and the target mask sequence, background partitioning and blending are performed, and the distorted foreground image is pasted back into the blended background to output a background-anti-distortion target video, including: In the non-human portrait area, the background pixels in the original video are retained, and the background pixels in the original human portrait area are used to complete the background in the complete background video. At the junction of the original human portrait area and the non-human portrait area, a pixel-weighted gradient method is used to make a smooth transition to obtain a mixed background. The deformed foreground of the portrait is pasted back into the blended background, and edge feathering, color consistency correction and shadow reconstruction are performed based on the target mask sequence; The synthesized video frame sequence is processed with frame timing smoothing, contrast fine-tuning, and noise suppression to output a target video that meets playback requirements.

10. A background anti-distortion system for video human body shape processing, characterized in that, include: Processor and memory; The processor and memory are connected via a communication bus: The processor is used to call and execute the program stored in the memory; The memory is used to store a program, which is at least used to execute the background anti-distortion method for video human body processing as described in any one of claims 1-9.

Images