A video processing method, device, storage medium and electronic device

Abstract

The application discloses a video processing method and device, a storage medium and an electronic device, relates to the field of artificial intelligence generated content and computer vision composite technology, and generates a region calibration marker video that can be accurately detected and positioned by a computer vision system in a target region according to a conditional guidance mechanism. In this way, the randomness of the diffusion model is strictly limited in a non-target region through active calibration guidance, pixel-level accurate embedding of specified content is realized, the content distortion problem in the full-modal generation scheme is eliminated, and the definition of the generated AIGC video is improved. According to the adaptive fusion of the specified video according to the frame-by-frame coordinate set, the specified video content is subjected to perspective geometric transformation and seamlessly fused to the target region in a light-adaptive manner, and the geometric accuracy, edge continuity and color consistency meet the visual reality requirements in three dimensions, and meanwhile, the reliability of each frame output is guaranteed through a quality evaluation mechanism, and the stability of the AIGC video in the time sequence dimension is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence-generated content and computer vision combined technology, and more specifically, to a video processing method, apparatus, storage medium and electronic device. Background Technology

[0002] In recent years, large-scale AIGC video generation models have been rapidly iterating. Their core architecture is a spatiotemporal attention network (DiT / U-Net) based on a denoising diffusion model or flow matching, which can generate high-quality video content based on text prompts, reference images, or video clips.

[0003] Existing AIGC video generation technologies have the following drawbacks: 1) The denoising process of the denoising diffusion model is probabilistic sampling. Even if a full-modal reference input is provided, the diffusion model still performs perceptual regeneration of the reference content during inference rather than direct copying, resulting in text deformation, logo blurring, color shift, etc. in the target area, which cannot meet the requirements of pixel-level accurate restoration, resulting in low clarity of AIGC videos; 2) The inherent inter-frame random noise of the denoising diffusion model can also cause content flickering and edge drift in the target area between adjacent frames, which cannot guarantee the stability of the video temporal dimension.

[0004] Therefore, improving the clarity of generated AIGC videos and enhancing the temporal stability of AIGC videos are the problems that this application urgently needs to solve. Summary of the Invention

[0005] In view of this, this application discloses a video processing method, apparatus, storage medium, and electronic device, aiming to improve the clarity of generated AIGC videos and enhance the temporal stability of AIGC videos.

[0006] To achieve the above objectives, the disclosed technical solution is as follows:

[0007] This application discloses a video processing method, the method comprising:

[0008] Based on the conditional guidance mechanism, generate a region-marked video in the target region of the specified video;

[0009] Perform multi-path heterogeneous detection frame by frame on the video of the calibrated region to obtain the coordinate estimate of the target region;

[0010] The coordinates of the target area are estimated and tracked by a preset filtering and tracking mechanism to obtain a frame-by-frame coordinate set;

[0011] The specified video is adaptively fused based on the frame-by-frame coordinate set to obtain a fused video frame.

[0012] The quality of the fused video frames is evaluated to obtain a target fused video that meets the quality requirements.

[0013] A second aspect of this application discloses a video processing apparatus, the apparatus comprising:

[0014] The generation unit is used to generate a region-labeled video in the target region of a specified video according to a conditional guidance mechanism.

[0015] Heterogeneous detection unit is used to perform multi-channel heterogeneous detection frame by frame on the area calibration and marking video to obtain the coordinate estimation of the target area;

[0016] The tracking unit is used to track the coordinate estimation of the target area through a preset filtering tracking mechanism to obtain a frame-by-frame coordinate set;

[0017] An adaptive fusion unit is used to adaptively fuse a specified video according to the frame-by-frame coordinate set to obtain a fused video frame.

[0018] An evaluation and acquisition unit is used to perform quality evaluation on the fused video frames in order to obtain a target fused video that meets the quality requirements.

[0019] A third aspect of this application discloses a storage medium including stored instructions, wherein, when the instructions are executed, the device in which the storage medium is located is controlled to perform the video processing method as described in the first aspect.

[0020] The fourth aspect of this application discloses an electronic device including a memory and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors using the video processing method described in the first aspect.

[0021] As can be seen from the above technical solutions, this application discloses a video processing method, apparatus, storage medium, and electronic device. The method generates a region calibration marker video in the target area of ​​a specified video according to a conditional guidance mechanism, performs multi-path heterogeneous detection frame by frame on the region calibration marker video to obtain coordinate estimates of the target area, tracks the coordinate estimates of the target area through a preset filtering and tracking mechanism to obtain a frame-by-frame coordinate set, adaptively fuses the specified video according to the frame-by-frame coordinate set to obtain a fused video frame, and performs quality evaluation on the fused video frame to obtain a target fused video that meets the quality conditions.

[0022] Through the above scheme, during the video generation inference stage, the diffusion model is actively guided by a conditional guidance mechanism to generate region-labeled videos that can be accurately detected and located by the computer vision system in the target area. This active labeling guide strictly limits the randomness of the diffusion model to non-target areas, while providing a reliable frame-by-frame spatial reference for subsequent fusion. This achieves pixel-level precise embedding of specified content, completely eliminating content distortion problems in the full-modal generation scheme and improving the clarity of the generated AIGC video. Furthermore, a preset filtering and tracking mechanism tracks the coordinate estimation of the target area, ensuring a smooth frame-by-frame coordinate set even under interference conditions such as camera movement, perspective changes, and inherent inter-frame jitter in the diffusion model. Adaptive fusion of the specified video based on the frame-by-frame coordinate set performs perspective geometric transformation on the specified video content and seamlessly integrates it into the target area using an adaptive lighting approach. This achieves visual realism in three dimensions: geometric accuracy, edge continuity, and color consistency. Simultaneously, a quality evaluation mechanism ensures the reliability of each frame output, improving the temporal stability of the AIGC video. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating a video processing method disclosed in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of the overall processing pipeline for AIGC video fixed content fusion disclosed in an embodiment of this application;

[0026] Figure 3 This is a comparative diagram of the region calibration schemes disclosed in the embodiments of this application;

[0027] Figure 4 This is a schematic diagram of the dual-path detection and Kalman inter-frame tracking algorithm disclosed in an embodiment of this application;

[0028] Figure 5 This is a schematic diagram of the perspective transformation calculation and multi-mode adaptive fusion engine disclosed in the embodiments of this application;

[0029] Figure 6 This is a schematic diagram of the structure of a video processing apparatus disclosed in an embodiment of this application;

[0030] Figure 7 This is a schematic diagram of the structure of the electronic device disclosed in the embodiments of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0033] As the background technology indicates, existing AIGC video generation technologies still perform perceptual regeneration of reference content during inference rather than direct copying. This results in text distortion, logo blurring, and color shifts within the target area, failing to meet the requirements for pixel-level accurate reproduction and leading to low clarity in AIGC videos. Furthermore, the inherent inter-frame random noise in the denoising diffusion model can cause content flickering and edge drift in the target area between adjacent frames, failing to guarantee the stability of the video's temporal dimension.

[0034] To address the aforementioned issues, this application discloses a video processing method, apparatus, storage medium, and electronic device. In the video generation inference stage, a conditional guidance mechanism actively guides the diffusion model to generate region-labeled videos within the target area, which can be accurately detected and located by a computer vision system. This active labeling guide strictly limits the randomness of the diffusion model to non-target areas, while providing a reliable frame-by-frame spatial reference for subsequent fusion. This achieves pixel-level precise embedding of specified content, completely eliminating content distortion issues in the full-modal generation scheme and improving the clarity of the generated AIGC video. Furthermore, a preset filtering and tracking mechanism tracks the coordinate estimation of the target area, ensuring a smooth frame-by-frame coordinate set output even under interference conditions such as camera movement, perspective changes, and inherent inter-frame random jitter in the diffusion model. Adaptive fusion of the specified video based on the frame-by-frame coordinate set performs perspective geometric transformation on the specified video content and seamlessly integrates it into the target area using an adaptive lighting approach. Visual realism is achieved in three dimensions: geometric accuracy, edge continuity, and color consistency. Simultaneously, a quality evaluation mechanism ensures the reliability of each frame output, improving the temporal stability of the AIGC video. Specific implementation methods are described in the following embodiments.

[0035] It should be noted that the video processing method, apparatus, storage medium, and electronic device provided in this application can be used for the combination of artificial intelligence-generated content and computer vision, conditional video generation and inference guidance control technology based on the Denoising Diffusion Probabilistic Model (DDPM) / Flow Matching, multi-path heterogeneous detection and temporal tracking technology of video region features, video perspective transformation and multi-mode fusion synthesis technology based on the Homography Matrix, and automatic video fusion quality evaluation technology based on the Structural Similarity Index Measure (SSIM) / Peak Signal to Noise Ratio (PSNR). This application can be applied to the fields of smart broadcasting and interactive network television (IPTV) content production, commercial advertising creative production, virtual studios and film and television auxiliary production, game asset synthesis, etc. The above are merely examples and do not limit the application fields of the video processing method, apparatus, storage medium, and electronic device provided in this application.

[0036] refer to Figure 1 The image shows a video processing method disclosed in an embodiment of this application. The video processing method mainly includes the following steps:

[0037] S101: Generate a region labeling video in the target region of the specified video according to the conditional guidance mechanism.

[0038] In S101, during the AIGC video generation inference stage, the Conditional Generation Module (CGM) of the AIGC video fixed content fusion overall processing pipeline generates a region-labeled video in the target region of the specified video according to the conditional guidance mechanism.

[0039] The specific workflow of the AIGC video fixed content fusion overall processing pipeline is as follows: Figure 2 As shown. Figure 2 The process includes four stages: Stage 1: AIGC condition-guided generation, generating initial video frames with fixed content; Stage 2: Multi-channel region detection and tracking, detecting target regions and stabilizing tracking across frames; Stage 3: Perspective-adaptive fusion synthesis, geometric correction, and seamless fusion of fixed content; Stage 4: QARM quality assessment output, quality scoring, and obtaining the target fused video. Figure 2It demonstrates the complete data flow and module relationships from AIGC condition-guided generation, multi-channel region detection and tracking, perspective adaptive fusion synthesis to QARM quality assessment output.

[0040] Among them, the conditional guidance mechanism includes chromaticity marker guidance mechanism and geometric marker guidance mechanism, etc.

[0041] The target region refers to the area that can be accurately detected and located by a computer vision system.

[0042] The labeled video is a video used to label regions that are recognized by a computer vision system.

[0043] During the video generation inference stage, the model is guided to generate region-labeled videos in the target region through the aforementioned chroma labeling and geometric labeling guidance mechanisms. Specifically, as follows... Figure 3 As shown. Figure 3 A comparative diagram of regional calibration schemes is shown.

[0044] Figure 3 In the middle, the left side shows the generation effect of the chromatic green screen solution and HSV segmentation annotation; the right side shows the four-corner positioning diagram and corner detection annotation of the ArUco geometric marking solution.

[0045] Figure 3 On the left, H represents hue; S represents saturation; V represents brightness; the red circle is the corner marker; the green area is the target area for the green screen. Figure 3 The black matrix on the right is marked with ArUco; the dashed rectangle is the calibration boundary.

[0046] The specific process of generating region-labeled videos in the target area of ​​a specified video based on the conditional guidance mechanism is shown in A1-A3.

[0047] A1: Determine the carrier scene type of the specified video.

[0048] Among them, the carrier scene types include frontal planar carrier scenes and carrier scenes with perspective distortion.

[0049] A2: If the carrier scene type is a frontal planar carrier scene, construct a region chroma fill control network, and generate a region calibration mark video containing pure color chroma values ​​in the target area of ​​the specified video through the region chroma fill control network.

[0050] In A2, the target region mask map is used as a spatial condition to guide the region chroma fill control network to generate a region labeling video containing pure color chroma values ​​in the target region of the specified video.

[0051] A2 involves a chroma marking guidance mechanism. This mechanism works as follows: If the scene type is a frontal planar scene, a region chroma fill control network based on ControlNet is constructed. The target region mask drawn by the user on the reference frame is used as a spatial condition signal. During the denoising iteration of the diffusion model, chroma consistency constraints are applied to the target region to guide it to stably present a pure green screen color value conforming to the Rec.709 standard. The pure green screen color value in the Rec.709 standard has a hue (H) of approximately 120 degrees, a saturation (S) greater than 90%, and a value (V) greater than 70%. This chroma marking guidance mechanism is suitable for frontal planar scene scenes with regular edges and minimal perspective distortion.

[0052] It should be noted that a frontal planar carrier scene with regular edges and minimal perspective distortion refers to a video carrier area that simultaneously meets the following three conditions:

[0053] Condition (1) Geometric edge feature rules: The boundary of the target area is composed of straight line segments, presenting a regular rectangular or approximately rectangular geometric shape, and the edge direction is clear and can be stably extracted by the binarization algorithm. Typical examples include: flat panel displays in indoor conference rooms, outdoor building wall advertising spaces, LCD advertising screens in elevator cars, column-type advertising boards on subway platforms, and flat poster areas in shopping mall windows, etc.

[0054] Condition (2) The viewing angle is close to orthogonal projection: the angle between the camera optical axis and the normal of the target carrier plane is less than 15 degrees, so that the shape of the target area in the picture is close to its real rectangular shape, the distortion of the length ratio of the four sides is less than 10%, and the deviation of the parallelism of opposite sides is less than 5 degrees. Under this condition, the target area can be approximated as an affine transformation relationship, and high-precision color area extraction can be achieved without complex perspective correction.

[0055] Condition (3) Good carrier planarity: The carrier surface is a flat and rigid plane without obvious curvature, wrinkles or unevenness. The entire target area is in the same geometric plane, thus ensuring that the green screen chromaticity fill presents a uniform color response throughout the area, avoiding local color deviation or uneven reflection caused by surface undulation.

[0056] To facilitate understanding, let's use common objects from daily life as analogies. The so-called "front-facing flat surface" with regular edges and minimal perspective distortion, such as square, flat screens or advertising spaces directly facing the camera, are examples:

[0057] The television screen is photographed from the front in the family living room. The four sides of the television screen are straight. The television screen appears as a regular rectangle in the picture.

[0058] When the speaker is speaking directly to the camera, the projection screen or conference screen hanging on the wall of the conference room appears as a standard rectangular area in the picture.

[0059] When a passenger stands facing the LCD advertising screen on the inner wall of the elevator car, the edges of the screen are straight and the shape is regular when the passenger takes a picture.

[0060] When viewed from the front of a subway station platform or airport waiting area, the freestanding flat advertising lightbox appears as a regular rectangle.

[0061] When photographed from the front, the posters and printed advertisements pasted flat in the shopping mall windows have clear edges and right angles.

[0062] The computer monitor on the desk is directly facing the camera; the screen of the computer monitor is a standard rectangle in the image.

[0063] A3: If the carrier scene type is a carrier scene with perspective distortion, the labeled keyframes are used as image condition input to guide the presentation of a region calibration mark video containing the detected geometric mark pattern at a preset position in the target area.

[0064] Specifically, a predefined marker code is superimposed at a preset position in the target area of ​​a specified video reference keyframe, and the labeled keyframe is used as the image condition input. Through this image condition input, the region calibration mark video containing the detected geometric mark pattern is presented at the preset position in the target area.

[0065] The preset positions of the target area of ​​the specified video are the four corners of the target area.

[0066] Predefined tag codes include, but are not limited to, machine vision reference tag codes (ArUco). The dictionary of ArUco tag codes is DICT_4X4_50, ID 0-3, and side length 32px.

[0067] In A3, a geometric marker guidance mechanism is involved. This mechanism involves overlaying predefined ArUco marker codes (e.g., dictionary DICT_4X4_50, ID 0-3, side length 32px, etc.) at the four corners of the target area in a reference keyframe. The annotated keyframe is used as the image condition input for an image processing adapter (IP-Adapter), guiding the generated video to stably display geometric marker patterns at the four corners of the target area that can be recognized by the detector. This geometric marker guidance mechanism is suitable for carrier scenes with perspective distortion, and the coordinates of the four corners of the target area can be directly used for homography matrix calculation.

[0068] It should be noted that a carrier scene with perspective distortion refers to a scene where the target carrier area presents a clearly non-rectangular perspective projection shape in the video image due to factors such as camera angle, carrier posture, or the geometric characteristics of the carrier itself. Specifically, it covers the following three typical scenarios:

[0069] (1) Shooting scenes from an oblique angle: The angle between the camera's optical axis and the normal to the plane of the carrier is greater than 15 degrees (usually in the range of 20 to 60 degrees), causing the originally rectangular carrier area to appear as an irregular quadrilateral (trapezoidal or general convex quadrilateral) in the picture, with opposite sides no longer parallel and the length of the far side being significantly shorter than the length of the near side. Typical examples include: building wall advertisements shot from an oblique angle on both sides of the street, roadside billboards passing by the side windows of moving vehicles, light-emitting diode displays (LEDs) on high-rise buildings shot from a low angle, and desktop screens on exhibition stands shot from a top-down angle.

[0070] (2) Non-frontal scene of carrier posture: The carrier itself is rotated or tilted relative to the camera, such as the promotional stand that is tilted in the exhibition, the video screen of the handheld device (such as mobile phone, tablet, etc.) in the natural holding posture, and the presentation of the vehicle central control screen from the driver's perspective.

[0071] (3) Camera movement trajectory scene: During the camera movement, the relative pose of the camera and the carrier changes continuously, resulting in the same carrier area presenting a continuously changing perspective shape in different frames of the video, such as the carrier scene under typical film and television camera movement techniques such as surround shooting, push-pull shots, and panning shots.

[0072] In the above scenario, the four corner coordinates of the carrier area cannot be obtained by simple rectangle fitting. It is necessary to use ArUco geometric mark codes to provide four corner references with sub-pixel precision, and then use homography matrix (Homography) to model perspective transformation relationships to achieve precise geometric alignment of the specified content to the target area.

[0073] To facilitate intuitive understanding, consider scenarios involving perspective distortion, such as screens or advertising spaces that appear crooked or no longer regular rectangles. While the actual screen or advertising space is a perfect rectangle, due to the shooting angle or the object's posture, it appears as a trapezoid, rhombus, or irregular quadrilateral in the video. Common examples are as follows:

[0074] While walking down the street, I glanced at a roadside billboard. The billboard itself is square, but because the camera angle wasn't directly facing it, the billboard appeared as a trapezoid in the frame, with objects appearing larger in the foreground and smaller in the background.

[0075] The roadside LED screen, as seen from the passenger's perspective while driving, changes shape frame by frame as the car moves, changing from angled to upright and then back to angled.

[0076] Looking up at the exterior wall of a high-rise building, the advertisement appears as a trapezoid, narrower at the top and wider at the bottom, because it is viewed from below.

[0077] When you hold a phone or tablet at an angle, the screen itself is rectangular, but when you hold it casually, the image on the screen becomes a tilted quadrilateral.

[0078] The in-car center console screen looks like this from the driver's perspective. The driver is not facing the screen directly, so the screen appears tilted in the photo.

[0079] The promotional standees and roll-up banners at the exhibition were placed at an angle to attract visitors from different directions;

[0080] The camera pans around a television set, rotating around the screen as the screen gradually changes from a square shape to various different quadrilateral shapes.

[0081] S102: Perform multi-path heterogeneous detection frame by frame on the region calibration and marking video to obtain the coordinate estimation of the target region.

[0082] In S102, the multi-path heterogeneous detection and Kalman tracking module (DDKTM) of the AIGC video fixed content fusion overall processing pipeline performs multi-path heterogeneous detection frame by frame on the region calibration and marking video to obtain the coordinate estimation of the target region.

[0083] Among them, multi-path heterogeneous detection includes chromaticity path detection, geometric path detection, etc.

[0084] The specific process of performing multi-path heterogeneous detection frame by frame on the region calibration and marking video to obtain the coordinate estimation of the target region is shown in B1-B5.

[0085] B1: Extract the green screen binary mask in the color space using an adaptive threshold.

[0086] B2: Perform morphological erosion and dilation refinement on the green screen binary mask to obtain the refined green screen binary mask.

[0087] B3: Based on the refined green screen binary mask, the coordinates of the minimum circumscribed quadrilateral corner point of the target area are calculated through contour fitting.

[0088] The contour fitting calculation includes, but is not limited to, the minimum area bounding rectangle (minAreaRect) plus the convex hull.

[0089] B1-B3 describes the chroma path detection process. Specifically, chroma path detection involves: extracting a green screen binary mask by setting an adaptive threshold in the color space (Hue, Saturation, Value, HSV); refining it through morphological erosion (e.g., 3x3 kernel, 2 iterations) and dilation (5x5 kernel, 1 iteration); and then extracting the coordinates of the corner points of the minimum bounding quadrilateral of the target region using minAreaRect and convex hull fitting.

[0090] The HSV color space is a color model based on the intuitive characteristics of the human eye. It uses three dimensions—hue, saturation, and value—to describe color. Because it is closer to how humans subjectively perceive color, it is widely used in image processing, computer vision, and art design.

[0091] B4: Locate geometric markers (such as ArUco markers) using a geometric marker detection function, and refine the corner coordinates of the geometric markers to obtain sub-pixel precision corner coordinates.

[0092] Among them, the geometric marker detection functions include, but are not limited to, the cv2.aruco.detectMarkers function.

[0093] B5: Determine the confidence scores corresponding to the corner coordinates of the minimum circumscribed quadrilateral and the sub-pixel precision corner coordinates respectively, and perform weighted fusion of the corresponding corner coordinates based on the confidence scores to obtain the detection coordinate estimate of the target area.

[0094] B4-B5 involve geometric path detection. Geometric path detection calls `cv2.aruco.detectMarkers` to locate ArUco markers, outputs the corner coordinates with sub-pixel precision, and filters out false matches using Random Sample Consensus (RANSAC). RANSAC is used to estimate mathematical models from a set of observations, especially in the presence of significant noise and outliers (called false matches). RANSAC can be used in scenarios such as image matching, correspondence extraction in stereo vision, and plane fitting.

[0095] The two detection results (minimum bounding quadrilateral corner coordinates and sub-pixel precision corner coordinates) are weighted and fused according to their respective confidence scores to output the optimal quadrilateral coordinate estimate for the current frame, which is the coordinate estimate of the target region (optimal coordinate estimate).

[0096] S103: Track the coordinate estimation of the target area through a preset filtering tracking mechanism to obtain a frame-by-frame coordinate set.

[0097] In S103, the coordinate estimation of the target area is tracked through DDKTM to obtain a frame-by-frame coordinate set.

[0098] The preset filtering tracking mechanism includes, but is not limited to, Kalman filter inter-frame tracking. Kalman filter inter-frame tracking is preferred as the preset filtering tracking mechanism in this application.

[0099] Specifically, the process of tracking the coordinates of the target area through a preset filtering tracking mechanism to obtain a frame-by-frame coordinate set is shown in C1-C9.

[0100] C1: Establish a preset filter using a multidimensional state vector.

[0101] In C1, for example, an 8-dimensional state vector (four-point coordinates x1, y1, ..., x4, y4) is used to build a Kalman filter.

[0102] Preset filters include, but are not limited to, Kalman filters.

[0103] C2: Determine the detection confidence of the target region based on the confidence score.

[0104] Specifically, the detection coordinates and detection confidence of the target region in the current frame are obtained, and the state estimate of the previous frame is predicted by using a Kalman filter to obtain the predicted coordinates of the current frame.

[0105] The detection confidence is output from the aforementioned multi-path heterogeneous detection process. The observation noise covariance matrix of the Kalman filter is dynamically adjusted according to the detection confidence. The observation noise covariance matrix is ​​expressed as R_n=50(1-c_n)I_8, where R_n is the observation noise covariance matrix, c_n is the detection confidence of the nth frame, and I_8 is an 8th-order identity matrix.

[0106] C3: Based on the detection confidence level, the detection coordinates of the target area are updated using a preset filter to obtain the updated result.

[0107] If the detection confidence is greater than or equal to the preset confidence threshold, the estimated detection coordinates of the current frame are used as the observation input to perform measurement updates on the Kalman filter, and the update result is obtained. If the detection confidence is less than the preset confidence threshold, the detection of the current frame is determined to be a failure, the measurement update of the current frame is skipped, and the predicted coordinates output by the Kalman filter are used as the update result.

[0108] The preset reliability threshold should be set according to the actual situation, and this application does not impose specific restrictions.

[0109] C4: Calculate the optical flow result based on the corner points of the target region in adjacent video frames.

[0110] In C4, the optical flow results are obtained by tracking the corner displacement between adjacent frames in the target area using a preset optical flow estimation method.

[0111] Specifically, the optical flow results can be obtained by tracking corner displacements between adjacent frames (such as between 5 consecutive frames) using the Lucas-Kanade inter-frame estimation optical flow algorithm (3-layer pyramid, 15x15 window).

[0112] C5: Calculate the optical flow estimation bias based on the updated results and the optical flow results.

[0113] In C5, the optical flow estimation bias is calculated based on the coordinate difference between the updated result and the optical flow result.

[0114] C6: If the optical flow estimation deviation is greater than the preset error, the update result is corrected by the optical flow result, and the corrected update result is determined as the coordinate estimate of the target area of ​​the current frame.

[0115] The preset error can be 8 pixels (px), 9 pixels, etc. For example, when the optical flow estimation of the Kalman update result deviates from the optical flow result by more than 8 pixels, the Kalman update result is corrected with the optical flow result.

[0116] The process described above, from C1 to C6, is Kalman filter inter-frame tracking.

[0117] C7: If the optical flow estimation deviation is less than or equal to the preset error, the update result will be determined as the coordinate estimation of the target region in the current frame.

[0118] C8: Arrange the coordinate estimates of the target region in each frame according to the frame order to obtain the coordinate sequence.

[0119] The specific process of estimating the coordinates of the target region in each frame and arranging them according to frame order is as follows:

[0120] The coordinates of the nth frame are estimated as Q_n={(x_{n,1},y_{n,1}),(x_{n,2},y_{n,2}),(x_{n,3},y_{n,3}),(x_{n,4},y_{n,4})}, and arranged in the order of the first to the Nth frame to obtain the coordinate sequence {Q_1, Q_2, ..., Q_N}.

[0121] Where Q_n is the estimated tracking coordinates of the nth frame; n is an integer greater than or equal to 1 and less than or equal to N, where N is the total number of video frames contained in the region calibration video; x and y are the tracking coordinates of the nth frame.

[0122] C9: Perform adaptive length-shifted median filtering on the coordinate sequence to obtain a frame-by-frame coordinate set.

[0123] C9 involves coordinate temporal smoothing. Coordinate temporal smoothing: an adaptive length-shifting median filter is applied to the output coordinate sequence (the window length can be adaptively adjusted between 3 and 9 frames according to the inter-frame motion amplitude) to eliminate residual high-frequency jitter in the coordinates and output a smooth frame-by-frame coordinate set (such as a frame-by-frame quadrilateral coordinate set {Q_1, Q_2, ..., Q_N}).

[0124] The process of multi-path heterogeneous detection and Kalman filter inter-frame tracking is as follows: Figure 4 As shown. Figure 4 The demonstration shows the processing logic and data flow of each node in the detection layer (chromatic / geometric dual-path), tracking layer (Kalman prediction + optical flow-assisted tracking), and smoothing layer (temporal smoothing filtering, jitter suppression, and trajectory smoothing output).

[0125] Figure 4 In the detection layer, the input video frame is subjected to chroma path detection (HSV thresholding) in the chroma candidate region and geometric path detection (ArUco) in the geometric candidate region to obtain the chroma path detection and geometric path detection results;

[0126] The two detection results (minimum bounding quadrilateral corner coordinates and sub-pixel precision corner coordinates) of the chroma path detection and geometric path detection are fused together. That is, they are weighted and fused according to their respective confidence scores, and the optimal quadrilateral coordinate estimate of the current frame is output, which is the coordinate estimate of the target area (optimal coordinate estimate).

[0127] In the tracking layer, a preset filter is established using a multi-dimensional state vector; the detection coordinates of the target region are determined based on the confidence score, and the corresponding detection confidence is estimated.

[0128] Based on the detection confidence level, the estimated detection coordinates of the target area are updated using a preset filter to obtain the updated result;

[0129] Calculate optical flow results based on the corner points of the target region in adjacent video frames;

[0130] Based on the updated results and the optical flow results, the optical flow estimation bias is calculated;

[0131] If the optical flow estimation deviation is greater than the preset error, the update result is corrected by the optical flow result, and the corrected update result is determined as the coordinate estimate of the target region in the current frame;

[0132] If the optical flow estimation deviation is less than or equal to the preset error, the updated result will be determined as the coordinate estimation of the target region in the current frame;

[0133] The coordinate estimates of the target region in each frame are arranged in frame order to obtain a coordinate sequence;

[0134] An adaptive length-shifting median filter is applied to the coordinate sequence to stabilize the coordinates of the region, resulting in a frame-by-frame coordinate set.

[0135] S104: Adaptively fuse the specified video according to the frame-by-frame coordinate set to obtain the fused video frame.

[0136] In S104, the specified video is adaptively fused using the Photo-Adaptive Fusion Engine (PAFE) and the frame-by-frame coordinate set output by DDKTM to obtain the fused video frame.

[0137] Specifically, this involves adaptively fusing a specified video frame based on a set of frame-by-frame coordinates to obtain the fused video frames, such as D1-D7. Figure 5 As shown. Figure 5 This is a schematic diagram of the perspective transformation calculation and multi-mode adaptive fusion engine. Figure 5 This demonstrates the complete process of a specified video frame undergoing homography matrix calculation, Lie algebra temporal smoothing, perspective transformation (warpPerspective), and three-mode illumination sensing fusion to QARM quality verification.

[0138] Figure 5 It includes a perspective calculation module, a temporal smoothing module, a perspective transformation module, a lighting perception fusion engine, and a QARM quality verification module.

[0139] Figure 5 In this process, the specified video frames (the video frames to be fused and the coordinates of the stable region) undergo homography matrix calculation (homography lookup (four-point pair)), random sampling consistency robust estimation (outer point culling, interior point optimization), Lie algebra mapping (matrix logarithmic mapping), temporal interpolation smoothing (exponential mapping to restore the smoothing matrix), perspective transformation, etc., and are processed in different modes by the illumination-aware fusion engine. Among them, perspective transformation is used to map fixed content to the target region coordinate system. Specifically, through a frame-by-frame perspective transformation matrix sequence, perspective transformation is applied to the nth frame of the specified video content, and geometrically aligned video frames are output.

[0140] The perspective calculation module is used for homography matrix calculation and RANSAC robust estimation;

[0141] The timing smoothing module is used for Lie algebra mapping and timing difference smoothing;

[0142] The perspective transformation module is used for warpPerspective transformation;

[0143] For soft curved edge scenes, the lighting-aware fusion engine uses a Poisson fusion mode; for semi-transparent reflective scenes, it uses an alpha fusion mode; and for hard straight edge scenes, it uses a mask hard replacement fusion mode.

[0144] The process of adaptive selection based on illumination perception is as follows:

[0145] The system first performs brightness difference, local contrast, gradient distribution, texture complexity and edge continuity analysis on the geometrically aligned candidate video frames to determine whether the current area belongs to a scene with relatively uniform lighting and obvious brightness transition, or a scene with more complex texture and details.

[0146] Then, mode routing is performed based on these indicators. For uniform lighting and small inter-frame brightness differences, Alpha blending is selected, and global or local transparency weights are smoothly superimposed, resulting in high computational efficiency.

[0147] For areas with obvious splicing boundaries or significant changes in brightness or color, Poisson blending is selected to maintain the continuity of edge changes in the gradient domain and reduce seams.

[0148] For textures that are rich, detail-sensitive, or have high-frequency structures, frequency domain fusion is selected, and Laplacian pyramid layered fusion is used to achieve smooth transitions in low-frequency layers and preserve texture details in high-frequency layers.

[0149] "Light perception" does not simply detect brightness, but combines lighting differences, gradient continuity and texture complexity to automatically select the most suitable fusion strategy for different regions or different frame pairs; multi-mode partition fusion can also be used when necessary.

[0150] The QARM quality verification module is used for boundary consistency detection of SSIM structural similarity, color difference evaluation of color fidelity, and inter-frame jitter scoring of temporal stability.

[0151] D1: Calculates the sequence of frame-by-frame perspective transformation matrices for the specified video based on the frame-by-frame coordinate set.

[0152] In D1, the frame-by-frame perspective transformation matrix is ​​calculated using the four corners of the rectangle of the specified video frame as the source point set and the frame-by-frame coordinate set as the target point set. The target point set is represented by Q_n.

[0153] Using the four corners of a specified video frame rectangle as the source point set and Q_n as the target point set, call cv2.findHomography(RANSAC mode, reprojection threshold 3px) to calculate the frame-by-frame perspective transformation matrix. The frame-by-frame perspective transformation matrix is ​​represented by H_n.

[0154] D2: Project the frame-by-frame perspective transformation matrix sequence onto a preset linear space and smooth it, then map the smoothed frame-by-frame perspective transformation matrix sequence back to the homography matrix space to obtain a smoothed perspective transformation matrix sequence.

[0155] In D2, the frame-by-frame perspective transformation matrix sequence is projected onto a preset linear space through matrix logarithmic mapping, and weighted smoothing is performed in the preset linear space to map the frame-by-frame perspective transformation matrix sequence back to the matrix space, resulting in a smoothed perspective transformation matrix sequence.

[0156] The preset linear space includes, but is not limited to, the Lie algebra linear space.

[0157] The smooth perspective transformation matrix sequence is represented by H_n'.

[0158] The D2 process is homography matrix temporal smoothing. Homography matrix temporal smoothing involves projecting the H_n sequence onto the Lie algebra (se3 manifold) linear space using a matrix logarithmic mapping, applying Gaussian weighted smoothing (3-frame window, sigma=1.0), and then mapping it back to the matrix space to obtain H_n'. Specifically, for the estimated H_n sequence, the sequence is first scaled to eliminate the influence of the non-unique homogeneous scale of the homography matrix; then, it is projected onto the corresponding Lie algebra linear space using a matrix logarithmic mapping, and weighted smoothing is performed in this space using a Gaussian kernel with a window length of 3 and a standard deviation of 1.0; finally, the smoothing result is restored to the homography matrix space using a matrix exponential mapping, resulting in the temporally smoothed homography smoothed perspective transformation matrix sequence H_n'. This process effectively reduces projection jitter caused by frame-by-frame estimation errors and local noise, thereby improving image stability.

[0159] D3: Apply perspective transformation to the nth frame of the specified video using the nth smooth perspective transformation matrix in the smooth perspective transformation matrix sequence to obtain a geometrically aligned video frame.

[0160] Where n is an integer greater than or equal to 1; geometrically aligned video frames are represented by W_n.

[0161] D3 is the process of calculating the frame-by-frame homography matrix.

[0162] Frame-by-frame homography matrix calculation: Using the four corners of the specified video frame rectangle as the source point set and Q_n as the target point set, call cv2.findHomography(RANSAC mode, reprojection threshold 3px) to calculate the frame-by-frame perspective transformation matrix, that is, the nth smooth perspective transformation matrix (H_n') in the smooth perspective transformation matrix sequence. Then, apply the perspective transformation (such as warpPerspective transformation) to the nth frame of the specified video using the nth smooth perspective transformation matrix in the smooth perspective transformation matrix sequence to obtain the geometrically aligned video frame.

[0163] Where n is an integer greater than or equal to 1.

[0164] D4: Extract the color space statistical features of the geometrically aligned video frames, and use the color space statistical features as a lighting reference to perform tone mapping and color temperature matching on the geometrically aligned video frames to obtain the video frames to be fused.

[0165] Specifically, color space statistical features are extracted from the environmental sampling area outside the target area within a preset range. Using the LAB color space statistical features as a lighting reference, tone mapping and color temperature matching are performed on the geometrically aligned video frames to obtain the video frames to be fused.

[0166] The preset range can be 30px, 40px, etc. This application does not specify a particular preset range.

[0167] D5: Extract the edge characteristics of the target region in the video frame to be fused.

[0168] D6: Determine the fusion mode based on the edge characteristics of the target area.

[0169] The fusion modes include mask hard replacement, Poisson fusion (cv2.seamlessClone), and Alpha fusion.

[0170] The process from D4 to D6 is adaptive illumination-aware fusion. Adaptive illumination-aware fusion: Extract LAB color space statistical features from the environmental sampling area extending 30px outward from the target area as an illumination reference, perform Reinhard tone mapping on the transformed frame W_n to complete brightness alignment and apply color temperature matching.

[0171] The blending mode is selected based on the edge characteristics of the target area. For example, hard mask replacement is used for scenes with hard straight edges (green screen solution); cv2.seamlessClone Poisson blending is used for scenes with soft curved edges (rounded corner screen); and alpha blending (alpha adaptive calculation) is used for areas containing semi-transparent reflective areas.

[0172] D7: In fusion mode, adaptive fusion is performed on the video frames to be fused to obtain fused video frames.

[0173] The fusion mode is determined based on a comprehensive assessment of edge characteristics, lighting characteristics, and other factors, as detailed below:

[0174] (1) For scenes with straight edges, a mask hard replacement fusion mode is adopted:

[0175] Applicable conditions: The target area has sharp straight edges (such as flat panel displays, rectangular billboards, printed posters, etc.), obvious abrupt changes in edge gradient, and no obvious transition zone;

[0176] Specific implementation: Construct a binary mask M_n using the quadrilateral coordinates Q_n output by DDKTM (pixel values ​​within the target area are 1, and outside the target area are 0). Apply a Gaussian blur (sigma=0.5) with a width of 1 pixel to the edge of the mask to eliminate jagged edges; the fusion formula is I_out=M_n•W_n+(1-M_n)•I_bg.

[0177] Where I_out is the fused video frame; M_n is the binary mask; W_n is the specified video frame after perspective transformation; and I_bg is the background frame generated by AIGC. This mode has the lowest computational cost and maintains the best edge sharpness, making it suitable for applications with high requirements for edge clarity.

[0178] For scenes with hard straight edges, a mask hard replacement fusion mode is adopted. Based on the binary mask of the target area, the corresponding pixels in the background video frame are replaced with pixels in the foreground area of ​​the mask in the video frame to be fused in the spatial domain, while retaining the original pixels in the background area of ​​the mask, to obtain the fused video frame.

[0179] (2) For soft curved edge scenes, the Poisson blending mode is adopted:

[0180] Applicable conditions: The target area has curved or rounded rectangle edges (such as rounded corner screen mobile phones, curved TVs, oval decorative frames, curved LED screens, etc.), or there is a significant color / brightness difference between the target area and the background, and it is necessary to eliminate visible splicing seams;

[0181] Implementation details: The OpenCV `cv2.seamlessClone` interface is called, setting the blending mode to `NORMAL_CLONE` (preserving the source image texture) or `MIXED_CLONE` (automatically blending based on gradient intensity), with the geometric center of the target region as the blending center point. This method solves for the gradient field at the blending boundary based on the Poisson equation, allowing the color and brightness of the transformed frame `W_n` to automatically adapt to the gradient distribution of the surrounding background, fundamentally eliminating visible seams at the blending boundary. To maintain computational efficiency, the blending region is locally solved within a ROI range 20 pixels outside the circumscribed rectangle of the quadrilateral.

[0182] (3) For semi-transparent reflective scenes, use the alpha blending mode:

[0183] Applicable conditions: The target area contains semi-transparent or mirror-reflective components (such as glass curtain wall advertisements, reflections on polished screen surfaces, acrylic light boxes, projections on wet roads after rain, etc.), and the target content needs to be superimposed while preserving the original reflection / transmission effect;

[0184] Implementation details: Construct a spatially varying alpha(x, y), whose value is adaptively calculated based on the following three factors:

[0185] Luminance factor: Based on the channel luminance (V) of the target region in the original frame, the alpha value of the highlight region (V>200) is appropriately reduced (0.6-0.75) to preserve the original highlight features;

[0186] Saturation factor: Based on the S channel, low saturation regions (S<50, i.e., potentially reflective regions) are assigned a medium alpha value (0.7-0.8).

[0187] Gradient factor: Based on the Sobel gradient magnitude, the alpha value is increased to 0.85-0.95 in high gradient regions (near the edge) to avoid edge blurring.

[0188] The final fusion formula is I_out(x,y) = alpha(x,y)•W_n(x,y) + (1-alpha(x,y))•I_bg(x,y).

[0189] Where I_out(x,y) is the final fused output pixel value at coordinate (x,y); α(x,y) is the adaptive transparency blending coefficient at coordinate (x,y), with the value of alpha ranging from [0.6, 0.95], ensuring that the target content is clearly distinguishable while preserving the physical characteristics of the carrier surface; W_n(x,y) is the pixel value of the specified video at coordinate (x,y) after perspective transformation of the nth frame; and I_bg(x,y) is the pixel value of the background frame generated by AIGC at coordinate (x,y).

[0190] Automatic fusion mode selection mechanism: Before fusion, the system performs Canny edge detection and Hough line transformation analysis on the edge of the target area. Based on three indicators, the system automatically matches the optimal fusion mode without manual intervention: the proportion of straight segments (>80% is judged as a hard straight edge), curvature characteristics (including the judgment of arc fitting residuals), and brightness variance (high variance is judged as a reflective area).

[0191] Perspective transformation and multi-mode adaptive fusion, specifically as follows: Figure 5 As shown. Figure 5 This demonstrates the complete process of a specified video frame undergoing homography matrix calculation, Lie algebra temporal smoothing, warpPerspective transformation, three-mode illumination sensing fusion, and QARM quality verification.

[0192] S105: Perform quality assessment on the fused video frames to obtain a target fused video that meets the quality requirements.

[0193] In S105, the Automatic Quality Assessment and Reprocessing Module (QARM) calculates the Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Edge Continuity Score for each fused video frame. The SSIM, PSNR, and Edge Continuity Score are used to determine all quality-verified target fused videos that meet the quality requirements.

[0194] The specific process of performing quality assessment on the fused video frames to obtain the target fused video that meets the quality requirements is shown in E1-E4.

[0195] E1: Calculate the Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Edge Continuity Score for each frame of the fused video frame within the target region.

[0196] E2: If any frame's Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), or Edge Continuity Score fails to meet its corresponding preset threshold, the fused video frame is determined to be substandard. In this case, the following steps are executed in sequence: improve illumination adaptation accuracy, switch fusion mode, and re-execute multi-path heterogeneous detection.

[0197] The corresponding preset thresholds are set according to the actual situation, and this application does not impose specific limitations.

[0198] For example, if the similarity index quality index within the target area is greater than or equal to 0.85, the peak signal-to-noise ratio quality index is greater than or equal to 30dB, and the edge continuity score (based on the Canny gradient mutation amount) is greater than or equal to 15, the fused video frame is determined to meet the standards.

[0199] E3: If the fused video frames do not meet the standards after performing the above operations, perform interpolation replacement operations on adjacent qualified frames to obtain the target fused video that meets the quality requirements.

[0200] When any frame fails to meet the quality standard, QARM sequentially attempts to improve the illumination adaptation accuracy, switch the fusion mode, and re-execute multi-path heterogeneous detection (e.g., up to 3 reprocessing steps). If it still fails to meet the standard, it replaces it with adjacent qualified frames through interpolation. The final output is a target fused video that meets the quality conditions, i.e., all of which have been quality verified, and is encoded in H.265 or AV1 format.

[0201] E4: If the structural similarity index quality index, peak signal-to-noise ratio quality index, and edge continuity score of any frame all reach their respective preset thresholds, the fused video frame is determined to meet the standards, and the fused video frame that meets the standards is determined as the target fused video that meets the quality conditions.

[0202] For example, the similarity index (SSIM) quality index within the target area is greater than or equal to 0.85, the peak signal-to-noise ratio (PSNR) quality index is greater than or equal to 30dB, and the edge continuity score (based on the Canny gradient mutation amount, with a threshold greater than or equal to 15) is used to determine whether the fused video frame meets the standards.

[0203] To facilitate understanding of the video processing process in this solution, we will use IPTV promotional video production as Example 1 and dynamic content replacement of outdoor billboards as Example 2 to detail the specific implementation process of this invention:

[0204] Example 1: Precise embedding of IPTV promotional videos into television screens:

[0205] Step S1 (CGM Configuration): Obtain the user-uploaded 1920x1080 reference keyframes, mark the quadrilateral area of ​​the TV screen on the system interface, and the system automatically overlays ArUco markers (DICT_4X4_50, ID 0-3, side length 32px) at the four corners of the marked area. Construct a composite conditional vector of IP-Adapter image conditions and ControlNet mask guidance, input it into the Wan2.1 generative model (50 inference steps, CFG=7.5), and generate a 1920x1080 resolution, 24fps, 10-second guidance video with 240 frames, and the ArUco marker pattern is stably displayed at the four corners of the target.

[0206] Step S2 (DDKTM Dual-Path Detection): Run path B in parallel for 240 frames of video, calling cv2.aruco.detectMarkers to detect the coordinates of four marked corner points frame by frame, and output the detection confidence score. 232 frames were successfully detected (the success condition can be set to a confidence score greater than or equal to 0.6), and 8 frames were marked as frames to be completed due to insufficient detection confidence scores caused by slight occlusion or motion blur.

[0207] Step S3 (Kalman Tracking and Smoothing): Establish an 8-dimensional Kalman filter (Q=diag(0.1)) and initialize it with the detection result of the first frame. Replace the 8 frames to be completed with Kalman prediction values, and simultaneously start LK optical flow to track corner points in the preceding and following 5 frames. The optical flow deviation is within 3px, and the prediction is corrected with the optical flow result. Apply an adaptive window (5 frames) moving median filter to the complete 240-frame coordinate sequence, and the standard deviation of the inter-frame coordinates reaches 1.8px;

[0208] Step S4 (PAFE Perspective Transformation): Prepare video footage of a provincial IPTV system interface (1280x720, 24fps, 10 seconds). For the nth frame, use findHomography(RANSAC, reprojection threshold 3px) to calculate H_n with Q_n as the target point set; perform Lie algebra space Gaussian smoothing (3 frames window, sigma=1.0) on the H_n sequence to obtain H_n'; apply warpPerspective(frame_n, H_n', (1920, 1080)) to the specified nth frame of the video, and output the geometrically aligned transformed frame W_n.

[0209] Step S5 (Light-Aware Fusion): Extract the LAB mean values ​​(L_env approximately 85, a_env approximately -2, b_env approximately 5) from the environmental sampling area extending 30px outward from the target quadrilateral. Perform Reinhard tone mapping on W_n and apply a linear color temperature shift (a shift -2, b shift +5). Since the edges of the TV screen are hard straight lines, select the fusion mode (mask hard replacement) to write W_n into the target area using a precise mask.

[0210] Step S6 (QARM Quality Assessment): SSIM and PSNR were calculated frame by frame for each of the 240 frames. The average SSIM was 0.91 and the average PSNR was 33.5dB, both meeting the standards. Five frames had an SSIM slightly below 0.85 (due to short-term occlusion), triggering reprocessing: Poisson fusion mode was used for recombination, resulting in 4 frames meeting the standards, and 1 frame being replaced by interpolation from an adjacent frame. Ultimately, all 240 frames passed verification, and the final product was output in H.265 (8Mbps). The IPTV system interface was accurately displayed on the TV screen, with clear text, accurate colors, and stable inter-frame display, meeting compliance requirements.

[0211] Example 2: Dynamic content replacement for outdoor billboards:

[0212] The billboard scene features approximately a 15-degree perspective tilt and slow camera panning. A green screen solution combined with Kalman tracking was employed. The cue text explicitly describes the billboard screen as pure green, while a ControlNet green screen area mask is injected for guidance. After generating a video containing the perspective green screen billboard, path A uses HSV adaptive thresholding to extract frame-by-frame masks, and minAreaRect is used to fit and obtain the coordinates of the quadrilateral corner points. Kalman tracking, combined with LK optical flow smoothing, addresses coordinate drift caused by camera movement. PAFE uses a RANSAC homography matrix to perform frame-by-frame perspective transformation of the brand promotional video, and Alpha fusion (alpha=0.95) preserves the slight environmental reflections of the billboard frame. Reinhard lighting mapping matches the brightness and color temperature of the outdoor strong light environment. The final product's advertising video features clear and accurate brand content, achieving film-grade compositing quality standards.

[0213] The inventive aspects of this application are as follows:

[0214] (1) Pioneering active region labeling guidance mechanism in the AIGC generation stage: This mechanism proposes to actively guide the generation of labeling markers that can be accurately recognized by computer vision systems in the target region during the diffusion model inference stage using ControlNet or IP-Adapter, fundamentally resolving the core technical contradiction between the randomness of AIGC generation and the precise controllability of key regions. This "reserving CV anchor points in the generation stage" approach is groundbreaking in the field of AIGC video generation and synthesis fusion, providing a novel systematic methodology for solving such problems.

[0215] It should be noted that reserving CV anchor points during the generation stage is the core innovative mechanism of this solution, playing a crucial role in the entire processing pipeline. Its core value lies in: by actively injecting ControlNet chromaticity constraints or IP-Adapter geometric marker conditions during the diffusion model inference stage, the inherent uncontrollable randomness of AIGC is strictly limited to non-critical areas, forcing the target area to stably present predefined high-discrimination visual features (green screen color blocks or ArUco markers), transforming "post-event passive detection" into "pre-event active agreement" from the source. Specifically, this mechanism provides high signal-to-noise ratio recognition targets for the subsequent DDKTM detection module, enabling high-precision extraction with lightweight traditional CV algorithms; it provides sub-pixel-level four-corner coordinates for the PAFE fusion engine, ensuring the geometric accuracy of homography matrix calculation; it provides temporally continuous and high-confidence observation sequences for Kalman filtering and optical flow tracking, effectively suppressing inter-frame random jitter in the diffusion model; and it provides an objective and quantifiable spatial benchmark for QARM quality evaluation. This mechanism breaks down the barriers between AIGC generation and CV processing in the traditional process, and is the fundamental technical prerequisite for this solution to achieve high geometric accuracy, high temporal stability and high visual realism.

[0216] The "reserved CV anchor point" mechanism is functionally similar to a "green screen" or "marker sticker" in film and television shooting. The traditional approach is to wait for AI to generate the video and then let the algorithm guess which area is where the advertisement needs to be replaced. Since the content drawn by AI is different for each frame, this guessing is difficult and inaccurate. This solution, however, pre-determines during the AI's drawing process: "This area must be drawn as pure green, or the four corners must be marked with specific small square patterns." This way, after the video is generated, the algorithm only needs to find the green areas or the small squares, which is not only efficient but also highly accurate.

[0217] (2) A multi-path heterogeneous detection and Kalman multi-level joint tracking architecture suitable for AIGC generated video characteristics is proposed: a multi-path parallel detection confidence weighted fusion, Kalman prediction, LK optical flow assisted verification, and adaptive temporal smoothing multi-level joint tracking mechanism is designed, which is significantly better than the stability performance of traditional single detection schemes on AIGC generated videos.

[0218] (3) A homography matrix temporal smoothing method based on Lie algebra manifold space is proposed: The matrix logarithmic mapping is innovatively introduced to project the H matrix sequence onto the linear space of Lie algebra (se3 manifold) for Gaussian weighted smoothing and then map it back to the matrix space. Compared with direct matrix element smoothing, it has mathematical correctness and visual superiority, and is innovative in the field of video synthesis.

[0219] (4) A complete end-to-end AIGC video fixed content embedding engineering pipeline was constructed: the four modules of CGM generation guidance, DDKTM region detection and tracking, PAFE perspective adaptive fusion and QARM quality assessment were organically integrated, and the quality assessment-driven reprocessing formed a closed loop, filling the technical gap in the field of AIGC video production engineering.

[0220] This application addresses the engineering challenge of AIGC diffusion model-generated videos, where the randomness of generation leads to the inaccurate representation of specific planar carrier areas (TV screens, billboards, monitors, etc.) within the frame. By actively guiding the diffusion model during the AIGC video generation inference stage, this application ensures that the model generates region markers in the target area that can be accurately detected and located by a computer vision system. This strictly limits model randomness to non-critical areas and provides a reliable frame-by-frame spatial reference for subsequent fusion. The application performs frame-by-frame accurate detection and stable inter-frame tracking of the marker areas in the AIGC-generated video, outputting temporally smooth coordinate estimates of the target area even under interference from camera movement, perspective changes, and inherent inter-frame jitter in the diffusion model. Finally, the application performs perspective geometric transformation on the specified video content and seamlessly integrates it into the target area using an adaptive lighting approach, achieving visual realism in geometric accuracy, edge continuity, and color consistency. Furthermore, a quality assessment mechanism ensures the reliability of each frame's output.

[0221] The beneficial effects of this application are as follows:

[0222] (1) Fundamentally solve the problem of randomness in AIGC generation: By actively calibrating and guiding the generation stage, the randomness of the diffusion model is strictly limited to non-target areas, achieving pixel-level precise embedding of specified content and completely eliminating the content distortion problem in the full-modal generation scheme;

[0223] (2) Excellent timing stability: The Kalman + optical flow dual-path joint tracking mechanism, combined with the Lie algebra spatial homography matrix timing smoothing, can control the standard deviation of inter-frame coordinate deviation to within 2px;

[0224] (3) Strong robustness: Multi-path detection confidence weighted fusion and Kalman prediction completion ensure tracking continuity under interference conditions such as brief occlusion and overexposure;

[0225] (4) High degree of naturalness in fusion: Reinhard's adaptive lighting perception and multi-mode fusion strategy achieves professional film and television compositing level effects in three dimensions: geometric accuracy, edge continuity and color consistency.

[0226] (5) Zero-intrusion compatibility with existing AIGC platforms: As an independent post-processing pipeline, it can be seamlessly integrated with various mainstream platforms without modifying the underlying diffusion model weights.

[0227] (6) Universal framework, multi-scene coverage: It supports various types of planar carriers such as TV screens, billboards, mobile phone screens, and studio screens. For new scenes, only the detection parameters need to be adjusted, and there is no need to retrain.

[0228] In this embodiment, during the video generation inference stage, a conditional guidance mechanism actively guides the diffusion model to generate region-labeled videos that can be accurately detected and located by a computer vision system within the target area. This active labeling guides the randomness of the diffusion model to non-target areas, while providing reliable frame-by-frame spatial references for subsequent fusion. This achieves pixel-level precise embedding of specified content, completely eliminating content distortion issues in the full-modal generation scheme and improving the clarity of the generated AIGC video. Furthermore, a preset filtering and tracking mechanism tracks the coordinate estimation of the target area, ensuring a smooth set of frame-by-frame coordinates even under interference conditions such as camera movement, perspective changes, and inherent inter-frame jitter in the diffusion model. Adaptive fusion of the specified video based on the frame-by-frame coordinate set performs perspective geometric transformations on the specified video content and seamlessly integrates it into the target area using an adaptive lighting approach. This achieves visual realism in three dimensions: geometric accuracy, edge continuity, and color consistency. Simultaneously, a quality evaluation mechanism ensures the reliability of each frame output, improving the temporal stability of the AIGC video.

[0229] Based on the above embodiments Figure 1 The present application discloses a video processing method and a corresponding video processing apparatus, such as... Figure 6 As shown, the video processing device includes:

[0230] The generation unit 601 is used to generate a region labeling video in the target area of ​​a specified video according to a conditional guidance mechanism.

[0231] The heterogeneous detection unit 602 is used to perform multi-channel heterogeneous detection on the region calibration and marking video frame by frame to obtain the coordinate estimation of the target region;

[0232] The tracking unit 603 is used to track the coordinate estimation of the target area through a preset filtering tracking mechanism to obtain a frame-by-frame coordinate set;

[0233] The adaptive fusion unit 604 is used to adaptively fuse a specified video according to a frame-by-frame coordinate set to obtain a fused video frame.

[0234] The evaluation and acquisition unit 605 is used to perform quality evaluation on the fused video frames in order to obtain the target fused video that meets the quality conditions.

[0235] Furthermore, the generating unit 601 includes:

[0236] The first determining module is used to determine the carrier scene type of the specified video;

[0237] The first generation module is used to construct a region chroma fill control network if the carrier scene type is a frontal planar carrier scene, and generate a region calibration mark video containing pure color chroma values ​​in the target area of ​​the specified video through the region chroma fill control network.

[0238] The second generation module is used to guide the presentation of a region calibration video containing the detected geometric mark pattern at a preset position in the target area if the carrier scene type is a carrier scene with perspective distortion, by using the annotated keyframes as image condition input.

[0239] Furthermore, the heterogeneous detection unit 602 includes:

[0240] The extraction module is used to extract the green screen binary mask in the color space using an adaptive threshold.

[0241] The first processing module is used to perform morphological erosion and dilation refinement on the green screen binary mask to obtain the refined green screen binary mask.

[0242] The first calculation module is used to calculate the coordinates of the minimum bounding quadrilateral corner of the target area based on the refined green screen binary mask and through contour fitting.

[0243] The positioning and refinement module is used to locate geometric marks through the geometric mark detection function and refine the corner coordinates of the geometric marks to obtain sub-pixel precision corner coordinates;

[0244] The weighted fusion module is used to determine the confidence scores corresponding to the corner coordinates of the minimum circumscribed quadrilateral and the sub-pixel precision corner coordinates, respectively, and to perform weighted fusion on the corresponding corner coordinates based on the confidence scores to obtain the detection coordinate estimate of the target area.

[0245] Furthermore, the tracking unit 603 includes:

[0246] A module is established to create a preset filter using a multi-dimensional state vector.

[0247] The second determination module is used to determine the detection confidence level corresponding to the detection coordinates of the target area based on the confidence score;

[0248] The update module is used to update the estimated detection coordinates of the target area based on the detection confidence level using a preset filter, and obtain the update result.

[0249] The second calculation module is used to calculate the optical flow results based on the corner points of the target region in adjacent video frames;

[0250] The third calculation module is used to calculate the optical flow estimation deviation based on the update results and the optical flow results;

[0251] The correction module is used to correct the update result based on the optical flow result if the optical flow estimation deviation is greater than the preset error, and to determine the corrected update result as the coordinate estimate of the target area in the current frame.

[0252] The third determination module is used to determine the updated result as the coordinate estimate of the target region in the current frame if the optical flow estimation deviation is less than or equal to the preset error.

[0253] The arrangement module is used to arrange the coordinate estimates of the target region of each frame according to the frame order to obtain a coordinate sequence.

[0254] The second processing module is used to perform adaptive length-shifted median filtering on the coordinate sequence to obtain a frame-by-frame coordinate set.

[0255] Furthermore, the adaptive fusion unit 604 includes:

[0256] The third calculation module is used to calculate the frame-by-frame perspective transformation matrix sequence of the specified video based on the frame-by-frame coordinate set;

[0257] The fourth processing module is used to project the frame-by-frame perspective transformation matrix sequence onto a preset linear space and perform smoothing processing, and then map the smoothed frame-by-frame perspective transformation matrix sequence back to the homography matrix space to obtain a smoothed perspective transformation matrix sequence.

[0258] The perspective transformation module is used to apply perspective transformation to the nth frame of a specified video by using the nth smooth perspective transformation matrix in the smooth perspective transformation matrix sequence, so as to obtain a geometrically aligned video frame.

[0259] The mapping and matching module is used to extract the color space statistical features of the geometrically aligned video frames, and use the color space statistical features as a lighting reference to perform tone mapping and color temperature matching on the geometrically aligned video frames to obtain the video frames to be fused.

[0260] The extraction module is used to extract the edge features of the target region in the video frame to be fused;

[0261] The third determining module is used to determine the fusion mode based on the edge characteristics of the target area;

[0262] The adaptive fusion module is used to adaptively fuse the video frames to be fused in fusion mode to obtain fused video frames.

[0263] Furthermore, the evaluation acquisition unit 605 includes:

[0264] The third calculation module is used to calculate the structural similarity index quality index, peak signal-to-noise ratio quality index, and edge continuity score within the target area of ​​the fused video frame frame by frame.

[0265] The fourth determination module is used to determine that if any frame's structural similarity index quality index, peak signal-to-noise ratio quality index, and edge continuity score fails to reach its corresponding preset threshold, the fused video frame is deemed substandard. In this case, the module sequentially performs the following steps: improving illumination adaptation accuracy, switching fusion mode, and re-performing multi-path heterogeneous detection.

[0266] The operation module is used to perform interpolation replacement operation on adjacent qualified frames if the fused video frames do not meet the standards after the above operations are performed, so as to obtain the target fused video that meets the quality conditions.

[0267] The fifth determination module is used to determine that the fused video frame meets the standard if the structural similarity index quality index, peak signal-to-noise ratio quality index, and edge continuity score of any frame all reach their respective preset thresholds, and to determine the fused video frame that meets the standard as the target fused video that meets the quality conditions.

[0268] The beneficial effects of this device embodiment are the same as those of the above method embodiment, and can be referred to accordingly. They will not be repeated here.

[0269] This application also provides a storage medium that includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to perform the video processing method described above.

[0270] This application also provides an electronic device, the structural schematic diagram of which is shown below. Figure 7 As shown, it specifically includes a memory 701 and one or more instructions 702, wherein one or more instructions 702 are stored in the memory 701 and configured to be executed by one or more processors 703 to perform the above-mentioned video processing method.

[0271] The steps in the methods of the various embodiments of this application can be adjusted, combined, and deleted according to actual needs. It should be noted that, in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0272] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0273] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A video processing method, characterized in that, The method includes: Based on the conditional guidance mechanism, generate a region-marked video in the target region of the specified video; Perform multi-path heterogeneous detection frame by frame on the video of the calibrated region to obtain the coordinate estimate of the target region; The coordinates of the target area are estimated and tracked by a preset filtering tracking mechanism to obtain a frame-by-frame coordinate set; The specified video is adaptively fused based on the frame-by-frame coordinate set to obtain a fused video frame. The quality of the fused video frames is evaluated to obtain a target fused video that meets the quality requirements; The step of generating a region-marked video in a target region of a specified video according to a conditional guidance mechanism includes: Determine the carrier scene type of the specified video; If the carrier scene type is a frontal planar carrier scene, a regional chroma fill control network is constructed, and a regional calibration mark video containing pure color chroma values ​​is generated in the target area of ​​the specified video through the regional chroma fill control network. If the carrier scene type is a carrier scene with perspective distortion, the labeled keyframes are used as image condition input to guide the presentation of a region calibration mark video containing the detected geometric mark pattern at a preset position in the target area.

2. The method according to claim 1, characterized in that, Perform multi-path heterogeneous detection frame by frame on the video of the calibrated region to obtain the coordinate estimation of the target region, including: Extract the green screen binary mask in the color space using an adaptive threshold; The green screen binary mask is refined by morphological erosion and dilation to obtain the refined green screen binary mask. Based on the refined green screen binary mask, the coordinates of the minimum circumscribed quadrilateral corner point of the target area are calculated by contour fitting. Geometric markers are located using a geometric marker detection function, and the corner coordinates of the geometric markers are refined to obtain sub-pixel precision corner coordinates. The confidence scores corresponding to the corner coordinates of the minimum circumscribed quadrilateral and the sub-pixel precision corner coordinates are determined respectively. Based on the confidence scores, the corresponding corner coordinates of the two are weighted and fused to obtain the detection coordinate estimate of the target area.

3. The method according to claim 2, characterized in that, The coordinates of the target region are estimated and tracked through a preset filtering tracking mechanism to obtain a frame-by-frame coordinate set, including: A preset filter is established using a multidimensional state vector; The detection confidence level corresponding to the estimated detection coordinates of the target region is determined based on the confidence score. Based on the detection confidence level, the detection coordinates of the target region are updated using the preset filter to obtain the update result; Calculate optical flow results based on the corner points of the target region in adjacent video frames; The optical flow estimation deviation is calculated based on the updated result and the optical flow result; If the optical flow estimation deviation is greater than the preset error, the update result is corrected by the optical flow result, and the corrected update result is determined as the coordinate estimate of the target region in the current frame; If the optical flow estimation deviation is less than or equal to the preset error, the update result will be determined as the coordinate estimation of the target region in the current frame; The coordinate estimates of the target region in each frame are arranged in frame order to obtain a coordinate sequence; An adaptive length-shifting median filter is applied to the coordinate sequence to obtain a frame-by-frame coordinate set.

4. The method according to claim 3, characterized in that, Adaptive fusion of the specified video based on the frame-by-frame coordinate set yields fused video frames, including: Calculate the frame-by-frame perspective transformation matrix sequence of the specified video based on the frame-by-frame coordinate set; The frame-by-frame perspective transformation matrix sequence is projected onto a preset linear space and smoothed, and the smoothed frame-by-frame perspective transformation matrix sequence is mapped back to the homography matrix space to obtain a smoothed perspective transformation matrix sequence. By applying the nth smooth perspective transformation matrix in the sequence of smooth perspective transformation matrices to the nth frame of a specified video, a geometrically aligned video frame is obtained. Extract the color space statistical features of the geometrically aligned video frames, and use the color space statistical features as a lighting reference to perform tone mapping and color temperature matching on the geometrically aligned video frames to obtain the video frames to be fused. Extract the edge features of the target region in the video frames to be fused; The fusion mode is determined based on the edge characteristics of the target region; In the fusion mode, the video frames to be fused are adaptively fused to obtain fused video frames.

5. The method according to claim 1, characterized in that, The quality of the fused video frames is evaluated to obtain a target fused video that meets the quality requirements, including: Calculate the structural similarity index quality index, peak signal-to-noise ratio quality index, and edge continuity score within the target region of the fused video frame frame by frame; If any frame fails to meet its corresponding preset threshold in the structural similarity index quality index, peak signal-to-noise ratio quality index, and edge continuity score, the fused video frame is determined to be substandard. Then, the following steps are executed in sequence: improve illumination adaptive accuracy, switch fusion mode, and re-execute multi-path heterogeneous detection. If the fused video frames do not meet the standards after performing the above operations, interpolation replacement of adjacent qualified frames is performed to obtain the target fused video that meets the quality requirements. If the structural similarity index quality index, peak signal-to-noise ratio quality index, and edge continuity score of any frame all reach their respective preset thresholds, the fused video frame is determined to meet the standards, and the fused video frame that meets the standards is determined as the target fused video that meets the quality conditions.

6. A video processing apparatus, characterized in that, The device includes: The generation unit is used to generate a region-labeled video in the target region of a specified video according to a conditional guidance mechanism. Heterogeneous detection unit is used to perform multi-channel heterogeneous detection frame by frame on the area calibration and marking video to obtain the coordinate estimation of the target area; The tracking unit is used to track the coordinate estimation of the target area through a preset filtering tracking mechanism to obtain a frame-by-frame coordinate set; An adaptive fusion unit is used to adaptively fuse a specified video according to the frame-by-frame coordinate set to obtain a fused video frame. An evaluation and acquisition unit is used to perform quality evaluation on the fused video frames in order to obtain a target fused video that meets the quality conditions. The generation unit includes: The first determining module is used to determine the carrier scene type of the specified video; The first generation module is used to construct a region chroma fill control network if the carrier scene type is a frontal planar carrier scene, and generate a region calibration mark video containing pure color chroma values ​​in the target area of ​​the specified video through the region chroma fill control network. The second generation module is used to guide the presentation of a region calibration mark video containing the detected geometric mark pattern at a preset position in the target area if the carrier scene type is a carrier scene with perspective distortion, by using the annotated keyframes as image condition input.

7. A storage medium, characterized in that, The storage medium includes stored instructions, wherein, when the instructions are executed, the device containing the storage medium is controlled to perform the video processing method as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, It includes a memory, and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors to implement the video processing method as described in any one of claims 1 to 5.

Images