Video fusion method and system based on large model

Abstract

The invention relates to the technical field of video processing, in particular to a video fusion method and system based on a large model. The method comprises the following steps: acquiring a video stream for picture segmentation, identifying a moving target for each segmented picture, predicting a moving target track, matching a minimum pixel region, and determining a key frame; extracting features, and focusing an identification target in the key area; generating a static scene, aligning the recognized target and the scene, obtaining a dynamic shielding relationship, and outputting fused video data; the system comprises a dynamic picture division module, a multi-modal feature extraction module and a dynamic fusion reconstruction module. The method comprises the following steps: identifying a moving target for each segmented image, predicting a moving target track, matching a minimum pixel region, determining a key frame, focusing an identified target in the key region, aligning the identified target and a scene, obtaining a dynamic shielding relationship, and outputting fused video data; trajectory changes of the moving target are processed, and ghosting and dislocation of the fusion result are avoided.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a large model-based video fusion method and system. Background Art

[0002] Video fusion technology is a technology that uses intelligent processing and integration of video data from different sources, different perspectives or different modalities to generate a unified video output with richer information, more complete scenes and clearer semantics.

[0003] Currently, traditional video fusion methods rely on static three-dimensional models and cannot process the trajectory changes of moving targets in real time, resulting in ghosting and misalignment in the fusion results.

[0004] Therefore, it is very necessary to propose a video fusion method and system that can handle the trajectory changes of moving targets and avoid ghosting and misalignment in the fusion results. Summary of the Invention

[0005] The purpose of the present invention is to provide a video fusion method and system based on a large model, aiming to solve the technical problem in the prior art that the trajectory changes of moving targets cannot be processed in real time, resulting in ghosting and misalignment in the fusion results.

[0006] To achieve the above object, the present invention adopts a video fusion method based on a large model, comprising the following steps: Obtain the video stream and perform screen segmentation. For each segmented screen, identify the moving target, predict the moving target trajectory, match the minimum pixel area, and determine the key frame. Extract temporal, textual, and visual features respectively, focusing on the recognition target in the key area; Generate static scenes, align and identify targets and scenes, obtain dynamic occlusion relationships, and output fused video data.

[0007] Among them, in the steps of acquiring the video stream for screen segmentation, identifying the moving target for each segmented screen, predicting the moving target trajectory, matching the minimum pixel area, and determining the key frame: Use multiple cameras to capture real-time video streams and perform denoising and color correction on the original video; Segment the video stream in real time, output segmented frames, and identify moving targets in each frame; Get the motion vector for the moving target and predict the trajectory of the moving target.

[0008] After obtaining a motion vector for a moving target and predicting the trajectory of the moving target: Determine the overlap of target areas in adjacent split-screen frames, and select the frame with the highest overlap as the key frame.

[0009] Among them, in the steps of using multiple cameras to capture real-time video streams and performing denoising and color correction on the original video: Align timestamps of multiple cameras to eliminate time deviation.

[0010] Among them, in the steps of extracting time series, text, and visual features respectively and focusing on the recognition target in the key area: Extract temporal, textual, and visual features; Focus on identifying targets in key areas; Align temporal, textual, and visual features across modalities to generate a joint feature vector.

[0011] Among them, in the steps of extracting time series, text, and visual features: The periodic characteristics of sensor time series data are extracted, and the dimension is reduced through principal component analysis, the variance information is retained, and the time series characteristics are output.

[0012] Among them, in the steps of extracting time series, text, and visual features: Parse text data, generate semantic vectors, map keywords, and output text features.

[0013] Among them, in the steps of extracting time series, text, and visual features: Adaptive histogram equalization is performed on image data to extract deep features and output visual features.

[0014] Among them, in the steps of generating static scenes, aligning and identifying targets and scenes, obtaining dynamic occlusion relationships, and outputting fused video data: Construct static scenes and initially register video frames and static scenes; Analyze the occlusion order between targets through multiple image sequences, predict the occlusion area, and generate a transparency mask; Synthesize videos in real time and output fused video data.

[0015] The present invention also provides a large-model-based video fusion system, comprising a dynamic picture segmentation module, a multimodal feature extraction module, and a dynamic fusion reconstruction module; wherein: The dynamic picture segmentation module is used to obtain the video stream and perform picture segmentation, identify the moving target for each segmented picture, predict the moving target trajectory, match the minimum pixel area, and determine the key frame; The multimodal feature extraction module is used to extract temporal, textual, and visual features respectively, focusing on the recognition target in the key area; The dynamic fusion reconstruction module is used to generate a static scene, align and identify the target and the scene, obtain dynamic occlusion relationships, and output fused video data.

[0016] The present invention provides a large-model-based video fusion method and system, which adopts the dynamic picture segmentation module, the multimodal feature extraction module, and the dynamic fusion reconstruction module to perform the following steps: obtaining a video stream for picture segmentation, identifying a moving target for each segmented picture, predicting the moving target trajectory, matching the minimum pixel area, and determining a key frame; extracting time series, text, and visual features respectively, focusing on the identified target in the key area; generating a static scene, aligning the identified target and the scene, obtaining a dynamic occlusion relationship, and outputting fused video data; identifying a moving target for each segmented picture, predicting the moving target trajectory, matching the minimum pixel area, determining a key frame, focusing on the identified target in the key area, aligning the identified target and the scene, obtaining a dynamic occlusion relationship, and outputting fused video data; realizing the processing of trajectory changes of the moving target and avoiding ghosting and misalignment in the fusion result. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 It is a flowchart of the steps of the video fusion method based on a large model of the present invention.

[0019] Figure 2 It is a step flow chart of S100 of the present invention.

[0020] Figure 3 It is a step flow chart of S200 of the present invention.

[0021] Figure 4 It is a step flow chart of S300 of the present invention.

[0022] Figure 5 It is a structural principle diagram of the video fusion system based on a large model of the present invention.

[0023] Figure 6 It is a structural principle diagram of the electronic device of the present invention.

[0024] 401-dynamic picture segmentation module, 402-multimodal feature extraction module, 403-dynamic fusion reconstruction module. DETAILED DESCRIPTION

[0025] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0026] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0027] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0028] See also Figures 1 to 4 The present invention provides a video fusion method based on a large model, comprising the following steps: S100: Obtain a video stream and perform screen segmentation. For each segmented screen, identify a moving target, predict the moving target trajectory, match the minimum pixel area, and determine a key frame.

[0029] In this embodiment, the video stream is obtained and segmented, and the moving target is identified for each segmented image, the moving target trajectory is predicted, and the minimum pixel area is matched to determine the key frame. The specific process is: S101: Use multiple cameras to capture real-time video streams, align multi-camera timestamps to eliminate time deviation, and perform denoising and color correction on the original video; S102: Segment the video stream in real time, output segmented frames, and identify moving targets in each frame; S103: Obtaining a motion vector for the moving target and predicting the trajectory of the moving target; S104: Determine the overlap of target areas in adjacent segmented image frames, and select the frame with the highest overlap as a key frame.

[0030] In the above process, high-quality segmented images are extracted from multi-camera videos, moving targets are identified and trajectories are predicted, and finally key frames are determined to reduce redundant data.

[0031] Multi-camera video stream preprocessing and time synchronization: Use NTP (Network Time Protocol) or PTP (Precision Time Protocol) to align multi-camera timestamps and eliminate acquisition delays (error < 1ms).

[0032] Video enhancement, denoising: Remove sensor noise using non-local means denoising (NLM) or deep learning denoising models such as DnCNN.

[0033] Color correction: Use white balance algorithms (such as GrayWorld) to unify the color temperature of different cameras to avoid inconsistent target features due to lighting differences.

[0034] Real-time image segmentation and target recognition, image segmentation: YOLOv8-Seg (target detection + instance segmentation model) is used to segment the image in real time and output the mask (Mask) and bounding box (BoundingBox) of each target.

[0035] Moving target recognition: The YOLOv8-3D model is used to identify moving targets within the segmented mask area. The YOLOv8-3D model is a deep learning target detection framework that uses convolutional neural networks (CNNs) and attention mechanisms (such as CSPNet) to extract multi-scale features to achieve high-precision moving target recognition (such as vehicles and pedestrians).

[0036] Moving target trajectory prediction: Motion vector aggregation, taking the average of the pixel motion vectors within the bounding box of each moving target to generate the target-level motion vector.

[0037] Trajectory prediction: Short-term prediction: Use Kalman filter (KalmanFilter) combined with the uniform velocity model (CVModel) to predict the position of the next 2~3 frames.

[0038] Long-term prediction: For targets with complex motion patterns (such as turns), large time series models (such as TimeSformer) can be used to capture the long-term dependencies of moving targets and improve trajectory prediction accuracy. For example, using a Transformer encoder-decoder architecture can model the contextual relationships of target position sequences and predict the trajectory for the next 2 seconds with an error of <5%.

[0039] Overlap calculation: For adjacent segmented frames, calculate the overlap of the target area; select the frame with the highest overlap as the key frame (such as Overlap>0.8) to ensure that the key frame contains the most complete moving target information and reduce the amount of subsequent processing data.

[0040] S200: Extracting time series, text, and visual features respectively, and focusing on the recognition target in the key area.

[0041] In this embodiment, time series, text, and visual features are extracted separately, focusing on the recognition target in the key area. The specific process is as follows: S201: Extracting periodic features of sensor time series data, reducing the dimension through principal component analysis, retaining variance information, and outputting time series features; S202: Parse the text data, generate semantic vectors, map the keywords, and output text features; S203: performing adaptive histogram equalization on the image data, extracting deep features, and outputting visual features; S204: focusing and identifying the target in the key area; S205: Perform cross-modal alignment on temporal, textual, and visual features to generate a joint feature vector.

[0042] In the above process, complementary features are extracted from time series, text, and visual data, focusing on key areas (such as accident areas) to improve recognition accuracy.

[0043] Time series feature extraction: Periodic feature analysis: Perform Fourier transform (FFT) on sensor time series data (such as radar speed and GPS coordinates) to extract the main frequency component (such as vehicle driving cycle).

[0044] Dimensionality reduction processing uses principal component analysis (PCA) to retain the principal components with variance contribution rate >95%, compressing the 100-dimensional time series data to 10 dimensions to reduce the amount of calculation.

[0045] Text feature extraction: Semantic vector generation: Using the BERT-base model, text data (such as "accident ahead") is encoded into a 768-dimensional semantic vector to capture contextual associations (such as the semantic similarity between "accident" and "congestion").

[0046] Keyword mapping: Use Word2Vec to map keywords (such as "vehicle" and "pedestrian") to a 300-dimensional word vector space to supplement BERT's global semantic missing.

[0047] Visual feature extraction: Image enhancement, applying adaptive histogram equalization (CLAHE) to keyframes to improve the visibility of low-contrast areas (such as license plates in shadows).

[0048] Deep feature extraction uses ResNet-152 to extract image features and output a 2048-dimensional feature vector containing high-level semantics such as target category and spatial location.

[0049] Key Areas of Focus: Region selection locates key regions (e.g., collision points) in visual features based on keywords (e.g., “accident”) in text features or outliers (e.g., speed drops) in temporal features.

[0050] The attention mechanism dynamically adjusts the weights of key area feature channels through the SE module (Squeeze-and-Excitation) to increase their importance (for example, increasing the weight of accident area features by 30%).

[0051] Cross-modal alignment: Feature fusion: concatenates temporal (10 dimensions), text (768+300 dimensions), and visual (2048 dimensions) features into a 3118-dimensional vector, which is then input into the Transformer encoder for cross-modal attention calculation.

[0052] Joint feature generation outputs a 512-dimensional joint feature vector to achieve semantic association between time series, text, and vision, such as aligning the "accident" text with the colliding vehicles in the vision and the speed mutation in the time series.

[0053] S300: Generate a static scene, align and identify the target and scene, obtain dynamic occlusion relationships, and output fused video data.

[0054] In this embodiment, a static scene is generated, the target and scene are aligned and recognized, the dynamic occlusion relationship is obtained, and the fused video data is output. The specific process is as follows: S301: Constructing a static scene and initially registering the video frame and the static scene; S302: Analyze the occlusion order between objects through multiple image sequences, predict the occlusion area, and generate a transparency mask; S303: Synthesize the video in real time and output the fused video data.

[0055] In the above process, a unified static scene is constructed, dynamic targets are aligned, occlusion relationships are processed, and an artifact-free fused video is generated.

[0056] Static scene construction: Initial registration: select the first keyframe as the static scene reference, and align the remaining frames to the reference coordinate system through SIFT feature matching or deep learning registration (such as LoFTR).

[0057] Scene fusion, performs weighted averaging fusion on the registered frames, with the weight inversely proportional to the time difference between frames (e.g., weight of the closest frame = 0.7, weight of the next closest frame = 0.3), to reduce motion blur.

[0058] Dynamic occlusion relationship analysis: Occlusion order prediction: By analyzing the relative position changes between targets through multiple image sequences (such as 5 consecutive frames), the occlusion order is determined (such as vehicle A occluding vehicle B first).

[0059] Transparency mask generation: Generates a transparency mask for the occluded area. Mask 0 indicates complete occlusion, and mask 1 indicates complete visibility. For example, if vehicle B is 50% occluded by vehicle A, its mask is 0.5.

[0060] Real-time video synthesis: Layered rendering uses the static scene as the background layer and the dynamic target as the foreground layer, and superimposes each layer in the order of occlusion.

[0061] Fusion output: Apply bilateral filtering to the superimposed video to eliminate splicing traces, and output 4K resolution, 30fps fused video to ensure smooth and jagged edges.

[0062] In the present invention, firstly, a video stream is acquired for screen segmentation, a moving target is identified for each segmented screen, the moving target trajectory is predicted, the minimum pixel area is matched, and a key frame is determined; then, timing, text, and visual features are extracted respectively, and the identified target in the key area is focused; finally, a static scene is generated, the identified target and the scene are aligned, a dynamic occlusion relationship is obtained, and fused video data is output; by identifying the moving target for each segmented screen, predicting the moving target trajectory, matching the minimum pixel area, determining the key frame, focusing on the identified target in the key area, aligning the identified target and the scene, a dynamic occlusion relationship is obtained, and fused video data is output; this realizes the processing of trajectory changes of the moving target and avoids ghosting and misalignment in the fusion result.

[0063] Corresponding to the aforementioned embodiment of the video fusion method based on a large model, the present application also provides an embodiment of a video fusion system based on a large model.

[0064] Figure 5 FIG. 1 is a block diagram of a video fusion system based on a large model according to an exemplary embodiment. Figure 5 The system may include: a dynamic picture segmentation module 401, a multimodal feature extraction module 402, and a dynamic fusion reconstruction module 403; wherein: The dynamic picture segmentation module 401 is used to obtain the video stream and perform picture segmentation, identify the moving target for each segmented picture, predict the moving target trajectory, match the minimum pixel area, and determine the key frame; The multimodal feature extraction module 402 is used to extract time series, text, and visual features respectively, focusing on the recognition target in the key area; The dynamic fusion reconstruction module 403 is used to generate a static scene, align and identify the target and the scene, obtain dynamic occlusion relationships, and output fused video data.

[0065] In this embodiment, the dynamic picture segmentation module 401 obtains the video stream to perform picture segmentation, identifies the moving target for each segmented picture, predicts the moving target trajectory, matches the minimum pixel area, and determines the key frame; the multimodal feature extraction module 402 extracts timing, text, and visual features respectively, and focuses on the identified target in the key area; the dynamic fusion reconstruction module 403 generates a static scene, aligns the identified target and the scene, obtains the dynamic occlusion relationship, and outputs the fused video data; by identifying the moving target for each segmented picture, predicting the moving target trajectory, matching the minimum pixel area, determining the key frame, and focusing on the identified target in the key area, aligning the identified target and the scene, obtaining the dynamic occlusion relationship, and outputting the fused video data; it realizes the processing of the trajectory changes of the moving target and avoids ghosting and misalignment in the fusion result.

[0066] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0067] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0068] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned large model-based video fusion method. Figure 6 As shown in the figure, a hardware structure diagram of a video fusion system based on a large model provided by an embodiment of the present invention is provided for any device with data processing capabilities, except Figure 6 In addition to the processor, memory, and network interface shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0069] Accordingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the large-model-based video fusion method as described above. The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0070] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed in this application.

[0071] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A video fusion method based on a large model, characterized in that: The steps include: Obtain the video stream and perform screen segmentation. For each segmented screen, identify the moving target, predict the moving target trajectory, match the minimum pixel area, and determine the key frame. Extract temporal, textual, and visual features respectively, focusing on the recognition target in the key area; Generate static scenes, align and identify targets and scenes, obtain dynamic occlusion relationships, and output fused video data.

2. The video fusion method based on a large model according to claim 1, characterized in that: In the steps of acquiring a video stream for screen segmentation, identifying a moving target for each segmented screen, predicting the moving target trajectory, matching the minimum pixel area, and determining the key frame: Use multiple cameras to capture real-time video streams and perform denoising and color correction on the original video; Segment the video stream in real time, output segmented frames, and identify moving targets in each frame; Get the motion vector for the moving target and predict the trajectory of the moving target.

3. The video fusion method based on a large model according to claim 2, characterized in that: After obtaining the motion vector for the moving target and predicting the trajectory of the moving target: Determine the overlap of target areas in adjacent split-screen frames, and select the frame with the highest overlap as the key frame.

4. The video fusion method based on a large model according to claim 2, characterized in that: In the steps of using multiple cameras to capture real-time video streams and denoising and color correcting the original videos: Align timestamps of multiple cameras to eliminate time deviation.

5. The video fusion method based on a large model according to claim 1, characterized in that: In the steps of extracting temporal, textual, and visual features respectively and focusing on the recognition target in the key area: Extract temporal, textual, and visual features; Focus on identifying targets in key areas; Align temporal, textual, and visual features across modalities to generate a joint feature vector.

6. The video fusion method based on a large model according to claim 5, characterized in that: In the steps of extracting temporal, textual, and visual features: The periodic characteristics of sensor time series data are extracted, and the dimension is reduced through principal component analysis, the variance information is retained, and the time series characteristics are output.

7. The video fusion method based on a large model according to claim 6, characterized in that: In the steps of extracting temporal, textual, and visual features: Parse text data, generate semantic vectors, map keywords, and output text features.

8. The video fusion method based on a large model according to claim 7, characterized in that: In the steps of extracting temporal, textual, and visual features: Adaptive histogram equalization is performed on image data to extract deep features and output visual features.

9. The video fusion method based on a large model according to claim 1, characterized in that: In the steps of generating static scenes, aligning and identifying targets and scenes, obtaining dynamic occlusion relationships, and outputting fused video data: Construct static scenes and initially register video frames and static scenes; Analyze the occlusion order between targets through multiple image sequences, predict the occlusion area, and generate a transparency mask; Synthesize videos in real time and output fused video data.

10. A large-model-based video fusion system, applied to the large-model-based video fusion method according to claim 1, characterized in that: It includes dynamic picture segmentation module, multimodal feature extraction module, and dynamic fusion reconstruction module; among which: The dynamic picture segmentation module is used to obtain the video stream and perform picture segmentation, identify the moving target for each segmented picture, predict the moving target trajectory, match the minimum pixel area, and determine the key frame; The multimodal feature extraction module is used to extract temporal, textual, and visual features respectively, focusing on the recognition target in the key area; The dynamic fusion reconstruction module is used to generate a static scene, align and identify the target and the scene, obtain dynamic occlusion relationships, and output fused video data.

Images