Video twinborn picture method based on video heterogeneous fusion and geometric perspective correction

Abstract

The invention discloses a video twinning picture method based on video heterogeneous fusion and geometric perspective correction, and relates to the crossing field of a digital twinning technology and a computer vision technology. Comprising the following steps of multi-source heterogeneous video acquisition, video preprocessing, geometric perspective correction, correction effect real-time verification and dynamic adjustment, feature level heterogeneous fusion, fusion feature adaptive weight calibration, video twinning mapping, interactive driven local picture enhancement and picture optimization and output. According to the video twinborn picture method based on video heterogeneous fusion and geometric perspective correction, the visual angle and geometric morphology of a multi-source video are unified through accurate geometric perspective correction, and correction precision is guaranteed through a real-time verification mechanism; feature-level heterogeneous fusion is carried out based on the corrected data, and core information is strengthened through dynamic weight calibration; and finally, the digital twin model is driven to realize real-time mapping, local enhancement is performed in response to interaction, and the quality of video twin pictures is improved.

Description

Technical Field

[0001] This invention relates to the intersection of digital twin technology and computer vision technology, specifically a video twin image method based on video heterogeneous fusion and geometric perspective correction. Background Technology

[0002] As the core visual carrier of digital twin technology, video twins map the dynamic visual information of physical scenes to digital space, providing support for subsequent perception, decision-making, and control. The limitations of single technologies have driven the coordinated development of "heterogeneous fusion and perspective correction": while heterogeneous fusion can integrate multimodal information, it does not solve the spatial misalignment caused by geometric distortion; perspective correction can correct single-view distortion, but it cannot eliminate the modal differences of multi-source data. The generation of video twin images mainly relies on multi-source video acquisition and data fusion technologies, and existing technologies still have the following shortcomings: Multi-source video acquisition often employs different shooting angles (such as tilted or overhead views), and camera pose shifts and lens distortion result in perspective distortion in the original video. Existing single correction methods based on homography matrices are ill-suited for complex scenes with significant depth variations, and local distortions persist even after correction, affecting the consistency between virtual and physical space. Furthermore, multi-source videos exhibit modal differences (visible light, infrared) and parameter differences (resolution, frame rate). Traditional fusion algorithms (such as wavelet transform and simple CNN fusion) are mostly pixel-level fusions, prone to artifacts and information conflicts, and fail to consider the specific characteristics of the corrected data for targeted fusion, leading to fragmented information and poor visual coherence in the fused image. In existing technologies, the correction and fusion processes are mutually exclusive. Independent and lacking collaborative optimization, the long data processing chain leads to delayed twin image updates, failing to meet the real-time mapping requirements of dynamic scenes. It exhibits poor adaptability to environmental changes (lighting, occlusion) in different scenarios (such as factory workshops and open ports), making it difficult to stably output high-quality twin images. Furthermore, the twin images often have a globally uniform resolution, resulting in insufficient detail clarity when users focus on specific target areas, failing to meet the needs of precise observation. The correction process lacks a real-time verification mechanism, making it difficult to detect residual local distortions, affecting the stability of spatial consistency. The lack of dynamic adjustment of modal weights based on scene type and data quality results in core information (such as infrared temperature information in emergency rescue) not being enhanced, and non-critical information being redundant. Summary of the Invention

[0003] The purpose of this invention is to provide a video twin image method based on heterogeneous video fusion and geometric perspective correction, in order to solve the problems of existing video twin image methods mentioned in the background art, such as prominent geometric distortion, poor heterogeneous fusion effect, virtual-physical synchronization lag, insufficient adaptability to scene, insufficient support for details after interaction, lack of closed-loop verification of correction accuracy, and fixed fusion weight.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a video twin image method based on video heterogeneous fusion and geometric perspective correction, wherein the video twin image method includes the following steps: S1. Multi-source heterogeneous video acquisition: Deploy video acquisition devices with different modalities and parameters to acquire visible light video, infrared video and depth video streams of physical scenes, forming a multi-source heterogeneous video dataset; S2. Video preprocessing: Spatiotemporal synchronization alignment, noise reduction, and format standardization are performed on the multi-source heterogeneous video dataset to obtain the video stream to be processed. S3. Geometric perspective correction: Based on camera intrinsic and extrinsic parameter estimation and local mesh transformation algorithm, perspective distortion correction is performed on each frame of the video stream to be processed, and a standard viewpoint video stream without geometric distortion is output. S31: Real-time verification and dynamic adjustment of correction effect. Calculate the reprojection error and structural similarity (SSIM) of the corrected video frame. When the index exceeds the preset threshold, dynamically encrypt the local mesh and recalculate the homography transformation matrix. Iterate and optimize until the accuracy requirements are met.

[0005] S4: Feature-level heterogeneous fusion: Extract multimodal features from the standard viewpoint video stream, and perform weighted fusion of the multimodal features through an attention mechanism and a Transformer fusion network to generate a unified feature map; S41: Adaptive weight calibration of fusion features. Based on scene type (industrial monitoring / emergency rescue, etc.) and real-time data quality (such as light intensity and occlusion ratio), the fusion weights of each modality feature are dynamically adjusted through a reinforcement learning model to optimize the information relevance of the feature map. S5: Video twin mapping, based on the unified feature map, drives the digital twin model to perform real-time state updates, generating a video twin image that is consistent with the physical scene in time and space and has complete information; S51: Interactive-driven local image enhancement, responding to user interaction commands (click / selection), locating target areas in the virtual scene, performing super-resolution reconstruction and detail enhancement on the feature map of the target area, and outputting local high-definition enhanced images; S6: Image optimization and output, performing edge enhancement and artifact removal on the video twin image, outputting a high-fidelity twin image and supporting real-time interactive feedback.

[0006] In this embodiment, step S1 involves deploying multiple types of video acquisition devices, including high-definition visible light cameras, infrared thermal imaging cameras, and depth cameras, to cover the entire physical scene without blind spots. The visible light cameras are used to capture texture details, the infrared cameras are used to capture areas with abnormal temperatures, and the depth cameras are used to acquire spatial depth information. The acquisition devices support parameter configuration (frame rate, resolution) to ensure that multi-source video data covers dynamic changes in the scene and multi-dimensional information.

[0007] In this embodiment, in step S2, the spatiotemporal synchronization alignment of video preprocessing adopts timestamp matching and inter-frame interpolation algorithms to eliminate frame delay and frame rate differences between multi-source videos; the noise reduction processing adopts an adaptive median filtering algorithm to retain image details while removing environmental noise.

[0008] In this embodiment, S3, geometric perspective correction includes: S3A: Predicts camera intrinsic parameters (focal length, principal point) and extrinsic parameters (rotation matrix, translation vector) using the VGGT deep learning model. S3B: The improved APAP algorithm is used to construct local image grids, and the homography transformation matrix of each grid is calculated in combination with camera parameters; S3C: Based on the homography transformation matrix, perform grid-by-grid perspective correction on the image, and achieve pixel resampling through bilinear interpolation to obtain a standard viewpoint image.

[0009] In this embodiment, in step S31, the reprojection error calculation is based on the coordinate deviation of feature points in the images before and after correction, with a preset error threshold of ≤1.5 pixels; the structural similarity (SSIM) preset threshold is ≥0.95; the local mesh densification adopts an adaptive density strategy, and the mesh density in the area where the reprojection error exceeds the standard is increased to 2-3 times the base density, with the number of iterations not exceeding 3 to ensure real-time performance.

[0010] In this embodiment, in step S4, the multimodal feature extraction uses a lightweight CNN network to extract the texture features of visible light video, the temperature features of infrared video, and the spatial features of depth video respectively; the Transformer fusion network learns feature weights through a cross-modal attention mechanism to achieve adaptive fusion of multimodal features.

[0011] In this embodiment, in step S41, the scene type is automatically determined by preset labels or image content recognition, and the data quality is quantitatively evaluated by light intensity sensor data and occlusion detection algorithm; the reinforcement learning model uses "maximizing information recognition" as the reward function, and the adjustment range of visible light texture feature weights is 0.3-0.8, infrared temperature features are 0.1-0.5, and depth spatial features are 0.1-0.4.

[0012] In this embodiment, in step S5, the digital twin mapping adopts a "feature-driven - state update" mechanism, which uses a unified feature map to drive the geometric shape, texture information and dynamic changes of the virtual scene in real time, thereby achieving millisecond-level synchronization between the virtual and physical worlds.

[0013] In this embodiment, in step S51, the user interaction command supports three triggering methods: mouse click, touch selection, and voice wake-up; the target area localization uses a coordinate mapping algorithm to associate the virtual scene with the physical scene location; the super-resolution reconstruction uses a lightweight ESRGAN network, and the detail enhancement is achieved through an adaptive contrast enhancement algorithm, increasing the local image resolution to twice the original resolution, with a processing latency of ≤50ms.

[0014] In this embodiment, in step S6, the image optimization uses a guided filtering algorithm to remove fusion artifacts and a Laplacian operator to enhance edges, thereby improving the clarity and visual coherence of the twin images.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This video twin image method based on heterogeneous video fusion and geometric perspective correction unifies the perspective and geometric shape of multi-source videos through precise geometric perspective correction, and ensures the accuracy of correction through a real-time verification mechanism. Then, feature-level heterogeneous fusion is performed based on the corrected data, and core information is strengthened through dynamic weight calibration. Finally, the digital twin model is driven to achieve real-time mapping and respond to interactions to perform local enhancement, thereby improving the quality of the video twin image and better serving reality.

[0016] Furthermore, by improving the APAP algorithm and automatically estimating camera parameters, and combining the newly added real-time verification and dynamic adjustment steps for the correction effect, this invention reduces the spatial error of the corrected image, stabilizes the reprojection error within 1.5 pixels, significantly improves spatial consistency compared to existing technologies, and completely solves the problem of residual local distortion.

[0017] Furthermore, by combining feature-level heterogeneous fusion with cross-modal attention mechanism and adding an adaptive weight calibration step, the core feature recognition of multimodal information is improved, the fusion artifact rate is reduced, and the adaptability of twin images in different scenarios (industrial / rescue) and environmental conditions is improved.

[0018] Furthermore, through a collaborative framework of "correction-verification-fusion-calibration-mapping-enhancement" and an incremental update mechanism, data processing efficiency is improved, with twin screen update latency ≤100ms and local enhancement processing latency ≤50ms, fully meeting the needs of dynamic scenes and interactions.

[0019] Furthermore, through interactive-driven local enhancement, the detail resolution of the target area is improved, and the recognition accuracy of key information such as equipment fault points and abnormal temperature areas is improved, significantly enhancing the practical value and user experience of the twin image. Furthermore, through the synergistic effect of adaptive denoising, dynamic mesh encryption, and weight calibration, the method's ability to resist interference in complex environments such as changes in lighting, occlusion, and large depth differences is improved, and the stability of image quality is further enhanced. By setting up high-fidelity, highly interactive twin screens for output, supporting multi-terminal display and precise interaction, it provides more comprehensive visual support for intelligent decision-making, remote control, and emergency response, and is applicable to multiple fields such as smart factories, port scheduling, and emergency rescue. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the video twin image method based on video heterogeneous fusion and geometric perspective correction of the present invention; Figure 2 This is a schematic diagram illustrating the deployment of multiple types of video acquisition devices according to the present invention; Figure 3 This is a schematic diagram illustrating the spatiotemporal synchronization alignment of multi-source heterogeneous video datasets according to the present invention. Figure 4 This is a schematic diagram of the geometric perspective correction of the present invention; Figure 5 This is a schematic diagram illustrating the real-time verification and dynamic adjustment of the correction effect of the present invention; Figure 6 This is a schematic diagram of the multimodal feature extraction and fusion network of the present invention; Figure 7 This is a schematic diagram of the adaptive weight calibration of fused features according to the present invention; Figure 8 This is a schematic diagram of the video twin mapping of the present invention; Figure 9 This is a schematic diagram of the interactive-driven local image enhancement of the present invention; Figure 10 This is a schematic diagram illustrating the image optimization and output of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] This invention provides a video twin image method based on video heterogeneous fusion and geometric perspective correction. Its core is to unify the perspective and geometric shape of multi-source videos through precise geometric perspective correction, and to ensure the correction accuracy through a real-time verification mechanism. Then, based on the corrected data, feature-level heterogeneous fusion is performed, and core information is strengthened through dynamic weight calibration. Finally, the digital twin model is driven to achieve real-time mapping and respond to interactions to perform local enhancement, thereby improving the quality of the video twin image and better serving reality.

[0023] Example 1: To better understand the above technical solution, the following will provide a detailed description of the technical solution in conjunction with the accompanying drawings and specific implementation methods. (Refer to...) Figure 1 As shown, Figure 1 This is a video twin image method based on video heterogeneous fusion and geometric perspective correction. The video twin image method based on video heterogeneous fusion and geometric perspective correction includes the following steps: In step S1, multi-source heterogeneous video acquisition is performed by deploying video acquisition devices with different modalities and parameters to acquire visible light video, infrared video, and depth video streams of the physical scene, forming a multi-source heterogeneous video dataset.

[0024] Specifically, multiple types of video acquisition devices are deployed, including high-definition visible light cameras, infrared thermal imaging cameras, and depth cameras, to cover the entire area of ​​the physical scene without blind spots. Among them, visible light cameras are used to capture texture details, infrared cameras are used to capture areas with abnormal temperatures, and depth cameras are used to acquire spatial depth information. The acquisition devices support parameter configuration (frame rate, resolution) to ensure that multi-source video data covers dynamic changes in the scene and multi-dimensional information.

[0025] It should be noted that the high-definition visible light camera has 8 megapixels, supports 4K / 1080P resolution switching, and an adjustable frame rate of 15-60fps; the infrared thermal imaging camera has a resolution of 640×512, a temperature measurement range of -20℃ to 500℃, and a thermal sensitivity of ≤50mK; the depth camera is a ToF depth camera, with a measurement distance of 0.3-10m, a depth accuracy of ±2%, and a frame rate of 10-30fps. The installation locations are planned according to the principle of "dense deployment in key areas + supplementary global coverage". Figure 2As shown in the diagram, this is a schematic diagram of the deployment of multiple types of video acquisition equipment. The functional division of each type of camera is clearly defined: high-definition visible light cameras are responsible for capturing scene texture details (such as wear marks on equipment surfaces, material color and state) and dynamic behaviors (such as personnel operation trajectories, material transportation processes); infrared thermal imaging cameras capture abnormal temperature areas in real time (such as motor overheating, material spontaneous combustion hazards), and output temperature distribution heat map data; ToF depth cameras accurately acquire spatial three-dimensional coordinate information (such as equipment spacing, material stacking height, obstacle spatial position), providing depth support for geometric perspective correction. All acquisition devices support remote parameter configuration and... Real-time status monitoring is achieved by connecting to an edge computing gateway via industrial Ethernet. The frame rate (upgraded to 60fps for high-speed motion scenes and reduced to 15fps for static monitoring scenes), resolution (switching to 4K mode for detailed observation and 1080P mode for global monitoring), and acquisition triggering method (continuous acquisition / event-triggered acquisition) can be flexibly adjusted according to the dynamic needs of the scenario (such as high-speed production periods and emergency response scenarios). At the same time, it supports multi-device timestamp synchronization calibration (synchronization error ≤10ms) to ensure the consistency of multi-source video streams in the time dimension, and finally form a multi-source heterogeneous video dataset covering texture, temperature, and spatial information.

[0026] In step S2, video preprocessing is performed on the multi-source heterogeneous video dataset, including spatiotemporal synchronization alignment, noise reduction, and format standardization, to obtain the video stream to be processed. This ensures that the video stream input to subsequent processes has temporal consistency, low noise interference, format uniformity, and data validity.

[0027] refer to Figure 3 As shown, Figure 3 This is a schematic diagram of spatiotemporal synchronization alignment of multi-source heterogeneous video datasets. Specifically, the spatiotemporal synchronization alignment of video preprocessing adopts timestamp matching and inter-frame interpolation algorithms to eliminate frame delay and frame rate differences between multi-source videos; the noise reduction process adopts an adaptive median filtering algorithm to remove environmental noise while preserving image details. Spatiotemporal synchronization alignment: Frame matching is performed based on the timestamp of the acquisition device. For videos with inconsistent frame rates, a linear interpolation algorithm is used to supplement missing frames and eliminate frame delay of multi-source videos (synchronization error is controlled within 50ms). Denoising: An adaptive median filtering algorithm is used to dynamically adjust the size of the filtering window based on the difference in pixel gray values, eliminating environmental noise (such as light reflection and device jitter) while preserving image edges and details; Format standardization: Videos of different resolutions are scaled to a preset size (e.g., 1920×1080) and converted to a unified data format (e.g., YUV420) to provide a unified input for subsequent processing.

[0028] It should be noted that spatiotemporal synchronization alignment is achieved by establishing a global timeline based on the high-precision timestamps of the acquisition devices (synchronization accuracy ≤10ms), and using a two-stage algorithm of "timestamp coarse matching + inter-frame motion compensation interpolation" to align multi-source videos. First, timestamp matching is used to locate the same-source frames of the multi-source videos (error ≤20ms). For videos with inconsistent frame rates (e.g., visible light 30fps, infrared 25fps, depth 15fps), an adaptive linear interpolation algorithm is used to supplement missing frames. Motion compensation interpolation is used to preserve dynamic continuity in high-speed motion areas (e.g., material conveyor belts), while ordinary linear interpolation is used to reduce computational overhead in static areas. For video streams with frame delays, the frame sequence order is dynamically adjusted by calculating the inter-frame time difference, ultimately controlling the spatiotemporal synchronization error of the multi-source videos within 20ms, ensuring spatiotemporal consistency for subsequent correction and fusion. Layered denoising: A "modal adaptation + layered denoising" strategy is adopted, selecting differentiated denoising algorithms based on the noise characteristics of different video modalities. For visible light video, an adaptive median filtering algorithm is used, dynamically adjusting the filtering window size (3×3~7×7) according to the variance of pixel gray values. While removing salt-and-pepper noise and Gaussian noise caused by light reflection and device jitter, key information such as device outline and texture details is preserved through an edge protection mechanism. For infrared video, a bilateral filtering algorithm is used, combining gray-level similarity and spatial proximity weights to suppress artifacts caused by thermal noise and environmental interference, while maintaining the continuity of temperature gradient changes. For depth video, a confidence-based bilateral filtering is used, utilizing the confidence map output by the depth camera to focus on denoising low-confidence pixels (such as blurred edge areas) and lightly filtering high-confidence pixels (such as planar areas) to ensure the accuracy and integrity of depth data. Format standardization: A "resolution adaptive scaling + unified format conversion" scheme is adopted. First, based on the computing power and accuracy requirements of subsequent processing, a target resolution (such as 1920×1080 or 3840×2160) is preset. Video streams with different original resolutions are scaled using bilinear interpolation (normal areas) or Lanczos interpolation (edge ​​areas) algorithms to avoid jagged edges or blurring after scaling. Then, all video streams are uniformly converted to YUV420 format (balancing image quality and storage overhead), and pixel values ​​are normalized (mapped to the [0,1] interval). At the same time, the data storage format is unified as H.265 encoded video files to reduce the bandwidth consumption of data transmission and processing, and to provide standardized input for subsequent geometric perspective correction and feature fusion. Data quality screening: A new data validity verification step has been added. By calculating the signal-to-noise ratio (SNR), edge preservation (EPI), and information entropy (IE) of each frame image, invalid frames that are severely blurred (SNR≤20dB), have lost edges (EPI≤0.6), or lack information (IE≤5) are removed. For consecutive invalid frames (≥3 frames), the acquisition device parameters are reconfigured (such as increasing exposure and adjusting focus) to ensure that the input video stream to be processed has stable high-quality characteristics and avoid invalid data affecting the subsequent processing effect.

[0029] In step S3, geometric perspective correction is performed. Based on camera intrinsic and extrinsic parameter estimation and local mesh transformation algorithm, perspective distortion correction is performed on each frame of the video stream to be processed, and a standard viewpoint video stream without geometric distortion is output.

[0030] refer to Figure 4 As shown, this diagram illustrates geometric perspective correction. Specifically, geometric perspective correction includes: S3A: Predicts camera intrinsic parameters (focal length, principal point) and extrinsic parameters (rotation matrix, translation vector) using the VGGT deep learning model. Specifically, it uses the VGGT deep learning model to automatically predict camera intrinsic parameters, focal length f, principal point coordinates (u0, v0), and extrinsic parameters (rotation matrix R, translation vector t) after inputting preprocessed video frames, without requiring manual calibration, thus improving adaptability. S3B: An improved APAP algorithm is used to construct a local image grid. The homography transformation matrix of each grid is calculated in combination with camera parameters. Specifically, the improved APAP algorithm is used to divide the image into M×N uniform grids (such as 10×10). The homography transformation matrix H of each grid is calculated based on camera parameters. The grid is dynamically densified for areas with large depth changes (such as increasing the grid density of edge areas by 2 times) to ensure local correction accuracy. S3C: Based on the homography transformation matrix, the image is subjected to grid-by-grid perspective correction, and pixel resampling is achieved through bilinear interpolation to obtain a standard viewpoint image. Specifically, perspective transformation is performed on each grid based on the homography transformation matrix, and the pixel grayscale value is recalculated using a bilinear interpolation algorithm to avoid jagged edges in the corrected image. Finally, a standard viewpoint video stream (such as top view or front view) without geometric distortion is output. It should be noted that, based on the three-level correction framework of "accurate camera parameter estimation - adaptive local mesh construction - precise mesh-by-mesh transformation", perspective distortion and lens distortion are jointly corrected for each frame of the video stream to be processed, outputting a standard-view video stream with uniform geometry and accurate spatial position (such as a preset global top view or regional front view), specifically including: Adaptive estimation of camera intrinsic and extrinsic parameters: An improved VGGT deep learning model (integrating attention mechanism and multi-scale feature extraction) is adopted. Preprocessed video frames and corresponding depth data are input as auxiliary constraints to automatically predict camera intrinsic and extrinsic parameters. Intrinsic parameters include focal length f (horizontal / vertical focal lengths are estimated separately, with an accuracy of ±1%), principal point coordinates (u0, v0), radial distortion coefficients (k1, k2, k3), and tangential distortion coefficients (p1, p2), comprehensively covering the effects of inherent lens distortion. Extrinsic parameters include rotation matrix R (a 3×3 matrix describing camera pose) and translation vector t (a 3×1 vector describing camera spatial position). The model training phase incorporates camera calibration datasets from industrial scenarios (such as workshops and ports), supporting real-time parameter updates in dynamic scenes (recalibrated every 10 frames to adapt to slight camera displacement). High-precision calibration can be completed without manual intervention, with intrinsic parameter estimation error ≤2% and extrinsic parameter pose angle error ≤0.5°, significantly improving adaptability to complex scenes. Local mesh adaptive construction and homography matrix calculation (using an improved APAP algorithm to construct a two-layer mesh structure of "globally uniform mesh + locally dynamic refinement"): First, a basic uniform grid is adaptively divided based on the image resolution. For example, a 1920×1080 resolution image is divided into a 15×15 grid. The grid size is adjusted proportionally with the image size to ensure balanced global coverage. Combining the 3D spatial information provided by the depth video, the depth variance of each grid region is calculated. When the depth variance is ≥0.5m (preset threshold, which can be adjusted as needed), it is identified as a region with large depth variation (such as equipment edges, material stacking areas, and areas with high and low scene elevations). The grid in this region is dynamically densified (the density of the densified grid is 3 times the base density, such as 15×15→45×45), while preserving the topological connection relationship between grids to avoid local distortion. Based on the camera intrinsic and extrinsic parameters estimated by S3A, combined with the 3D coordinates of each grid vertex, the independent homography transformation matrix H (3×3 matrix) of each grid is derived through the perspective projection principle. The RANSAC algorithm is used to remove outliers (such as coordinate deviation points caused by occlusion) in the grid vertices to ensure the calculation accuracy of the homography matrix, with a matrix error ≤0.001. Per-mesh perspective transformation and pixel resampling: Perspective transformation is performed grid by grid: Based on the homography transformation matrix H of each grid, coordinate mapping is performed on all pixels within the grid, converting tilted and distorted pixel coordinates into target coordinates under the standard viewpoint, achieving precise correction at the grid level; for grid boundary areas, a neighborhood grid transformation matrix fusion strategy is adopted to avoid boundary breaks or overlaps. Pixel resampling optimization: A hybrid algorithm combining bilinear interpolation and bicubic interpolation is used to recalculate the grayscale value of the transformed pixel coordinates. Bilinear interpolation is used to balance efficiency and accuracy in flat areas (such as the ground and equipment planes), while bicubic interpolation is used to preserve detail sharpness in edge areas (such as equipment outlines and material edges). At the same time, anti-aliasing processing is introduced to avoid jagged edges or moiré patterns in the corrected image. Standard viewpoint output: After all grids are corrected, they are stitched together to form a complete standard viewpoint image. Coordinate normalization is used to ensure the viewpoint uniformity of multi-source videos (e.g., all are corrected to a top view in the geodetic coordinate system, with a pixel to physical space ratio of 1px:5cm), ultimately forming a continuous standard viewpoint video stream without geometric distortion.

[0031] In step S31, the correction effect is verified and dynamically adjusted in real time. The reprojection error and structural similarity (SSIM) of the corrected video frame are calculated. When the index exceeds the preset threshold, the local mesh is dynamically encrypted and the homography transformation matrix is ​​recalculated. The optimization is iterated until the accuracy requirements are met. refer to Figure 5 As shown in the figure, this diagram illustrates the real-time verification and dynamic adjustment of the correction effect. Specifically, the calculation of indicators involves: extracting key feature points (such as device corner points and area boundary points) from video frames before and after correction, calculating the reprojection error (coordinate deviation value) of the feature points, and simultaneously calculating the structural similarity (SSIM) between the corrected image and the standard viewpoint reference image; threshold judgment: the preset reprojection error threshold is ≤1.5 pixels and the SSIM threshold is ≥0.95. If either indicator fails to meet the requirements, it is determined that the local correction has failed.

[0032] Specifically, the reprojection error calculation is based on the coordinate deviation of feature points in the images before and after correction, with a preset error threshold of ≤1.5 pixels; the structural similarity (SSIM) preset threshold is ≥0.95; local mesh refinement adopts an adaptive density strategy, increasing the mesh density of areas with excessive reprojection error to 2-3 times the base density, and the number of iterations is no more than 3 to ensure real-time performance; The dynamic optimization is as follows: an adaptive mesh densification strategy is adopted for the failed area, the mesh density of the area is increased to 2-3 times the base density, the homography transformation matrix is ​​recalculated and correction is performed, the number of iterations is not more than 3 (to avoid affecting real-time performance), until both indicators meet the preset requirements, and a high-precision standard viewpoint video stream is output.

[0033] It should be noted that a closed-loop mechanism of "precise feature point matching - dual-index joint verification - local adaptive optimization" is adopted to ensure the geometric accuracy and visual consistency of the corrected video frames. The specific process is as follows: Intelligent extraction and matching of key feature points: Based on the improved SIFT algorithm (integrating edge detection and corner response enhancement), stable key feature points are automatically extracted from video frames before and after correction. Rigid feature points such as device corners, regional boundary turning points, and center points of fixed landmarks are prioritized (≥50 points per frame, repetition rate ≥90%), while avoiding the selection of variable feature points such as material surfaces and shadow areas. A one-to-one correspondence between feature points before and after correction is established through the FLANN matching algorithm, and the RANSAC algorithm is used to eliminate mismatched points to ensure the accuracy of coordinate deviation calculation. Precise calculation and grading of dual indicators: Reprojection error calculation: For the matched feature point pairs, calculate the Euclidean distance between the actual coordinates and the theoretical projected coordinates of the corrected feature points to obtain the single feature point reprojection error. Take the average error of all valid feature points as the frame-level reprojection error; at the same time, count the proportion of feature points exceeding the threshold in a single frame (abnormal proportion threshold ≤ 5%) to avoid local extreme errors affecting the overall judgment. The preset reprojection error threshold is ≤ 1.5 pixels (industrial scene) / ≤ 2.0 pixels (outdoor large scene), and the abnormal proportion threshold is ≤ 5%.

[0034] Structural Similarity (SSIM) Calculation: A standard viewpoint reference image (such as a scene BIM model rendering or a high-precision calibration photograph) is selected as the baseline. A block-level SSIM calculation method (block size 16×16) is used to calculate brightness similarity, contrast similarity, and structural similarity separately, and then weighted to obtain the global SSIM value. Simultaneously, local SSIM calculations are performed in densely populated areas and areas with large depth variations (local threshold ≥ 0.93) to avoid global metrics masking local distortions. The preset global SSIM threshold is ≥ 0.95, and the local SSIM threshold is ≥ 0.93.

[0035] Grading judgment rules: If the frame-level reprojection error is ≤ the threshold and the abnormal proportion is ≤ 5%, the global SSIM is ≥ 0.95 and the local SSIM is ≥ 0.93, it is judged as qualified for correction; if any indicator is not met, the feature point clustering area with excessive positioning error (i.e., the local correction failure area) is marked with the boundary coordinates of the area for subsequent optimization. Local adaptive optimization and iterative control: Precise location of failure areas: Based on the coordinate distribution of abnormal feature points, the minimum bounding rectangle of the local correction failure area is delineated using a connected component analysis algorithm, thus clarifying the optimization range (avoiding global recalculation and reducing computational power consumption). Adaptive mesh refinement: The refinement strength is dynamically adjusted based on the depth variance and error magnitude of the failure area. When the reprojection error is 1.5-2.0 pixels (or local SSIM 0.90-0.93), the mesh density is increased to twice the base density. When the reprojection error is >2.0 pixels (or local SSIM <0.90), the mesh density is increased to three times the base density, and the mesh is further subdivided in the error peak area (minimum mesh size ≤10×10 pixels). Iterative optimization execution: The homography transformation matrix is ​​recalculated for the encrypted local mesh. Secondary correction is performed only on the failure area, while the correction results of the non-failure area remain unchanged. After each iteration, the index calculation and judgment are repeated. The maximum number of iterations is set to 3 (the processing delay of a single iteration is ≤30ms). If the threshold is still not met after 3 iterations, the camera parameters are re-estimated (S3A step) and the correction process is restarted to ensure the correction accuracy in extreme scenarios. Results and Log Recording: Optimized video frames that meet the dual requirements are directly output to subsequent processes; the iterative optimization process, indicator change data, grid encryption parameters, etc. are logged to support subsequent scene parameter tuning and fault tracing; at the same time, the correction pass rate of consecutive frames is monitored in real time (≥98%). If 3 consecutive frames are unqualified, the device status self-check is triggered (such as whether the camera is offset or the lens is contaminated), which improves the robustness of the system.

[0036] In step S4, feature-level heterogeneous fusion is performed to extract multimodal features from the standard viewpoint video stream. The multimodal features are then weighted and fused using an attention mechanism and a Transformer fusion network to generate a unified feature map.

[0037] refer to Figure 6 As shown in the figure, this is a schematic diagram of a multimodal feature extraction and fusion network. Specifically, multimodal feature extraction uses a lightweight CNN network to extract texture features from visible light video, temperature features from infrared video, and spatial features from depth video. Specifically, this includes texture feature maps (dimensions 512×H×W) from visible light video, temperature feature maps (dimensions 256×H×W) from infrared video, and spatial feature maps (dimensions 256×H×W) from depth video under standard viewpoints, thereby reducing computational overhead while ensuring feature extraction accuracy. The Transformer fusion network learns feature weights through a cross-modal attention mechanism to achieve adaptive fusion of multimodal features. Specifically, it calculates the correlation weights of different modal features through a self-attention mechanism, and performs weighted summation on texture features, temperature features, and spatial features. The feature weights of key regions (such as equipment failure points and densely populated areas) are dynamically increased to achieve adaptive fusion of "important information enhancement and redundant information suppression" and generate a unified feature map (dimension 512×H×W).

[0038] It should be noted that a three-tiered architecture of "modal-specific feature extraction - cross-modal association modeling - key information enhancement fusion" is adopted to accurately extract the core features of multimodal videos and achieve adaptive fusion, generating a unified feature map with complete information and clear hierarchy. The specific process is as follows: Multimodal feature extraction and enhancement: For visible light, infrared, and depth video from the standard viewpoint, a lightweight, customized CNN network (based on MobileNetV3 with the introduction of a channel attention module, SE) is used to extract dedicated features for each, balancing extraction accuracy and computational efficiency. Visible light video feature extraction: A feature extraction branch is constructed by three layers of depthwise separable convolution and two layers of bottleneck convolution, which focuses on capturing scene texture details (such as scratches on the surface of the device and material texture distribution) and color features (such as material qualification color marks and abnormal warning colors), and outputs a texture feature map with dimensions of 512×H×W; an edge enhancement module is integrated into the convolution process to enhance the expression of key structural features such as device outline and region boundaries, and improve the feature response value.

[0039] Infrared video feature extraction: A temperature-sensing convolutional branch is designed. The first two convolutional layers are used to suppress thermal noise, and the last three convolutional layers focus on temperature gradient changes and abnormal high-temperature areas (such as motor overheating and material spontaneous combustion hazards). The output is a temperature feature map with dimensions of 256×H×W. The feature map is normalized to temperature (mapped to the [0,1] interval), and the feature values ​​of high-temperature areas are weighted and amplified (weight coefficient 1.5-2.0) to enhance the identification of abnormal temperature information.

[0040] Deep video feature extraction: Construct a spatial geometric feature extraction branch, combine 3D convolution and pooling operations to extract three-dimensional spatial information of the scene (such as device spacing, material stacking height, and obstacle spatial coordinates), and output a spatial feature map with dimensions of 256×H×W; introduce a spatial attention module to enhance the features of depth abrupt regions (such as device edges and ground drop areas), while suppressing redundant features in depth blurred regions (such as distant backgrounds), thereby improving the accuracy of spatial features.

[0041] Cross-modal feature association modeling and weight learning are implemented by constructing a Transformer fusion network with a dual attention mechanism to achieve dynamic association and adaptive weight allocation of multimodal features. Intramodal self-attention modeling: Self-attention calculation is performed on the feature map of each modality. By calculating the spatial correlation of pixels in the feature map, the feature representation of local key areas (such as equipment failure points and densely populated areas) is strengthened, and the feature weights of redundant areas such as the background are suppressed, so that important information within the modality is more prominent.

[0042] Cross-modal mutual attention modeling: Construct a cross-modal attention matrix to calculate the semantic correlation among texture features, temperature features, and spatial features (such as the spatial correspondence between high-temperature areas and equipment fault points), and quantify the contribution of different modal features to global information; for example, when infrared features detect high-temperature anomalies, increase the association weight between the texture features (used to identify fault types) and spatial features (used to locate fault locations) corresponding to that area.

[0043] Dynamic weight generation: Based on the calculation results of the cross-modal attention matrix, combined with softmax normalization, global and local weights of each modality feature are generated. The global weight is used to balance the overall contribution of the three modalities (default texture feature 0.5, temperature feature 0.25, spatial feature 0.25, which can be dynamically adjusted). The local weight is used to accurately allocate the modality feature ratio of each pixel position, so as to achieve adaptive adjustment of "whichever modality is needed where it is dominant".

[0044] Multimodal feature weighted fusion and map generation employs a fusion strategy of "global weighting + local enhancement" to generate a unified feature map. Basic weighted fusion: The texture feature map, temperature feature map, and spatial feature map are summed pixel by pixel according to dynamically generated global and local weights to initially integrate multimodal information; among them, the 512-dimensional texture feature map is split and stitched through channels to achieve dimensionality matching with the 256-dimensional temperature feature map and the 256-dimensional spatial feature map to ensure the feasibility of fusion operation.

[0045] Key region enhancement fusion: Key regions (equipment failure points, densely populated areas, high temperature abnormality areas, and spatial obstacle areas) in the feature map are identified in real time through the target detection algorithm (lightweight YOLOv8). The fusion features of these regions are then enhanced by a second weighting (enhancement coefficient 1.2-1.8) to further strengthen the feature expression of core information.

[0046] Feature map optimization: Batch normalization is performed on the fused feature maps to eliminate feature distribution shifts caused by modal differences; redundant channels are compressed through 1×1 convolution to maintain the feature dimension of 512×H×W, and finally a unified feature map with complete information, key prominence, and no modal conflicts is generated, providing high-quality feature input for subsequent video twin mapping.

[0047] In step S41, the fusion feature adaptive weight calibration is performed. Based on the scene type (industrial monitoring / emergency rescue, etc.) and real-time data quality (such as light intensity and occlusion ratio), the fusion weight of each modality feature is dynamically adjusted through a reinforcement learning model to optimize the information relevance of the feature map.

[0048] refer to Figure 7As shown, this figure is a schematic diagram of adaptive weight calibration of fused features. Specifically, the scene type is automatically determined by preset labels or image content recognition, and the data quality is quantitatively evaluated by light intensity sensor data and occlusion detection algorithm. The reinforcement learning model uses "maximizing information recognition" as the reward function, and the adjustment range of visible light texture feature weights is 0.3-0.8, infrared temperature features are 0.1-0.5, and depth spatial features are 0.1-0.4. Specifically, scene and data quality assessment: The scene type is automatically determined by preset scene labels (such as "smart factory" and "emergency rescue") or image content recognition algorithms; the ambient lighting conditions are quantified by light intensity sensor data, and the occlusion ratio of the image is calculated by occlusion detection algorithm to comprehensively assess the data quality; Dynamic weight adjustment: Construct a reinforcement learning model with "maximizing information recognition" as the reward function (e.g., in emergency rescue scenarios, the recognition weight of infrared temperature features is increased), and dynamically adjust the weights of each modality feature: the weight of visible light texture features is adjusted from 0.3 to 0.8 (higher value is taken when there is sufficient light), infrared temperature features from 0.1 to 0.5 (higher value is taken in high temperature abnormal scenarios), and depth spatial features from 0.1 to 0.4 (higher value is taken when there is high spatial positioning requirements); Feature map optimization: The adjusted weights are applied to the unified feature map to enhance the feature representation of core information, suppress the interference of redundant information, and generate a more targeted optimized feature map.

[0049] It should be noted that the scene and data quality are refined for evaluation: Scene type determination adopts a dual confirmation mechanism of "preset labels + image content recognition" - the preset labels are configured by the user in advance according to the application scenario (such as "smart factory - production line monitoring" and "emergency rescue - fire scene"). At the same time, a lightweight image classification network (based on the improvement of EfficientNet-B0) is used to analyze key elements in video frames (such as equipment type, flame characteristics, and personnel distribution), automatically verifying and correcting scene labels, with an accuracy rate of ≥95%; The data quality assessment constructs a multi-dimensional quantitative index system. The light intensity is mapped to 0-10 levels through sensor data (level 0 is complete darkness, level 10 is strong direct light), and the occlusion ratio is calculated by segmenting foreground occluders (such as dust, obstacles, and personnel) through semantic segmentation algorithms (accurate to 1%). At the same time, image clarity (quantified by Laplacian variance, threshold ≥100 is clear) and temperature signal-to-noise ratio (specific to infrared video, threshold ≥30dB is effective) indicators are added to form a four-dimensional data quality vector, which comprehensively reflects the effectiveness of multimodal data; Reinforcement Learning Model Optimization and Precise Weight Control: The reinforcement learning model adopts the DQN (DeepQ-Network) architecture, using scene type labels and four-dimensional data quality vectors as state inputs, and combining the weights of each modality feature as the action output. The reward function, "maximizing information recognition," is calculated by weighting the key information detection rate (such as fault area identification rate and high temperature point location accuracy) with the modal conflict value (multimodal feature semantic bias) (reward value range 0-10). A multi-scene sample library (covering video data under different lighting, occlusion, and temperature conditions) is integrated into the model training phase, supporting online incremental learning. The weight adjustment strategy is updated in real time. The weight adjustment adopts the "global benchmark + local compensation" mode. The global benchmark weight is preset according to the scenario type (e.g., smart factory scenario: visible light 0.6, infrared 0.2, depth 0.2; emergency rescue scenario: visible light 0.3, infrared 0.5, depth 0.2). The local compensation weight is dynamically fine-tuned according to the data quality. For example, when the light intensity is ≤3, the visible light weight is reduced by 0.1-0.2 and the infrared weight is increased by 0.05-0.15. When the occlusion ratio is ≥30%, the depth weight is increased by 0.05-0.1. This ensures that the weight adjustment not only meets the scenario requirements but also adapts to the real-time data status.

[0050] In step S5, video twin mapping is performed, and the digital twin model is driven by the unified feature map to perform real-time state updates, generating a video twin image that is consistent with the physical scene in time and space and has complete information.

[0051] refer to Figure 8 As shown in the figure, this is a schematic diagram of video twin mapping. Specifically, digital twin mapping adopts a "feature-driven - state update" mechanism, which uses a unified feature map to drive the geometric shape, texture information and dynamic changes of the virtual scene in real time, achieving millisecond-level synchronization between virtual and physical. Specifically, based on the geometric structure and physical properties of the digital twin model, an optimized feature map mapping relationship is established between the virtual scene and the virtual scene: texture features drive the surface texture update of the virtual scene, temperature features are mapped to the thermal imaging overlay of the virtual scene, and spatial features drive the geometric shape and dynamic position update of the virtual model; an "incremental update" mechanism is adopted, which only processes the features of the changing areas, and controls the update delay of the twin image within 100ms, so as to achieve real-time synchronization between virtual and physical.

[0052] It should be noted that the multi-dimensional feature-virtual scene precise mapping mechanism establishes a one-to-one correspondence mapping rule between "feature dimensions and virtual attributes". Texture features are precisely fitted to the surface mesh of the digital twin model through UV mapping technology, restoring the texture details of the equipment (such as wear marks and markings) and the appearance features of the material (such as color and particle texture), with the mapping error controlled within 1 pixel. Temperature features are converted into a pseudo-color overlay layer through a thermal imaging visualization algorithm, and a red-yellow-green three-color mapping scheme is matched according to the temperature gradient (-20℃—500℃). Flashing warning signs are automatically added to areas with abnormal high temperatures, and the transparency of the overlay layer can be dynamically adjusted (30%—70%) without obscuring the basic texture information. Spatial features are mapped to the virtual scene coordinate system through a coordinate transformation algorithm, driving the model's geometric shape update (such as changes in material stacking height and movement trajectory of equipment parts) and dynamic position calibration (such as real-time coordinate synchronization of mobile devices), ensuring that the spatial position deviation between the virtual scene and the physical entity is ≤5cm. Incremental update and hierarchical synchronization optimization strategy: A hierarchical mechanism of "global coarse synchronization + local fine update" is adopted. At the global level, multimodal features and virtual scenes are coarsely synchronized at the millisecond level (latency ≤50ms) based on timestamp alignment. At the local level, a feature difference detection algorithm (calculating the cosine similarity of feature maps of adjacent frames, and judging the change area by similarity <0.95) is used to accurately locate dynamically changing areas (such as material transfer, equipment start-up and shutdown, personnel movement). Only the feature data of the area is parsed and the model is updated, avoiding the waste of computing power caused by global redrawing. For high-frequency changing areas (such as material flow on conveyor belts), an inter-frame prediction algorithm is used to generate the predicted model state in advance. Combined with real-time feature data, the deviation is corrected, and the update latency is further compressed to within 30ms. For low-frequency changing areas (such as static equipment surfaces), the update frequency is reduced (updated once every 3 frames) to balance real-time performance and computing power overhead, and finally the virtual-physical spatiotemporal synchronization error of the whole scene is ≤100ms.

[0053] In step S51, the interactive-driven local image enhancement responds to user interaction commands (click / selection), locates the target area in the virtual scene, performs super-resolution reconstruction and detail enhancement on the feature map of the target area, and outputs a local high-definition enhanced image.

[0054] refer to Figure 9As shown in the figure, this is a schematic diagram of interactive-driven local image enhancement. Specifically, user interaction commands support three triggering methods: mouse click, touch selection, and voice wake-up. The virtual coordinates of the target area are extracted through the command parsing module. The target area localization uses a coordinate mapping algorithm to associate the virtual scene with the physical scene position, specifically: accurately associating the virtual coordinates with the physical scene coordinates; locking the corresponding feature map region; super-resolution reconstruction uses a lightweight ESRGAN network, and detail enhancement is achieved through an adaptive contrast enhancement algorithm, increasing the local image resolution to twice the original resolution, with a processing latency of ≤50ms. The local enhancement process specifically involves: using a lightweight ESRGAN network to perform super-resolution reconstruction on the feature map of the target region, increasing the resolution to twice the original resolution; and enhancing details within the region (such as the texture of device parts and temperature gradient changes) through an adaptive contrast enhancement algorithm, while maintaining visual consistency with the overall image. The enhanced image output is as follows: the local enhanced image is displayed in the form of a floating window or overlay, with a processing latency of ≤50ms, and does not affect the real-time update of the global twin image.

[0055] It should be noted that the interactive command intelligent parsing and target area precise positioning are as follows: The interactive command parsing module supports multimodal command fusion processing. Mouse click commands accurately extract the click coordinates and the region of interest with a radius of 10-30 pixels (which can be adjusted as needed). Touch selection commands generate irregular target area boundaries through coordinate fitting. Voice wake-up commands combine natural language processing technology to parse target objects (such as "1# motor" and "material stacking area") and automatically match the coordinates of the virtual scene. The target area positioning adopts a "virtual-physical dual coordinate calibration" mechanism. Based on the BIM coordinate system of the digital twin model, a mapping relationship is established between the BIM coordinate system and the geodetic coordinate system of the physical scene. The coordinate deviation is corrected through an error compensation algorithm (positioning error ≤ 3cm). At the same time, the geometric correction parameters of step S3 and the feature map area information of step S4 are linked to achieve precise alignment of the virtual area, feature area and physical area, ensuring the uniqueness and accuracy of the enhanced object. Enhanced algorithm optimization and visual consistency assurance: Super-resolution reconstruction employs an improved lightweight ESRGAN network, which reduces computational cost while maintaining reconstruction accuracy by pruning redundant convolutional layers and quantizing model parameters. It also incorporates prior modal features of the target region (such as texture features guiding texture detail reconstruction and temperature features constraining thermal imaging clarity) to avoid artifacts during reconstruction. Detail enhancement utilizes a "multi-dimensional adaptive enhancement" strategy: texture details are enhanced with a Laplacian pyramid enhancement algorithm to improve edge sharpness; temperature information is optimized for temperature gradient discriminability through histogram equalization; and spatial structures are enhanced with a contour extraction algorithm to strengthen geometric shapes. To ensure visual consistency, brightness, contrast, and color saturation parameters of the global image are automatically extracted during enhancement processing, and the visual parameters of the locally enhanced image are calibrated accordingly. Feathering edge processing technology eliminates the sense of separation between the enhanced area and the global image, ensuring that floating windows or overlays blend naturally with the overall twin image.

[0056] In step S6, the video twin is optimized and output by performing edge enhancement and artifact removal on the video twin, outputting a high-fidelity twin that supports real-time interactive feedback.

[0057] refer to Figure 10 As shown in the figure, this diagram illustrates the image optimization and output process. Specifically, the image optimization uses a guided filtering algorithm to remove fusion artifacts and a Laplacian operator to enhance edges, thereby improving the clarity and visual coherence of the twin images. The artifact removal process involves using a guided filtering algorithm, guided by the texture features of visible light video, to filter the initial twin image generated by the optimized feature map, thereby eliminating ghost artifacts and edge blurring generated during the fusion process. Edge enhancement specifically involves calculating the image edge gradient using the Laplacian operator to enhance key edges such as device outlines and region boundaries, thereby improving image clarity.

[0058] The specific output of the image is as follows: the optimized twin image is output in the form of a video stream, supporting multiple methods such as direct HDMI connection display and cloud platform push. At the same time, it provides an API interface to support real-time interactive feedback (such as clicking on a device in the virtual scene to return the operating parameters of its physical entity).

[0059] It should be noted that the multi-dimensional image optimization and detail preservation processing are as follows: Artifact removal adopts a "hierarchical guided filtering" strategy, setting differentiated filtering parameters for different types of artifacts (ghost artifacts generated by fusion, edge artifacts from super-resolution reconstruction, and color artifacts from cross-modal superposition). For ghost artifacts, a large window guided filtering (window size 9×9) is used, and for edge artifacts, a small window edge-preserving filtering (window size 3×3) is used. At the same time, visible light texture features are used as a guide, combined with the semantic consistency constraints of infrared temperature features and depth space features to ensure that key modal information is not lost during the filtering process. Edge enhancement incorporates the Canny edge detection algorithm on the basis of the Laplacian operator. First, it accurately locates key edges such as device outlines and region boundaries, and then achieves targeted enhancement through dynamic gradient enhancement coefficients (1.5-2.0 coefficient for edge regions and 1.0 coefficient for non-edge regions), avoiding noise amplification caused by global enhancement, while preserving the natural texture of the image. Multi-terminal adaptation output and interactive feedback upgrade: The screen output supports "resolution adaptive + multi-protocol compatibility" mode. It automatically matches different resolutions such as 2K / 4K / 1080P according to the hardware performance and bandwidth conditions of the output terminal (monitoring screen, mobile APP, cloud platform). The video stream encoding adopts H.265 / H.266 dual protocol switching to ensure image quality while reducing bandwidth consumption (4K image quality bandwidth requirement ≤10Mbps). Real-time interactive feedback builds a closed loop of "visual interaction - data feedback - command issuance". When the user clicks on the virtual scene device, in addition to returning the operating parameters (temperature, speed, energy consumption), it also supports the association and retrieval of the device's historical operating data curves and fault warning records. At the same time, it provides a remote control command interface (such as device start / stop, parameter adjustment). The command issuance delay is ≤100ms. The feedback results are superimposed and displayed in real time in the corresponding area of ​​the twin screen, realizing full-process interactive support of "visualization - queryability - controllability".

[0060] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0061] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video twin image method based on heterogeneous video fusion and geometric perspective correction, characterized in that: The video twinning method includes the following steps: S1. Multi-source heterogeneous video acquisition: Deploy video acquisition devices with different modalities and parameters to acquire visible light video, infrared video and depth video streams of physical scenes, forming a multi-source heterogeneous video dataset; S2. Video preprocessing: Spatiotemporal synchronization alignment, noise reduction, and format standardization are performed on the multi-source heterogeneous video dataset to obtain the video stream to be processed. S3. Geometric perspective correction: Based on camera intrinsic and extrinsic parameter estimation and local mesh transformation algorithm, perspective distortion correction is performed on each frame of the video stream to be processed, and a standard perspective video stream without geometric distortion is output. S31: Real-time verification and dynamic adjustment of correction effect, calculate the reprojection error and structural similarity of the corrected video frame, and when the index exceeds the preset threshold, dynamically encrypt the local mesh and recalculate the homography transformation matrix, iteratively optimize until the accuracy requirements are met. S4: Feature-level heterogeneous fusion: Extract multimodal features from the standard viewpoint video stream, and perform weighted fusion of the multimodal features through an attention mechanism and a Transformer fusion network to generate a unified feature map; S41: Adaptive weight calibration of fusion features. Based on scene type and real-time data quality, the fusion weights of each modality feature are dynamically adjusted through a reinforcement learning model to optimize the information relevance of the feature map. S5: Video twin mapping, based on the unified feature map, drives the digital twin model to perform real-time state updates, generating a video twin image that is consistent with the physical scene in time and space and has complete information; S51: Interactive-driven local image enhancement, responding to user interaction commands, locating target areas in virtual scenes, performing super-resolution reconstruction and detail enhancement on the feature maps of target areas, and outputting local high-definition enhanced images; S6: Image optimization and output, performing edge enhancement and artifact removal on the video twin image, outputting a high-fidelity twin image and supporting real-time interactive feedback.

2. The video twin image method based on video heterogeneous fusion and geometric perspective correction according to claim 1, characterized in that: The S1 deploys multiple types of video acquisition devices, including high-definition visible light cameras, infrared thermal imaging cameras, and depth cameras, to cover the entire area of ​​the physical scene without blind spots. The visible light camera is used to capture texture details, the infrared camera is used to capture areas with abnormal temperatures, and the depth camera is used to obtain spatial depth information. The acquisition equipment supports parameter configuration to ensure that multi-source video data covers dynamic changes in the scene and multi-dimensional information.

3. The video twin image method based on video heterogeneous fusion and geometric perspective correction according to claim 1, characterized in that: In step S2, the spatiotemporal synchronization alignment of video preprocessing adopts timestamp matching and inter-frame interpolation algorithms to eliminate frame delay and frame rate differences between multi-source videos; the noise reduction processing adopts an adaptive median filtering algorithm to retain image details while removing environmental noise.

4. The video twin image method based on video heterogeneous fusion and geometric perspective correction according to claim 1, characterized in that: S3, geometric perspective correction includes: S3A: Predicts camera intrinsic and extrinsic parameters using the VGGT deep learning model; S3B: The improved APAP algorithm is used to construct local image grids, and the homography transformation matrix of each grid is calculated in combination with camera parameters; S3C: Based on the homography transformation matrix, perform grid-by-grid perspective correction on the image, and achieve pixel resampling through bilinear interpolation to obtain a standard viewpoint image.

5. The video twin image method based on video heterogeneous fusion and geometric perspective correction according to claim 1, characterized in that: In step S31, the reprojection error calculation is based on the coordinate deviation of feature points in the images before and after correction, with a preset error threshold of ≤1.5 pixels and a preset structural similarity threshold of ≥0.

95. The local mesh densification adopts an adaptive density strategy, increasing the mesh density of areas with excessive reprojection error to 2-3 times the base density, and the number of iterations is no more than 3 to ensure real-time performance.

6. The video twin image method based on video heterogeneous fusion and geometric perspective correction according to claim 1, characterized in that: In step S4, multimodal feature extraction uses a lightweight CNN network to extract texture features from visible light video, temperature features from infrared video, and spatial features from depth video, respectively. Transformer fusion networks learn feature weights through a cross-modal attention mechanism, enabling adaptive fusion of multimodal features.

7. The video twin image method based on video heterogeneous fusion and geometric perspective correction according to claim 1, characterized in that: In S41, the scene type is automatically determined by preset labels or image content recognition, and the data quality is quantitatively evaluated by light intensity sensor data and occlusion detection algorithm. The reinforcement learning model uses "maximizing information recognition" as the reward function, and adjusts the weights of visible light texture features by 0.3-0.8, infrared temperature features by 0.1-0.5, and depth spatial features by 0.1-0.

4.

8. The video twin image method based on video heterogeneous fusion and geometric perspective correction according to claim 1, characterized in that: The S5 digital twin mapping adopts a "feature-driven - state update" mechanism, which uses a unified feature map to drive the geometric shape, texture information and dynamic changes of the virtual scene in real time, achieving millisecond-level synchronization between the virtual and physical worlds.

9. The video twin image method based on video heterogeneous fusion and geometric perspective correction according to claim 1, characterized in that: The S51 supports three triggering methods for user interaction commands: mouse click, touch selection, and voice wake-up; target area localization uses a coordinate mapping algorithm to associate the virtual scene with the physical scene location; super-resolution reconstruction uses a lightweight ESRGAN network, and detail enhancement is achieved through an adaptive contrast enhancement algorithm, increasing the local image resolution to twice the original resolution, with a processing latency of ≤50ms.

10. The video twin image method based on video heterogeneous fusion and geometric perspective correction according to claim 1, characterized in that: In step S6, the image optimization uses a guided filtering algorithm to remove fusion artifacts and a Laplacian operator to enhance edges, thereby improving the clarity and visual coherence of the twin images.

Images