Fusion system based on three-dimensional video

Abstract

The invention discloses a three-dimensional video fusion system, which can realize real-time alignment and registration of a video image and a three-dimensional model by integrating a video stream acquisition module, a sensor data acquisition module and an image processing algorithm, thereby improving the real-time performance and accuracy of the system. Through a depth image processing technology, visual consistency of a video image and a three-dimensional model is optimized, accurate butt joint of the image and the model is ensured, and a visual inconsistency phenomenon is eliminated. According to the method, the video image and the three-dimensional model data are registered by fusing sensor data such as a GPS (Global Positioning System), an IMU (Inertial Measurement Unit) and the like and combining an image processing algorithm, so that the video image and the three-dimensional model data are aligned in the same coordinate system, and spatial synchronization is realized. Accurate integration of sensor data enables the method to maintain high precision and high real-time performance in a dynamic scene.

Description

Technical Field

[0001] The present invention belongs to the field of video fusion, and specifically relates to a three-dimensional video fusion system. Background Art

[0002] With the continuous development of three-dimensional modeling and video technology, three-dimensional video fusion technology has been widely applied in multiple fields, such as disaster rescue, military command, urban planning, etc. However, the existing three-dimensional video fusion systems still face challenges in terms of real-time performance, accuracy, and system integration. Traditional three-dimensional models and video streams mostly rely on static data display, lacking efficient fusion processing of real-time video streams and three-dimensional model data, which results in the system being unable to provide sufficient real-time performance and precision when facing complex dynamic scenes.

[0003] In the prior art, the fusion processing of video streams and three-dimensional models usually uses simple registration and matching algorithms, lacking precise processing of depth information and spatial synchronization. Therefore, when fusing video images and three-dimensional models, obvious visual inconsistencies may occur, resulting in image overlap or misalignment, affecting the overall fusion effect. The prior art also does not fully utilize image processing algorithms (such as depth image processing, edge detection, interpolation methods, etc.) to optimize the visual consistency between images and three-dimensional models, resulting in poor accuracy and realism of the fusion results.

[0004] In view of this, the present invention is specifically proposed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a three-dimensional video fusion system, which solves the problems mentioned in the above background art.

[0006] To solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0007] A three-dimensional video fusion system includes a video stream acquisition module that obtains multiple video streams from a video source in real time through a data acquisition module, and performs denoising, color correction, and image enhancement on the video streams;

[0008] A model loading module that loads three-dimensional model data through a storage module;

[0009] An alignment and registration module that registers the video image and the three-dimensional model data by using sensor data and image processing algorithms, aligns them in the same coordinate system, and synchronizes the space of the video image and the three-dimensional model;

[0010] A transformation and mapping module that maps the pixel points in the video image to the surface of the three-dimensional model according to the registration result by using a transformation matrix;

[0011] The fusion module optimizes the fusion effect according to the depth information of the video image through deep image processing technology, and adjusts the visual consistency between the video image and the 3D model;

[0012] The real-time rendering and display module performs real-time rendering on the fused image through the graphics processing module, and uses the display module to display the fused 3D sand table scene;

[0013] The annotation and information editing module performs annotation and editing in the fused sand table scene through the user input device.

[0014] Optionally, it further includes a storage module, including an Akitio disk array hard disk cabinet and a Western Digital 16TB enterprise-class helium hard disk HC550, which provides high-capacity, efficient storage and data redundancy backup functions, and supports the loading and management of data in multiple formats, including satellite images, topographic maps, aerial maps, and BIM models.

[0015] Optionally, the steps of registering the video image and the 3D model data by using sensor data (such as GPS, IMU) and image processing algorithms to align them in the same coordinate system and synchronize the space between the video image and the 3D model are as follows:

[0016] Obtain real-time sensor data through the sensor module. The sensor data includes GPS data, IMU inertial measurement unit data, and other positioning information, which is used to provide the spatial position information of the video stream and the 3D model;

[0017] Image feature extraction: Extract feature points from the video image, and use image processing algorithms (such as SIFT, SURF, ORB, etc.) to detect and match the feature points in the image for subsequent registration processes;

[0018] Transformation matrix calculation: According to the obtained sensor data (such as GPS position and IMU data) and the matching results of the image feature points, calculate the transformation matrix between the video image and the 3D model, and its expression is: Among them, is the coordinate in the video image coordinate system, is the coordinate in the 3D model coordinate system, M is the rotation matrix, indicating the rotation relationship between the video image and the 3D model, is the translation vector, indicating the translation between the two coordinate systems.

[0019] Registration and spatial synchronization: Apply the calculated transformation matrix to map each pixel point in the video image to the 3D model coordinate system, so that the video image and the 3D model are aligned in the same coordinate system.

[0020] Optionally, according to the registration result, the steps of mapping the pixel points in the video image to the surface of the 3D model by using the transformation matrix are as follows:

[0021] According to the calculated transformation matrix, each pixel point in the video image is mapped to the surface coordinate system of the 3D model through matrix operations. For any pixel point (x′, y′) in the video image, using the transformation matrix M and the translation vector t, the pixel point coordinates (x′, y′) are mapped to the corresponding coordinates (X, Y, Z) on the surface of the 3D model through the following formula, and its expression is: Among them, are the coordinates on the surface of the 3D model, the pixel point coordinates in the video image, M is the rotation matrix, t x , t y , t z is the translation vector;

[0022] Mapping result verification and optimization. After completing the pixel point mapping, check the accuracy of each mapping result and perform optimization processing to ensure the precise docking between the video image and the surface of the 3D model. For the parts that cannot be accurately mapped, apply edge detection and interpolation methods to correct and refine the mapping results, and further improve the fusion quality of the image and the model.

[0023] Optionally, the steps of applying edge detection and interpolation methods to correct and refine the mapping results are as follows:

[0024] Use an edge detection algorithm to process each pixel in the video image, identify and extract the edge information in the image, ensure that the details of the image can be accurately processed during the mapping process, and reduce the influence of low-quality areas;

[0025] Fill in the pixels that cannot be accurately mapped through an interpolation algorithm to ensure that all pixel points can be correctly mapped to the surface of the 3D model. The interpolation algorithms include bilinear interpolation, bicubic interpolation, or nearest neighbor interpolation;

[0026] By analyzing the mapping results after edge detection and interpolation, further correct the mapping errors, and use the information of surrounding pixels for weighted averaging or local optimization to improve the mapping accuracy;

[0027] Refine the corrected mapping results through an image enhancement algorithm, and use methods such as adaptive histogram equalization to enhance the details.

[0028] Optionally, through depth image processing technology, the steps of optimizing the fusion effect according to the depth information of the video image and adjusting the visual consistency between the video image and the 3D model are as follows:

[0029] Extract the depth information from the video image, use a depth sensor or calculate the depth value of each pixel point through a stereo vision algorithm, and generate a depth map. The depth map contains the distance information between each pixel point and the camera;

[0030] Denoise and correct the depth map to remove depth deviations caused by sensor errors or environmental noise, and use Gaussian filtering to reduce the noise of depth information;

[0031] Fuse the depth information with the 3D model. According to the depth map information, adjust the display effect of each pixel point in the video image. Adjust the transparency, brightness, contrast, etc. of the pixels through the depth information to make the depth information of the video image consistent with the surface of the 3D model. Its expression is: I′ = I·(1 - α·d), where I′ is the pixel value of the adjusted video image, I is the pixel value of the original video image, α is the weighting coefficient of the depth information, and d is the depth value of the pixel point in the video image, representing the distance between the pixel and the surface of the 3D model;

[0032] Compare the spatial information of the depth map with the 3D model, and adjust the perspective and lighting of the objects in the video image to make them consistent with the spatial position and texture effect of the 3D model. Its expression is: V = V model +β·(d model -d video ), where V is the optimized visual consistency value, V model is the original visual data of the 3D model, β is the visual consistency adjustment coefficient, d model and d video are the depth information of the 3D model and the object in the video image respectively.

[0033] Optionally, the video source includes but is not limited to cameras, monitoring devices, drones, etc. The denoising algorithm is the median filtering algorithm. When performing color correction, use white balance adjustment and color reconstruction algorithms to correct the color of the video image, automatically adjust the brightness, contrast, and saturation of the video stream to ensure the accuracy and consistency of the image color, and solve the color deviation problem caused by light changes. Image enhancement is achieved through histogram equalization and sharpening processing to enhance the details and edges of the image, make the objects in the video stream clearer, and increase the visibility of the details in the video stream.

[0034] After adopting the above technical solutions, the present invention has the following beneficial effects compared with the prior art. Of course, any product implementing the present invention does not necessarily need to achieve all the advantages described below at the same time:

[0035] 1. By integrating the video stream acquisition module, sensor data acquisition module, and image processing algorithm, the present invention can achieve real-time alignment and registration of the video image and the 3D model, thereby improving the real-time performance and accuracy of the system. Through depth image processing technology, optimize the visual consistency between the video image and the 3D model, ensure the precise docking of the image and the model, and eliminate visual inconsistency.

[0036] 2. The present invention integrates sensor data such as GPS and IMU (Inertial Measurement Unit), and combines image processing algorithms to register video images with 3D model data, aligning them in the same coordinate system and achieving spatial synchronization. The precise integration of sensor data enables the present invention to maintain high precision and high real-time performance even in dynamic scenarios.

[0037] 3. The present invention adjusts the display effect of each pixel point in the video image through image processing technology based on depth information, optimizes the fusion effect, and ensures the depth consistency between the video image and the 3D model. Further, the mapping result is corrected and refined through edge detection and interpolation methods, improving the fusion quality of the image and the model, and making the final image clearer and more delicate.

[0038] The following further describes in detail the specific embodiments of the present invention with reference to the accompanying drawings. Description of the Drawings

[0039] The following drawings in the description are only some embodiments. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. In the accompanying

[0040] In the figures:

[0041] Figure 1 is a block diagram of a 3D video fusion system.

[0042] It should be noted that these drawings and text descriptions are not intended to limit the scope of the concept of the present invention in any way, but to illustrate the concept of the present invention to those skilled in the art by referring to specific embodiments. Specific Embodiments

[0043] The present invention will now be further described in detail with reference to the accompanying drawings.

[0044] Please refer to Figure 1 As shown, in this embodiment, a 3D video fusion system is provided, including a video stream acquisition module that real-time obtains multiple video streams from a video source through a data acquisition module, and performs denoising, color correction, and image enhancement on the video streams;

[0045] a model loading module that loads 3D model data through a storage module;

[0046] an alignment and registration module that registers the video image with the 3D model data by using sensor data and image processing algorithms, aligning them in the same coordinate system and achieving spatial synchronization between the video image and the 3D model;

[0047] a transformation and mapping module that maps the pixel points in the video image to the surface of the 3D model according to the registration result by using a transformation matrix;

[0048] The fusion module optimizes the fusion effect according to the depth information of the video image through deep image processing technology, and adjusts the visual consistency between the video image and the 3D model;

[0049] The real-time rendering and display module performs real-time rendering on the fused image through the graphics processing module, and uses the display module to display the fused 3D sand table scene;

[0050] The annotation and information editing module performs annotation and editing in the fused sand table scene through the user input device.

[0051] This embodiment further includes a storage module, including an ARESIS disk array hard disk cabinet and a Western Digital 16TB enterprise-class helium hard disk HC550, which provides large-capacity, efficient storage and data redundancy backup functions, and supports the loading and management of data in multiple formats, including satellite images, topographic maps, aerial maps, and BIM models.

[0052] The steps of registering the video image and the 3D model data through the use of sensor data (such as GPS, IMU) and image processing algorithms to align them in the same coordinate system and synchronize the space between the video image and the 3D model are as follows:

[0053] Obtain real-time sensor data through the sensor module. The sensor data includes GPS data, IMU inertial measurement unit data, and other positioning information, which is used to provide the spatial position information of the video stream and the 3D model;

[0054] Image feature extraction: Extract feature points from the video image, and use image processing algorithms (such as SIFT, SURF, ORB, etc.) to detect and match the feature points in the image for subsequent registration processes;

[0055] Transformation matrix calculation: According to the obtained sensor data (such as GPS position and IMU data) and the matching results of the image feature points, calculate the transformation matrix between the video image and the 3D model, and its expression is: Among them, is the coordinate in the video image coordinate system, is the coordinate in the 3D model coordinate system, M is the rotation matrix, indicating the rotation relationship between the video image and the 3D model, is the translation vector, indicating the translation between the two coordinate systems.

[0056] Registration and spatial synchronization: Apply the calculated transformation matrix to map each pixel point in the video image to the 3D model coordinate system, so that the video image and the 3D model are aligned in the same coordinate system.

[0057] In this embodiment, according to the registration result, the step of mapping the pixel points in the video image to the surface of the three-dimensional model using the transformation matrix is as follows:

[0058] According to the calculated transformation matrix, each pixel point in the video image is mapped to the surface coordinate system of the three-dimensional model through matrix operations. For any pixel point (x′, y′) in the video image, using the transformation matrix M and the translation vector t, the pixel point coordinates (x′, y′) are mapped to the corresponding coordinates (X, Y, Z) on the surface of the three-dimensional model through the following formula, and its expression is: Where, are the coordinates on the surface of the three-dimensional model, the pixel point coordinates in the video image, M is the rotation matrix, t x t y t z is the translation vector;

[0059] Mapping result verification and optimization. After completing the pixel point mapping, check the accuracy of each mapping result and perform optimization processing to ensure the precise docking between the video image and the surface of the three-dimensional model. For the parts that cannot be accurately mapped, apply edge detection and interpolation methods to correct and refine the mapping result, and further improve the fusion quality of the image and the model.

[0060] In this embodiment, the steps of applying edge detection and interpolation methods to correct and refine the mapping result are as follows:

[0061] Use the edge detection algorithm to process each pixel in the video image, identify and extract the edge information in the image, ensure that the details of the image can be accurately processed during the mapping process, and reduce the influence of low-quality areas;

[0062] Fill in the pixels that cannot be accurately mapped through the interpolation algorithm to ensure that all pixel points can be correctly mapped to the surface of the three-dimensional model. The interpolation algorithm includes bilinear interpolation, bicubic interpolation, or nearest neighbor interpolation;

[0063] By analyzing the mapping result after edge detection and interpolation, further correct the mapping error, and use the information of surrounding pixels for weighted average or local optimization to improve the mapping accuracy;

[0064] Refine the corrected mapping result through the image enhancement algorithm, and use methods such as adaptive histogram equalization to enhance the details.

[0065] In this embodiment, through the depth image processing technology, the steps of optimizing the fusion effect according to the depth information of the video image and adjusting the visual consistency between the video image and the three-dimensional model are as follows:

[0066] Extract depth information from video images, calculate the depth value of each pixel using a depth sensor or through a stereo vision algorithm, and generate a depth map that contains the distance information between each pixel and the camera;

[0067] Denoise and correct the depth map to remove depth biases caused by sensor errors or environmental noise, and use Gaussian filtering to reduce the noise of depth information;

[0068] Fuse the depth information with the 3D model. According to the depth map information, adjust the display effect of each pixel in the video image. Adjust the transparency, brightness, contrast, etc. of the pixels through the depth information to make the depth information on the surface of the video image consistent with that of the 3D model. The expression is: I′ = I·(1 - α·d), where I′ is the adjusted video image pixel value, I is the original video image pixel value, α is the weighting coefficient of the depth information, and d is the depth value of the video image pixel, representing the distance between the pixel and the surface of the 3D model;

[0069] Compare the depth map with the spatial information of the 3D model to adjust the perspective and lighting of the objects in the video image to make them consistent with the spatial position and texture effect of the 3D model. The expression is: V = V model +β·(d model -d video ), where V is the optimized visual consistency value, V model is the original visual data of the 3D model, β is the visual consistency adjustment coefficient, d model and d video are the depth information of the 3D model and the objects in the video image respectively.

[0070] In this embodiment, the video source includes but is not limited to cameras, monitoring devices, drones, etc. The denoising algorithm is the median filtering algorithm. When performing color correction, white balance adjustment and color reconstruction algorithms are used to correct the color of the video image, automatically adjust the brightness, contrast, and saturation of the video stream, ensure the accuracy and consistency of the image color, and solve the color deviation problem caused by light changes. Image enhancement is achieved through histogram equalization and sharpening to enhance the details and edges of the image, make the objects in the video stream clearer, and increase the visibility of the details in the video stream.

[0071] In this embodiment, the storage module supports an offline working mode and can load and process data without a network connection to adapt to the task requirements in complex field environments.

[0072] In this embodiment, the system integrates multiple hardware devices, such as a portable three-proof data processing terminal, a smart touch screen, and an external camera, to ensure the high efficiency and reliability of the system in various application scenarios such as disaster rescue and field exercises.

[0073] In this embodiment, the step of loading the three-dimensional model data further includes loading data from different sources, including satellite images, aerial orthophotos, artificial models and other formats of data, so as to ensure that the fused scene covers a complete three-dimensional environment.

[0074] In this embodiment, the step of depth image processing and layer fusion includes using an image correction algorithm based on depth information to optimize the visual information in the video stream and the three-dimensional model, so that the lighting and shadow effects in the video image are consistent with those in the three-dimensional environment.

[0075] In this embodiment, the step of annotation and information editing further includes that the user performs graphic annotation through a mouse or a touch screen. The annotation supports shapes such as points, lines, surfaces, circles, etc., and can edit the attribute information of the annotation in real time, such as position, video link, text content.

[0076] Existing three-dimensional video fusion systems usually rely on static data display and traditional image processing methods, and cannot effectively process the real-time fusion between video streams and three-dimensional models. Common existing technical solutions include the following:

[0077] 1. Fusion of static images and three-dimensional models:

[0078] Most existing systems rely on static images and pre-loaded three-dimensional model data. The video stream and the three-dimensional model data are usually separated and cannot achieve real-time synchronization and fusion. Image processing generally relies on predefined registration algorithms, such as registration methods based on feature point matching (such as SIFT, SURF algorithms). These methods can achieve good results in fixed scenes, but in dynamic scenes, especially in terms of the synchronization and spatial consistency between real-time video streams and three-dimensional models, the accuracy and stability are poor, and they cannot provide sufficient real-time feedback.

[0079] 2. Rough alignment of video images and three-dimensional models:

[0080] Some systems use sensor data such as GPS and IMU to roughly align video images with three-dimensional models. However, due to the simple calculation method, it is often difficult to ensure the precise synchronization of video streams and three-dimensional models when facing large-scale and complex dynamic scenes. Even when using image registration algorithms, it is still difficult to maintain high-precision spatial synchronization in dynamically changing scenes.

[0081] 3. Limitations of depth image processing technology:

[0082] Some systems have started to attempt deep image processing to optimize the fusion effect of video images and 3D models. However, existing technologies often rely too much on depth sensors (such as lidar), ignoring the precise mapping relationship between depth information and video images, resulting in poor visual consistency of the final image. Moreover, most methods fail to comprehensively utilize information such as light changes and object movements in the video stream to improve the fusion effect.

[0083] Comparative experiment under the same conditions

[0084] To comprehensively evaluate the beneficial effects of the proposed solution of the present invention and conduct a comparative experiment with the existing technical solution, the following same conditions need to be set:

[0085] Hardware environment

[0086] Use the same type of hardware devices for the experiment, including computers and graphics processing units (GPUs) with the same configuration. For example, use a graphics processor with 8GB of video memory on the T1000.

[0087] Use hard disk arrays of the same specification for the storage device, such as the Acasis disk array hard disk cabinet and the Western Digital 16TB enterprise-class helium hard disk HC550, to ensure consistent storage and data reading speeds.

[0088] Use a portable three-proof data processing terminal and an intelligent touch screen for data display and operation.

[0089] Data source

[0090] Adopt the same sources of 3D model data and video stream data to ensure the unity of experimental data.

[0091] Use multiple video stream sources, including real-time video data from drones, vehicle-mounted cameras, or ground cameras.

[0092] Conduct experiments in dynamic scenarios to ensure the complexity of the real scenario.

[0093] Algorithm and processing flow

[0094] In the comparative experiment, both solutions use the same registration algorithm (such as SIFT, SURF) for the preliminary alignment of video images and 3D models.

[0095] In the steps of deep image processing and layer fusion, the existing solution uses standard depth image correction and image enhancement algorithms, while the proposed solution of the present invention combines depth information to optimize the image fusion effect and perform spatial synchronization.

[0096] Test scenario

[0097] Test in dynamic environments such as disaster rescue simulations and field exercises, where the scenarios include complex terrain changes and the appearance of dynamic objects (such as pedestrians, vehicles, etc.).

[0098] The changes in the test scenarios include changes in the illumination of video images, changes in the viewing angle, and continuous changes in the scene depth.

[0099] Table 1: Data table of comparative experiments

[0100]

[0101]

[0102] As shown in Table 1, the solution of the present invention is significantly superior to the prior art solutions in terms of fusion accuracy, processing time, visual consistency, and user operation experience. Specifically:

[0103] The fusion accuracy of the solution of the present invention is 95%, which is improved by 10% - 15% compared with the prior art solutions. The processing time per frame of the solution of the present invention is 1.2 seconds, which is significantly better than the prior art solutions and is improved by 52%. The visual consistency score of the solution of the present invention is 90%, which is improved by 15% - 25% compared with the prior art solutions. The solution of the present invention has a shorter response time and a smoother operation experience, greatly improving the user's operation efficiency.

[0104] The present invention is not limited to the above embodiments. Anyone should know that structural changes made under the inspiration of the present invention, as long as they have the same or similar technical solutions as the present invention, fall within the protection scope of the present invention. The technologies, shapes, and structures not detailedly described in the present invention are all well-known technologies.

Claims

1. A three-dimensional video fusion system, characterized in that, Including: A video stream acquisition module that obtains multiple video streams from a video source in real time through a data acquisition module, and performs denoising, color correction, and image enhancement on the video streams. A model loading module that loads 3D model data through a storage module. An alignment and registration module that registers the video image with the 3D model data by using sensor data and image processing algorithms, aligns them in the same coordinate system, and synchronizes the space of the video image and the 3D model. A transformation mapping module that maps the pixel points in the video image to the surface of the 3D model by using a transformation matrix according to the registration result. A fusion module that optimizes the fusion effect according to the depth information of the video image through depth image processing technology, and adjusts the visual consistency between the video image and the 3D model. A real-time rendering and display module that renders the fused image in real time through a graphics processing module, and uses a display module to display the fused 3D sand table scene. A labeling and information editing module that performs labeling and editing in the fused sand table scene through a user input device.

2. The three-dimensional video fusion system according to claim 1, wherein, It also includes a storage module, and the storage module includes an Accelsior disk array hard disk cabinet and a Western Digital 16TB enterprise-class helium hard disk HC550.

3. The three-dimensional video fusion system according to claim 1, wherein The steps of registering the video image with the 3D model data by using sensor data and image processing algorithms, aligning them in the same coordinate system, and synchronizing the space of the video image and the 3D model are as follows: Obtain real-time sensor data through a sensor module. The sensor data includes GPS data, IMU inertial measurement unit data, and other positioning information. The other positioning information includes, but is not limited to: magnetometer data, visual positioning system data, radar data, ultra-wideband positioning data, sonar or ultrasonic positioning data, wireless network positioning data, which are used to provide the spatial position information of the video stream and the 3D model. Image feature extraction: Extract feature points from the video image, and use image processing algorithms to detect and match the feature points in the image for subsequent registration processes. Transformation matrix calculation: Based on the acquired sensor data and the matching results of image feature points, calculate the transformation matrix between the video image and the 3D model, and its expression is as follows: Wherein, is the coordinate in the video image coordinate system, is the coordinate in the 3D model coordinate system, M is the rotation matrix, representing the rotation relationship between the video image and the 3D model, is the translation vector, representing the translation between the two coordinate systems; Registration and spatial synchronization: Apply the calculated transformation matrix to map each pixel point in the video image to the 3D model coordinate system, so that the video image and the 3D model are aligned in the same coordinate system.

4. A three-dimensional video fusion system according to claim 1, characterized in that, The steps of mapping the pixel points in the video image to the surface of the 3D model by using a transformation matrix according to the registration result are as follows: According to the calculated transformation matrix, each pixel point in the video image is mapped to the surface coordinate system of the 3D model through matrix operations. For any pixel point (x′, y′) in the video image, using the transformation matrix M and the translation vector t, the pixel point coordinates (x′, y′) are mapped to the corresponding coordinates (X, Y, Z) on the surface of the 3D model through the following formula, and its expression is: Wherein, are the coordinates on the surface of the 3D model, are the pixel point coordinates in the video image, M is the rotation matrix, t x , t y , t z is the translation vector; Verification and optimization of mapping results: After completing the pixel point mapping, check the accuracy of each mapping result and perform optimization processing. For the unmapped parts, apply edge detection and interpolation methods to correct and refine the mapping results.

5. A three-dimensional video fusion system according to claim 1, characterized in that The steps of applying edge detection and interpolation methods to correct and refine the mapping results are as follows: Process each pixel in the video image by using an edge detection algorithm to identify and extract the edge information in the image. Then, fill in the unmapped pixels through an interpolation algorithm. The interpolation algorithm includes bilinear interpolation, bicubic interpolation, or nearest neighbor interpolation. Further correct the mapping error by analyzing the mapping results after edge detection and interpolation, perform weighted averaging by using the information of surrounding pixels. Then, refine the corrected mapping results through an image enhancement algorithm and adopt adaptive histogram equalization to enhance the details.

6. The three-dimensional video fusion system according to claim 1, wherein, The steps of optimizing the fusion effect and adjusting the visual consistency between the video image and the 3D model according to the depth information of the video image through deep image processing technology are as follows: Extract depth information from the video image, use a depth sensor or calculate the depth value of each pixel point through a stereo vision algorithm, and generate a depth map, where the depth map contains the distance information between each pixel point and the camera; Denoise and correct the depth map, remove the depth deviation caused by sensor errors or environmental noise, and use Gaussian filtering to reduce the noise of the depth information; Fuse the depth information with the 3D model. According to the depth map information, adjust the display effect of each pixel point in the video image. Adjust the transparency, brightness, and contrast of the pixel through the depth information to make the depth information on the surface of the video image consistent with the 3D model. Its expression is: I′ = I·(1 - α·d), where I′ is the adjusted video image pixel value, I is the original video image pixel value, α is the weighting coefficient of the depth information, and d is the depth value of the video image pixel point, representing the distance between the pixel and the surface of the 3D model; By comparing the spatial information of the depth map with that of the 3D model, the perspective and illumination of the objects in the video image are adjusted to make them consistent with the spatial position and texture effect of the 3D model. The expression is: V = V model +β·(d model -d υideo ), where V is the optimized visual consistency value, V model is the original visual data of the 3D model, β is the visual consistency adjustment coefficient, and d model and d υideo are the depth information of the 3D model and the objects in the video image respectively.

7. The three-dimensional video fusion system according to claim 1, wherein, The video source includes but is not limited to cameras, monitoring devices, and drones. The denoising algorithm is the median filtering algorithm. When performing color correction, the white balance adjustment and color reconstruction algorithm are used to correct the color of the video image, automatically adjust the brightness, contrast, and saturation of the video stream, and image enhancement is achieved through histogram equalization and sharpening processing.

Images