Video fusion ecological environment monitoring system
By working together in the data acquisition, registration, and display modules, the problems of isolated video footage and static 3D scenes have been solved. This has enabled the deep integration of ecological environment monitoring videos with 3D terrain, providing more intuitive and three-dimensional decision support for ecological environment management and enhancing the perception capabilities of ecological environment big data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-09
- Publication Date
- 2026-03-10
AI Technical Summary
Existing ecological and environmental monitoring methods cannot effectively integrate video footage with three-dimensional terrain, resulting in decision-makers being unable to fully understand the environmental information surrounding the monitoring video. Furthermore, the lack of dynamic elements in the three-dimensional scene makes it difficult to meet the intelligent decision-making needs of ecological and environmental management.
It employs a data acquisition module, a 3D spatial registration module, a video fusion display module, and a spatial service publishing module. It collects monitoring video streams through cameras, combines them with real-scene 3D models and high-precision remote sensing data to achieve deep fusion of video and 3D space. It uses Kalman filtering and particle swarm optimization algorithms for dynamic registration, and WebGL technology for texture mapping and feature point matching to provide dynamic display and data sharing of 3D scenes.
It achieves deep integration of ecological environment monitoring videos with 3D terrain, simulates the real ecological environment, expands the observation perspective, reduces modeling costs, provides more intuitive and three-dimensional decision support, and enhances the perception and acquisition capabilities of ecological environment big data.
Smart Images
Figure CN121644767A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of ecological environment monitoring technology, and specifically to a video fusion ecological environment monitoring system. Background Technology
[0002] Currently, in the process of ecological and environmental management, the number of environmental video monitoring points in multiple business areas, including atmosphere, water, soil, groundwater, greenhouse gases, noise, ecological status, rural environment, and nuclear and radiation, is increasing year by year, and the coverage is constantly expanding, thus initially forming an ecological and environmental monitoring network. However, the existing monitoring methods have obvious shortcomings: for decision-makers, when watching monitoring videos, they can only see the scene within the video frame, lacking understanding of the surrounding environment beyond the frame, which is often of important reference value for ecological and environmental emergency decision-making; at the same time, current 3D GIS technology is mostly used to reconstruct static scene models, lacking dynamic elements, and cannot truly restore the ecological scene combining static and dynamic elements, making it difficult to meet the business needs of intelligent ecological and environmental management.
[0003] To address the aforementioned issues, there is an urgent need for a technical solution that can deeply integrate ecological and environmental monitoring videos with three-dimensional terrain data, thereby revitalizing massive amounts of ecological and environmental video monitoring data and providing more intuitive and comprehensive technical support for ecological and environmental management decisions. Summary of the Invention
[0004] The purpose of this invention is to provide a video fusion-based ecological environment monitoring system to solve the problems of isolated video images and static three-dimensional scenes in traditional ecological environment monitoring.
[0005] The basic solution provided by this invention includes a data acquisition module, a three-dimensional spatial registration module, a video fusion and display module, and a spatial service publishing module. The data acquisition module includes cameras deployed in the monitoring area for collecting thematic elements of atmospheric environment, water environment, and ecological environment. The data acquisition module collects the identity information and monitoring parameters of the cameras and captures real-time monitoring video streams. The monitoring parameters include three-dimensional spatial coordinates, horizontal viewing angle, vertical viewing angle, projection distance, azimuth angle, and pitch angle. The three-dimensional spatial registration module is used to collect real-scene three-dimensional model data of the monitoring area, load the three-dimensional spatial scene constructed by the real-time monitoring video stream and the real-scene three-dimensional model, and complete the spatial registration of the monitoring video and the three-dimensional space through texture mapping, camera tracking and dynamic adjustment. The video fusion and display module is used to receive high-precision DEM data and high-precision remote sensing satellite image data, construct a basic three-dimensional spatial scene based on the DEM data and remote sensing satellite image data, and then overlay oblique photogrammetry model, building white model data and ecological environment data on the basic three-dimensional spatial scene. The fused data is then connected to the three-dimensional spatial registration module to realize the playback of monitoring video and the fusion display of video and model. The spatial service publishing module, based on the registration data, integrates the 3D real-scene 3D model around the monitoring video and publishes 3D fusion scene services, business topic services, and registration file services, realizing online sharing of 3D terrain, remote sensing images, tilt models, and ecological environment topic elements.
[0006] The workflow of this invention is as follows: First, the data acquisition module completes camera selection, information collection, and real-time video stream capture; then, the 3D spatial registration module, based on the real-scene model and real-time video stream, completes spatial registration and outputs registration data through texture mapping, camera tracking, and dynamic adjustment algorithms; the spatial service publishing module publishes various 3D spatial services based on the registration data; finally, the video fusion and display module, based on the published services and related data, realizes 3D scene construction, monitoring video playback, and fusion display of video and model, providing users with query, browsing, and analysis functions.
[0007] Through the collaborative work of various modules, deep integration of ecological environment monitoring videos and 3D terrain has been achieved. This solves the problems of isolated video footage, inability to acquire surrounding environmental information, and static 3D scenes in traditional monitoring. It can simulate and restore the real ecological environment monitoring environment, effectively reducing the cost of updating and modeling outdoor 3D models, expanding the observation perspective, making the videos more three-dimensional, and bringing the 3D scenes to life. This enriches the application of new ecological environment big data infrastructure in the ecological environment field, provides necessary technical support for improving the perception and acquisition capabilities of ecological environment big data, and provides a more intuitive and comprehensive technical guarantee for carrying out ecological environment management decision analysis.
[0008] Furthermore, the three-dimensional spatial registration module performs the following spatial registration process: acquiring camera API interface data, using a Kalman filter algorithm to filter the video data, monitoring camera attitude changes in real time, dynamically acquiring the camera's vertical angle, horizontal angle, and magnification, and dynamically adjusting the video projection surface.
[0009] Furthermore, the three-dimensional spatial registration module uses the WebGL dynamic texture mapping algorithm for texture mapping and generates a closely fitting three-dimensional video projection surface based on the camera pose and the image magnification.
[0010] Furthermore, the three-dimensional spatial registration module uses a particle swarm optimization algorithm to dynamically adjust the video projection surface. By combining the real-scene three-dimensional model with the monitoring video footage, it dynamically adjusts the camera's latitude and longitude, elevation, horizontal angle, pitch angle, vertical angle, and projection distance parameters to ensure accurate registration between the monitoring video and the three-dimensional space.
[0011] Furthermore, the spatial service publishing module includes the setting of service access authorization and service sharing scope. Service access authorization obtains service access permissions by carrying a user token or transmitting service request parameters. After parsing the user role through the map service, the spatial service request is forwarded. The service sharing scope splits spatial data according to time sequence and district / county dimensions, and shares it online according to private, public, and district / county dimensions.
[0012] Furthermore, the video fusion display module includes a video configuration submodule and a video display submodule. The video configuration submodule is used to access camera management, video 3D registration, and thematic element management functions, including camera list display, adding / deleting / editing camera data, preset point management, 3D map display, point picking, video cropping, configuring viewing angle, video preview, model preview, video uploading, and video registration functions. The video display submodule provides monitoring video query, detail browsing, and video fusion display functions based on configuration information, including monitoring video wall display, camera detail display, preset point list display, 3D map display, video fusion display, and thematic data display.
[0013] Furthermore, the video surveillance cameras selected by the data acquisition module include multiple atmospheric environment monitoring cameras, multiple water environment monitoring cameras, and at least one natural ecology monitoring camera.
[0014] Furthermore, the video fusion display module automatically follows the changes in the camera lens, adjusts the fusion display effect through manual customization, and uses a dynamic calibration algorithm based on feature point matching for video calibration.
[0015] Furthermore, the video-fusion-based ecological environment monitoring system also includes a database module for storing spatial map data such as 3D terrain, remote sensing images, tilt models, and building white models, as well as business monitoring data, management behavior data, and monitoring video streams. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of the structure of the video fusion-based ecological environment monitoring system in the embodiment; Figure 2 This is a schematic diagram of the three-dimensional spatial data registration process in the embodiment; Figure 3 This is a schematic diagram of the video fusion process in the embodiment; Figure 4This is one of the video frames after 3D fusion. Detailed Implementation
[0017] The following detailed description illustrates the specific implementation method: Appendix Figure 1 The video fusion-based ecological environment monitoring system shown includes a data acquisition module, a three-dimensional spatial registration module, a spatial service publishing module, a video fusion display module, and a database module. The data acquisition module selects eight video surveillance cameras based on the ecological environment monitoring network, including four atmospheric environment monitoring cameras, three water environment monitoring cameras, and one natural ecology monitoring camera.
[0018] Collect basic information and monitoring parameters from 8 cameras, and capture real-time monitoring video streams. The monitoring parameters include three-dimensional spatial coordinates, horizontal viewing angle, vertical viewing angle, projection distance, azimuth angle, and pitch angle.
[0019] Outlier detection was performed on the collected monitoring parameters using the 3σ criterion; the mean was calculated for each parameter. and standard deviation If the data exceeds If the value falls outside the specified range, it is considered an outlier and corrected. Upon inspection, no outliers were found in any of the monitored parameters in this embodiment.
[0020] Real-time video streams from each camera are captured using the RTSP protocol. The video stream parameters are: resolution 1920×1080, frame rate 25fps, and encoding format H.264. The captured video streams are then subjected to integrity checks to ensure there is no packet loss or screen tearing.
[0021] Collect real-scene 3D model data of the monitored area, load the real-time monitoring video stream and the real-scene 3D model to construct a 3D spatial scene. Taking camera number 1 as an example, load the real-time video stream of camera 1 and the real-scene model data within a 5km radius around it to construct a 3D spatial scene of the area.
[0022] like Figure 2 As shown, the 3D spatial registration module collects real-scene model data such as 3D terrain, remote sensing images, tilt models, and environmental themes of the demonstration area, loads the real-time monitoring video stream and the real-scene model to construct a 3D spatial scene, completes the spatial registration of the monitoring video and the 3D space through texture mapping, camera tracking and dynamic adjustment of relevant algorithm models, and outputs the monitoring video with 3D spatial fusion in JSON format.
[0023] The video stream from each camera is processed by texture mapping, and the video image is accurately mapped onto the 3D projection surface as a dynamic texture. The texture mapping process adopts a WebGL-based dynamic texture mapping algorithm.
[0024] Texture coordinate transformation formula: Let the video pixel coordinates be... The video resolution is The 3D projection surface texture coordinates are The conversion formula is: ; Because the WebGL texture coordinate origin is in the bottom left corner and the video coordinate origin is in the top left corner, the v-axis needs to be flipped: Projection surface generation formula: Given camera attitude parameters (azimuth angle) Pitch angle Roll angle ) and screen magnification The camera position coordinates are The distance between the projection surface and the camera is Then the coordinates of the four vertices of the projection plane The calculation is as follows: Calculate the coordinates of the center of the projection plane in the camera coordinate system
[0025] Calculate the half-width of the projected plane in the camera coordinate system Half-height ,in A horizontal perspective. This is a vertical perspective.
[0026] The four vertices of the projection plane in the camera coordinate system: , , ,
[0027] Transform vertices in camera coordinate system to world coordinate system using the pose matrix: Let the attitude matrix be set From azimuth Pitch angle Roll angle The calculation yielded:
[0028] in , ,
[0031] Vertex in world coordinate system: .
[0032] In this embodiment, taking camera number 1 as an example, the basic parameters of camera 1 are obtained: horizontal viewing angle. Vertical perspective azimuth Pitch angle Roll angle Magnification Camera position coordinates Distance between the projection surface and the camera .
[0033] The half-width of the projected plane in the camera coordinate system is calculated. Half-height .
[0034] The four vertices of the projection plane in the camera coordinate system are calculated as follows: , , , .
[0035] Calculate the attitude matrix :
[0036]
[0037]
[0038]
[0039] Vertex calculation in world coordinate system:
[0040]
[0041]
[0042] .
[0043] Texture coordinate transformation: for pixel coordinates in a video frame (Center pixel of video), video resolution , Then texture coordinates , .
[0044] The video feed is rendered as a dynamic texture onto the calculated projection surface using WebGL technology, completing the texture mapping for camera 1. Similarly, the texture mapping process is completed for the other seven cameras.
[0045] The camera tracking process employs a Kalman filter algorithm. Based on camera API interface data, it monitors camera pose changes in real time, dynamically acquires the camera's vertical angle, horizontal angle, and magnification, and dynamically adjusts the video projection surface to ensure the accuracy and consistency of video fusion.
[0046] The dynamic adjustment process employs the Particle Swarm Optimization (PSO) algorithm. Since there are often errors between the camera's basic information such as latitude, longitude, and elevation and its attitude information (such as pitch and horizontal angles), the relevant parameters of the camera are dynamically adjusted by combining the real-scene 3D model with the monitoring video footage to achieve accurate registration between the monitoring video and the 3D space.
[0047] Fitness function: Let the adjustment parameter vector be... ,in Longitude Latitude For elevation, For horizontal angles, It is a vertical angle. For projection distance, It is the azimuth angle. The pitch angle is used to evaluate registration accuracy, and is defined as:
[0048] in This represents the number of feature points used to match the video footage with the 3D model. For parameters Video Feature Points The corresponding projection points in the 3D model Feature points in a 3D model The actual coordinates, It is the Euclidean distance function. The larger the value, the higher the registration accuracy.
[0049] Particle position update formula:
[0050] Particle velocity update formula:
[0051] in For inertial weights, , For acceleration coefficient, , A random number between [0,1] For particles In dimensions The optimal position of an individual on the [top] For the entire particle swarm in dimension The globally optimal position.
[0052] Set the particle swarm size for camera number 1. ; dimension ( ); Inertia weight The initial value is 0.9, which decreases linearly to 0.4 during iteration; the acceleration factor... Maximum number of iterations The parameter search range is shown in the table below:
[0053] Twenty matching feature points were selected between the video footage and the 3D model. During the iteration process, the fitness function value changed as follows:
[0054] When the iteration reaches 100 times, the optimal parameter vector is obtained: Based on these optimal parameters, the relevant parameters of camera 1 are adjusted to achieve precise registration. Similarly, the dynamic adjustment and precise registration of the other seven cameras are completed. Finally, the registration data for each camera is output in JSON format via a standard service interface.
[0055] like Figure 3 As shown, the video fusion display module integrates high-precision DEM data and high-precision remote sensing satellite image data to construct a basic three-dimensional spatial scene, overlays oblique photography model, architectural white model data and ecological environment thematic elements, and integrates the real-time monitoring video stream and registration data to realize the playback of monitoring video and the fusion display of video and model.
[0056] High-precision DEM data is stitched and cropped to retain the DEM data of the demonstration area; remote sensing satellite image data is orthorectified, stitched, and cropped, and then registered with the DEM data.
[0057] Scene construction: Based on WebGL technology, DEM data and remote sensing satellite imagery data are loaded into the 3D engine to construct a basic 3D terrain scene, with the terrain exaggeration factor set to 1.0 and the image transparency set to 1.0.
[0058] Oblique photography model overlay: The oblique model data of the monitoring area of 8 cameras is loaded into the basic 3D space scene. Coordinate registration is used to ensure that the model is aligned with the basic terrain scene, and the model transparency is set to 0.9.
[0059] Building white model overlay: Load the building white model data of the monitoring area of 8 cameras and the surrounding 1km, register it with the basic scene and tilted model, set the model color to gray and the transparency to 0.8.
[0060] Overlay of ecological and environmental thematic elements: Load thematic elements such as the distribution data of air quality monitoring stations, water environment monitoring section data, and boundary data of natural ecological reserves, and use different icons and colors for identification.
[0061] Dynamic calibration processing: A dynamic calibration algorithm based on feature point matching is used to calibrate the fused display effect of each camera in real time. Taking camera 1 as an example, SIFT feature points of video frames and 3D models are extracted every 10 frames for matching and error calculation. When the average distance error exceeds 0.5 meters, the projection surface parameters are adjusted.
[0062] like Figure 4 As shown, this is one of the video frames after 3D fusion. Dynamically changing video data is fused into a 3D model to provide decision-makers with a comprehensive visual understanding of the monitored area and surrounding geographical data.
[0063] The video configuration submodule is used by system administrators for backend maintenance, enabling functions such as displaying a camera list, adding / deleting / editing camera data, and managing preset points. Administrators can add cameras through this submodule, inputting basic camera information and monitoring parameters. The system automatically completes the registration process, including texture mapping, camera tracking, and dynamic adjustments.
[0064] The video display submodule is for use by personnel in the water supply, air, ecology, and big data application centers. It enables functions such as querying surveillance videos, browsing details, and displaying fused video feeds. Users can query surveillance videos by region, camera type, and other criteria, view detailed camera parameters and registration information, observe the fused video and model display effect in a 3D spatial scene, and switch preset points to view surveillance footage from different angles.
[0065] The database module stores spatial map data such as 3D terrain, remote sensing imagery, oblique models, and architectural white models, as well as business monitoring data and management behavior data, and monitoring video streams and equipment parameter data, enabling efficient storage and service access for various types of data. It uses PostgreSQL+PostGIS databases to store spatial data, MySQL databases to store business data and equipment parameter data, and MongoDB databases to store monitoring video stream data.
[0066] The spatial service publishing module is used to publish 3D fusion scene services, business topic services, and registration file services, enabling online sharing of 3D terrain, remote sensing images, oblique models, and environmental topic data.
[0067] The above are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.
Claims
1. An eco-environment monitoring system with video fusion, characterized in that, The system comprises a data acquisition module, a three-dimensional space registration module, a video fusion display module and a space service publishing module. The data acquisition module comprises cameras arranged in a monitoring area for acquiring atmospheric environment, water environment and ecological environment parameters. The three-dimensional space registration module is configured to collect real scene three-dimensional model data of the monitoring area, load a three-dimensional space scene constructed by the real-time monitoring video stream and the real scene three-dimensional model, and complete space registration of the monitoring video and the three-dimensional space through texture mapping, camera tracking and dynamic adjustment. The video fusion display module is configured to receive high-precision DEM data and high-precision remote sensing satellite image data, construct a basic three-dimensional space scene according to the DEM data and the remote sensing satellite image data, superimpose a tilt photography model, a building white model data and ecological environment data on the basic three-dimensional space scene, and realize calling and playing of the monitoring video, and fusion display of the video and the model by inputting the fused data into the three-dimensional space registration module. The space service publishing module is configured to publish three-dimensional fusion scene services, business thematic services and registration file services based on the registration data, and realize online sharing of three-dimensional terrain, remote sensing image, tilt model and ecological environment data by fusing three-dimensional real scene three-dimensional models around the monitoring video.
2. The video-fused ecosystem monitoring system of claim 1, wherein: The three-dimensional space registration module is configured to acquire camera API interface data, filter video data by using a Kalman filtering algorithm, listen to camera posture changes in real time, dynamically acquire camera vertical angle, horizontal angle and magnification, and dynamically adjust a video projection surface.
3. The video-fused ecosystem monitoring system of claim 2, wherein: The three-dimensional space registration module is configured to perform texture mapping by using a WebGL dynamic texture mapping algorithm, and generate a close three-dimensional video projection surface by using a camera posture and a picture magnification.
4. The video-fused ecosystem monitoring system of claim 3, wherein: The three-dimensional space registration module is configured to dynamically adjust the video projection surface by using a particle swarm optimization algorithm, dynamically adjust longitude and latitude, elevation, horizontal angle, pitch angle, vertical angle and projection distance parameters of the camera by combining the real scene three-dimensional model with the monitoring video picture, and realize accurate registration of the monitoring video and the three-dimensional space.
5. The video-fused ecosystem monitoring system of claim 4, wherein: The space service publishing module comprises service access authorization and service sharing range setting.
6. The video-fused ecosystem monitoring system of claim 5, wherein: The video configuration submodule is configured to access camera management, video three-dimensional registration and thematic element management functions, and comprises camera list display, adding, deleting and editing camera data, preset point management, three-dimensional map display, pickup point, video cropping, configuration angle, video preview, model preview, video superposition and video registration functions. The video display submodule is configured to realize three-dimensional video display, video playback, video fusion display and video sharing functions. The video display submodule provides monitoring video query, detail browsing, and video fusion display functions based on configuration information; The system comprises a monitoring video wall display, a camera detail display, a preset point list display, a three-dimensional map display, a video fusion display, and a special data display.
7. The video-fused ecosystem monitoring system of claim 6, wherein: The video monitoring cameras selected by the data acquisition module comprise a plurality of atmospheric environment monitoring cameras, a plurality of water environment monitoring cameras, and at least one natural ecology monitoring camera.
8. The video-fused ecosystem monitoring system of claim 7, wherein: The video fusion display module automatically follows the camera lens changes, adjusts the fusion display effect through manual custom switching, and adopts a dynamic calibration algorithm for feature point matching to calibrate the video.
9. The video-fused ecosystem monitoring system of claim 1, wherein: The system further comprises a database module for storing spatial map data of three-dimensional terrain, remote sensing images, tilt models, and building white models, as well as business monitoring data, management behavior data, and monitoring video streams.