Video stream and rendering simulation fusion method for coping with water conservancy factor evolution scene
By accessing water conservancy monitoring video streams and using computer vision and deep learning models to identify hydrological elements, combined with GLSL custom shaders to generate dynamic effects, the real-time synchronization problem in the scenario of water conservancy element evolution was solved, achieving efficient rendering and accurate simulation, and improving the real-time performance and decision support capabilities of flood control command.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-18
- Publication Date
- 2026-03-27
AI Technical Summary
Existing technologies are insufficient to achieve real-time synchronization and efficient rendering of water conservancy elements in the context of their evolution, resulting in inadequate accuracy and real-time performance in water conservancy disaster prevention and mitigation work.
By accessing water conservancy monitoring video streams compatible with mainstream streaming media protocols, using computer vision and deep learning models to identify hydrological elements, and combining GLSL custom shaders to generate dynamic effects on the GPU, the system then overlays and blends these effects with the real-time video stream in the rendering engine to achieve dynamic simulation of water conservancy elements.
It significantly improves the intuitiveness and real-time nature of flood evolution simulation, provides accurate "four predictions" support, achieves seamless integration of physical site and simulation effect, and improves the decision-making accuracy of flood control command.
Smart Images
Figure CN121744657A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the field of water conservancy digital twinning, and particularly relates to a video stream and rendering simulation fusion method for coping with water conservancy element evolution scenarios. BACKGROUND
[0003] The periodic update of the current model still faces challenges and it is difficult to achieve real-time synchronization in a true sense. In this context, video digital twinning, as a feasible path to realize real-time mapping, has attracted widespread attention. Its technical essence lies in the deep fusion of video stream and spatio-temporal information to build a dynamic link between the physical world and the digital space. Although traditional video monitoring is mainly used for dynamic target observation and is difficult to directly measure accurately, the industry has begun to explore video and three-dimensional model overlay rendering technology and has initially realized functions such as Internet of Things data loading and water conservancy element evolution simulation. By fusing real-time video pictures into the twinning scene, the water and work conditions can be intuitively presented, significantly improving the sense of presence and decision-making accuracy of flood control command. In the future, the deep fusion of video digital twinning, BIM, flow field simulation and AI large models will be the key direction to realize "second-level perception, real-time simulation and autonomous regulation" of all elements of water conservancy. Based on video stream, water conservancy environment data is loaded, superimposed and rendered, and the data of the corresponding water conservancy elements are rendered and simulated to realize the emergency command of disaster prevention and mitigation of water conservancy management departments. SUMMARY
[0004] The purpose of the present application is to provide a video stream and rendering simulation fusion method for coping with water conservancy element evolution scenarios, to realize a digital video twinning rendering application of real water conservancy projects and river basins based on on-site video monitoring and superimposed modeling of actual scene twinning, and to render and fuse water conservancy elements in video stream. The method aims to significantly alleviate the large-scale investment of hardware resources in water conservancy projects, river basins and other large-scale scenarios through new technical applications, and to realize the visualization rendering of the real physical world in a new way, supporting the work of benefiting and reducing disaster in the water conservancy industry.
[0005] To achieve the above purpose, the technical solution of the present application is as follows: a video stream and rendering simulation fusion method for coping with water conservancy element evolution scenarios, comprising:
[0006] By compatible with mainstream streaming media protocols, access and decode water conservancy monitoring video stream, obtain continuous original video frame sequence;
[0007] Using computer vision and deep learning model to analyze the semantics of video frames, automatically identify and extract key hydrological elements and their dynamic change range, and obtain hydrological element boundary data;
[0008] The extracted hydrological element boundary data is dynamically generated on the GPU through parallel computing by using a custom shader based on GLSL, and the simulation effect is parameterized and adjusted according to the evolution degree of the element.
[0009] The dynamic effect obtained by the custom shader based on GLSL is superimposed and fused with the real-time video stream and the hydrological element boundary data by using a rendering engine, and the evolution process of the water conservancy element is dynamically simulated.
[0010] Further, the mainstream streaming media protocols include RTSP, RTMP and HLS.
[0011] Further, by being compatible with mainstream streaming media protocols, the water conservancy monitoring video stream is accessed and decoded to obtain a continuous sequence of original video frames, and the specific mode is as follows:
[0012] Streaming media protocol processing: a high-performance streaming media server including ZLMediaKit is integrated to support direct access and conversion of mainstream streaming media protocols including RTSP, RTMP and HLS;
[0013] Real-time decoding and frame extraction: the FFmpeg library is used for video stream decoding, that is, first, a network connection with the video stream is established, then the video format is parsed, and finally the compressed video data is decoded into a continuous sequence of original video frames.
[0014] Further, the video frames are analyzed semantically by using computer vision and deep learning models to automatically identify and extract key hydrological elements and their dynamic change ranges to obtain hydrological element boundary data, and the specific mode is as follows:
[0015] Image preprocessing: the video frames in the decoded sequence of original video frames are operated including denoising, contrast enhancement and color correction;
[0016] Water body boundary identification: an image segmentation method is used to preliminarily distinguish water body and non-water body areas, and a deep learning semantic segmentation method is further used to accurately extract the water body contour;
[0017] Dynamic change analysis: for scenes including river embankment collapse and rapid water level rise, the change of the boundary needs to be analyzed, and the optical flow method is used to estimate the light and dark contrast of the pixel points in the image sequence, calculate the flow rate or identify the deformation displacement of the dam contour.
[0018] Further, in the image preprocessing process, for water conservancy scenes, water surface reflection and interference caused by rain and snow weather need to be handled.
[0019] Further, the image segmentation method uses an edge detection algorithm or a threshold-based segmentation method to preliminarily distinguish water body and non-water body areas.
[0020] Further, the deep learning semantic segmentation method adopts network structures including U-Net and DeepLabv3+ for pixel-level classification, accurately identifies the water body boundary in a complex background including vegetation shelter and building shadow, and outputs a probability map of each pixel belonging to the water body. By setting a threshold, a binary mask is generated to accurately extract the water body contour.
[0021] Further, the basic constraint equation of the optical flow method is based on the image gray invariance assumption:
[0022] I(x,y,t) = I(x+dx, y+dy, t+dt)
[0023] The above formula indicates that when a pixel point is located at position (x, y) at time t, the brightness is I(x, y, t), and after a time dt, the pixel point moves to position (x + dx, y + dy) at time t + dt, and the brightness remains unchanged;
[0024] By Taylor expansion and ignoring high-order terms, we have:
[0025] I_x * u + I_y * v + I_t = 0
[0026] I_x represents the partial derivative of brightness with respect to x, the horizontal brightness change;
[0027] I_y represents the partial derivative of brightness with respect to y, the vertical brightness change;
[0028] I_t represents the partial derivative of brightness with respect to time, the brightness change over time;
[0029] Since the brightness is constant, I_x dx + I_y dy + I_t dt = 0.
[0030] Further, the extracted hydrological feature boundary data is dynamically generated by parallel computing on the GPU using a custom shader based on GLSL, and the parameters are adjusted according to the evolution degree of the element. The specific way is:
[0031] Shader type and division of labor:
[0032] (1) Vertex shader: responsible for processing the geometric coordinates of the hydrological feature boundary point;
[0033] (2) Fragment shader: executed on each pixel, dynamically calculates the final color and transparency of the corresponding pixel according to the input hydrological feature boundary data;
[0034] Effect simulation principle:
[0035] (1) Data-driven: The extracted hydrological element boundary data is transmitted into the shader in the form of texture or Uniform variable;
[0036] (2) Physical effect simulation: In the fragment shader, use noise function, wave equation or particle system principle to simulate the texture, wave and flow feeling of water flow.
[0037] Further, the dynamic effects obtained by the custom shader based on GLSL are superimposed and fused with real-time video stream and hydrological element boundary data using the rendering engine to dynamically simulate the evolution process of water conservancy elements. The specific way is:
[0038] Video background layer: The real-time decoded video frame is pasted as a texture on a quadrilateral aligned with the screen as the background of rendering;
[0039] Simulation effect layer: The hydrological element evolution effect processed by the custom shader based on GLSL is rendered to another layer, and the rendering depth of this layer needs to be set to ensure the correct spatial relationship with the video background layer;
[0040] Alpha blending: Use the blending function of GPU to superimpose the simulation effect layer and the video background layer; the blending function is set as:
[0041] FinalColor = SrcColor * SrcAlpha + DestColor * (1 - SrcAlpha)
[0042] Where SrcColor is the color of the simulation effect layer, DestColor is the color of the video background layer, and SrcAlpha is the transparency of the simulation effect layer;
[0043] Coordinate system and space-time synchronization: Through camera calibration technology, the mapping relationship between 2D pixel coordinates of video image and 3D world coordinates is established; at the same time, the frame rate of video stream is synchronized with the frame rate of shader rendering to avoid tearing or delay, ensuring the real-time and authenticity of the evolution simulation.
[0044] Compared with the prior art, the method has the following beneficial effects: the method constructs a complete technical link from real-time perception to dynamic simulation, significantly improves the intuitiveness, real-time performance and decision support capability of flood evolution simulation. The application supports mainstream streaming media protocols such as RTSP, RTMP and HLS, realizes the wide compatibility and stable access of water conservancy monitoring video, and provides a real scene picture base for simulation. Further, based on computer vision and deep learning model, the key hydrological elements (such as water body boundary) can be intelligently identified from the video stream and the change range thereof is dynamically extracted, which provides data basis for accurate simulation, and some advanced practices have shown that AI video flow measurement technology can effectively extract water surface features and calculate flow rate. On this basis, the custom shader written by GLSL can generate specific and realistic visual effects (such as flood submergence and water flow evolution) based on the extracted hydrological element boundary data. Finally, in the rendering engine, the special effects generated by the shader are superimposed and fused with the real-time video stream and the hydrological element boundary for rendering, so as to dynamically simulate the evolution process of the water conservancy element, realize the seamless integration of the physical site and the simulation effect, and provide intuitive and accurate 'four predictions' (forecast, early warning, rehearsal and plan) support for flood control command. The method combines streaming media technology, AI recognition and real-time computer graphics, and embodies the advanced idea of realizing the 'perception-decision-simulation' closed loop of digital twin technology in the field of water conservancy. BRIEF DESCRIPTION OF DRAWINGS
[0045] Figure 1 The technical scheme architecture of the application is shown in the figure.
[0046] Figure 2 The intelligent identification and dynamic boundary extraction results of hydrological elements are shown in the figure.
[0047] Figure 3 The non-fusion effect diagram is shown in the figure.
[0048] Figure 4 The fusion effect diagram is shown in the figure. DETAILED DESCRIPTION
[0049] The technical scheme of the application will be specifically described below with reference to the drawings.
[0050] The application provides a video stream and rendering simulation fusion method for water conservancy element evolution scenarios, which comprises the following steps:
[0051] By compatible mainstream streaming media protocol, access and decode water conservancy monitoring video stream, obtain continuous original video frame sequence;
[0052] Using computer vision and deep learning model to perform semantic analysis on video frames, automatically identifying and extracting key hydrological elements and their dynamic change range, obtaining hydrological element boundary data;
[0053] A custom GLSL-based shader is used to dynamically generate simulation effects on the GPU through parallel computing of the extracted hydrological element boundary data, and the parameters are adjusted according to the degree of evolution of the elements.
[0054] The rendering engine is used to overlay and fuse dynamic effects obtained by custom shaders based on GLSL with real-time video streams and boundary data of hydrological elements to dynamically simulate the evolution process of water conservancy elements.
[0055] The following is a detailed implementation process of the present invention.
[0056] like Figure 1 As shown, this invention provides a method for fusing video streams and rendering simulations in scenarios involving the evolution of water conservancy elements. It acquires video streams from video equipment deployed at project sites and constructs a dynamic simulation system for water conservancy elements based on real-time video streams. The system first accesses and decodes water conservancy monitoring video streams by being compatible with mainstream streaming media protocols such as RTSP and RTMP, obtaining a continuous sequence of raw video frames. Subsequently, it uses computer vision and deep learning models to perform semantic analysis on the video frames, automatically identifying and extracting key hydrological elements (such as water body boundaries and riverbanks) and their dynamic change ranges, paying particular attention to contour deformation caused by emergencies such as riverbank collapses, thus achieving a shift from passive monitoring to active perception. To achieve realistic evolution simulation effects, the core of the solution includes the development of a custom shader based on the OpenGL Shading Language (GLSL). The shader program dynamically generates simulation effects (such as water flow impact, soil erosion, and crack propagation) on the GPU through parallel computation based on the extracted hydrological element boundary data, and can be parametrically adjusted according to the degree of element evolution. Finally, the rendering engine uses the real-time video image as a background layer, overlaying and fusing it with the dynamic effects processed by the shader. By employing precise coordinate mapping and alpha blending technologies, the simulation seamlessly integrates with the real-world scenario, enabling real-time and dynamic visualization of the evolution of water resources elements on the monitoring screen. This significantly enhances the intuitiveness of flood control command and the accuracy of decision-making. The solution integrates streaming media technology, AI recognition, and real-time computer graphics, constructing a complete technological chain from data perception to visual presentation, aiming to improve the intelligence and visualization level of water resources monitoring.
[0057] To achieve efficient visualization of large-scale twin models, the core technical path of the method in this invention is as follows:
[0058] 1. Supports mainstream streaming media protocols such as RTSP, RTMP and HLS, accesses and decodes video streams in real time, and obtains the original frame sequence.
[0059] To achieve real-time acquisition of water conservancy monitoring videos, the system needs to build an access layer that supports multiple streaming media protocols. The core technologies lie in the selection and configuration of the streaming media server, and an efficient decoding and frame extraction process. First, streaming media protocol processing is implemented by integrating high-performance streaming media servers such as ZLMediaKit to support direct access and conversion of mainstream protocols such as RTSP, RTMP, and HLS. These servers can effectively handle video streams from IPCs, NVRs, and other devices from different manufacturers, and resolve protocol differences and compatibility issues. Next, real-time decoding and frame extraction using the FFmpeg library are crucial steps. This process first establishes a network connection with the video stream, then parses the video format, and finally decodes the compressed video data into a continuous sequence of raw frames (usually YUV or RGB format), preparing for subsequent image processing.
[0060] 2. Based on computer vision and deep learning models, intelligently identify key hydrological elements (such as water body boundaries) and dynamically extract their range of change.
[0061] Intelligent identification and dynamic boundary extraction of hydrological elements: This stage serves as a bridge connecting physical video and digital simulation. The core is to use computer vision and deep learning algorithms to extract hydrological information from video frames.
[0062] (1) Image preprocessing: Denoising, contrast enhancement, and color correction are performed on the decoded video frames to improve the accuracy of subsequent recognition. For water conservancy scenarios, special handling of water surface reflection and interference caused by rain and snow may be required.
[0063] (2) Water body boundary identification:
[0064] 1) Traditional image segmentation methods: Edge detection algorithms (such as the Canny operator) or threshold-based segmentation methods (such as the Otsu algorithm) can be used to initially distinguish water and non-water regions. The gradient calculation of the Canny operator can be expressed as:
[0065] G = √(Gx² + Gy²)
[0066] Where Gx and Gy are approximate gradient values of the image in the x and y directions, respectively.
[0067] 2) Deep Learning Semantic Segmentation: Employing network structures such as U-Net and DeepLabv3+ for pixel-level classification, this method can more accurately identify water body boundaries in complex backgrounds (such as vegetation occlusion and building shadows). The model outputs a probability map of each pixel belonging to "water body," and a binarized mask is generated by setting a threshold to accurately extract the water body contour.
[0068] (3) Dynamic Change Analysis: For scenarios such as embankment collapse and rapid water level rise, it is necessary to analyze the changes in the boundary. Optical flow methods (such as the Lucas-Kanade algorithm or the Farnebäck algorithm) can be used to estimate the contrast of pixels in the image sequence, thereby calculating the flow velocity or identifying the deformation displacement of the embankment outline. The basic constraint equation of the optical flow method is based on the assumption of image grayscale invariance:
[0069] I(x,y,t) = I(x+dx, y+dy, t+dt)
[0070] The above formula means that when a pixel is located at position (x, y) at time t, its brightness is I(x, y, t). After time dt, that is, when the pixel moves to position (x + dx, y + dy) at time t+dt, its brightness remains unchanged.
[0071] By Taylor expansion and ignoring higher-order terms, we get:
[0072] I_x * u + I_y * v + I_t = 0
[0073] I_x represents the partial derivative of brightness with respect to x, representing the change in horizontal brightness;
[0074] I_y represents the partial derivative of brightness with respect to y, perpendicular to the change in brightness;
[0075] I_t represents the partial derivative of brightness with respect to time, which is the change in brightness over time;
[0076] Since the brightness remains constant, I_x dx + I_y dy + I_t dt = 0.
[0077] like Figure 2 The image shows the extraction results.
[0078] 3. Based on the extracted hydrological element boundary data, a custom shader is written using GLSL to achieve specific visualization effects.
[0079] Shaders are small programs that run on the GPU and are used to implement various complex visual effects. This invention uses GLSL to write custom shaders to simulate effects such as flooding and water impact.
[0080] (1) Shader types and division of labor:
[0081] 1) Vertex Shader: Primarily responsible for processing the geometric coordinates of boundary points of hydrological elements. It can receive boundary vertex data from the CPU and perform coordinate transformations (from world coordinates to clip coordinates).
[0082] 2) Fragment Shader: This is the core of effect generation. It operates on each pixel, dynamically calculating the final color and transparency of that pixel based on the input parameters (such as water depth, flow rate, and boundary information obtained from recognition results).
[0083] (2) Effect simulation principle:
[0084] 1) Data-driven: The hydrological element boundaries, deformation range, and calculated velocity field data extracted in the second stage are passed to the shader in the form of textures or Uniform variables. For example, a "depth map" can be generated and passed as a texture.
[0085] 2) Physics simulation: In the fragment shader, noise functions (such as Perlin Noise), wave equations, or particle system principles can be used to simulate the texture, waves, and flow of water. For example, by combining time and flow velocity parameters with a noise function, realistic water surface ripples can be dynamically generated.
[0086] / / Simplified GLSL fragment shader code example (simulating color gradients based on water depth)
[0087] uniform sampler2D u_depthMap; / / Water depth texture
[0088] uniform float u_time; / / Uniform time
[0089] varying vec2 v_texCoord; / / Texture coordinates
[0090] void main() {
[0091] float depth = texture2D(u_depthMap, v_texCoord).r; / / Get the current pixel water depth
[0092] vec3 waterColor = mix(vec3(0.1, 0.3, 0.8), vec3(0.0, 0.1, 0.5),depth); / / Mix color based on water depth (from light blue to dark blue)
[0093] / / You can add noise-based ripple simulation here
[0094] gl_FragColor = vec4(waterColor, 0.8); / / Output color, alpha value controls transparency
[0095] }
[0096] 4. In the rendering engine, GLSL shader effects are overlaid and blended with real-time video streams and hydrological element boundaries to dynamically simulate the evolution of water conservancy elements.
[0097] The rendering engine seamlessly blends realistic video footage with computer-generated simulation effects to achieve dynamic evolution simulation. Rendering engine and scene management: A high-performance engine such as WebGL frameworks (e.g., Three.js, Cesium) or Unity / Unreal Engine can be used as the foundation for visualization. The engine is responsible for creating 3D scenes, managing cameras, lighting, and the rendering process.
[0098] (1) Layer blending technology:
[0099] 1) Video background layer: The real-time decoded video frames are pasted as textures onto a quadrilateral aligned with the screen to serve as the rendering background.
[0100] 2) Simulation Effects Layer: This layer renders the evolution effects of hydrological elements processed by custom shaders (such as flood inundation range and water flow vectors) onto another layer. The rendering depth (Z-Buffer) of this layer needs to be finely set to ensure the correct spatial relationship with the video scene.
[0101] 3) Alpha blending: Utilizes the GPU's blending capabilities to overlay the effect layer onto the video background layer. The blending function is typically set as follows:
[0102] FinalColor = SrcColor * SrcAlpha + DestColor * (1 - SrcAlpha)
[0103] SrcColor is the color of the effect layer (foreground), DestColor is the color of the video background layer, and SrcAlpha is the transparency of the effect layer. By controlling the Alpha value, a semi-transparent blending of the effect can be achieved, making the simulated water flow appear to be "overlaid" on the real scene, rather than completely obscuring it.
[0104] (2) Coordinate system synchronization with time and space: This is the key to achieving accurate fusion. It is necessary to establish a mapping relationship between the 2D pixel coordinates of the video image and the 3D world coordinates through camera calibration technology. At the same time, it is essential to ensure that the frame rate of the video stream is synchronized with the frame rate of the shader rendering to avoid tearing or delay, thereby ensuring the real-time performance and realism of the evolution simulation.
[0105] Figure 3 This is the unblended effect image. Figure 4 This is a fusion effect diagram obtained by the method of the present invention.
[0106] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.
Claims
1. A method for fusing video streams and rendering simulations in scenarios involving the evolution of water conservancy elements, characterized in that, include: By being compatible with mainstream streaming media protocols, the system accesses and decodes water conservancy monitoring video streams to obtain continuous sequences of raw video frames. Using computer vision and deep learning models to perform semantic analysis on video frames, key hydrological elements and their dynamic change ranges are automatically identified and extracted to obtain hydrological element boundary data. A custom GLSL-based shader is used to dynamically generate simulation effects on the GPU through parallel computing of the extracted hydrological element boundary data, and the parameters are adjusted according to the degree of evolution of the elements. The rendering engine is used to overlay and fuse dynamic effects obtained by custom shaders based on GLSL with real-time video streams and boundary data of hydrological elements to dynamically simulate the evolution process of water conservancy elements.
2. The method for fusing video streams and rendering simulations in the context of water conservancy element evolution as described in claim 1, characterized in that, Mainstream streaming media protocols include RTSP, RTMP, and HLS.
3. The method for fusing video streams and rendering simulations in the context of water conservancy element evolution as described in claim 1, characterized in that, By being compatible with mainstream streaming media protocols, the system accesses and decodes water conservancy monitoring video streams to obtain continuous sequences of raw video frames. The specific method is as follows: Streaming media protocol processing: Integrates a high-performance streaming media server including ZLMediaKit to support direct access and conversion of mainstream streaming media protocols including RTSP, RTMP and HLS; Real-time decoding and frame extraction: The FFmpeg library is used to decode the video stream, that is: first, a network connection with the video stream is established, then the video format is parsed, and finally the compressed video data is decoded into a continuous sequence of raw video frames.
4. The method for fusing video streams and rendering simulations in the context of water conservancy element evolution as described in claim 1, characterized in that, Semantic analysis of video frames is performed using computer vision and deep learning models to automatically identify and extract key hydrological elements and their dynamic variation ranges, obtaining hydrological element boundary data. The specific method is as follows: Image preprocessing: Performing operations on the video frames in the decoded original video frame sequence, including noise reduction, contrast enhancement, and color correction; Water body boundary recognition: Image segmentation methods are used to initially distinguish water bodies from non-water bodies, and deep learning semantic segmentation methods are further used to accurately extract the water body contours; Dynamic change analysis: For scenarios including riverbank collapse and rapid water level rise, it is necessary to analyze the changes in the boundary, use optical flow to estimate the brightness and contrast of pixels in the image sequence, calculate the flow velocity or identify the deformation and displacement of the dam outline.
5. The method for fusing video streams and rendering simulations in the context of water conservancy element evolution as described in claim 4, characterized in that, In the image preprocessing process, for water conservancy scenarios, it is necessary to handle the interference caused by water surface reflection and rain and snow.
6. The method for fusing video streams and rendering simulations in the context of water conservancy element evolution as described in claim 4, characterized in that, Image segmentation methods employ edge detection algorithms or threshold-based segmentation to initially distinguish between water and non-water areas.
7. The method for fusing video streams and rendering simulations in the context of water conservancy element evolution as described in claim 4, characterized in that, Deep learning semantic segmentation methods employ network structures including U-Net and DeepLabv3+ for pixel-level classification, accurately identifying water body boundaries in complex backgrounds including vegetation occlusion and building shadows. The network structure outputs a probability map of each pixel belonging to the water body, and a binarized mask is generated by setting a threshold to accurately extract the water body contour.
8. The method for fusing video streams and rendering simulations in the context of water conservancy element evolution as described in claim 4, characterized in that, The fundamental constraint equations of the optical flow method are based on the assumption of image grayscale invariance: I(x,y,t) = I(x+dx, y+dy, t+dt) The above formula means that when a pixel is located at position (x, y) at time t, its brightness is I(x, y, t). After time dt, that is, when the pixel moves to position (x + dx, y + dy) at time t+dt, its brightness remains unchanged. By Taylor expansion and ignoring higher-order terms, we get: I_x * u + I_y * v + I_t = 0 I_x represents the partial derivative of brightness with respect to x, representing the change in horizontal brightness; I_y represents the partial derivative of brightness with respect to y, perpendicular to the change in brightness; I_t represents the partial derivative of brightness with respect to time, which is the change in brightness over time; Since the brightness remains constant, I_x dx + I_y dy + I_t dt = 0.
9. The method for fusing video streams and rendering simulations in the context of water conservancy element evolution as described in claim 1, characterized in that, A custom GLSL-based shader is used to dynamically generate simulation effects on the GPU through parallel computation of the extracted hydrological element boundary data, and the parameters are adjusted according to the degree of evolution of the elements. Specifically: Shader types and their functions: (1) Vertex shader: responsible for processing the geometric coordinates of the boundary points of hydrological elements; (2) Fragment shader: Executed on each pixel, dynamically calculating the final color and transparency of the corresponding pixel based on the incoming hydrological feature boundary data; Effect simulation principle: (1) Data-driven: The extracted hydrological element boundary data is passed into the shader in the form of texture or Uniform variable; (2) Physical effect simulation: In the fragment shader, noise function, wave equation or particle system principle are used to simulate the texture, waves and flow of water.
10. The method for fusing video streams and rendering simulations in the context of water conservancy element evolution as described in claim 1, characterized in that, The rendering engine utilizes a custom GLSL-based shader to overlay and fuse dynamic effects with real-time video streams and hydrological element boundary data, dynamically simulating the evolution of hydraulic elements. The specific method is as follows: Video background layer: Real-time decoded video frames are pasted as textures onto a quadrilateral aligned with the screen to serve as the rendered background; Simulation Effects Layer: This layer renders the evolution of hydrological elements processed by a custom GLSL shader to another layer. The rendering depth of this layer needs to be set to ensure the correct spatial relationship with the video background layer. Alpha blending: Uses the GPU's blending capabilities to overlay the simulation effect layer onto the video background layer; the blending function is set to: FinalColor = SrcColor * SrcAlpha + DestColor * (1 - SrcAlpha) Where SrcColor is the color of the simulation effect layer, DestColor is the color of the video background layer, and SrcAlpha is the transparency of the simulation effect layer; Coordinate system one is synchronized with time and space: Through camera calibration technology, a mapping relationship between the 2D pixel coordinates of video images and the 3D world coordinates is established; at the same time, the frame rate of the video stream is synchronized with the frame rate of the shader rendering to avoid tearing or delay, and to ensure the real-time performance and realism of the evolution simulation.