Method and system for video compression and transmission of public network monitoring

Abstract

The application discloses a public network monitoring video compression and transmission method and system, and belongs to the technical field of video processing, which comprises the following steps: identifying dynamic foreground targets in an original video stream through target detection and dividing the original video stream into continuous segments; dividing the segments into three categories according to whether the segments contain dynamic foreground targets and whether the segments contain disturbances that do not conform to natural light and shadow rules; encoding the first two categories of segments with different compression rates; only extracting the starting frame, the ending frame and the light and shadow parallax parameter sequence of the third category of segments for encapsulation and transmission; calling a building information model on the server side, performing physical light simulation based on the timestamp and the parameter sequence, and driving a pre-trained latent diffusion model based on the condition, so as to iteratively generate reconstructed intermediate frames that conform to the physical rules based on the starting frame and the ending frame, and to synthesize a complete video stream, wherein the application realizes the maximum compression of a static monitoring scene, saves bandwidth and storage costs, and guarantees the continuity of the reconstructed video.

Description

Technical Field

[0001] This invention relates to video acquisition and transmission. More specifically, this invention relates to a method and system for video compression and transmission in public network surveillance. Background Technology

[0002] Public network surveillance refers to remote monitoring systems that transmit video data via public networks (such as 4G / 5G, satellite links, etc.). It is widely used in scenarios such as urban road network traffic monitoring, safety supervision of large construction sites, and status inspection of field facilities (such as oil fields, power grids, and nature reserves). The core characteristics of this type of surveillance are that camera deployment locations often have limited network conditions, high bandwidth costs, and require centralized management and video playback across geographical areas.

[0003] In practical applications, many public network surveillance cameras employ highly singular and fixed shooting angles and scenes, such as consistently focusing on construction site entrances / exits, traffic intersections, fixed equipment, or specific area boundaries. In these scenarios, the surveillance footage remains static or quasi-static for the vast majority of the time, with no visible target activity such as vehicles or personnel. However, the changes in light and shadow caused by the movement of natural light sources continuously occur within the footage. These changes strictly adhere to the laws of physical optics and, from an information theory perspective, possess extremely high predictability and structure. Yet, in traditional video streams, they are still encoded and transmitted as random pixel fluctuations, continuously consuming network bandwidth.

[0004] Existing technologies primarily address this problem from two directions. One is to adopt more efficient video coding standards, improving compression ratios through enhanced prediction, transform, and entropy coding. The other is to employ intelligent analysis combined with frame skipping or bitrate adaptive strategies, such as reducing the frame rate or bitrate during static periods. However, both methods remain within the traditional paradigm of "transmitting pixel values," and their compression efficiency improvements are nearing their limits.

[0005] Therefore, it is necessary to propose a completely new video compression and transmission method. This method can intelligently identify video content and perform parametric modeling and description of changes caused by natural light sources that conform to physical laws. This allows for the transmission of only a very small number of physical parameters instead of massive amounts of pixel data. Finally, at the receiving end, based on prior knowledge and physical laws, a complete video sequence can be reconstructed with high quality. Summary of the Invention

[0006] One objective of this invention is to provide a video compression and transmission method and system for public network surveillance. By identifying video content, the method performs parametric modeling and description of video segments that change in accordance with physical laws due to natural light sources. When transmitting these video segments, the method extracts their start and end frames and physical parameters describing the laws of light and shadow changes for transmission. On the server side, the method uses Building Information Modeling (BIM) and generative artificial intelligence to reconstruct the video according to physical laws, thereby achieving a higher compression ratio while ensuring the physical authenticity and continuity of the video content.

[0007] To achieve these objectives and other advantages according to the present invention, according to one aspect of the present invention, a method for video compression and transmission for public network surveillance is provided, comprising the following steps:

[0008] S1. The camera captures the original video stream of the monitored scene and records the timestamp for each video frame. It calls the preset target detection model to identify the static background area and dynamic foreground target in the video frame of the original video stream and divides the original video stream into continuous video segments.

[0009] S2. The video segment containing the dynamic foreground target is classified into the first type of segment; for the video segment that does not contain the dynamic foreground target, inter-frame pixel analysis is performed. If it is determined that there is a dynamic disturbance that does not conform to the law of pure natural light and shadow change, it is classified into the second type of segment; if it is determined that there is no dynamic disturbance, it is classified into the third type of segment.

[0010] S3. For the first type of segment and the second type of segment, the first encoding rule and the second encoding rule with a higher compression ratio are used for compression respectively; for the third type of segment, the start frame and the end frame are extracted, and the natural light and shadow change pattern of the static background area is analyzed. The light and shadow difference parameter sequence that characterizes the natural light and shadow change pattern in the entire video segment is calculated, and the start frame, the end frame and the light and shadow difference parameter sequence are encapsulated together into a parameterized data packet.

[0011] S4. The compressed first type of segment, the second type of segment, and the parameterized data packet are transmitted to the server. The server calls the building information model and performs physical illumination simulation calculation based on the timestamp and photoluminescence difference parameter sequence. Using the pre-trained latent diffusion model, guided by the simulation calculation results and based on the start frame and end frame, the latent diffusion model iteratively generates a reconstructed third type of segment that conforms to physical laws. The first type of segment, the second type of segment, and the reconstructed third type of segment are synthesized into a reconstructed video stream according to the timeline.

[0012] Preferably, the inter-frame pixel analysis includes the following steps:

[0013] S21. For video segments that do not contain the dynamic foreground target, the dense optical flow algorithm is used to calculate the pixel motion vectors of consecutive adjacent video frames to obtain the dense optical flow field.

[0014] S22. Based on the dense optical flow field, calculate the average motion amplitude of all pixel motion vectors, and statistically analyze the proportion of pixel motion directions that are consistent with the global dominant direction.

[0015] S23. Compare the average motion amplitude with a preset amplitude threshold, and compare the directional consistency ratio with a preset consistency threshold.

[0016] If the average motion amplitude is lower than the amplitude threshold and the directional consistency ratio is higher than the consistency threshold, then the video segment is determined to have no dynamic disturbance and belongs to the third type of segment.

[0017] Otherwise, the video clip is determined to have dynamic disturbances and belongs to the second type of clip.

[0018] Preferably, the method for obtaining the photochromic aberration parameter sequence includes the following steps:

[0019] A1. Based on the registration relationship between the building information model and the video footage established through visual calibration, one or more anchoring areas corresponding to the surface of the building information model are determined in the third type of segment.

[0020] A2. For each anchored area, based on its known surface material properties and geometric normal, calculate the theoretical brightness value of the area under direct natural light at each time stamp. Compare the theoretical brightness value of the anchored area with the actual brightness value in the corresponding video frame, and calculate a time-varying attenuation coefficient function that characterizes the change of the overall intensity of the natural light source over time.

[0021] A3. Discretize the time-varying attenuation coefficient function according to the timestamp, and the resulting sequence is the optical aberration parameter sequence.

[0022] Preferably, the time-varying decay coefficient function is obtained by solving the following optimization problem:

[0023]

[0024] in, This is a sequence of theoretical brightness values. This is a sequence of actual brightness values. Let be the time-varying decay coefficient function to be solved. λ To smooth the constraint weights, This represents the gradient operator.

[0025] Preferably, the anchoring area is composed of fixed surfaces such as the building facade and the ground, and the total area of ​​the anchoring area is not less than 10% of the total area of ​​the video frame.

[0026] Preferably, the target detection model is a target detection neural network model based on the YOLO architecture and fine-tuned with monitoring scene data. The target detection model is configured to process the decoded video frames and output the category, location information and corresponding detection mask of the dynamic foreground target. The static background area is obtained by removing all areas covered by the detection mask from the video frames.

[0027] Preferably, the method for reconstructing the third type of fragment includes the following steps:

[0028] S41. The server calculates the incident direction vector of the natural light source based on the timestamp and the geographical coordinates of the monitored scene. Combining the surface material properties, geometric normals and photochromic aberration parameter sequence of the anchored area, the server calculates the reconstructed brightness change sequence of the anchored area. The server encodes the start frame, end frame, timestamp sequence and the reconstructed brightness change sequence into a unified spatiotemporal condition vector.

[0029] S42. The server uses a pre-trained latent diffusion model, takes the spatiotemporal condition vector as the global condition input, and encodes the start frame and end frame into the latent space through the forward diffusion process as the start and end constraints of the generation process. Through a guided iterative denoising process, a complete sequence of intermediate frame latent representations is generated.

[0030] S43. Decode the intermediate frame latent representation sequence into a video frame sequence in pixel space, and synthesize it with the start frame and end frame in chronological order to form the reconstructed third type of segment.

[0031] Preferably, the latent diffusion model includes a variational autoencoder, a noise prediction network based on a U-Net architecture, and a denoising sampling scheduler;

[0032] The training method for the latent diffusion model is as follows: a training dataset is constructed, in which each sample includes a real surveillance video clip, the corresponding timestamp and geographic coordinates, and a theoretical illumination distribution sequence rendered based on a building information model; the model is trained under supervision using the theoretical illumination distribution sequence and timestamp as conditions and the real video clip as the target, and the loss function used for training includes image reconstruction loss and physical consistency loss.

[0033] Preferably, a consistency function is introduced to dynamically guide each step of the iterative denoising process, including the following steps:

[0034] S421. During each denoising sampling step, the current intermediate latent variables are... z t Map back to pixel space to obtain a temporary reconstructed image, and extract temporary pixel brightness values ​​of the anchoring region registered with the building information model from the temporary reconstructed image;

[0035] S422. Calculate the difference between the temporary pixel brightness value and the theoretical brightness value at the corresponding time, and construct a physical consistency loss based on this difference:

[0036]

[0037] in, N This represents the total number of pixels within the anchored area. i For pixel index, For temporary pixel brightness values ​​in the anchored area, To reconstruct the brightness value sequence for the anchored region, This results in a loss of consistency.

[0038] S423, Apply the consistency loss to the current intermediate latent variables. z t Perform backpropagation to obtain the guiding gradient. By using guided gradient correction to adjust the denoising sampling step, the latent variables for the next time step are obtained:

[0039]

[0040] in, This is the denoising network for the latent diffusion model. C Let be the spatiotemporal condition vector. η For guiding strength coefficient, , , To predetermine noise scheduling parameters, It is random noise. This is a potential variable for the next step.

[0041] The present invention also provides a video compression and transmission system for public network monitoring, applied to the above method, including an edge terminal system deployed at the monitoring site and a server terminal system deployed in the data center;

[0042] The edge terminal system includes:

[0043] The camera is configured to capture raw video streams of the monitored scene;

[0044] An edge processing unit, communicatively connected to the camera, decodes the original video stream, calls a target detection model to identify dynamic foreground targets in the video frames and divides the video into segments; performs inter-frame pixel analysis to classify video segments that do not contain dynamic foreground targets; performs compression encoding on the first and second types of segments; extracts the start and end frames of the third type of segments and calculates the photodiode parameter sequence to encapsulate it into a parameterized data packet; and transmits the processed data to the server subsystem.

[0045] The server terminal system includes:

[0046] The database stores building information models and pre-trained generative diffusion models corresponding to the monitoring scenarios.

[0047] The server is communicatively connected to the edge terminal system. The reconstruction module receives the first type of segments, the second type of segments, and parameterized data packets; it calls the building information model and performs physical illumination simulation calculations based on timestamps and photoelectric difference parameter sequences; using the simulation results as conditions, it drives the generative diffusion model to iteratively generate the reconstructed third type of segments based on the start frame and the end frame; and it combines the received first type of segments, the second type of segments, and the generated reconstructed third type of segments into a complete reconstructed video stream according to the timeline.

[0048] The present invention has at least the following beneficial effects:

[0049] First, this invention targets static scene monitoring, which constitutes the majority of public network surveillance in areas such as construction sites, road networks, and the field. By identifying video segments containing only natural light and shadow variations as third-category segments and transmitting only their start and end frames and light and shadow difference parameter sequences, it achieves extreme compression of such segment data. This directly solves the core contradiction between limited bandwidth resources and high storage costs in public network surveillance, and can significantly reduce the operating costs of long-term monitoring.

[0050] Secondly, this invention performs a three-level classification of video streams, including foreground target identification and dynamic disturbance analysis, and applies standard encoding, high-compression encoding, and parameterized description to different categories of segments. This enables the monitoring system to adaptively allocate encoding and network resources. This processing mechanism optimizes the overall transmission and computational efficiency of the system while ensuring that critical dynamic information is not lost, enhancing the system's practicality and deployability in complex monitoring scenarios.

[0051] Third, in scenarios requiring event retrospective analysis and evidence collection from surveillance videos, such as construction site safety supervision, traditional high-compression encoding can disrupt the continuity of light and shadow changes. This invention, by reconstructing the third type of segment on the server side based on the building information model and transmitted light and shadow difference parameters in accordance with physical laws, ensures that the light and shadow evolution of the static background in the synthesized video has temporal continuity and physical authenticity, thus improving the credibility and usability of video evidence.

[0052] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0054] The present invention will now be described in further detail so that those skilled in the art can implement it based on the description.

[0055] It should be understood that terms such as “having,” “comprising,” and “including” as used herein do not exclude the presence or addition of one or more other elements or combinations thereof.

[0056] like Figure 1 As shown, this invention provides a method for video compression and transmission in public network surveillance, characterized by comprising the following steps:

[0057] S1. The camera captures the raw video stream of the monitored scene and records a timestamp for each video frame. A pre-set target detection model is invoked to identify static background areas and dynamic foreground targets within the video frames of the raw video stream. The raw video stream is then divided into continuous video segments. Specifically, industrial-grade network cameras supporting High Dynamic Range (HDR) are typically selected to ensure clear capture of the raw video stream in both strong sunlight and low-light conditions. The raw video stream can be encapsulated in H.264 or H.265 format, and each video frame is precisely timestamped by an edge processing unit such as a smart NVR or edge computing box. This timestamp information originates from the system clock synchronized with a Network Time Protocol (NTP) server, with millisecond-level accuracy, thus providing a foundation for subsequent time-series-based analysis and simulation.

[0058] Object detection models refer to neural network models that have been trained and optimized before system deployment and are stored in edge processing units. The core principle of object detection models lies in using deep convolutional neural networks to extract multi-level features from input images and, through a specific detection head structure, predicting the bounding box coordinates and the probability of the target object's category on these feature maps. Currently, directly usable model architectures are mainly divided into two categories: single-stage detectors and two-stage detectors. Single-stage detectors, such as the YOLO (You Only Look Once) series and SSD (Single Shot MultiBox Detector), have the advantage of extremely high speed. They divide the input image into a grid at once, and each grid cell directly predicts the bounding box and category of the target associated with that location, achieving end-to-end detection, suitable for edge computing scenarios with stringent real-time requirements. Two-stage detectors, such as the Faster R-CNN series, first generate a series of candidate regions where targets may exist, and then perform fine-grained classification and bounding box regression on the features of these candidate regions. Both single-stage and two-stage detectors follow a standardized training process: first, they are pre-trained on a large, general-purpose dataset to acquire general object representation capabilities; then, the model is fine-tuned using a smaller, labeled dataset collected from specific monitoring scenarios (such as traffic intersections and industrial parks) to optimize its detection performance for specific targets of interest, such as "vehicles," "pedestrians," and "safety helmets." Finally, the trained model can be converted into a format suitable for edge processor deployment using model compression techniques and then integrated into the software system of the edge processing unit.

[0059] In one technical solution, the target detection model is a target detection neural network model based on the YOLO architecture and fine-tuned with monitoring scene data. The target detection model is configured to process decoded video frames, outputting the category, location information, and corresponding detection mask of dynamic foreground targets. The static background region is obtained by removing all regions covered by the detection mask from the video frames. The YOLO architecture is suitable for real-time video analysis on edge processing units. Before actual deployment, a YOLO model pre-trained on a general dataset (e.g., YOLOv5 or YOLOv8) is fine-tuned using specific image data collected from the target monitoring scene (e.g., roads, construction sites). This process aims to improve the model's accuracy and robustness in recognizing key dynamic foreground targets (e.g., vehicles, pedestrians, construction machinery) in the scene and to filter out meaningless interfering targets. After processing each frame of image, the target detection model not only outputs the bounding box and category label of each detected target but also simultaneously generates a coarse detection mask corresponding to the bounding box. This mask can be obtained by running a lightweight semantic segmentation branch within the bounding box region or by employing a simple fill and feathering algorithm, and it defines the approximate area occupied by the target at the pixel level. Subsequently, the system overlays the masks of all detected targets onto the image to form a comprehensive foreground region mask. This method provides a clear and stable static background region input for subsequent inter-frame pixel analysis and lighting pattern calculation with low computational overhead.

[0060] Dividing the original video stream into consecutive video segments can be achieved using a hybrid triggering mechanism to balance the accuracy of the logic with the processing efficiency of the system.

[0061] First, the object detection model determines in real time whether a dynamic foreground object exists in each video frame. When a dynamic foreground object is detected to "appear" or "disappear," this event is immediately used as a mandatory segmentation boundary. This event-based segmentation ensures that each video segment containing an active object is completely and independently segmented in time, facilitating subsequent high-quality encoding.

[0062] Secondly, for video segments that do not contain any dynamic foreground targets, a fixed-duration segmentation method is more efficient and stable due to the lack of clear event boundaries. In one embodiment, the system divides such continuous static footage at fixed intervals of 10 to 30 seconds (e.g., 15 seconds). This means that a static surveillance video lasting up to 5 minutes without any targets will be regularly divided into approximately 20 consecutive second- or third-class segments.

[0063] The advantage of this hybrid strategy is that all dynamic events can be fully encapsulated and will not be accidentally cut off in the middle of the segment, providing rule-driven stability. For long static scenes, it avoids false triggering of segmentation due to minor noise or light flicker, while generating static segments of uniform length that are easy to batch process later. In addition, the fixed duration timer is a simple and reliable module, which, together with the complex event detection logic, constitutes a robust and practical video stream preprocessing front end.

[0064] S2. The video segments containing the dynamic foreground target are divided into first-class segments; for video segments that do not contain the dynamic foreground target, inter-frame pixel analysis is performed. If it is determined that there is dynamic disturbance that does not conform to the laws of pure natural light and shadow changes, it is divided into second-class segments; if it is determined that there is no dynamic disturbance, it is divided into third-class segments. Specifically, the system intelligently classifies the divided video segments. The core basis is whether there is motion within the segment and whether the motion strictly follows the physical laws of natural light and shadow changes. The division of the three types of video segments determines the completely different compression and processing strategies.

[0065] The first category of segments refers to video clips containing clearly defined dynamic foreground objects. For example, in road surveillance, this might show a car driving across the entire frame; or in construction site surveillance, it might show a worker walking and working within a designated area. These segments contain the dynamic events most relevant to the surveillance, and therefore require high-fidelity encoding to preserve them completely.

[0066] The second type of segment refers to segments that, while not containing the aforementioned identifiable targets, exhibit dynamic disturbances within their static background areas that do not conform to the purely natural laws of light and shadow variation. Examples include the continuous waving of a flag, the violent swaying of a tree in the wind, or birds occasionally flying past the camera. These movements are not the subject of the monitoring, and their patterns do not conform to the globally consistent light and shadow movement patterns dominated by the continuous changes in the azimuth of natural light sources (sun, moon). These segments still need to be transmitted, but can be compressed at a higher rate.

[0067] The third type of segment refers to video clips that contain no dynamic foreground objects and whose background area only exhibits pure natural light and shadow changes. A typical scene is: on an empty street in the afternoon, the shadows of buildings and the ground slowly and evenly lengthen; at this time, the motion vectors of all pixels in the picture show a high degree of consistency. This motion is driven entirely by the positional changes of natural light sources (sun, moon), and is information with strong determinism that can be accurately described by physical models.

[0068] The inter-frame pixel analysis includes the following steps:

[0069] S21. For video segments that do not contain the dynamic foreground target, the dense optical flow algorithm is used to calculate pixel motion vectors for consecutive adjacent video frames to obtain a dense optical flow field. Specifically, for video segments that have been determined not to contain dynamic foreground targets, the system needs to analyze the subtle motion of their background regions. The dense optical flow algorithm is a classic computer vision method for accomplishing this task. Mature algorithms that can be used include the Farneback algorithm, which approximates motion by fitting a polynomial expansion of the pixel neighborhood, thereby obtaining a dense and continuous motion vector field. The dense optical flow field provides panoramic motion information, laying the foundation for subsequent statistical feature analysis. The calculated dense optical flow field is a data structure of the same size as the video frame, where each position stores a two-dimensional motion vector.

[0070] Dense optical flow algorithm analysis is performed within the edge processing unit (EPU). The software modules within the EPU can be written in C++ or Python and rely on optimized visual computing libraries, such as the video analysis module in the open-source computer vision library OpenCV. In practice, the software module reads consecutive video frames from the memory buffer that have been decoded in step S1 and calls functions for core calculations. Taking the Farneback algorithm as an example, a series of key parameters need to be set during the call, which collectively determine the balance between computational accuracy and speed. These parameters include the "pyramid scaling factor" (usually set to 0.5), the "number of pyramid layers" (e.g., 3 layers), the "number of iterations per layer" (e.g., 3 times), the "neighborhood window size" (e.g., 15 pixels), and the "Gaussian standard deviation." These parameters need to be pre-tuned for the edge processor's computing power and the typical resolution of the surveillance video to ensure a sufficiently stable optical flow field output while meeting real-time requirements.

[0071] The calculated dense optical flow field is represented in memory as a two-channel floating-point matrix with the same width and height as the original image. The first channel stores the horizontal displacement component of each pixel, and the second channel stores the vertical displacement component. This two-channel floating-point matrix serves as the direct data basis for subsequent feature extraction.

[0072] S22. Based on the dense optical flow field, calculate the average motion amplitude of all pixel motion vectors and statistically analyze the proportion of pixel motion directions that are consistent with the global dominant direction. Specifically, the system extracts two core statistical features to quantify the nature of the motion. The first feature is the average motion amplitude, which is the average value of the magnitudes of all pixel motion vectors in the optical flow field. This value reflects the intensity of the overall motion of the image. The second feature is the proportion of direction consistency, which statistically analyzes the proportion of pixels whose motion direction has an angle less than a certain set angle with the dominant direction. This value represents the most prevalent motion direction in the image.

[0073] The calculation process for the average motion amplitude is as follows: traverse all pixels in the optical flow field, calculate the Euclidean magnitude of the motion vector of each pixel, then sum all the magnitude values ​​and divide by the total number of pixels to obtain the global average motion amplitude, which is expressed in pixels.

[0074] The calculation process for the orientation consistency ratio is as follows: First, the system determines the global dominant direction using a vector synthesis method. This involves summing the motion vectors of all pixels in the optical flow field to form a sum vector, the direction of which is the global dominant direction. Next, the system calculates the orientation angle of each pixel's own motion vector. Then, an angle tolerance threshold (e.g., 15 degrees) is set, and the angle difference between each pixel's orientation and the global dominant direction is compared. During the comparison, the system performs cyclic normalization of the angle difference to ensure accurate calculation. If the orientation difference of a pixel is less than or equal to the tolerance threshold, the pixel is determined to be consistent with the dominant direction. Finally, the system counts the number of pixels with consistent orientations, divides this number by the total number of pixels, and the resulting percentage is the orientation consistency ratio. The higher this ratio, the more coordinated and consistent the image motion, and the more likely it is caused by a single physical factor (such as the movement of the sun).

[0075] S23. The average motion amplitude is compared with a preset amplitude threshold, and the directional consistency ratio is compared with a preset consistency threshold. If the average motion amplitude is lower than the amplitude threshold and the directional consistency ratio is higher than the consistency threshold, the video segment is determined to have no dynamic disturbance and belongs to the third category of segments; otherwise, the video segment is determined to have dynamic disturbance and belongs to the second category of segments. Specifically, the amplitude threshold is usually set to a small value (e.g., 1 to 5 pixels) to filter out small, meaningless movements caused by image noise or compression distortion. The consistency threshold is set to a relatively high value (e.g., 85%). Only when the average motion amplitude is sufficiently low and the motion direction is highly consistent is the motion of the current segment determined to be global, smooth, and conforming to the laws of natural light and shadow changes, thus belonging to the third category of segments. Conversely, if the average amplitude is too high or the consistency is too low, dynamic disturbance is determined to exist, and the segment belongs to the second category of segments.

[0076] For example, a 5-minute video recorded the entire process of a cloud drifting across the sky, creating and eventually removing a shadow on the ground. This process can be divided into three stages, and the system's analysis and judgment are as follows:

[0077] The first phase lasts for half a minute: the cloud front begins to block sunlight, causing a significant and rapid decrease in brightness in some areas of the image, while the rest remains bright. During this phase, the boundary between light and shadow shifts significantly between adjacent video frames. The dense optical flow algorithm calculates a vector field with high motion amplitude because of the drastic changes in pixel brightness values; simultaneously, due to the irregular shadow edges and their intrusion from one side, the consistency of optical flow direction is low, causing motion vectors in different areas of the image to point in different directions. At this point, the average motion amplitude exceeds a preset low amplitude threshold, and the proportion of directional consistency is lower than a preset high consistency threshold. Therefore, the system classifies this 1-minute video segment as a second-type segment, indicating the presence of dynamic perturbation.

[0078] The second phase lasted four minutes: the cloud cover completely covered the monitored area, and the ground was generally in stable shadow. The overall brightness of the image was uniform. Although there were still slight shifts in shadow position due to the extremely slow apparent motion of the sun, the changes between adjacent frames were extremely weak. The optical flow field showed that the motion vector amplitude of all pixels was extremely low and the direction of movement was highly uniform, with a directional consistency ratio higher than the consistency threshold. Therefore, the system determined that this four-minute video segment was a third-type segment, meaning there was no dynamic disturbance and only pure natural light and shadow changes.

[0079] The third phase lasts for half a minute. The clouds move away, sunlight shines on the earth again, and the image brightens rapidly. This process is the opposite of the first phase but similar in nature: the boundary between light and shadow moves rapidly, causing the optical flow field to again exhibit a high average amplitude of motion and low directional consistency. The system classifies this final half minute as a second-type segment.

[0080] Before actual system deployment, the specific values ​​of the amplitude threshold and consistency threshold can be determined through an offline calibration process to adapt to the unique lighting environment, camera characteristics, and background features of the monitoring point. During the calibration process, engineers need to record two sample videos at the target point. The first video is a pure background sample, requiring no moving objects or dynamic disturbances in the scene, and the recording duration should cover different weather conditions and time periods to fully capture the range of pure natural light and shadow changes. The second video is a disturbance sample, which should intentionally include typical dynamic disturbances that may occur in the scene, such as leaves being blown by the wind, flags fluttering in the distance, or birds occasionally flying across the frame.

[0081] After collecting pure background samples and disturbed samples, the samples are processed and analyzed using the same algorithm as the online system. For pure background samples, the system calculates the average motion amplitude and directional consistency ratio of all video frame pairs and plots statistical distribution maps for each. Based on this distribution, initial threshold values ​​are set using statistical principles. For example, the 95th percentile of the average motion amplitude distribution is set as the initial value for the amplitude threshold, and the 5th percentile of the directional consistency ratio distribution is set as the initial value for the consistency threshold. The purpose of this method is to ensure that the thresholds can encompass most smooth and consistent natural lighting changes, classifying them as third-category segments.

[0082] The initial threshold value is applied to perturbed samples for classification testing. If a large number of perturbed segments are misclassified as Class III segments, it indicates that the current threshold is too lenient. The amplitude threshold should be appropriately increased or the consistency threshold decreased to improve system sensitivity. Conversely, if some pure background samples are misclassified as Class II segments, it indicates that the threshold is too strict and should be fine-tuned. Through several iterations, a set of thresholds that achieve high classification accuracy on both pure background and perturbed samples is finally determined as the final parameters for this location. These parameters will be permanently stored in the configuration file of the edge processing unit for online system access, thereby completing customized calibration for this specific scenario.

[0083] Offline calibration is an automated, low-intervention engineering deployment process. It utilizes the inherent recording and storage capabilities of the monitoring system to acquire sample data. All core computations are completed during idle periods of the backend server or edge processing unit, requiring no real-time manual analysis. The calibration software can be packaged as a standardized tool; operators only need to trigger execution and review the final generated threshold configuration file to optimize parameters for specific locations.

[0084] The amplitude and consistency thresholds used in inter-frame pixel analysis are not fixed empirical values, but rather parameters that can be flexibly configured according to the specific monitoring scenario's customer needs and environmental characteristics. In practical applications, users can set different threshold combinations through offline calibration processes or online adjustments to control the strictness of the third-category segment judgment. For example, in scenarios requiring a higher compression ratio, the consistency threshold can be appropriately relaxed or the amplitude threshold increased, allowing more static video segments to be classified into the third category; conversely, in scenarios sensitive to dynamic disturbances, the thresholds can be tightened to ensure that more segments containing uncertain motion are classified into the second category for regular compression. This adjustable threshold design allows the system to adaptively balance compression efficiency and information retention accuracy, enhancing the practicality and flexibility of the solution.

[0085] S3. For the first type of segment and the second type of segment, compress them using the first encoding rule and the second encoding rule with a higher compression ratio, respectively. For the third type of segment, extract its start frame and end frame, analyze the natural light and shadow change pattern of the static background area, calculate the light and shadow aberration parameter sequence that characterizes the natural light and shadow change pattern within the entire video segment, and encapsulate the start frame, end frame, and the light and shadow aberration parameter sequence together into a parameterized data packet.

[0086] For the first type of segment containing dynamic foreground targets, which carries the most valuable activity event information in surveillance and security, a coding rule prioritizing fidelity is required. This first coding rule can be implemented based on mature video coding standards (such as H.264 / AVC, H.265 / HEVC, or AV1). In a preferred embodiment, the rule is configured with a higher constant bitrate (CBR) or variable bitrate (VBR) mode, employing a shorter group of pictures (GOP) length and a strategy of periodically inserting keyframes (I-frames) to ensure clarity in fast-moving scenes and smoothness in random playback. The coding parameter settings aim to preserve the detailed features of the target and the continuity of its motion trajectory, providing a high-quality video source for post-event review and analysis.

[0087] For the second category of segments identified as having dynamic disturbances, although their content lacks critical targets, it still contains unpredictable local movements, such as sudden bird flight or falling objects, as well as sudden light disturbances. Therefore, higher compression efficiency needs to be pursued while maintaining acceptable visual quality. The second encoding rule is based on the first encoding rule, achieving a higher compression rate through a series of parameter adjustments. Typical measures include: significantly reducing the encoding bitrate (e.g., reducing it to 30%-50% of the bitrate of the first category of segments), moderately increasing the quantization parameter (QP) to increase the compression step size, extending the GOP structure to reduce the number of keyframes, or using perceptual encoding tools to actively smooth out high-frequency spatial details and redundant temporal jitter. These strategies, while sacrificing some non-critical image detail, can significantly reduce data volume and are suitable for video content with a transmission priority lower than the first category of segments.

[0088] The effective information contained in the third type of segment is the continuous and precisely mathematically describable physical laws driving the changes in light and shadow. Therefore, this invention abandons the traditional paradigm of transmitting all frames, preserves the static appearance of the scene by extracting the start and end frames, and compactly represents the continuous evolution of the light and shadow state over the entire time period by solving the sequence of light and shadow aberration parameters.

[0089] In one technical solution, the method for obtaining the photochromic aberration parameter sequence includes the following steps:

[0090] A1. Based on the registration relationship between the building information model and the video frame established through visual calibration, one or more anchoring areas corresponding to the surface of the building information model are determined in the third type of segment. The anchoring areas are composed of fixed surfaces such as building facades and ground, and the total area of ​​the anchoring areas is not less than 10% of the total area of ​​the video frame.

[0091] Specifically, in actual monitoring system deployments, such as when providing full coverage of an industrial park or large construction site, the number of monitoring points often reaches dozens or even hundreds. The building information model (BIM) required by this invention can be a unified digital base for the entire factory area or project, rather than creating a separate model for each camera. A complete or partial BIM model of the park can be shared and reused by all relevant monitoring cameras within the jurisdiction. In existing engineering construction processes, the corresponding BIM model already exists during the project planning and design phase, and data can be obtained from the relevant design units. Even for older parks, the results obtained through a one-time full-area real-scene 3D scanning model can continuously serve all points. Providing BIM support for individual cameras does not constitute additional repetitive work or significant cost. To ensure the reliability of the preliminary preparation work, after obtaining the BIM model, the consistency between the BIM model and reality can be verified. Deployment personnel can directly compare the rendered BIM model with high-definition photos taken on site to quickly check whether the outlines, dimensions, and positional relationships of major buildings match. This eliminates deviations caused by model version errors or on-site construction changes, establishing a reliable foundation for subsequent accurate calibration.

[0092] After obtaining the shared Building Information Model (BIM), a mapping relationship is established between the video feed of each specific camera and the BIM. Deployers utilize permanent features with known coordinates in the BIM (such as building corners and vent vertices) or temporarily place several high-contrast visual markers (such as AprilTag calibration boards) in the scene. Then, a clear image containing these points is captured using the camera. Next, by running a standard, open-source "perspective N-point" calculation program (such as the solvePnP function in OpenCV), the system automatically calculates the camera's precise spatial position, orientation, and lens parameters. The entire process can be completed within ten minutes and can be guided by calibration software on a tablet.

[0093] During camera calibration, after the system calculates the camera parameters using the perspective N-point calculation program, it automatically performs reprojection verification: using the newly calculated parameters, it reprojects the feature points with known 3D coordinates onto the 2D image plane to obtain calculated pixel coordinates. These coordinates are then compared with the actual pixel coordinates of these points in the original image, either manually labeled or automatically identified, and the average deviation of all feature points is calculated as the reprojection error. A qualified calibration result requires this average reprojection error to be less than 2 pixels. If the error exceeds this threshold, the system will determine that the calibration has failed and prompt the user to check the accuracy of the feature point selection or re-acquire the image. This built-in quality control mechanism ensures that each camera obtains a mapping relationship that meets the accuracy requirements of subsequent geometric analysis.

[0094] Once the mapping relationship between the camera and the building information model is established, the system automatically selects a suitable anchoring area in the video frame based on this connection. After the anchoring area is selected, its definition is saved as the camera's configuration file. From then on, no further calibration is required during the device's daily operation. This configuration file allows the system to know in real time the real-world surface position and geometric normal corresponding to any pixel in the image, laying an indispensable physical and geometric foundation for the next step of accurately analyzing light and shadow changes.

[0095] Anchor areas should be prioritized for large, fixed planes that are clearly defined in the Building Information Model (BIM), clearly visible in the actual video footage, and whose surface properties can be assigned. Examples include the main facade of a building, the surface of an open parking lot, road surfaces, or the surface of large, fixed equipment. These areas should be physically stable over the long term, not easily obstructed by moving objects, and visually uniform in texture and color to ensure stable and reliable brightness values ​​extracted from the video. The total area of ​​all selected anchor areas projected onto the video frame should be no less than 10% of the total frame area.

[0096] By distributing anchoring areas across multiple large, fixed surfaces with varying spatial locations and normal directions (such as east-facing walls and horizontal ground), and ensuring their cumulative coverage of a sufficient proportion of the screen, the system effectively performs spatially distributed sampling and averaging of the scene's lighting conditions. A 10% threshold, based on engineering experience, ensures that the light and shadow variations retrieved from the video represent global changes in the sun's position, rather than local noise, thus providing a stable and reliable data foundation for high-quality reconstruction on the server side.

[0097] A2. For each anchored area, based on its known surface material properties and geometric normal, calculate the theoretical brightness value of the area under direct natural light at each time stamp. Compare the theoretical brightness value of the anchored area with the actual brightness value in the corresponding video frame to calculate a time-varying attenuation coefficient function that characterizes the overall intensity of the natural light source as time changes.

[0098] To calculate the theoretical brightness value of the anchored area, the system needs to extract the geometric and physical attributes of the anchored area from the Building Information Model (BIM). The system first obtains the average surface normal vector from the BIM. and the specified material diffuse reflectance ρ The diffuse reflectance is a typical empirical value assigned based on the surface material type (such as concrete, asphalt, glass), usually between 0.1 and 0.7. For each moment in the timestamp sequence... t The system will call a high-precision natural light source position calculation library (such as Solar Position Algorithm, SPA) to calculate the solar altitude angle and azimuth angle based on the precise geographic coordinates (latitude and longitude) and time of the monitored scene, and finally convert them into unit direction vectors in the world coordinate system. Subsequently, the system uses the registered building information model to emit a ray from the location of the natural light source towards the center of the anchored area. It then detects whether this ray intersects with other parts of the model before reaching the target surface. If an intersection occurs, the area is determined to be in a non-direct illumination state. If no intersection occurs, it is determined to be in a direct illumination state.

[0099] In one technical solution, if the anchoring area is determined to be under direct natural light, its theoretical brightness value sequence is composed of the direct light component and the ambient sky light component, and the calculation formula can be adopted. ,in To calculate the attenuation of the incident angle of direct light according to Lambert's cosine law, Represents the relative intensity of direct sunlight under ideal sunny conditions. This represents the relative intensity of diffused light from the sky. If the anchoring area is not under direct illumination, the calculation formula can be simplified to... The diffuse reflectance of the material in the above formula. ρ The values ​​are directly taken from the building information model material library. and A typical initial value can be estimated using geographic location, date, and time.

[0100] Actual brightness value sequence The acquisition of RGB values ​​is accomplished through video analysis. In one technical solution, the system extracts the RGB values ​​of all pixels within a pre-stored anchor region pixel mask. Subsequently, the color information is converted into single-channel luminance values ​​using a luminance conversion formula (e.g., Y = 0.299*R + 0.587*G + 0.114*B). The arithmetic mean of the luminance values ​​of all pixels within the anchor region is calculated and used as the actual observed luminance of the anchor region at that moment. To improve data reliability, the system performs a simple preprocessing step during calculation: for the brightness value of each pixel, outliers that may be extremely high or low due to sensor thermal noise are excluded (e.g., by using the three-standard-deviation principle) before averaging. This process is repeated for all timestamps. t This yields a sequence of theoretical brightness values. A time-aligned sequence of actual observations Each anchored region generates an independent set of such data pairs, providing multiple sets of observational data for subsequent calculations.

[0101] The time-varying decay coefficient function is obtained by solving the following optimization problem:

[0102]

[0103] in, This is a sequence of theoretical brightness values. This is a sequence of actual brightness values. Let be the time-varying decay coefficient function to be solved. λ To smooth the constraint weights, This represents the gradient operator. In the solution formula... It is a data fidelity term that enables the calculated... It can scale theoretical values ​​to closely fit actual observations. This is a time-series smoothing constraint. Based on the prior knowledge that atmospheric transmittance and cloud thickness do not change abruptly, mathematically, it ensures function smoothness by penalizing drastic changes between adjacent time points. Smoothing constraint weights λ The trade-off between quantization smoothness and fitting accuracy.

[0104] In one technical solution, the third type of fragment contains N timestamps. t 1 ,t 2 ,……t N The time-varying decay coefficient function to be determined Discretize into vectors , and It is also discretized into vectors accordingly. sum vector gradient operator The original optimization problem can be approximated by a first-order difference matrix D, transforming it into a discrete linear least squares problem:

[0105]

[0106] in, λ represents the element-wise multiplication of vectors. W is an optional diagonal weight matrix that can be used to assign different confidence levels to different data points in robustness estimation. Calibration of λ is typically performed offline during system deployment, using historical pure light and shadow video clips, testing on multiple candidate values, and selecting the value that yields the best solution. A sufficiently smooth value.

[0107] This discrete optimization problem can be transformed into a standard linear least squares form through further algebraic transformations. Where matrix A and vector b are derived from matrix A. Matrix D , The system is constructed using efficient and stable numerical algorithms for solving the sparse equations, such as direct methods based on Cholesky decomposition of sparse matrices or iterative methods using preprocessed conjugate gradients (PCG). These algorithms have highly optimized implementations in scientific computing libraries (such as Eigen and SciPy). For a typical 15-second, 30fps segment (N=450), solving this sparse equation system can be completed within tens of milliseconds on modern edge computing chips (such as ARM A72 or equivalent computing cores). The computational load is far lower than that of motion estimation modules in video coding, ensuring the engineering feasibility of the solution.

[0108] A3. Discretize the time-varying attenuation coefficient function according to timestamps; the resulting sequence is the photoelectric aberration parameter sequence. After completing the optimization solution in step A2, the obtained time-varying attenuation coefficient function... Internally, the computer has represented it as the optimal numerical solution at a series of discrete time points. τ(t1),τ(t2),...,τ(tn) The system extracts and saves the function's values ​​at these predetermined timestamps in chronological order, naturally generating a sequence of light and shadow aberration parameters of length n, each corresponding to a timestamp. This sequence is a compact array of floating-point numbers that fully records the smooth trajectory of natural light intensity relative to an ideal clear-sky model over the duration of the video segment, thus providing the core driving data for subsequent physical simulation and video reconstruction on the server side.

[0109] After parameter calculation, the system enters the data encapsulation stage to generate the final parameterized data packet to be transmitted. Structurally, the parameterized data packet typically includes a fixed header, a payload data area, and an optional checksum. The header records key metadata, such as: the data packet version number, the unique identifier and time range of the corresponding original video segment, the image encoding format (e.g., JPEG) and size of the start and end frames, and the data type, length, and sampling interval of the photochromic aberration parameter sequence. The payload data area sequentially stores the binary image data of the start and end frames encoded with a high compression ratio (e.g., JPEG or WebP format), and the photochromic aberration parameter sequence stored directly as a compact floating-point array. To further reduce the number of bytes transmitted, this floating-point sequence can be processed using a common lossless compression algorithm (e.g., ZIP compression) before encapsulation.

[0110] S4. The compressed first type of segment, the second type of segment, and the parameterized data packet are transmitted to the server. The server calls the building information model and performs physical illumination simulation calculation based on the timestamp and light-diffraction difference parameter sequence. Using a pre-trained latent diffusion model, guided by the simulation calculation results and based on the start frame and end frame, the latent diffusion model iteratively generates a reconstructed third type of segment that conforms to physical laws. The first type of segment, the second type of segment, and the reconstructed third type of segment are synthesized into a reconstructed video stream according to the timeline.

[0111] After front-end classification, compression, and encapsulation are completed, the system enters the data transmission phase. The edge processing unit transmits the encoded first-type and second-type segments, along with the encapsulated parameterized data packets, to the remote central server via a network interface, through the public internet or a dedicated data link. This transmission process is typically based on a reliable transport protocol (such as TCP) to ensure data integrity and order. All data is associated with a unique timestamp and classification tag, enabling the server to clearly distinguish the data type upon receipt and place it in the corresponding processing queue, preparing it for subsequent decoding, parsing, and video reconstruction processes.

[0112] The method for reconstructing the third type of fragment on the server includes the following steps:

[0113] S41. The server calculates the incident direction vector of the natural light source based on the timestamp and the geographical coordinates of the monitored scene. Combining the surface material properties, geometric normals, and photochromatic aberration parameter sequence of the anchored area, the server calculates the reconstructed brightness change sequence of the anchored area. The server encodes the start frame, the end frame, the timestamp sequence, and the reconstructed brightness change sequence into a unified spatiotemporal condition vector.

[0114] First, the server loads pre-stored data uniquely corresponding to the surveillance camera from the storage system, including: precise geographic coordinates, a complete building information model, and a list of anchoring areas and their geometric attributes determined after calibration with the video image on the building information model.

[0115] Subsequently, the server proceeds to calculate the physical state reconstruction layer. Its input is the transmitted timestamp sequence {t1, t2, ..., t...}. N The calculation process for the light and shadow difference parameter sequence undergoes several optimizations to ensure efficiency: the server first pre-calculates and caches the continuous change function of the natural light source's orientation within the day based on the geographic coordinates and date, and obtains the unit incident direction vector of the natural light source at each moment through fast table lookup and interpolation during actual calculation.

[0116] Next, the server processes each anchored area in parallel, using the building information model to perform rapid ray casting to determine the direct sunlight status at each moment, and calculates its theoretical brightness value sequence. A model that closely approximates real physics can be used here: , Represents the relative intensity of direct sunlight under ideal sunny conditions. Represents the relative intensity of diffuse light from the sky, and is pre-calibrated as a scene-dependent constant.

[0117] Finally, the optical aberration parameter sequence is applied to the theoretical brightness value sequence, and the reconstructed brightness change sequence is obtained by element-wise multiplication.

[0118] Next, data from different sources and of different types are encoded and fused into a unified, machine-understandable spatiotemporal condition vector to ensure that each type of information is fully extracted and effectively expressed.

[0119] First, visual feature encoding is performed. The start and end frames are the fixed visual boundaries of the reconstructed video, and the system needs to deeply understand them. To this end, the system uses a convolutional neural network with shared weights to process these two frames separately. This convolutional neural network does not need to be trained from scratch; it can be built based on a mature backbone network pre-trained on a large-scale general-purpose image dataset, such as ResNet-34 or EfficientNet. In practice, the fully connected classification layer at the end of the original network is removed, retaining its powerful feature extraction capabilities. After an RGB image is input into the network, a high-dimensional feature map is output from the last convolutional layer. This map is then represented as a fixed-length visual feature vector (e.g., a 512-dimensional or 1024-dimensional vector) through global pooling. This visual feature vector densely encodes the global semantic information and local texture details of the image, such as the outlines of buildings, the texture of roads, and the proportion of sky, providing stable visual context anchors for the generation process.

[0120] Secondly, in the temporal feature encoding stage, the system needs to transform the discrete timestamp sequence into a time position encoding vector representation with order awareness. The standard sine-cosine position encoding method from the Transformer model can be used to generate a unique vector for each time position, with different dimensions assigned sine and cosine function values ​​of different frequencies. The core characteristic of this design is that it enables the model to explicitly perceive the linear order and intervals of time; the relative relationship between any two time points can be measured through the geometric relationship of their corresponding encoding vectors, thus providing accurate time coordinates for the dynamic generation process.

[0121] To reconstruct a brightness change sequence, this scalar time-series signal can be elevated to a high-dimensional feature containing physical dynamics. This can be achieved by designing a lightweight multilayer perceptron. The input to this network is not a single brightness value, but a concatenation of the brightness scalar at each time point and its corresponding time position encoding vector. Through multilayer nonlinear transformations, the network learns to extract and summarize the continuous dynamic patterns and state evolution laws in natural light and shadow changes from the "time-brightness" data pairs, and outputs a physical feature vector that characterizes the essence of the physical process.

[0122] Finally, the visual feature vector, temporal location encoding vector, and physical feature vector generated above are fused into a longer joint vector through vector concatenation. This joint vector can then be input into an additional, small fusion network for feature interaction and dimensionality reduction, ultimately outputting a fixed-dimensional, information-dense spatiotemporal conditional vector. The convolutional networks, location encoding, multilayer perceptrons, and feature fusion techniques relied upon throughout the encoding process are all existing mature components in the field of deep learning. Their combination ensures that multi-source information is effectively structured into conditional signals that can be directly utilized by the generative model.

[0123] Although this technical solution involves multi-stage processing, its overall computation is within a controllable and efficient range.

[0124] In one specific implementation, calculating the orientation of natural light sources is a typical astromechanical problem. The model is fixed, and the calculation result depends only on latitude, longitude, date, and time. Therefore, the system can pre-calculate and store high-precision solar vectors for each monitoring point at any time of the year during the deployment phase or using offline tasks, transforming the complex trigonometric function solutions at runtime into millisecond-level memory lookups and linear interpolation. Similarly, based on the complex 3D occlusion judgment using the Building Information Model, the "shadow mask" for each anchored area at all times of the day can be pre-calculated for a fixed camera viewpoint and the annual solar trajectory, and the state can be directly read at runtime. This transforms the core of the physical reconstruction layer from "real-time simulation" to "real-time query."

[0125] Convolutional neural networks for visual feature encoding can utilize mature industry models such as ResNet-34, whose forward inference process on GPUs takes only milliseconds and is supported by numerous open-source inference frameworks like ONNX Runtime for optimization. Temporal feature encoding uses a sine-cosine function, a parameter-free deterministic computation with extremely low overhead. Lightweight multilayer perceptrons used to encode physical signals for reconstructed brightness variation sequences have negligible computational cost due to their fixed input dimensions and shallow network structure. The same applies to small fusion networks. The total time consumption of these components is far lower than the computational cost of a single iteration of the subsequent generative model.

[0126] For example, a surveillance camera at a city intersection captures footage of an empty road in the afternoon. After intelligent analysis, the system identifies it as a 30-second segment of a third type, without any vehicles or pedestrians. The original video frame rate is 30 frames per second. The edge processing unit encapsulates the start frame image, key frame images, compressed light and shadow difference parameter sequence, and precise timestamp metadata into a parameterized data packet, which is then sent to a cloud server via the public network. Upon receiving the data packet, the server utilizes the pre-stored building information model and pre-calculated solar radiation database of the intersection to rapidly synthesize the reconstructed brightness sequence of the anchored area (such as a specific area of ​​the road surface) within tens of milliseconds. Pre-trained models are used to quickly extract visual features: the start and end frames are extracted into 1024-dimensional visual feature vectors using a standard, pre-trained ResNet-34 model (with the classification head removed) within the PyTorch framework; the timestamp sequence is encoded into a time-position vector using sine-cosine position encoding; and the reconstructed brightness change sequence is encoded into a physical feature vector using a lightweight multilayer perceptron. All feature vectors are fused into a spatiotemporal conditional vector. The server invokes a custom-trained video interpolation model based on a publicly available diffusion model architecture (such as the latent diffusion model of Stable Diffusion). This model uses spatiotemporal conditional vectors as global guidance and the latent representations of the start and end frames as fixed boundaries. Iteratively generates coherent intermediate frames through approximately 50 steps of DDIM sampling. This process takes about 2 seconds on a single NVIDIA V100 GPU.

[0127] S42. The server uses a pre-trained latent diffusion model, takes the spatiotemporal condition vector as the global condition input, and encodes the start frame and end frame into the latent space through the forward diffusion process as the start and end constraints of the generation process. Through a guided iterative denoising process, a complete sequence of intermediate frame latent representations is generated.

[0128] In this technical solution, the core architecture and training paradigm of the latent diffusion model belong to a mature technical system in the current field of generative artificial intelligence. The latent diffusion model reduces the computational burden of high-resolution image generation by performing denoising diffusion in a low-dimensional, dense latent space, and has become one of the mainstream frameworks for image and video synthesis. Its specific implementation is highly modular and selective.

[0129] The latent diffusion model adopts a mainstream modular architecture, with its core components including a variational autoencoder (VAE), a noise prediction network, and a denoising sampling scheduler. These can be directly adopted or adapted from existing open-source implementations. The variational autoencoder is used for efficient and high-fidelity conversion between high-dimensional pixel space and low-dimensional latent space. The system can directly integrate publicly available pre-trained models such as the KL-F8 VAE, which have been trained on massive image data and can stably encode the start and end frames into latent representations as anchor points for the generation process, ultimately decoding the denoised latent sequence into a high-resolution image. The noise prediction network typically uses a U-Net structure with skip connections, its core function being to predict noise at each diffusion step. To incorporate control information, this network receives and fuses spatiotemporal conditional vectors through a standard cross-attention mechanism. Such U-Net architectures and their conditional injection methods have complete and efficient implementations in well-known open-source projects such as Stable Diffusion, eliminating the need for development from scratch. The denoising sampling scheduler determines the generation path from noise to data. To balance quality and speed, the system prioritizes deterministic samplers such as DDIM, which allows traditional thousand-step sampling to be compressed to 20-50 steps, increasing inference speed by one to two orders of magnitude with almost no loss of visual fidelity. These components together form a rapidly deployable generation pipeline based on mature components.

[0130] The model's training data can be generated efficiently programmatically, greatly reducing reliance on massive amounts of real surveillance footage that are difficult to obtain. Specifically, using a registered building information model, the system simulates the sun's trajectory at different seasons, dates, and times based on the scene's geographic coordinates within 3D rendering engines such as Blender and Unreal Engine. It also randomly generates a sequence of coefficients to simulate cloud attenuation, and then batch-renders static scene video clips containing only natural light and shadow variations as "realistic" training targets. Simultaneously, the rendering engine can output the theoretical illumination distribution map for each frame. This method can automatically generate training sample pairs of arbitrary size and perfect annotation, solving the problem of data scarcity for specific scenes.

[0131] The real video clips required for training, along with their corresponding timestamps, geographic coordinates, and building information models, can be directly acquired from historical surveillance footage of the target application scenario (such as a construction site or a road), covering all-weather changes in different seasons and weather conditions. For a completely new scenario, surveillance video of a specific period can also be collected first as the basic training set.

[0132] The model's training strategy adheres to practical principles, employing a two-stage approach to improve efficiency and effectiveness. The first stage is general pre-training: using large-scale, diverse data generated by the aforementioned rendering methods, or combining this with publicly available video datasets, a basic conditional video interpolation model is trained, enabling it to generate intermediate transition frames under given first and last frames and simple temporal conditions. The second stage is targeted fine-tuning: using a small amount of data actually collected or rendered in high-fidelity scenarios within the target surveillance environment, the pre-trained model is fine-tuned. The key to this stage is the introduction of a physical consistency loss function. This involves adding a constraint term to the conventional noise prediction mean square error loss, forcing the model to generate brightness changes in the anchored region that strictly adhere to theoretical calculations. Through this targeted guidance, the model effectively constrains its general generation capabilities to conform to the physical laws of the specific scene.

[0133] In the final deployment and inference phases, computational efficiency is ensured through multiple means. First, the finely tuned model itself possesses a high degree of target specificity, thus achieving satisfactory generation quality with fewer sampling steps (e.g., 30 steps). Second, the entire denoising and generation process can be executed in high parallelism on modern GPUs, with the system processing all latent variable predictions at once, rather than generating frame by frame. Furthermore, the encoding and decoding operations of the variational autoencoder, as well as the attention computation in U-Net, can be accelerated using optimizations and mixed-precision techniques within deep learning frameworks. After comprehensive optimization, the total time to reconstruct a 30-second video clip on a single commercial GPU can be controlled within a few seconds. This computational cost is negligible compared to the savings in long-term bandwidth leasing costs and is entirely within the affordability range of data centers.

[0134] Each step of the iterative denoising process introduces a consistency function for dynamic guidance. The physical guidance mechanism introduced in the iterative denoising process is a method that injects external physical laws as hard constraints into the AI ​​generation loop in real time. Its purpose is to ensure that the direction of each generation converges towards a result that conforms to physical facts. This includes the following steps:

[0135] S421. During each denoising sampling step, the current intermediate latent variables are... z t Mapping back to pixel space yields a temporary reconstructed image. Temporary pixel brightness values ​​of the anchoring region registered with the building information model are extracted from this temporary reconstructed image. During each denoising sampling step, the system incorporates the current noisy intermediate latent variables. z tThe decoder portion of the variational autoencoder built into the latent diffusion model temporarily maps back to the pixel space to obtain a temporary reconstructed image. Subsequently, based on a pre-established registration relationship between the building information model and the temporary reconstructed image, the system locates the pixel positions corresponding to the anchored regions in the temporary image and quickly calculates the brightness values ​​of all pixels within that region, using these as the temporary pixel brightness values ​​for the current generation state. .

[0136] S422. Calculate the difference between the temporary pixel brightness value and the theoretical brightness value at the corresponding time, and construct a physical consistency loss based on this difference:

[0137]

[0138] in, N This represents the total number of pixels within the anchored area. i For pixel index, For temporary pixel brightness values ​​in the anchored area, For the reconstructed brightness value sequence of the anchored region, For consistency loss, The deviation of the currently generated image from the target in a key physical dimension (brightness) is measured numerically.

[0139] The aforementioned physical consistency loss employs a pixel-by-pixel comparison method, which theoretically yields the finest guiding gradient. However, its computational complexity is directly proportional to the number of pixels in the anchored region. In practical engineering deployments, to balance guiding accuracy and computational efficiency, a comparison of the average brightness of the anchored region can be used as an alternative, i.e.:

[0140]

[0141] in This is the average temporary brightness value of all pixels within the anchored area. The average reconstructed brightness value of the anchored area is used. The simplified method significantly improves the generation efficiency while ensuring the correctness of the physical guidance direction. It is especially suitable for typical monitoring scenarios where the anchored area has uniform texture and consistent brightness variation.

[0142] S423, Apply the consistency loss to the current intermediate latent variables. z t Perform backpropagation to obtain the guiding gradient. Correct the standard denoising sampling step:

[0143]

[0144] in, This is the denoising network for the latent diffusion model. C Let be the spatiotemporal condition vector.η For guiding strength coefficient, , , To predetermine noise scheduling parameters, It is random noise. This serves as a latent variable for the next step. The system utilizes the automatic differentiation function of the deep learning framework to calculate the physical consistency loss. For the current latent variables z t gradient This gradient indicates how to fine-tune. z t This is necessary to reduce physical errors. Subsequently, this gradient is used to correct the noise direction predicted by the diffusion model itself. Specifically, it is derived from the noise network predicted by the diffusion model. In the middle, subtract a factor derived from the gradient and the guiding strength coefficient. η The determined quantity forms the corrected noise estimate. This operation is equivalent to applying a pulling force in the direction the model is creating, pointing it towards a region that is "more in line with the laws of physics." Finally, the corrected... Substituting the samples into the standard diffusion model sampling formula (such as DDIM), the next step is to calculate the cleaner latent variables. This process is performed in each iteration of the sampling loop to ensure that the final output is strictly constrained to the trajectory of physical laws.

[0145] S43. The intermediate frame latent representation sequence is decoded into a video frame sequence in pixel space, and synthesized with the start frame and end frame in chronological order to form the reconstructed third-type segment. After the guided iterative denoising process is completed, the system obtains a final latent representation sequence that smoothly transitions from the latent representation of the start frame to the latent representation of the end frame. This sequence is completely mapped back to the high-resolution pixel space by the decoder of the variational autoencoder, generating all intermediate video frames. These frames are arranged in chronological order and seamlessly connected with the start frame and end frame, which serve as fixed boundaries, to synthesize a complete, high-fidelity reconstructed third-type segment.

[0146] After processing all segments, the system performs final video stream synthesis. This step, based on the precise timestamp metadata carried by each video segment, rigorously and seamlessly splices and assembles the compressed and encoded first and second-type segments with the newly generated reconstructed third-type segments on the server side along the timeline. To ensure playback continuity, the system performs necessary decoding and re-encoding at segment junctions to unify the encoding format, frame rate, and resolution of all segments, ultimately outputting a single, continuous reconstructed video stream file. This file completely reproduces all visual information from the original monitoring period, where the light and shadow changes in static background areas have physical continuity, and dynamic foreground targets and disturbances are also preserved, thus achieving high-fidelity video reconstruction at an extreme compression ratio.

[0147] The primary application scenario of this technical solution is video storage and post-event playback. In this scenario, the server-side video reconstruction process is triggered on demand: when a user needs to access a complete surveillance video containing a specific time range, the system identifies the third type of segments contained in that video segment and only uses the Building Information Model (BIM) and diffusion model to reconstruct these segments. The reconstructed third type of segments are then combined with the first and second type segments stored concurrently to form a complete video stream for the user to view. A large number of unaccessed third type segments are stored in the database as parameterized data packets until they expire and are deleted. This mechanism of "extreme compression during storage and on-demand reconstruction of the compressed portions during retrieval" fundamentally alleviates the computational pressure that generative models may face during large-scale concurrent reconstruction at the data center level. For a single post-event investigation, a reconstruction delay of several seconds to tens of seconds is entirely within acceptable limits; therefore, this solution has high engineering feasibility and commercial value.

[0148] A video compression and transmission system for public network monitoring, applied to the above method, includes an edge terminal system deployed at the monitoring site and a server terminal system deployed in a data center; the two subsystems communicate and exchange data via the public Internet or a dedicated data link.

[0149] The edge terminal system is an intelligent front-end that performs real-time sensing, analysis, and primary compression. Its hardware core is an edge processing unit with integrated computing power. This unit communicates directly with one or more industrial-grade cameras. The cameras are configured to acquire raw video streams of the monitored scene and typically have high dynamic range characteristics to adapt to changes in outdoor lighting. The edge processing unit has built-in dedicated video decoding hardware and AI acceleration modules. Its software stack is configured to sequentially execute the following tasks: First, decode the raw video stream; second, call a pre-set target detection model to perform real-time analysis on each frame, identify dynamic foreground targets, and segment the video accordingly; next, perform inter-frame pixel analysis on segments that do not contain dynamic targets to complete the fine classification of second and third category segments; then, call the corresponding encoders to compress the first and second category segments respectively; finally, for the third category segments, extract their start and end frames, calculate the photoelectric aberration parameter sequence in parallel, and encapsulate these data into compact parameterized data packets. All processed data (compressed video segments and parameter packets) are assigned time stamps and transmitted to the server terminal system via a network module.

[0150] The server subsystem is the central hub for deep computing and video reconstruction. Its core consists of one or more high-performance video reconstruction servers connected to a centralized database. This database stores building information model (BIM) files corresponding to each monitored scene, as well as pre-trained generative diffusion model (GDM) parameter files for each scene. The reconstruction service program on the server is configured to: receive and parse data packets from the edge; retrieve the corresponding BIM model from the database based on the timestamp and scene ID in the parameter packets; perform high-precision physical lighting simulation calculations based on the timestamp and shadow difference parameter sequence to reproduce the physical conditions driving the changes in light and shadow; use these physical conditions as strong guidance to drive the GDM model, and generate physically consistent reconstructed third-type segments through an iterative denoising process, using the start and end frames as immutable visual references; finally, decode, synchronize, and seamlessly stitch all received segments and generated reconstructed segments according to their global timestamps to synthesize a complete and coherent reconstructed video stream for storage, viewing, or further analysis. This system combines intelligent preprocessing at the edge with generative reconstruction based on physical models in the cloud, enabling the transformation of surveillance video streams from "end-to-end transmission" to "on-demand synthesis".

[0151] The number of devices and processing scale described herein are for the purpose of simplifying the description of the invention. Applications, modifications, and variations of the invention will be readily apparent to those skilled in the art.

[0152] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details.

Claims

1. A method for video compression and transmission in public network surveillance, characterized in that, Includes the following steps: S1. The camera captures the original video stream of the monitored scene and records the timestamp for each video frame. It calls the preset target detection model to identify the static background area and dynamic foreground target in the video frame of the original video stream and divides the original video stream into continuous video segments. S2. The video segment containing the dynamic foreground target is classified into the first type of segment; for the video segment that does not contain the dynamic foreground target, inter-frame pixel analysis is performed. If it is determined that there is a dynamic disturbance that does not conform to the law of pure natural light and shadow change, it is classified into the second type of segment; if it is determined that there is no dynamic disturbance, it is classified into the third type of segment. S3. For the first type of segment and the second type of segment, the first encoding rule and the second encoding rule with a higher compression ratio are used for compression respectively; for the third type of segment, the start frame and the end frame are extracted, and the natural light and shadow change pattern of the static background area is analyzed. The light and shadow difference parameter sequence that characterizes the natural light and shadow change pattern in the entire video segment is calculated, and the start frame, the end frame and the light and shadow difference parameter sequence are encapsulated together into a parameterized data packet. The method for obtaining the optical-visual aberration parameter sequence includes the following steps: A1. Based on the registration relationship between the building information model and the video footage established through visual calibration, one or more anchoring areas corresponding to the surface of the building information model are determined in the third type of segment. A2. For each anchored area, based on its known surface material properties and geometric normal, calculate the theoretical brightness value of the area under direct natural light at each time stamp. Compare the theoretical brightness value of the anchored area with the actual brightness value in the corresponding video frame, and calculate a time-varying attenuation coefficient function that characterizes the change of the overall intensity of the natural light source over time. A3. Discretize the time-varying attenuation coefficient function according to the timestamp, and the resulting sequence is the photoelectric aberration parameter sequence; S4. The compressed first type of segment, the second type of segment, and the parameterized data packet are transmitted to the server. The server calls the building information model and performs physical illumination simulation calculation based on the timestamp and photoluminescence difference parameter sequence. Using the pre-trained latent diffusion model, guided by the simulation calculation results and based on the start frame and end frame, the latent diffusion model iteratively generates a reconstructed third type of segment that conforms to physical laws. The first type of segment, the second type of segment, and the reconstructed third type of segment are synthesized into a reconstructed video stream according to the timeline.

2. The video compression and transmission method for public network surveillance as described in claim 1, characterized in that, The inter-frame pixel analysis includes the following steps: S21. For video segments that do not contain the dynamic foreground target, the dense optical flow algorithm is used to calculate the pixel motion vectors of consecutive adjacent video frames to obtain the dense optical flow field. S22. Based on the dense optical flow field, calculate the average motion amplitude of all pixel motion vectors, and statistically analyze the proportion of pixel motion directions that are consistent with the global dominant direction. S23. Compare the average motion amplitude with a preset amplitude threshold, and compare the directional consistency ratio with a preset consistency threshold. If the average motion amplitude is lower than the amplitude threshold and the directional consistency ratio is higher than the consistency threshold, then the video segment is determined to have no dynamic disturbance and belongs to the third type of segment. Otherwise, the video clip is determined to have dynamic disturbances and belongs to the second type of clip.

3. The video compression and transmission method for public network surveillance as described in claim 1, characterized in that, The time-varying decay coefficient function is obtained by solving the following optimization problem: in, This is a sequence of theoretical brightness values. This is a sequence of actual brightness values. Let be the time-varying decay coefficient function to be solved. λ To smooth the constraint weights, This represents the gradient operator.

4. The video compression and transmission method for public network surveillance as described in claim 1, characterized in that, The anchoring area consists of fixed surfaces such as the building facade and the ground, and the total area of ​​the anchoring area is not less than 10% of the total area of ​​the video frame.

5. The video compression and transmission method for public network surveillance as described in claim 1, characterized in that, The target detection model is a target detection neural network model based on the YOLO architecture and fine-tuned with monitoring scene data. The target detection model is configured to process the decoded video frames and output the category, location information and corresponding detection mask of the dynamic foreground target. The static background area is obtained by removing all areas covered by the detection mask from the video frame.

6. The video compression and transmission method for public network surveillance as described in claim 1, characterized in that, The method for reconstructing the third type of fragment includes the following steps: S41. The server calculates the incident direction vector of the natural light source based on the timestamp and the geographical coordinates of the monitored scene. Combining the surface material properties, geometric normals and photochromic aberration parameter sequence of the anchored area, the server calculates the reconstructed brightness change sequence of the anchored area. The server encodes the start frame, end frame, timestamp sequence and the reconstructed brightness change sequence into a unified spatiotemporal condition vector. S42. The server uses a pre-trained latent diffusion model, takes the spatiotemporal condition vector as the global condition input, and encodes the start frame and end frame into the latent space through the forward diffusion process as the start and end constraints of the generation process. Through a guided iterative denoising process, a complete sequence of intermediate frame latent representations is generated. S43. Decode the intermediate frame latent representation sequence into a video frame sequence in pixel space, and synthesize it with the start frame and end frame in chronological order to form the reconstructed third type of segment.

7. The video compression and transmission method for public network surveillance as described in claim 6, characterized in that, The latent diffusion model includes a variational autoencoder, a noise prediction network based on U-Net architecture, and a denoising sampling scheduler. The training method for the latent diffusion model is as follows: a training dataset is constructed, in which each sample includes a real surveillance video clip, the corresponding timestamp and geographic coordinates, and a theoretical illumination distribution sequence rendered based on a building information model; the model is trained under supervision using the theoretical illumination distribution sequence and timestamp as conditions and the real video clip as the target, and the loss function used for training includes image reconstruction loss and physical consistency loss.

8. The video compression and transmission method for public network surveillance as described in claim 6, characterized in that, Each step in the iterative denoising process introduces a consistency function for dynamic guidance, including the following steps: S421. During each denoising sampling step, the current intermediate latent variables are... z t Map back to pixel space to obtain a temporary reconstructed image, and extract temporary pixel brightness values ​​of the anchoring region registered with the building information model from the temporary reconstructed image; S422. Calculate the difference between the temporary pixel brightness value and the theoretical brightness value at the corresponding time, and construct a physical consistency loss based on this difference: in, N This represents the total number of pixels within the anchored area. i For pixel index, For temporary pixel brightness values ​​in the anchoring area, To reconstruct the brightness value sequence for the anchored region, This results in a loss of consistency. S423, Apply the consistency loss to the current intermediate latent variables. z t Perform backpropagation to obtain the guiding gradient. By using guided gradient correction to adjust the denoising sampling step, the latent variables for the next time step are obtained: in, This is the denoising network for the latent diffusion model. C Let be the spatiotemporal condition vector. η For guiding strength coefficient, , , To predetermine noise scheduling parameters, It is random noise. This is a latent variable for the next time step.

9. A video compression and transmission system for public network surveillance, applied to the video compression and transmission method for public network surveillance as described in any one of claims 1 to 8, characterized in that, This includes edge terminal systems deployed at monitoring sites and server terminal systems deployed in data centers; The edge terminal system includes: The camera is configured to capture raw video streams of the monitored scene; An edge processing unit, communicatively connected to the camera, decodes the original video stream, calls a target detection model to identify dynamic foreground targets in the video frames and divides the video into segments; performs inter-frame pixel analysis to classify video segments that do not contain dynamic foreground targets; performs compression encoding on the first and second types of segments; extracts the start and end frames of the third type of segments and calculates the photodiode parameter sequence to encapsulate it into a parameterized data packet; and transmits the processed data to the server subsystem. The server terminal system includes: The database stores building information models and pre-trained generative diffusion models corresponding to the monitoring scenarios. The server is communicatively connected to the edge terminal system. The reconstruction module receives the first type of segments, the second type of segments, and parameterized data packets; it calls the building information model and performs physical illumination simulation calculations based on timestamps and photoelectric difference parameter sequences; using the simulation results as conditions, it drives the generative diffusion model to iteratively generate the reconstructed third type of segments based on the start frame and the end frame; and it combines the received first type of segments, the second type of segments, and the generated reconstructed third type of segments into a complete reconstructed video stream according to the timeline.

Images