Camera patrol video quality diagnosis and optimization method and system and storage medium

Abstract

The application provides a camera patrol video quality diagnosis and optimization method and system and a storage medium. The method comprises the following steps: acquiring original video stream collected by a camera, device metadata associated with the camera, and network performance data, analyzing the original video stream to extract video content quality features; constructing a heterogeneous node set based on the video content quality features, the device metadata, and the network performance data, and constructing a dynamic multi-modal spatio-temporal feature graph with edges representing the bearing or spatio-temporal correlation between nodes as a heterogeneous edge set; inputting the dynamic multi-modal spatio-temporal feature graph into a spatio-temporal graph neural network to capture the propagation path of faults in the heterogeneous node set and output the abnormal probability of each node; taking the abnormal probability and the propagation path as a query condition to match and cooperatively reason in a pre-constructed knowledge graph to determine the root cause and the corresponding evidence chain causing the video quality to decrease; and matching an optimization strategy based on the root cause. The application improves the reliability and operation and maintenance efficiency of video monitoring.

Description

Technical Field

[0001] This application relates to the field of video surveillance technology, and in particular to methods, systems and storage media for diagnosing and optimizing video quality during camera inspections. Background Technology

[0002] Current video surveillance systems are widely used in security patrols, smart cities, and other scenarios. To ensure the input quality of subsequent intelligent analysis (such as facial recognition and behavior analysis), effective monitoring of camera video quality is necessary. Existing technical solutions mainly employ manual spot checks or passive monitoring methods based on simple network transmission indicators (such as packet loss rate and latency). Manual spot checks are inefficient and cannot achieve 24 / 7 full coverage; while relying solely on network indicators can reflect transmission layer anomalies, it cannot determine whether the video content itself is identifiable. Quality issues such as image blurring, occlusion, and abnormal brightness are difficult to detect in a timely manner. More importantly, existing technologies can only detect the surface symptoms of problems and cannot correlate and analyze the intrinsic relationship between video content, device status, and network performance. This leads to difficulties in locating the root cause of the fault, a disconnect between diagnosis and repair, long problem repair cycles, and seriously affects the reliability of the monitoring system. Summary of the Invention

[0003] In order to improve the reliability and operational efficiency of video surveillance, embodiments of this application provide a method, system, and storage medium for diagnosing and optimizing video quality during camera inspections.

[0004] Firstly, it provides methods for diagnosing and optimizing the quality of camera-based surveillance video, including: The system acquires raw video streams collected by cameras deployed at the inspection site, along with device metadata and network performance data associated with the cameras, and performs in-depth analysis on the raw video streams to extract multi-dimensional video content quality features. Based on the video content quality features, the device metadata, and the network performance data, a dynamic multimodal spatiotemporal feature graph is constructed, with video stream nodes, device nodes, and network nodes as heterogeneous node sets, and edges representing the carrying or spatiotemporal relationships between nodes as heterogeneous edge sets. The dynamic multimodal spatiotemporal feature map is input into the spatiotemporal graph neural network to capture the propagation path of the fault among the nodes in the heterogeneous node set and output the anomaly probability of each node. Using the anomaly probability and the propagation path as query conditions, matching and collaborative reasoning are performed in a pre-constructed knowledge graph containing causal relationships between entities within the monitoring system to determine the root cause of the video quality degradation and the evidence chain supporting the root cause. The optimization strategy is based on the root cause matching predefined.

[0005] In some embodiments, acquiring the raw video stream captured by cameras deployed at the patrol site, device metadata associated with the cameras, and network performance data includes: The system accesses the camera's real-time video stream to extract the raw video stream, collects the camera's operating status through the device management protocol to obtain device metadata, and measures the transmission metrics of the link where the camera is located through a network probe to obtain network performance data.

[0006] In some embodiments, the deep analysis of the original video stream to extract multi-dimensional video content quality features includes: Obtain keyframe images from the original video stream; The shared features of the keyframe images are extracted by learning the backbone network of a convolutional neural network through multi-task learning; The shared features are processed by the sharpness evaluation branch of the multi-task learning convolutional neural network to output a sharpness score; The shared features are processed by the occlusion segmentation branch of the multi-task learning convolutional neural network to output a pixel-level occlusion mask map for distinguishing foreground objects from abnormal occlusions. The shared features are processed by the brightness and color anomaly classification branch of the multi-task learning convolutional neural network to output the classification probabilities of brightness and color anomalies. The shared features are processed by the freeze detection branch of the multi-task learning convolutional neural network to output the freeze duration for determining the freeze. The sharpness score, pixel-level occlusion mask image, classification probability, and freeze duration are used together as multi-dimensional video content quality features.

[0007] In some embodiments, the construction of a dynamic multimodal spatiotemporal feature graph based on the video content quality features, the device metadata, and the network performance data, with video stream nodes, device nodes, and network nodes as heterogeneous node sets, and edges representing the carrying or spatiotemporal relationships between nodes as heterogeneous edge sets, includes: The video stream node, device node, and network node of each camera are obtained. The attributes of the video stream node include the video content quality characteristics, the attributes of the device node include the device metadata, and the attributes of the network node include the network performance data. Construct a first bearer relationship edge for connecting the video stream node and the device node, and a second bearer relationship edge for connecting the device node and the network node; Based on the overlap of field of view and the similarity of image content between different cameras in the same scene, the spatiotemporal correlation edges connecting different video stream nodes are dynamically calculated and constructed. Aggregate all nodes, first carrier relationship edges, second carrier relationship edges, and spatiotemporal association edges within the current time window to generate a dynamic multimodal spatiotemporal feature map.

[0008] In some embodiments, the spatiotemporal graph neural network includes spatiotemporal convolutional layers and sequence learning layers, and the step of inputting the dynamic multimodal spatiotemporal feature map into the spatiotemporal graph neural network to capture the propagation path of the fault among nodes in the heterogeneous node set and outputting the anomaly probability of each node includes: The graph sequence composed of multiple dynamic multimodal spatiotemporal feature maps within a continuous time window is input into the spatiotemporal convolutional layer to aggregate the neighborhood features of each node in the spatiotemporal dimension and extract the change pattern of node attributes in the time dimension to obtain the node latent state features that fuse spatiotemporal dependencies. The hidden state features of the nodes are input into the sequence learning layer to capture the temporal propagation pattern of fault features in the graph sequence and generate hypotheses about the propagation path of faults between nodes. The propagation path hypothesis and the hidden state of the nodes are fed into the anomaly classification layer to calculate the probability that each video node, device node and network node belongs to the anomaly category, and output the anomaly probability to indicate the health status of each node.

[0009] In some embodiments, using the anomaly probability and the propagation path as query conditions to perform matching and collaborative reasoning in a pre-constructed knowledge graph containing causal relationships between entities within the monitoring system to determine the root cause of video quality degradation and the chain of evidence supporting the root cause includes: The nodes with an anomaly probability higher than a preset threshold and the propagation path are transformed into structured queries on the knowledge graph to match candidate root cause entities. The candidate root cause entities are compared with the propagation path to verify causal consistency and filter out at least one root cause that conforms to the fault propagation logic. Based on the predefined causal links in the knowledge graph, the root cause is traced backward to generate a readable chain of evidence pointing from the root cause to the original anomalous node.

[0010] In some embodiments, the optimization strategy based on the root cause matching predefined includes: Query the pre-built strategy knowledge base to match at least one candidate optimization strategy corresponding to the root cause and the evidence chain; The execution conditions and control interface types corresponding to the candidate optimization strategies are analyzed to select optimization strategies that conform to the current device status and network topology.

[0011] In some embodiments, the method further includes: In response to the completion of the optimization strategy, the video content quality characteristics of the camera within a preset time period are continuously monitored to generate a posterior quality score. The posterior quality score, the evidence chain, and the root cause are encapsulated together as a feedback case, and the feedback case is input into the rule learning layer of the knowledge graph to extract new causal association rules. The knowledge graph is incrementally updated using the new causal association rules to optimize its reasoning ability.

[0012] Secondly, a video quality diagnosis and optimization system for camera patrols is provided. This system includes: an acquisition module, a construction module, a diagnosis module, and an optimization module; wherein... The acquisition module is used to acquire the original video stream collected by the camera deployed at the inspection site, the device metadata associated with the camera, and the network performance data, and to perform in-depth analysis on the original video stream to extract multi-dimensional video content quality features; The construction module is used to construct a dynamic multimodal spatiotemporal feature graph based on the video content quality features, the device metadata, and the network performance data. The graph consists of a heterogeneous set of video stream nodes, device nodes, and network nodes, and a heterogeneous set of edges representing the carrying or spatiotemporal relationships between nodes. The diagnostic module is used to input the dynamic multimodal spatiotemporal feature map into the spatiotemporal graph neural network to capture the propagation path of the fault among the nodes in the heterogeneous node set and output the abnormal probability of each node. The abnormal probability and the propagation path are used as query conditions to perform matching and collaborative reasoning in a pre-constructed knowledge graph containing causal relationships between entities within the monitoring system to determine the root cause of the video quality degradation and the evidence chain supporting the root cause. The optimization module is used to implement a predefined optimization strategy based on the root cause matching.

[0013] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon that can run on a processor, wherein when the computer program is executed by the processor, it implements the camera patrol video quality diagnosis and optimization method as described in the first aspect.

[0014] Using the above method, this application obtains raw video streams collected by cameras deployed at the inspection site, along with device metadata and network performance data associated with the cameras. Deep analysis of the raw video streams is then performed to extract multi-dimensional video content quality features. Based on the video content quality features, device metadata, and network performance data, a dynamic multimodal spatiotemporal feature graph is constructed, with video stream nodes, device nodes, and network nodes as heterogeneous node sets, and edges representing the carrying or spatiotemporal relationships between nodes as heterogeneous edge sets. This dynamic multimodal spatiotemporal feature graph is input into a spatiotemporal graph neural network to capture the propagation path of faults among nodes in the heterogeneous node set and output the anomaly probability of each node. The anomaly probability and propagation path are used as query conditions to perform matching and collaborative reasoning in a pre-constructed knowledge graph containing causal relationships between entities within the monitoring system to determine the root causes of video quality degradation and the evidence chains supporting these root causes. A predefined optimization strategy based on root cause matching is then applied to improve the reliability and operational efficiency of video surveillance. Attached Figure Description

[0015] Figure 1 This is a flowchart of the camera inspection video quality diagnosis and optimization method provided in this application.

[0016] Figure 2 This is a flowchart of a method for performing deep analysis on raw video streams to extract multi-dimensional video content quality features, as provided in this application.

[0017] Figure 3 This application provides a method block diagram for inputting dynamic multimodal spatiotemporal feature maps into a spatiotemporal graph neural network to capture the propagation path between nodes in a heterogeneous node set and output the anomaly probability of each node.

[0018] Figure 4 This is a flowchart illustrating the process of diagnosing and optimizing the video quality of camera inspections provided in this application.

[0019] Figure 5 This is a schematic diagram of the connection of the camera patrol video quality diagnosis and optimization system provided in the embodiments of this application. Detailed Implementation

[0020] To better understand the purpose, technical solutions, and advantages of this application, it has been described and illustrated below with reference to the accompanying drawings and embodiments. However, those skilled in the art should understand that this application can be implemented without these details. It will be apparent to those skilled in the art that various modifications can be made to the embodiments disclosed in this application, and the general principles defined in this application can be applied to other embodiments and application scenarios without departing from the principles and scope of this application. Therefore, this application is not limited to the illustrated embodiments, but is consistent with the broadest scope claimed in this application.

[0021] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart of the camera inspection video quality diagnosis and optimization method provided in this application. Figure 1 As shown, the method for diagnosing and optimizing the video quality of camera inspections includes the following steps: Step S100: Obtain the raw video stream collected by the camera deployed at the inspection site, the device metadata associated with the camera, and the network performance data. Perform in-depth analysis on the raw video stream to extract multi-dimensional video content quality features.

[0023] This application describes the process from the diagnostic and optimization perspective. First, it requires real-time acquisition and standardized preprocessing of multi-source heterogeneous data to provide a high-quality data foundation for subsequent in-depth analysis and graph model construction. This includes acquiring raw video streams from cameras deployed at the inspection site, device metadata associated with the cameras, and network performance data. This involves accessing the cameras' real-time video streams to extract the raw video streams, simultaneously collecting the cameras' operational status through device management protocols to obtain device metadata, and measuring the transmission metrics of the links where the cameras are located using network probes to obtain network performance data.

[0024] Specifically, the system first accesses raw video streams in real time through cameras deployed at the inspection site, using RTSP or GB / T 28181 protocols to retrieve real-time video stream data. Simultaneously, it periodically collects camera metadata via ONVIF protocol or device SDK, including online status, CPU temperature, pan / tilt position, focal length, aperture value, and last cleaning or restart time. Furthermore, it collects network performance data related to the camera's link from network devices such as switches and routers via network probes or SNMP protocol, including port traffic, packet error rate, packet loss rate, latency jitter, and round-trip time. After collection, this multi-source data undergoes preprocessing: keyframes are extracted from the video stream and scaled to a standard size; numerical data is normalized; categorical data is encoded; and all data is aligned by camera ID and timestamp to form a unified data view. This achieves comprehensive, real-time data acquisition of video streams, device status, and network performance, providing a high-quality, multi-dimensional, and time-aligned data foundation for subsequent video content quality feature extraction and multimodal spatiotemporal map construction. By collaboratively collecting and fusing multi-source heterogeneous data, the comprehensiveness and accuracy of fault diagnosis have been significantly improved, avoiding misjudgments or omissions caused by data silos, and laying a solid data foundation for subsequent accurate root cause analysis and closed-loop optimization.

[0025] After completing the acquisition and preprocessing of multi-source heterogeneous data, in-depth analysis of the video stream content is required to extract quality features. Figure 2 This is a flowchart illustrating the method provided in this application for deep analysis of raw video streams to extract multi-dimensional video content quality features. For example... Figure 2 As shown, by designing a multi-task learning convolutional neural network, key quality features for subsequent diagnosis can be efficiently and accurately extracted from the video stream. The deep analysis of the original video stream to extract multi-dimensional video content quality features includes the following steps: Step S101: Obtain keyframe images from the original video stream.

[0026] Step S102: Extract shared features of keyframe images by learning the backbone network of a multi-task convolutional neural network.

[0027] Step S103: The sharpness evaluation branch of the multi-task learning convolutional neural network processes shared features to output a sharpness score.

[0028] Step S104: The occlusion segmentation branch of the convolutional neural network is processed through multi-task learning to output a pixel-level occlusion mask map for distinguishing foreground objects from abnormal occlusions.

[0029] Step S105: The brightness and color anomaly classification branches of the multi-task learning convolutional neural network process shared features to output the classification probabilities of brightness and color anomalies.

[0030] Step S106: Process shared features through the screen freeze detection branch of the multi-task learning convolutional neural network to output the freeze duration for determining screen stagnation.

[0031] Step S107: The sharpness score, pixel-level occlusion mask image, classification probability, and freeze duration are used together as multi-dimensional video content quality features.

[0032] Specifically, to reduce computational overhead without sacrificing diagnostic reliability, a keyframe image is extracted from the original, continuous video stream at preset intervals. The keyframe contains complete image information, sufficient to support subsequent precise analysis of quality metrics such as image sharpness, occlusion, and brightness. The extracted keyframe images are uniformly scaled to a fixed standard size to ensure input data consistency. Then, to address the high resource consumption and slow inference speed issues caused by deploying an independent model for each quality metric in traditional solutions, a multi-task learning convolutional neural network is employed. This network includes a shared backbone network, designed as a lightweight and efficient convolutional neural network, such as EfficientNet-B0 or ​​MobileNetV3. This backbone network receives the preprocessed keyframe images as input and, through its deep convolutional and pooling layers, automatically learns and outputs a shared feature map that represents the global and local information of the image. This shared feature map encapsulates basic visual information such as color, texture, edges, and shape, providing a unified and rich feature foundation for subsequent refined analysis of multiple different tasks. Next, the shared feature map is fed into a specially designed sharpness evaluation branch. This branch typically consists of several fully connected layers or a combination of global average pooling layers and fully connected layers. Its core task is to learn the mapping from shared features to image sharpness. This branch is trained to output a sharpness score S_sharp with a value between 0 and 1. To achieve accurate evaluation, the training data for this branch not only includes blurred images synthesized using algorithms such as Gaussian blur and motion blur, but more importantly, it also incorporates a large number of blurred video clips from real surveillance scenes caused by factors such as defocusing, fast movement, and rain / fog. This allows the trained model to better distinguish different types of blur, thus outputting a sharpness score that is more consistent with human visual perception in practical applications.

[0033] Next, a semantic segmentation-based occlusion segmentation branch was designed, employing a decoder based on architectures such as U-Net or DeepLabV3+. The input to this branch is also a shared feature map extracted from the backbone network, and its output is a pixel-level occlusion mask map M_occ with the same size as the original image. In this mask map, the value of each pixel represents the probability (or a binarized 0 / 1 label) that the pixel belongs to an anomalous occlusion. During training, this branch is explicitly taught to distinguish between normal, permissible semantic targets (such as people, vehicles, and animals) and anomalous occlusions that truly obstruct the surveillance view, thus significantly improving the accuracy and practicality of occlusion detection. The brightness and color anomaly classification branch is responsible for diagnosing the photoelectric conversion quality of the image. Its input is a shared feature map, and its main structure consists of several fully connected layers, ultimately outputting a multi-dimensional classification probability vector through a Softmax layer. Each dimension of this vector corresponds to a predefined brightness and color state category, including but not limited to normal, overexposed, underexposed, color cast (reddish), color cast (greenish), color cast (blueish), and black and white mode. This branch is trained using a large number of surveillance videos containing various brightness anomalies and color distortions, enabling it to sensitively detect image quality issues caused by camera white balance failure, improper switching of infrared modes, direct sunlight, or insufficient lighting. Unlike the single-frame-based branches mentioned earlier, the image freeze detection branch requires information across frames. Therefore, its design is more specialized. It not only receives shared features from the current keyframe but also acquires features from the previous few keyframes through a short-term feature buffer. By comparing the content hash (pHash) differences and optical flow information between consecutive frames, it determines whether the image is frozen. If the image content is highly similar and persists for more than a preset time threshold, it is determined to be frozen, and the duration T_freeze is output. This design effectively distinguishes between normal stillness caused by a static scene (such as a deserted area at night) and abnormal freezes caused by encoder jamming or transmission interruption. Finally, through the collaborative work of the above six branches, a rich and high-dimensional set of video content quality features is extracted for each video stream and each diagnostic cycle. These features are no longer isolated, simple alarm signals, but rather refined digital expressions from deep learning models that can precisely describe the intrinsic quality of video footage. Together, they constitute the core attributes of video stream nodes in subsequent multimodal spatiotemporal graphs. Thus, through a multi-task learning framework, multiple quality indicators can be extracted in parallel using only a single backbone network, significantly reducing the computational and storage overhead of model deployment, improving analysis speed, and meeting the real-time requirements of large-scale monitoring scenarios.Secondly, each task branch is specifically designed and trained for specific quality defects (such as sharpness, occlusion, brightness, color, and freeze). In particular, the occlusion segmentation branch's fine distinction between foreground objects and abnormal occlusions, and the freeze detection branch's temporal analysis capabilities, make the extracted video content quality features more accurate and robust, avoiding the false positives and false negatives common in traditional methods. These high-quality, multi-dimensional features provide a solid data foundation for subsequent correlation analysis and accurate root cause diagnosis in heterogeneous spatiotemporal graphs.

[0034] Step S200: Based on video content quality features, device metadata, and network performance data, construct a dynamic multimodal spatiotemporal feature map with video stream nodes, device nodes, and network nodes as heterogeneous node sets, and edges representing the carrying or spatiotemporal relationships between nodes as heterogeneous edge sets.

[0035] After obtaining the refined content quality characteristics of each video stream, along with the corresponding device metadata and network performance data, these data remain isolated information silos. To reveal the propagation paths of faults across different stages (acquisition, transmission, and processing) and the complex relationships hidden behind video quality faults, it is necessary to connect these heterogeneous data, i.e., construct a dynamic multimodal spatiotemporal feature graph that maps monitoring entities and their logical relationships in the physical world into a unified graph structure. The construction of this dynamic multimodal spatiotemporal feature graph, based on video content quality characteristics, device metadata, and network performance data, with video stream nodes, device nodes, and network nodes as heterogeneous node sets, and edges representing the inter-node carrying or spatiotemporal relationships as heterogeneous edge sets, includes the following steps: Step S201: Obtain the video stream node, device node, and network node of each camera. The attributes of the video stream node include video content quality characteristics, the attributes of the device node include device metadata, and the attributes of the network node include network performance data.

[0036] Step S202: Construct a first bearer relationship edge for connecting video stream nodes and device nodes, and a second bearer relationship edge for connecting device nodes and network nodes.

[0037] Step S203: Dynamically calculate and construct spatiotemporal association edges connecting different video stream nodes based on the overlap of field of view and the similarity of image content between different cameras in the same scene.

[0038] Step S204: Aggregate all nodes, first bearing relationship edges, second bearing relationship edges, and spatiotemporal association edges within the current time window to generate a dynamic multimodal spatiotemporal feature map.

[0039] Specifically, the nodes in the diagram need to be defined first. For a camera undergoing inspection, three logical nodes are created. The first is the video stream node, representing the video data stream generated by the camera. It is a logical entity, and its attributes are set to the multi-dimensional video content quality features that dynamically change over time, extracted in step S100, such as the current sharpness score, pixel-level occlusion mask map, brightness and color anomaly classification probability, and frame freeze duration. These attributes describe the quality of the data itself. The second is the device node, representing the physical camera that generates the video stream. Its attributes are set to device metadata collected through the device management protocol, such as online status, CPU temperature, pan / tilt angle, focal length, and last cleaning time. These attributes describe the health status of the source. The third is the network node, representing the network port or link carrying the video stream transmission. Its attributes are set to network performance data measured by network probes, such as packet loss rate, latency jitter, round-trip time, and port traffic. These attributes describe the transmission quality of the channel. In this way, the originally isolated video quality features, device status data, and network performance indicators are organically attached to three different types of but interconnected nodes, forming a complete digital characterization of a certain video stream in the monitoring system from the source to the data and then to the channel.

[0040] After defining the nodes, edges need to be constructed to connect them and express the logical relationships between them. First, a first bearer edge is constructed to connect the video stream node and the device node corresponding to the same camera. This edge signifies that the video stream is generated and encoded by this physical device. The edge's attributes can include relatively static information such as encoding format and resolution. Second, a second bearer edge is constructed to connect the device node and the network node. This edge signifies access, meaning the physical device is connected to this network port and data is transmitted through this network link. The edge's attributes can include information such as IP address and VLAN ID. Through these two bearer edges, a complete, directed dependency link is established from data (video stream node) to the source (device node) and then to the channel (network node). This allows the subsequent model to clearly understand that if a video quality problem occurs, its impact may propagate backward along this link, or the fault may originate from any link in this link.

[0041] In many patrol scenarios, the fields of view of multiple cameras may overlap, such as four cameras at an intersection or two adjacent cameras in a long corridor. When a moving target (such as a suspect or vehicle) crosses the monitored area, it will appear in the images of multiple cameras sequentially. To capture this cross-camera spatiotemporal correlation, this step dynamically calculates and constructs spatiotemporal correlation edges connecting the video stream nodes of different cameras. This involves continuously analyzing the video streams of different cameras in the same scene. First, the overlap of the fields of view between each pair of cameras is calculated using pre-defined camera calibration information or real-time image feature matching. More importantly, deep learning-based target re-identification technology is used to detect and track key targets (such as people and vehicles) in the footage in real time. When a target is detected in the image of camera A, and the same target is detected in the image of camera B within a subsequent time window, a spatiotemporal correlation edge is dynamically constructed between the video stream nodes of cameras A and B. The weight of this edge can be dynamically assigned based on the spatiotemporal continuity of the target's appearance and its similarity score. This edge is dynamic; it appears as the target moves and automatically disappears after a period of time if there is no longer any correlation. In this way, the originally isolated, single-channel video stream nodes are connected into a spatiotemporal relationship network that can reflect the flow of targets and the evolution of events in the real world.

[0042] Finally, using a fixed time window as a cycle, the video stream nodes, device nodes, network nodes, and all static first and second bearer relationship edges and dynamically generated spatiotemporal association edges corresponding to all active cameras within that window are collected and aggregated to form a snapshot of the graph structure at the current moment, namely the dynamic multimodal spatiotemporal feature graph. As time progresses, new graph snapshots are continuously generated, forming a graph sequence. This graph sequence not only contains the state (node ​​attributes) of each entity within the system at each moment, but more importantly, it captures the propagation structure of faults along the device-network-data link and the transfer path of events across camera spaces through bearer relationship edges and dynamic spatiotemporal association edges. This provides structured and semantically rich input for subsequent spatiotemporal graph neural networks to perform in-depth fault mode mining and propagation path tracing. This elevates the video surveillance system from a simple collection of devices into a computationally and reasoning-capable complex relational network. Firstly, it achieves deep fusion of multi-source data. By defining heterogeneous nodes and edges, it unifies the originally fragmented video content, device status, and network performance data into a single graph model, laying the foundation for correlation analysis. Secondly, a dynamic cross-camera spatiotemporal correlation is introduced. By calculating field-of-view overlap and target re-identification in real time, spatiotemporal correlation edges are constructed, enabling the model to perceive the spatial shift and temporal continuation of events. This is crucial for detecting complex faults such as interruptions in multi-camera relay tracking or the disappearance of targets in overlapping areas. Finally, the generated graph structure clearly reveals the potential propagation paths of faults, allowing subsequent diagnostic models to move beyond a blind man's groping in the dark regarding a single camera. Instead, they can take a holistic view, understanding how an anomaly at one node affects other nodes through relational edges, thereby accurately locating the root cause of the fault.

[0043] Step S300: Input the dynamic multimodal spatiotemporal feature map into the spatiotemporal graph neural network to capture the propagation path of the fault among the nodes in the heterogeneous node set and output the anomaly probability of each node.

[0044] After obtaining the graph sequence composed of dynamic multimodal spatiotemporal feature maps, a deep learning model is needed to capture the propagation pattern of faults among heterogeneous nodes, thereby accurately locating the source of the anomaly. To this end, a specially designed spatiotemporal graph neural network is introduced, which includes spatiotemporal convolutional layers and sequence learning layers. Figure 3 This is a block diagram of the method provided in this application for inputting dynamic multimodal spatiotemporal feature maps into a spatiotemporal graph neural network to capture the propagation paths between nodes in a heterogeneous node set and output the anomaly probability of each node. Figure 3 As shown, inputting a dynamic multi-module spatiotemporal feature map into a spatiotemporal graph neural network to capture the propagation path between nodes in a heterogeneous node set and output the anomaly probability of each node includes the following steps: Step S301: Input the graph sequence composed of multiple dynamic multimodal spatiotemporal feature maps within a continuous time window into the spatiotemporal convolutional layer to aggregate the domain features of each node in the spatiotemporal dimension and extract the change patterns of node attributes in the time dimension to obtain the node latent state features that fuse spatiotemporal dependencies.

[0045] Step S302: Input the hidden state features of the nodes into the sequence learning layer to capture the temporal propagation pattern of fault features in the graph sequence to generate the hypothesis of the propagation path of the fault between nodes.

[0046] Step S303: The propagation path hypothesis and the hidden state of the nodes are fed into the anomaly classification layer to calculate the probability that each video node, device node and network node belongs to the anomaly category, and output the anomaly probability used to indicate the health status of each node.

[0047] Specifically, the spatiotemporal convolutional layer employs a Graph Convolutional Network (GCN) to perform spatial domain message passing on graph snapshots at each time slice. This allows each node to aggregate attribute information from its domain nodes (e.g., device nodes aggregate network nodes, video stream nodes aggregate device nodes), thereby capturing the spatial propagation path of faults along the bearer relationships. Simultaneously, this layer introduces temporal convolutions or gated recurrent units (GRUs) to convolve or recursively process the historical hidden states of the same node along the time dimension, extracting the trend patterns of node attributes (e.g., continuous decrease in sharpness score, sudden increase in packet loss rate) over time. Through this spatiotemporal joint message passing mechanism, the hidden state features of each node not only contain the spatial influence from related nodes at the current moment but also characterize the dynamic changes of its own attributes within the historical window, providing a rich representational basis for subsequent analysis of the temporal causal chain of faults.

[0048] Sequence learning layers are typically constructed from bidirectional long short-term memory networks (Bi-LSTM) or Transformers, with each node's input being a sequence of hidden state vectors within a continuous time window. By learning the dependencies between node states at different time steps, this layer can identify the start and end times, duration, and transition order across nodes for anomalous features. For example, if the packet loss rate hidden state sequence of a network node fluctuates significantly before the frame rate hidden state sequence of a video stream node, the sequence learning layer can infer a temporal causal hypothesis that network congestion leads to a decrease in frame rate. Based on this, and combined with the connections between nodes in the graph structure, the model can generate one or more directed propagation path hypotheses from the source node of the anomaly to the affected node, providing structured candidate directions for root cause localization.

[0049] The anomaly classification layer typically employs a Multilayer Perceptron (MLP) combined with a Softmax or Sigmoid function. Its input is a concatenated vector of node latent state features and corresponding propagation path embeddings. This layer trains an anomaly classifier for each node type, outputting the probability value of a node belonging to categories such as normal, warning, or abnormal. For example, for video stream nodes, the classifier assigns the probability of a clarity anomaly based on its latent state features and the influence of historical propagation paths; for network nodes, it assigns the probability of a high packet loss anomaly. Ultimately, the anomaly probability distribution of all nodes in the entire graph structure is output, thus achieving a quantitative assessment of the monitoring system's health status from a global perspective, providing accurate input features for subsequent root cause inference based on knowledge graphs. Through the synergistic effect of spatiotemporal convolutional layers and sequence learning layers, it can not only aggregate the spatial correlations of multimodal nodes but also capture the propagation patterns of fault features in the temporal dimension, ultimately accurately outputting the anomaly probability of each node. This enables diagnostic models to move beyond threshold judgments for single-point indicators and instead automatically identify fault sources and propagation paths from system-level spatiotemporal correlations. This significantly improves the accuracy and robustness of root cause localization in complex scenarios and provides reliable data support for subsequent interpretable strategy generation and closed-loop optimization.

[0050] Step S400: Using the anomaly probability and propagation path as query conditions, match and perform collaborative reasoning in a pre-built knowledge graph containing causal relationships between entities within the monitoring system to determine the root cause of the video quality degradation and the evidence chain supporting the root cause.

[0051] After completing the anomaly propagation analysis based on spatiotemporal graph neural networks, the anomaly probability distribution and fault propagation path hypotheses for each node have been obtained. However, these are essentially inferences at the correlation level and have not yet been combined with professional knowledge in the monitoring field and practical fault experience. Therefore, introducing knowledge graphs for causal-level collaborative reasoning can accurately map the symptoms discovered by the data to actionable root causes and generate interpretable chains of evidence. Specifically, using anomaly probabilities and propagation paths as query conditions, matching and collaborative reasoning are performed in a pre-built knowledge graph containing causal relationships between entities within the monitoring system to determine the root causes of video quality degradation and the supporting chains of evidence. This includes the following steps: Step S401: Nodes and propagation paths with anomaly probabilities higher than a preset threshold are transformed into structured queries on the knowledge graph to match candidate root cause entities.

[0052] Step S402: Perform causal consistency verification on the candidate root cause entities and propagation paths to filter out at least one root cause that conforms to the fault propagation logic.

[0053] Step S403: Based on the predefined causal links in the knowledge graph, the root cause is traced backward to generate a readable chain of evidence pointing from the root cause to the original abnormal node.

[0054] Specifically, firstly, based on the anomaly probabilities output by the spatiotemporal graph neural network, abnormal nodes with probabilities higher than a threshold in the video stream, devices, or network nodes are selected. These nodes, along with the generated propagation path hypotheses, are then transformed into a query language recognizable by the knowledge graph. Query conditions include the type of abnormal node, attribute characteristics, and the temporal order of influence between nodes. The knowledge graph engine retrieves all entities and relationships matching these query conditions from the graph database, initially identifying candidate root cause entities that conform to the symptom description. Then, based on predefined causal rules in the knowledge graph (e.g., dirty lens - blurry image, and dirty lens does not affect network nodes), each candidate root cause is verified to reasonably explain the entire propagation path obtained in step S303. For example, if the propagation path shows that network node packet loss increases first, followed by a decrease in the frame rate of video stream nodes, then the candidate root cause "dirty lens" is excluded because it cannot explain the anomaly of network nodes, while network congestion is retained because it perfectly matches the propagation path. Through this causal consistency constraint, the range of candidate root causes is narrowed down to the root cause that best fits the fault propagation logic.

[0055] After finally identifying the root cause, the causal chain stored in the knowledge graph, such as focus motor failure - autofocus failure - blurry image, is used to trace back from the root cause entity to the node where the anomaly was initially detected. During the tracing process, the node attributes, edge relationships, and predefined rules on which each step of reasoning is based are recorded, forming a structured reasoning path. Subsequently, a natural language template is used to convert this path into a human-readable description, such as the evidence chain: Because the cumulative working time of the focus motor of the device node exceeded the lifespan threshold, the autofocus of the video stream node failed, leading to a continuous decline in the sharpness score; at the same time, the network nodes' various indicators were normal, ruling out transmission interference. According to the knowledge graph rules of focus motor aging - focus failure - blurry image, the final cause was determined to be lens defocusing. This deeply integrates data-driven anomaly detection with knowledge-driven causal reasoning, not only accurately locating the root cause of the fault but also outputting a clear and traceable evidence chain. It overcomes the drawbacks of black-box reasoning in traditional AI models, enabling operations and maintenance personnel to intuitively understand the origin of diagnostic conclusions. This greatly improves the trust and adoption rate of automatic diagnostic results, providing a solid decision-making basis for subsequent accurate closed-loop optimization.

[0056] Step S500: Based on the predefined optimization strategy of root cause matching.

[0057] After accurately identifying the root cause of video quality degradation and generating an interpretable chain of evidence, it needs to be translated into specific operational actions to complete the diagnostic optimization loop. The predefined optimization strategy based on root cause matching includes the following steps: Step S501: Query the pre-built strategy knowledge base to match at least one candidate optimization strategy corresponding to the root cause and the evidence chain.

[0058] Step S502: Analyze the execution conditions and control interface types corresponding to the candidate optimization strategies, and select the optimization strategies that match the current device status and network topology.

[0059] Specifically, the identified cause and its evidence chain are used as the query key for retrieval in the policy knowledge base. This knowledge base pre-stores the mapping relationship between diagnostic root causes and optimization actions, such as mapping the root cause of lens contamination to candidate strategies like initiating automatic cleaning or generating a cleaning work order. Then, the execution conditions of each candidate strategy are analyzed; for example, automatic cleaning requires the device to support ONVIF cleaning commands. Simultaneously, the current device status (whether it is online) and network topology (whether the management channel is reachable) are checked, ultimately selecting the optimal executable strategy. For instance, if the device supports remote cleaning and is online, initiating automatic cleaning is prioritized; otherwise, generating a manual work order is selected. This transforms abstract diagnostic conclusions into specific, executable optimization instructions, achieving an automated closed loop from problem discovery to problem resolution, significantly shortening fault repair time and improving operational efficiency.

[0060] Preferably, this application further includes the following steps: In step S600, in response to the completion of the optimization strategy, the video content quality characteristics of the camera within a preset time period are continuously monitored to generate a posterior quality score.

[0061] Step S700: The posterior quality score, the evidence chain, and the root cause are encapsulated together as a feedback case, and the feedback case is input into the rule learning layer of the knowledge graph to extract new causal association rules.

[0062] Step S800: Incrementally update the knowledge graph using the new causal association rules to optimize the reasoning ability of the knowledge graph.

[0063] Specifically, after the automated actuator completes the camera bitrate adjustment or work order closure, it does not terminate the monitoring of the camera but instead opens a preset observation window. Within this window, the video quality intelligent diagnostic center continues to extract quality features such as the clarity score and frame rate of the video stream at a high frequency, and recalculates a posterior quality score based on the original scoring rules. This score objectively quantifies the actual repair effect of the optimization action on video quality and serves as the data foundation for subsequent knowledge evolution. Then, the posterior quality score, the previously diagnosed root causes, and the evidence chain supporting the conclusion are encapsulated into a complete feedback case. Subsequently, this case is fed into the rule learning layer of the knowledge graph. This layer uses graph pattern matching or inductive logic programming techniques to analyze the commonalities and differences between the new case and existing historical cases in the graph. If it is found that the optimization strategy corresponding to a certain root cause can effectively improve the posterior score in multiple similar scenarios, a new causal association rule with higher confidence can be abstracted and mined. Finally, the extracted new causal association rule is incrementally written into the original knowledge graph in the form of triples or rule statements, realizing the real-time expansion and iteration of the knowledge base. This makes the fault modes and causal logic stored in the knowledge graph richer and more accurate. When faced with similar anomaly propagation paths again, the knowledge graph can perform root cause matching more quickly and accurately, thereby optimizing the overall reasoning capability. This transforms each successful operational experience into structured knowledge stored in the knowledge graph, enabling the diagnostic model to continuously learn from practice. This not only improves the accuracy of subsequent diagnoses but also allows for adaptive responses to newly emerging complex fault modes, achieving automated construction of operational knowledge and significantly enhancing the overall intelligence level and long-term evolution value of the solution.

[0064] Figure 4 This is a flowchart illustrating the camera inspection video quality diagnosis and optimization process provided in this application. For example... Figure 4As shown, the process begins with video stream access and data preprocessing. First, real-time video streams from cameras deployed at the inspection site are accessed and preprocessed, including decoding and image standardization, to provide a unified data format for subsequent analysis. Then, the process enters the parallel quality feature extraction stage. On one hand, the preprocessed video data is fed into the image content analysis engine, which uses a deep learning model to extract image content layer quality features such as sharpness score, occlusion area, brightness mean variance, and color histogram in parallel. On the other hand, the bitstream and network analysis unit analyzes the video transmission protocol in real time, extracting encoding and network performance indicators such as frame rate, bit rate, I-frame interval, network packet loss rate, and latency. After feature extraction, the process moves to the multi-source feature fusion and root cause diagnosis stage. The extracted image quality features, device metadata, and network performance data are input into the feature fusion and root cause inference engine. This engine, based on preset knowledge rules or classification models, performs correlation analysis on multi-dimensional information to infer the most likely root cause of video quality degradation. The next step is the quality assessment and alarm triggering process. The quality quantification and alarm generation unit calculates a comprehensive video quality score based on the diagnostic conclusions. When the score falls below a preset threshold or triggers a serious defect rule, it generates structured alarm information containing a problem description, root cause, confidence level, and evidence. Following this, the process moves to the strategy matching and optimization execution step. After receiving the alarm, the strategy optimization execution module queries a pre-built strategy knowledge base based on the diagnostic root cause, matches the corresponding optimization strategy, and sends control commands to the camera or network device via standard protocols or APIs to execute optimization actions such as dynamically adjusting the bitrate, triggering cleaning, or generating maintenance work orders. Finally, the process enters the effect verification and closed-loop feedback step. After the optimization strategy is executed, the quality indicators of the original problematic video stream are continuously monitored. If the quality returns to normal within a specified time, the alarm is turned off; otherwise, the alarm is escalated or a backup strategy is tried. Simultaneously, the complete case of this diagnosis and optimization is fed back to the system for subsequent iterative optimization of diagnostic rules and strategy libraries, thus forming a complete monitoring-diagnosis-location-optimization closed loop.

[0065] Figure 5 This is a schematic diagram of the connection of the camera patrol video quality diagnosis and optimization system provided in an embodiment of this application. Figure 5 As shown, the camera patrol video quality diagnosis and optimization system includes: an acquisition module, a construction module, a diagnosis module, and an optimization module.

[0066] The system comprises several modules: **Acquisition Module:** Acquires raw video streams from cameras deployed at the inspection site, along with associated device metadata and network performance data. It performs in-depth analysis of the raw video streams to extract multi-dimensional video content quality features. **Construction Module:** Constructs a dynamic multimodal spatiotemporal feature graph based on video content quality features, device metadata, and network performance data. This graph uses video stream nodes, device nodes, and network nodes as heterogeneous node sets, and edges representing the spatiotemporal relationships between nodes as heterogeneous edge sets. **Diagnosis Module:** Inputs the dynamic multimodal spatiotemporal feature graph into a spatiotemporal graph neural network to capture the propagation path of faults among nodes in the heterogeneous node set and outputs the anomaly probability of each node. Using the anomaly probability and propagation path as query conditions, it performs matching and collaborative reasoning within a pre-built knowledge graph containing causal relationships between entities within the monitoring system to determine the root cause of video quality degradation and the supporting evidence chain. **Optimization Module:** Matches predefined optimization strategies based on the root cause.

[0067] The other functions performed by the above-mentioned acquisition module, construction module, diagnosis module, and optimization module, as well as the technical details of each function, are the same as or similar to the corresponding features in the camera inspection video quality diagnosis and optimization method described above, so they will not be repeated here.

[0068] This application also provides a computer storage medium storing a computer program that, when run on a computer, enables the computer to execute the steps in the camera patrol video quality diagnosis and optimization method described above.

[0069] It should be understood that although the steps in the flowcharts in the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order requirement for the execution of these steps, and they can be performed in other orders.

[0070] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for diagnosing and optimizing video quality during camera inspections, characterized in that, The method includes: The system acquires raw video streams collected by cameras deployed at the inspection site, along with device metadata and network performance data associated with the cameras, and performs in-depth analysis on the raw video streams to extract multi-dimensional video content quality features. Based on the video content quality features, the device metadata, and the network performance data, a dynamic multimodal spatiotemporal feature graph is constructed, with video stream nodes, device nodes, and network nodes as heterogeneous node sets, and edges representing the carrying or spatiotemporal relationships between nodes as heterogeneous edge sets. The dynamic multimodal spatiotemporal feature map is input into the spatiotemporal graph neural network to capture the propagation path of the fault among the nodes in the heterogeneous node set and output the anomaly probability of each node. Using the anomaly probability and the propagation path as query conditions, matching and collaborative reasoning are performed in a pre-constructed knowledge graph containing causal relationships between entities within the monitoring system to determine the root cause of the video quality degradation and the evidence chain supporting the root cause. The optimization strategy is based on the root cause matching predefined.

2. The method according to claim 1, characterized in that, The acquisition of raw video streams collected by cameras deployed at the inspection site, device metadata associated with the cameras, and network performance data includes: The system accesses the camera's real-time video stream to extract the raw video stream, collects the camera's operating status through the device management protocol to obtain device metadata, and measures the transmission metrics of the link where the camera is located through a network probe to obtain network performance data.

3. The method according to claim 1, characterized in that, The deep analysis of the original video stream to extract multi-dimensional video content quality features includes: Obtain keyframe images from the original video stream; The shared features of the keyframe images are extracted by learning the backbone network of a convolutional neural network through multi-task learning; The shared features are processed by the sharpness evaluation branch of the multi-task learning convolutional neural network to output a sharpness score; The shared features are processed by the occlusion segmentation branch of the multi-task learning convolutional neural network to output a pixel-level occlusion mask map for distinguishing foreground objects from abnormal occlusions. The shared features are processed by the brightness and color anomaly classification branch of the multi-task learning convolutional neural network to output the classification probabilities of brightness and color anomalies. The shared features are processed by the freeze detection branch of the multi-task learning convolutional neural network to output the freeze duration for determining the freeze. The sharpness score, pixel-level occlusion mask image, classification probability, and freeze duration are used together as multi-dimensional video content quality features.

4. The method according to claim 3, characterized in that, The construction of a dynamic multimodal spatiotemporal feature graph based on the video content quality features, device metadata, and network performance data, with video stream nodes, device nodes, and network nodes as heterogeneous node sets, and edges representing the carrying or spatiotemporal relationships between nodes as heterogeneous edge sets, includes: The video stream node, device node, and network node of each camera are obtained. The attributes of the video stream node include the video content quality characteristics, the attributes of the device node include the device metadata, and the attributes of the network node include the network performance data. Construct a first bearer relationship edge for connecting the video stream node and the device node, and a second bearer relationship edge for connecting the device node and the network node; Based on the overlap of field of view and the similarity of image content between different cameras in the same scene, the spatiotemporal correlation edges connecting different video stream nodes are dynamically calculated and constructed. Aggregate all nodes, first carrier relationship edges, second carrier relationship edges, and spatiotemporal association edges within the current time window to generate a dynamic multimodal spatiotemporal feature map.

5. The method according to claim 4, characterized in that, The spatiotemporal graph neural network includes spatiotemporal convolutional layers and sequence learning layers. The step of inputting the dynamic multimodal spatiotemporal feature map into the spatiotemporal graph neural network to capture the propagation path of the fault among nodes in the heterogeneous node set and outputting the anomaly probability of each node includes: The graph sequence composed of multiple dynamic multimodal spatiotemporal feature maps within a continuous time window is input into the spatiotemporal convolutional layer to aggregate the neighborhood features of each node in the spatiotemporal dimension and extract the change pattern of node attributes in the time dimension to obtain the node latent state features that fuse spatiotemporal dependencies. The hidden state features of the nodes are input into the sequence learning layer to capture the temporal propagation pattern of fault features in the graph sequence and generate hypotheses about the propagation path of faults between nodes. The propagation path hypothesis and the hidden state of the nodes are fed into the anomaly classification layer to calculate the probability that each video node, device node and network node belongs to the anomaly category, and output the anomaly probability to indicate the health status of each node.

6. The method according to claim 5, characterized in that, The step of using the anomaly probability and the propagation path as query conditions to perform matching and collaborative reasoning in a pre-constructed knowledge graph containing causal relationships between entities within the monitoring system to determine the root cause of video quality degradation and the chain of evidence supporting the root cause includes: The nodes with an anomaly probability higher than a preset threshold and the propagation path are transformed into structured queries on the knowledge graph to match candidate root cause entities. The candidate root cause entities are compared with the propagation path to verify causal consistency and filter out at least one root cause that conforms to the fault propagation logic. Based on the predefined causal links in the knowledge graph, the root cause is traced backward to generate a readable chain of evidence pointing from the root cause to the original anomalous node.

7. The method according to claim 6, characterized in that, The optimization strategy based on the predefined root cause matching includes: Query the pre-built strategy knowledge base to match at least one candidate optimization strategy corresponding to the root cause and the evidence chain; The execution conditions and control interface types corresponding to the candidate optimization strategies are analyzed to select optimization strategies that conform to the current device status and network topology.

8. The method according to claim 7, characterized in that, The method further includes: In response to the completion of the optimization strategy, the video content quality characteristics of the camera within a preset time period are continuously monitored to generate a posterior quality score. The posterior quality score, the evidence chain, and the root cause are encapsulated together as a feedback case, and the feedback case is input into the rule learning layer of the knowledge graph to extract new causal association rules. The knowledge graph is incrementally updated using the new causal association rules to optimize its reasoning ability.

9. A video quality diagnosis and optimization system for camera patrol, characterized in that, The system includes: an acquisition module, a construction module, a diagnosis module, and an optimization module; wherein, The acquisition module is used to acquire the original video stream collected by the camera deployed at the inspection site, the device metadata associated with the camera, and the network performance data, and to perform in-depth analysis on the original video stream to extract multi-dimensional video content quality features; The construction module is used to construct a dynamic multimodal spatiotemporal feature graph based on the video content quality features, the device metadata, and the network performance data. The graph consists of a heterogeneous set of video stream nodes, device nodes, and network nodes, and a heterogeneous set of edges representing the carrying or spatiotemporal relationships between nodes. The diagnostic module is used to input the dynamic multimodal spatiotemporal feature map into the spatiotemporal graph neural network to capture the propagation path of the fault among the nodes in the heterogeneous node set and output the abnormal probability of each node. The abnormal probability and the propagation path are used as query conditions to perform matching and collaborative reasoning in a pre-constructed knowledge graph containing causal relationships between entities within the monitoring system to determine the root cause of the video quality degradation and the evidence chain supporting the root cause. The optimization module is used to implement a predefined optimization strategy based on the root cause matching.

10. A computer-readable storage medium storing a computer program executable on a processor, characterized in that, When the computer program is executed by the processor, it implements the camera patrol video quality diagnosis and optimization method as described in any one of claims 1 to 8.

Images