Root cause analysis methods, apparatus, equipment, media and products for abnormal video frames

By acquiring multimodal data sequences to construct feature representations, and utilizing lightweight student models and root cause localization models, the problem of identifying the root cause of video shooting equipment failures was solved, enabling accurate localization and timely maintenance when video quality is poor.

CN123123558APending Publication Date: 2026-11-17CHINA MOBILE (XIONGAN) ICT CO LTD +3
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611272690.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-20
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies make it difficult to accurately identify the root cause of video shooting equipment malfunctions, resulting in the inability to perform timely equipment maintenance when video quality is poor.

Method used

By acquiring multimodal data sequences, constructing multimodal feature representations, and using a pre-trained lightweight student model to identify abnormal video frames, fault analysis is performed by combining structured causal graphs and root cause localization models with intervention reasoning, and a root cause analysis report is generated.

Benefits of technology

It enables precise identification of the root cause of video shooting equipment failure when video quality is poor, and provides accurate maintenance suggestions, thereby improving the timeliness and effectiveness of equipment maintenance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123558A_ABST
    Figure CN123123558A_ABST
Patent Text Reader

Abstract

This invention discloses a method, apparatus, device, medium, and product for root cause analysis of abnormal video frames. The method includes: acquiring a multimodal data sequence and constructing a multimodal feature representation corresponding to the multimodal data sequence; inputting the multimodal feature representation into a pre-trained first model and outputting at least one abnormal video frame in the video to be detected based on the first model; and performing root cause localization processing on each abnormal video frame based on a pre-trained second model to generate a root cause analysis report. This invention solves the problem in existing technologies where it is difficult to accurately locate the root cause of a video shooting device's malfunction when the video quality is poor. It enables timely identification of abnormal video frames in the video when the video shooting device produces poor quality footage, and accurately locates the root cause of the video shooting device's malfunction based on these abnormal video frames, thus providing accurate maintenance suggestions for the video shooting device's troubleshooting.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, medium, and product for root cause analysis of abnormal video frames. Background Technology

[0002] Video quality inspection is a video service quality assurance measure for the entire video business scenario, playing a key supporting role in stabilizing video services and optimizing user viewing experience.

[0003] Currently, video quality inspection methods mainly rely on image quality assessment based on visual features, continuity detection based on temporal sequence and motion state, transmission diagnosis based on data stream and protocol layers, or logical judgment based on rule engines. However, these methods struggle to accurately identify abnormal images in videos and cannot promptly alert maintenance users to perform equipment maintenance when video quality is poor. Summary of the Invention

[0004] This invention provides a method, apparatus, device, medium, and product for root cause analysis of abnormal video frames, in order to solve the problem in the prior art that it is difficult to accurately locate the root cause of the video shooting equipment failure when the video quality is poor.

[0005] In a first aspect, embodiments of the present invention provide a root cause analysis method for abnormal video frames, comprising: A multimodal data sequence is acquired, and a multimodal feature representation corresponding to the multimodal data sequence is constructed; wherein, the multimodal data sequence includes a video frame sequence of the video to be detected, and a device status data sequence of the target device of the video to be detected; The multimodal feature representation is input into a pre-trained first model, and at least one abnormal video frame in the video to be detected is output based on the first model. The root cause analysis report is generated by performing root cause localization processing on each abnormal video frame based on the pre-trained second model.

[0006] Secondly, embodiments of the present invention also provide a root cause analysis device for abnormal video frames, comprising: A construction module is used to acquire a multimodal data sequence and construct a multimodal feature representation corresponding to the multimodal data sequence; wherein, the multimodal data sequence includes a video frame sequence of the video to be detected and a device status data sequence of the target device of the video to be detected; The detection module is used to input the multimodal feature representation into a pre-trained first model and output at least one abnormal video frame in the video to be detected based on the first model. The report generation module is used to perform root cause localization processing on each abnormal video frame based on a pre-trained second model and generate a root cause analysis report.

[0007] Thirdly, embodiments of the present invention also provide an electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the root cause analysis method for abnormal video frames according to any embodiment of the present invention.

[0008] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute and implement the root cause analysis method for abnormal video frames as described in any embodiment of the present invention.

[0009] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the root cause analysis method for abnormal video frames as described in any of the embodiments of the present invention.

[0010] The technical solution of this invention involves acquiring a multimodal data sequence and constructing a multimodal feature representation corresponding to the multimodal data sequence; inputting the multimodal feature representation into a pre-trained first model and outputting at least one abnormal video frame in the video to be detected based on the first model; and performing root cause localization processing on each abnormal video frame based on a pre-trained second model to generate a root cause analysis report. In this technical solution, a multimodal data sequence corresponding to a video shooting device is acquired. The multimodal data sequence includes at least a video frame sequence of the video to be detected obtained by the video shooting device or an image sequence with temporal information extracted from the video to be detected, as well as a device status data sequence of the video shooting device. Feature extraction is then performed on the multimodal data sequence to obtain a multimodal feature representation. Furthermore, anomalies in the multimodal feature representation are identified using a pre-constructed first model. Abnormal video frames are then extracted from the video to be detected based on these multimodal feature representations. These abnormal video frames are then input into a second model with root cause analysis capabilities. Root cause analysis is performed on the abnormal information in these abnormal video frames using the second model. The failure points of the video shooting equipment are determined based on the root cause analysis results of each abnormal video frame, and the root cause analysis report serves as the basis for maintaining the video shooting equipment. This solves the problem in existing technologies where it is difficult to accurately locate the root cause of video shooting equipment failures when video quality is poor, leading to delays in timely maintenance. It achieves the effect of timely identification of abnormal video frames in the video when video shooting equipment produces poor quality footage, and accurately locating the root cause of the video shooting equipment failure based on these abnormal video frames, thus providing accurate maintenance suggestions for the video shooting equipment. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the content of the embodiments of the present invention and these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a root cause analysis method for abnormal video frames provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of a root cause analysis method for abnormal video frames provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a root cause analysis device for abnormal video frames provided in Embodiment 3 of the present invention; Figure 4This is a schematic diagram of the structure of an electronic device that implements the root cause analysis method for abnormal video frames according to embodiments of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention. The acquisition, transmission, storage, use, and processing of data in the technical solutions of this application comply with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, certain software, components, or models and other existing solutions in the industry may be mentioned. These should be considered as exemplary, and their purpose is only to illustrate the feasibility of implementing the technical solutions of this application, but it does not mean that the applicant has or necessarily used such solutions.

[0014] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solution disclosed herein all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to maintain user personal information security and network security. It should also be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solution disclosed herein are all conducted with the user's knowledge and consent, and comply with relevant privacy protection regulations.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0016] Before elaborating on this technical solution in detail, let's briefly introduce its application scenarios to help us understand it more clearly.

[0017] Video quality inspection refers to the real-time monitoring and evaluation of video clarity, smoothness, integrity, audio-visual synchronization, and the presence of abnormalities such as black screens, screen tearing, and stuttering at each stage of video content production, transmission, and playback, using automated or manual methods. This ensures that users can obtain a stable and high-quality viewing experience and is widely used in streaming media platforms, video conferencing systems, online education, live streaming platforms, and other fields. It is an important technical means to ensure video service quality and user experience.

[0018] Example 1 Figure 1 The flowchart of an abnormal video frame root cause analysis method provided in Embodiment 1 of the present invention is applicable to the analysis of multimodal feature representations associated with the video to be detected using a pre-trained lightweight student model to identify abnormal video frames in the video to be detected. Then, based on a pre-trained second model with root cause analysis function, root cause analysis is performed on the abnormal video frames to obtain a root cause analysis report. Based on the root cause analysis report, equipment maintenance is performed on the video shooting equipment that obtained the video to be detected. This method can be executed by an abnormal video frame root cause analysis device, which can be implemented in hardware and / or software. The abnormal video frame root cause analysis device can be configured in a computing device capable of executing the abnormal video frame root cause analysis method.

[0019] like Figure 1 As shown, the method includes: S110. Obtain multimodal data sequences and construct multimodal feature representations corresponding to the multimodal data sequences.

[0020] Multimodal data sequences refer to ordered sets of data that integrate temporal information from different modalities. A multimodal data sequence includes at least a sequence of video frames generated from the decomposition of the video to be detected; this sequence can also be a sequence of images labeled with timestamps. A multimodal data sequence includes at least a sequence of device status data from the target device that provided the video to be detected. Multimodal feature representations refer to data sets that extract and integrate various modal representation information from multimodal data sequences. Multimodal feature representations include at least visual temporal features extracted from the video frame sequence. Multimodal feature representations include at least device sensing features extracted from the target device status data sequence. In some cases, multimodal feature representations are dedicated feature datasets that uniformly organize multi-source temporal modal features for subsequent detection and analysis.

[0021] In a specific example, a target device, pre-deployed in the target scene, acquires video footage of the scene to be tested. Based on its built-in sensing components, the target device can collect real-time data on its own operation, environment, and camera-related indicators, thereby synchronously obtaining a device status data sequence. Simultaneously, by extracting each frame of the video to be tested, a video frame sequence with timestamp information can be generated. The target device supports all existing mainstream video protocols. Combining the video frame sequence with the device status data sequence yields the multimodal data sequence corresponding to the scene captured by the device. The device status data can be divided into three categories: first, network parameters, such as network bandwidth; second, environmental sensing parameters, such as ambient temperature and humidity; and third, camera encoding parameters, such as lens cleanliness and video encoding parameters.

[0022] Based on this, a multimodal feature representation corresponding to the multimodal data sequence is constructed, including: performing feature encoding on the multimodal data sequence based on a pre-constructed multimodal encoding module to obtain multimodal features to be processed; and performing spatiotemporal feature alignment processing on each multimodal feature to be processed to obtain a multimodal feature representation.

[0023] The multimodal coding module refers to a plug-in or tool for feature encoding of multimodal data sequences. In this technical solution, the multimodal data sequence includes a video frame sequence with temporal features and a device status data sequence of the target device. Based on this, the multimodal coding module can include two independent feature encoders: a first encoder for image feature encoding and a second encoder for feature encoding of the device status data sequence. The first encoder is suitable for feature encoding of image sequences, and the second encoder is suitable for feature encoding of device status data sequences.

[0024] Specifically, the first encoder can be applied to two main scenarios. First, when the encoding object is a sequence of continuous video frames directly decomposed from the video to be detected, for example, the first encoder can adopt a 3D-CNN structure. After encoding the continuous video frames of the video to be detected, the output is image feature information with a feature dimension of 1024. Second, when the encoding object is an image sequence with temporal features obtained by filtering video frames based on the video frames to be detected, the first encoder can adopt an improved ResNet-50 structure, in which case the output is image feature information with a feature dimension of 2048. The second encoder uses a fully connected network structure when encoding device status data sequences. The input is standardized sensor data, and the output is device feature information with a feature dimension of 128.

[0025] Based on this, the multimodal data sequences of each dimension are fused through a cross-modal attention mechanism to obtain the multimodal feature representation corresponding to the multimodal data sequence.

[0026] S120. Input the multimodal feature representation into the pre-trained first model, and output at least one abnormal video frame in the video to be detected based on the first model.

[0027] The first model is a lightweight student model obtained by inputting historical multimodal data sample sets into a pre-trained teacher model and a student model to be trained, respectively, and then converging after knowledge distillation training based on a pre-set distillation objective function within the training computation framework. The first model is used to identify anomalies in video frames of the video to be tested, thereby obtaining anomalous video frames. An anomalous video frame refers to a video frame whose image does not meet normal imaging standards due to abnormal equipment conditions or shooting environment of the target device. For example, an anomalous video frame may be a video frame in the video to be tested that is unclear in quality, incomplete in image, blurry, jittery, has color distortion, or is overexposed.

[0028] Specifically, the constructed multimodal feature representations are all input into the pre-constructed first model. This model is trained by knowledge distillation based on the multimodal historical data of labeled normal and abnormal samples. It simultaneously learns the visual features and supporting equipment status features corresponding to normal and various abnormal images. It can integrate image visual features and equipment operating condition features to achieve multi-dimensional joint discrimination. It uses the feature rules learned from multimodal data sequences of different dimensions to perform inference, so as to automatically identify abnormal video frames in the video to be detected based on the first model.

[0029] S130. Based on the pre-trained second model, perform root cause localization processing on each abnormal video frame and generate a root cause analysis report.

[0030] The second model is the root cause localization model after training convergence. It locates the root cause of faults in root cause-labeled samples based on structured causal graphs, intervention reasoning, and counterfactual analysis. The input data for the second model is a root cause-labeled sample set, which includes multiple root cause-labeled samples. Each root cause-labeled sample consists of anomaly images and their root cause information. The anomaly images are those identified and output by the trained first model, and the root cause information is obtained through manual annotation.

[0031] In practical applications, the system performs deep training of the second model during the training phase by constructing a "perception-cognition" collaborative closed loop. First, after receiving input containing multimodal feature representations of anomalous images and their corresponding device status data, the first model accurately captures at least one anomalous image from the video to be detected. Subsequently, an expert or manual annotation system combines the anomalous image with the device's operating status at the time (such as sensor parameters or control commands) to perform multi-dimensional customized annotations on the underlying causes of the visual anomalies, thereby constructing a high-quality root cause annotation sample set. This fusion of multimodal information provides a solid data foundation for subsequently revealing the deep connection between "visual appearance" and "physical essence."

[0032] Furthermore, the aforementioned root cause-labeled sample set is input into the second model, driving it to learn complex fault evolution logic. During training, the second model does not simply fit image features, but relies on structured causal graphs to analyze the dependencies between elements in the samples, and combines advanced cognitive mechanisms such as intervention reasoning and counterfactual analysis to deduce the true causal chain between equipment state changes and image anomalies. After sufficient training and convergence, the second model possesses the ability to reverse-engineer complex root causes from a single abnormal image.

[0033] Based on this, in practical applications, the second model can directly perform accurate root cause analysis on new abnormal video frames output by the first model. Optionally, the root cause analysis report output by the second model can not only cover the root cause analysis results of each abnormal video frame, but also, according to actual needs, further connect the temporal information between each abnormal video frame to provide cross-frame global root cause tracing or fault evolution trend analysis, thereby providing more in-depth guidance for operation and maintenance decisions.

[0034] Optionally, it also includes: generating a pending work order based on the root cause analysis report of all abnormal video frames, and sending the pending work order to the target processing platform.

[0035] The pending work order can be understood as a maintenance work order for the target device. The pending work order includes at least the video to be detected to which the abnormal video frame belongs, the target device of the video to be detected, the abnormality category of the abnormal video frame, the root cause analysis results, and processing suggestions. The target processing platform can be understood as a control platform used to prompt the user to perform maintenance on the target device corresponding to the abnormal video frame.

[0036] For example, when an abnormal video frame appears in the video to be detected, it is usually a sign of poor video quality from the target device. After obtaining the root cause analysis report, which includes information such as the target device requiring maintenance and potential device malfunctions, a work order can be generated based on the report. This allows users to perform maintenance on the target device according to the work order. For instance, the root cause analysis report might indicate that an abnormal video frame has a completely black screen. The root cause analysis result is that the target device's camera, which captured the abnormal video frame, has a malfunctioning infrared light, causing the screen to be completely black at night. Based on this, a work order is generated and dispatched to the target processing platform. The work order should include at least the anomaly category of the abnormal video frame, the corresponding target device, the root cause analysis result, and processing suggestions. For example, the processing suggestion might be "Replace the infrared light of the XX camera." Maintenance personnel can then perform maintenance on the target device based on this suggestion and provide feedback after processing: "Infrared light replaced, video restored. This model of light is nearing the end of its lifespan; batch inspection is recommended." This resolves the current fault and provides data for predictive maintenance, forming a closed-loop maintenance system.

[0037] In a specific example, a hybrid cloud deployment architecture is adopted to balance real-time performance and data security requirements. The cloud deployment includes a model training center and an intelligent operation and maintenance management suite. The model training center can be built on a GPU cluster for executing ADCT algorithm training and meta-course pre-training. The intelligent operation and maintenance management suite includes functions such as work order management and performance statistics, supporting multi-tenant and multi-role access control. Edge node deployment involves deploying edge servers in various administrative areas of the city, with each edge node deploying a lightweight inference engine and a multimodal data preprocessing module. A single edge node supports real-time analysis of 1000 channels of 1080P video, achieving horizontal scaling through load balancing. Terminal device integration includes front-end cameras connecting to edge nodes via mainstream communication protocols, synchronously uploading video streams and device status data; data transmission uses existing mainstream communication protocols, such as MQTT, to ensure data reliability in unstable network scenarios. Data preparation and model training can be based on the Q-Vid15 dataset to train a multimodal contrastive quality representation learning model (including spatiotemporal adversarial distillation) and a causal intervention diagnostic network. Edge node deployment and testing included deploying a lightweight inference engine on edge servers in each region, configuring data receiving interfaces and protocol conversion modules; the processing latency for a single 1080P video stream was tested to be 42ms, and there was no performance degradation when a single node concurrently processed 1000 video streams.

[0038] The technical solution of this invention involves acquiring a multimodal data sequence and constructing a multimodal feature representation corresponding to the multimodal data sequence; inputting the multimodal feature representation into a pre-trained first model and outputting at least one abnormal video frame in the video to be detected based on the first model; and performing root cause localization processing on each abnormal video frame based on a pre-trained second model to generate a root cause analysis report. In this technical solution, a multimodal data sequence corresponding to a video shooting device is acquired. The multimodal data sequence includes at least a video frame sequence of the video to be detected obtained by the video shooting device or an image sequence with temporal information extracted from the video to be detected, as well as a device status data sequence of the video shooting device. Feature extraction is then performed on the multimodal data sequence to obtain a multimodal feature representation. Furthermore, anomalies in the multimodal feature representation are identified using a pre-constructed first model. Abnormal video frames are then extracted from the video to be detected based on these multimodal feature representations. These abnormal video frames are then input into a second model with root cause analysis capabilities. Root cause analysis is performed on the abnormal information in these abnormal video frames using the second model. The failure points of the video shooting equipment are determined based on the root cause analysis results of each abnormal video frame, and the root cause analysis report serves as the basis for maintaining the video shooting equipment. This solves the problem in existing technologies where it is difficult to accurately locate the root cause of video shooting equipment failures when video quality is poor, leading to delays in timely maintenance. It achieves the effect of timely identification of abnormal video frames in the video when video shooting equipment produces poor quality footage, and accurately locating the root cause of the video shooting equipment failure based on these abnormal video frames, thus providing accurate maintenance suggestions for the video shooting equipment.

[0039] Example 2 Figure 2 The flowchart of a root cause analysis method for abnormal video frames provided in Embodiment 2 of the present invention includes, optionally, the following steps before acquiring the multimodal data sequence and constructing the multimodal feature representation corresponding to the multimodal data sequence: pre-constructing a first model and a second model.

[0040] like Figure 2 As shown, the method includes: S210, Pre-build the first and second models.

[0041] During the training phase, a first model is pre-built, including: acquiring a historical multimodal data sample set under the target business scenario; inputting the historical multimodal data sample set into the pre-trained teacher model and the student model to be trained, respectively, and training the student model based on the distillation objective function pre-deployed in the training computing framework to obtain the first model.

[0042] The historical multimodal data sample set includes multiple historical multimodal positive samples and multiple historical multimodal negative samples. Each historical multimodal sample is generated after extracting multimodal features based on the historical multimodal data sequence. The first model is a lightweight student model that has been trained and converged.

[0043] In a specific example, to ensure the effectiveness of model training and inference, a large-scale labeled dataset Q-Vid15 was constructed, containing 100,000 videos and covering 15 types of quality anomalies (blur, occlusion, overexposure, jitter, color distortion, etc.), as the basic data support for model training and validation.

[0044] Specifically, before constructing the first and second models, the acquired video to be detected undergoes data preprocessing through the following steps. The data preprocessing stage is performed in three steps to ensure the validity and consistency of the multimodal data: (1) Video and image preprocessing: A sampling frequency of 30 frames per second is obtained to ensure the integrity of the time series information; image frames are normalized to map pixel values ​​to the [0,1] interval, and histogram equalization is used to enhance contrast and improve the model's sensitivity to quality degradation; at the same time, a spatiotemporal feature alignment operation is introduced, and the spatiotemporal dependency relationship between features is constructed through the spatiotemporal graph structure extraction function \mathcal{G}(\cdot) to solve the problem of spatiotemporal feature misalignment in traditional processing. The spatiotemporal alignment loss is defined as: in, Let Ft represent the spatiotemporal alignment loss, and Fs represent the features extracted by the teacher model and the student model, respectively.

[0045] (2) Equipment status data preprocessing: The Z-score standardization method is used to convert sensor data into a distribution with a mean of 0 and a standard deviation of 1, eliminating the impact of dimensional differences on model training; for parameter differences of different brands and models of equipment, a normalization step for equipment parameters is added to ensure the compatibility of data from multiple types of equipment. The Z-score standardization calculation formula is as follows: in, This represents the normalization result. x These are the original equipment parameters. μ These are the average parameters for similar equipment. σ This represents the standard deviation of the corresponding parameter.

[0046] (3) Multimodal data alignment: A Transformer-based Self-Attention cross-modal attention mechanism is introduced to calculate the similarity matrix between image frame features, video temporal features, and device state features. Key fusion features are extracted using a max-pooling strategy to ensure the comparability of different modal data in the semantic space, providing high-quality input data for subsequent comparative learning and knowledge distillation. The cross-modal attention weight calculation expression is as follows: Where Q, K, and V represent the image modality query vector, the device status modality key vector, and the multimodal value vector, respectively, d k This is the scaling factor for the feature dimension.

[0047] Based on this, various types of data are encoded using multimodal encoders. For example, an image encoder with an improved ResNet-50 structure outputs a feature dimension of 2048; a video encoder with a 3D-CNN structure takes a continuous frame sequence as input and outputs a feature dimension of 1024; and a device status encoder with a fully connected network structure takes standardized sensor data as input and outputs a feature dimension of 128. After the three types of features are fused through a cross-modal attention mechanism, a unified multimodal feature representation is generated. Contrastive learning training: Positive and negative sample pairs are constructed for model training. Positive samples are defined as enhanced versions of the same image at different quality levels (e.g., the original sharp image and an image after Gaussian blurring), while negative samples are defined as images at different quality levels (e.g., a sharp image and a heavily compressed image). The contrastive loss function is used to optimize the model parameters, with the following formula: in, This represents the function for calculating cosine similarity. The temperature parameter (usually 0.07) is used as a loss function to teach the model features that are insensitive to changes in image content but sensitive to quality degradation.

[0048] Spatiotemporal Adversarial Distillation (STAD): This approach introduces a teacher-student model architecture, where the teacher model is a high-precision, complex model (89.7M parameters), and the student model is a lightweight model to be optimized (i.e., the first model). Knowledge transfer from the teacher model to the student model is achieved through minimax game theory. The distillation objective function is: in, For the loss of video quality inspection task, For spatiotemporal alignment loss: Where D is the discriminator, used to distinguish the feature distributions of the teacher model and the student model, and α and β are the balancing hyperparameters (with values ​​of 0.5 and 0.3, respectively). Ultimately, the detection accuracy of the student model is close to that of the teacher model, and the computational cost is reduced by 80%.

[0049] Optionally, before acquiring the multimodal data sequence and constructing the multimodal feature representation corresponding to the multimodal data sequence, the method further includes: pre-constructing a second model.

[0050] The process of pre-constructing the second model includes: obtaining a root cause labeled sample set; and constructing and training a causal intervention diagnostic network based on the root cause labeled sample set to obtain the second model.

[0051] The root cause annotation sample set includes multiple root cause annotation samples, each consisting of anomaly images and their root cause information. The anomaly images are those identified and output by the trained first model, while the root cause information is obtained through manual annotation. The second model is a converged root cause localization model that uses structured causal graphs, intervention reasoning, and counterfactual analysis to locate the root cause of faults within the root cause annotation samples. Building upon the example above, this module is constructed based on the Causal Intervention Diagnostic Network (CIDN), replacing the traditional Bayesian network, and achieving a technological leap from "anomaly detection" to "root cause localization." The specific design is as follows: Structured causal model construction: Define a causal graph G=(\mathcal{V},\mathcal{E}), where the node set\mathcal{V}=\{E,S,C,R\} (E represents environmental factors, such as light intensity, rain and snow; S represents image symptoms, such as blur, overexposure, color cast; C represents device status, such as lens cleanliness, focal length parameters, network bandwidth; R represents the root cause of quality abnormalities), and the edge set\mathcal{E} represents the causal relationship between nodes (such as "lens damage → blur" "insufficient network bandwidth → frame rate decrease").

[0052] Intervention reasoning and counterfactual analysis: For the detected abnormal symptom S=s, the contribution of each potential root cause to the abnormality is calculated using Do-Calculus theory, which can be expressed by the following formula: in, This indicates the contribution of the underlying root cause to the anomaly, where C represents the root cause category random variable; Let S represent the observed variable of abnormal symptoms and E represent the intermediate latent state variable, which are candidate root causes of type i. Let e ​​represent the posterior probability of taking the intermediate hidden state e when an anomaly s is observed.

[0053] At the same time, a counterfactual diagnosis is performed to assess "if the current root cause is considered..." Replace with "Changes in abnormal symptoms", the formula is: in, This represents the change in the abnormal contribution of the counterfactual diagnostic factor after replacing the root cause ci with cj. This represents the counterfactual hypothesis: if the current anomaly is solely caused by root cause cj, what is the root cause contribution probability observed at time s? Represents the real-world baseline, where ci is the current root cause and the root cause contribution probability when s is observed. Indicates the original root cause. This indicates the candidate root cause to be replaced.

[0054] The final output includes a root cause ranking list (including the percentage contribution of each root cause) and targeted repair suggestions (such as "lens contamination contributes 82%, it is recommended to clean the lens"). The root cause location accuracy can reach 89.3%.

[0055] Based on this, if the number of samples in the historical multimodal data sample set is less than the preset number of samples, the sample training difficulty coefficient corresponding to each historical multimodal sample is determined; according to the sample training difficulty coefficient, the student model is controlled to train the historical multimodal sample set from easy to difficult, and the parameters of the student model are fine-tuned based on the distillation objective function to update the first model.

[0056] Specifically, the module adopts the Meta-Course Learning (MCL) framework to replace the traditional MAML (Model-Agnostic Meta-Learning), achieving "rapid adaptation to small samples" in new scenarios. The core design is as follows: Course difficulty measurement: Based on the feature definition of the task loss surface, the sample difficulty is quantified and graded, and the formula is: In the formula, Let x be the gradient of the loss function with respect to sample x. This is the Hessian matrix (reflecting the difficulty of sample training). This is the balancing parameter (value 0.1).

[0057] Meta-course optimization: During the pre-training phase, multi-scenario datasets (such as urban security, factory workshops, traffic intersections, etc.) are collected, treating each scenario as an independent task; by optimizing the course scheduling strategy pi, the model achieves adaptive learning "from easy to difficult," with the optimization objective being: in, This indicates the course scheduling strategy distribution, controlling the sampling order of training samples from "easy to difficult". Indicates the distribution of tasks in the scene. Take the expected value above. This represents the optimal model parameters obtained by minimizing the training loss, given a course strategy π. This indicates that the expected training loss is taken on the sample set sampled by policy π. This represents the training set loss when the input sample is x and the parameter is θ.

[0058] This module is based on Neural Architecture Search (NAS) technology and employs the ParetoFrontier search strategy to find the optimal balance between model detection accuracy and computational complexity. The specific design is as follows: Model structure: The final generated MobileNetV3 structure has only 1.2M parameters (only 4.7% of the traditional ResNet-50 model) and 1.5G of computation (only 1 / 5 of the traditional method); it supports deployment on x86 architecture servers and domestic platforms and is compatible with the resource constraints of edge devices.

[0059] Performance metrics: A single node supports real-time concurrent analysis of 1000 channels of 1080P video, with a single-channel video processing latency of less than 50ms and an inference speed of 65fps (more than 3 times faster than traditional methods); the model's false alarm rate is controlled within 2%, meeting the real-time and accuracy requirements of city-level inspection of tens of thousands of video channels.

[0060] Building upon this, this module employs Neural Architecture Search (NAS) technology and the ParetoFrontier search strategy to find the optimal balance between model detection accuracy and computational complexity. The specific design is as follows: Model structure: The final generated MobileNetV3 structure has only 1.2M parameters (only 4.7% of the traditional ResNet-50 model) and 1.5G of computation (only 1 / 5 of the traditional method); it supports deployment on x86 architecture servers and domestic platforms and is compatible with the resource constraints of edge devices.

[0061] Performance metrics: A single node supports real-time concurrent analysis of 1000 channels of 1080P video, with a single-channel video processing latency of less than 50ms and an inference speed of 65fps (more than 3 times faster than traditional methods); the model's false alarm rate is controlled within 2%, meeting the real-time and accuracy requirements of city-level inspection of tens of thousands of video channels.

[0062] In a specific example, multimodal data (image frames, video streams, device status data) and the large-scale labeled dataset Q-Vid15 are input into the student model to be trained. The teacher model T, student model S, discriminator D, and course scheduler π are initialized. Based on the course scheduler π, sample batches are selected from the dataset (following a "from easy to difficult" sample selection strategy). The samples are then input into the multimodal encoders of the teacher model T and student model S respectively, and features F_t (teacher model) and F_s (student model) are extracted. The parameters of the student model S are frozen, and the parameters of the discriminator D are updated: maximizing the adversarial loss. in, Represents the adversarial branching loss in the spatiotemporal adversarial distillation structure. The multimodal fusion features output by the high-precision teacher model T with 89.7M parameters. To reduce the multimodal fusion features output by the lightweight student model S, D() is a feature discriminator network that distinguishes whether the input features originate from the teacher or student model. These represent the expected values ​​of the output feature sets for all teachers and students, respectively. , These are used to maximize the probability that the discriminator correctly identifies the characteristics of teachers and students, respectively. Freeze the parameters of the discriminator D and update the parameters of the student model S: minimize the total loss of spatiotemporal adversarial distillation. in, The overall objective loss function for spatiotemporal adversarial distillation is... This represents a loss in the fundamental task of video quality inspection. For multimodal spatiotemporal alignment loss, It is a spatiotemporal alignment loss balancing hyperparameter with a value of 0.5. It is the adversarial loss balancing hyperparameter with a value of 0.3.

[0063] Iteratively train the model until it converges (the F1 score on the validation set is consistently greater than 96%), save the optimized student model parameters, and obtain the first model.

[0064] Load the pre-trained causal graph model G=(V,E) and the multimodal quality detection model, and input the detection results of the real-time video stream (including abnormal symptoms S=s, such as "blur") and the corresponding camera device status data C and environmental data E. Calculate the intervention effect P(R\mid\text{do}(C=c_i),S=s) of each potential root cause c_i using Do-Calculus theory, and calculate CF_{c_i\toc_j} through counterfactual analysis; sort the root causes in descending order of intervention effect value, and generate a diagnostic report containing "root cause name - contribution percentage - repair suggestion"; push the diagnostic report to the intelligent operation and maintenance management suite, automatically generate maintenance work orders, and assign them to the corresponding operation and maintenance personnel.

[0065] The process involves loading the pre-trained base model and initial course scheduling strategy \pi; collecting a small number of samples (5-10 frames, including normal and abnormal scenes) from the new scene, and calculating the difficulty value of each sample based on the difficulty metric formula d(x); updating the course scheduling strategy \pi based on the sample difficulty values, and generating a training sample batch specific to the new scene. A small learning rate (\alpha=0.001) is used to fine-tune the base model parameters. in, The loss function for the new scene samples is used to evaluate the performance of the fine-tuned model on the new scene validation set. If the F1 score is greater than 95%, the adaptation is complete; otherwise, 2-3 additional frames are added and training is repeated.

[0066] S220. Obtain multimodal data sequences and construct multimodal feature representations corresponding to the multimodal data sequences.

[0067] S230. Input the multimodal feature representation into the pre-trained first model, and output at least one abnormal video frame in the video to be detected based on the first model.

[0068] S240. Based on the pre-trained second model, perform root cause localization processing on each abnormal video frame and generate a root cause analysis report.

[0069] The technical solution of this invention further includes, before acquiring the multimodal data sequence and constructing the multimodal feature representation corresponding to the multimodal data sequence, pre-constructing a first model and a second model. The first model has the function of identifying abnormal video frames in a video, and is a lightweight student model after training convergence. The second model is a model obtained by training based on the abnormal images output by the first model through root cause analysis. It can be used to perform root cause analysis on abnormal video frames and output a root cause analysis report to provide a maintenance basis for the equipment fault maintenance of the video shooting device that obtained the video. In this solution, the first model is trained based on a distillation objective function pre-deployed within the training computation framework to make the accuracy of the student model in identifying abnormal video frames close to the accuracy of the teacher model in identifying abnormal video frames. The lightweight model after training convergence is used as the first model. Furthermore, through the accurate identification of abnormal video frames in the video by the first model, abnormal video frames can be obtained, and then the second model performs further root cause analysis on the abnormal video frames. This invention solves the problem in existing technologies where it is difficult to accurately locate the root cause of video shooting equipment failure when the video quality is poor, thus hindering timely maintenance. It enables timely identification of abnormal video frames in the video when the video shooting equipment produces poor quality footage, and accurately locates the root cause of the failure based on these abnormal video frames, thereby providing accurate maintenance suggestions for the video shooting equipment.

[0070] Example 3 Figure 3 This is a schematic diagram of the structure of a root cause analysis device for abnormal video frames provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a construction module 310, used to acquire a multimodal data sequence and construct a multimodal feature representation corresponding to the multimodal data sequence; wherein, the multimodal data sequence includes a video frame sequence of the video to be detected, and a device status data sequence of the target device for the video to be detected; The detection module 320 is used to input multimodal feature representations into a pre-trained first model and output at least one abnormal video frame in the video to be detected based on the first model. The report generation module 330 is used to perform root cause localization processing on each abnormal video frame based on a pre-trained second model and generate a root cause analysis report.

[0071] The technical solution of this invention involves acquiring a multimodal data sequence and constructing a multimodal feature representation corresponding to the multimodal data sequence; inputting the multimodal feature representation into a pre-trained first model and outputting at least one abnormal video frame in the video to be detected based on the first model; and performing root cause localization processing on each abnormal video frame based on a pre-trained second model to generate a root cause analysis report. In this technical solution, a multimodal data sequence corresponding to a video shooting device is acquired. The multimodal data sequence includes at least a video frame sequence of the video to be detected obtained by the video shooting device or an image sequence with temporal information extracted from the video to be detected, as well as a device status data sequence of the video shooting device. Feature extraction is then performed on the multimodal data sequence to obtain a multimodal feature representation. Furthermore, anomalies in the multimodal feature representation are identified using a pre-constructed first model. Abnormal video frames are then extracted from the video to be detected based on these multimodal feature representations. These abnormal video frames are then input into a second model with root cause analysis capabilities. Root cause analysis is performed on the abnormal information in these abnormal video frames using the second model. The failure points of the video shooting equipment are determined based on the root cause analysis results of each abnormal video frame, and the root cause analysis report serves as the basis for maintaining the video shooting equipment. This solves the problem in existing technologies where it is difficult to accurately locate the root cause of video shooting equipment failures when video quality is poor, leading to delays in timely maintenance. It achieves the effect of timely identification of abnormal video frames in the video when video shooting equipment produces poor quality footage, and accurately locating the root cause of the video shooting equipment failure based on these abnormal video frames, thus providing accurate maintenance suggestions for the video shooting equipment.

[0072] Optionally, the root cause analysis device for abnormal video frames further includes: a first model building module for pre-building a first model; wherein, The first model construction module includes: a first acquisition unit, used to acquire a historical multimodal data sample set under the target business scenario before acquiring the multimodal data sequence and constructing the multimodal feature representation corresponding to the multimodal data sequence; wherein, the historical multimodal data sample set includes multiple historical multimodal positive samples and multiple historical multimodal negative samples, and each historical multimodal sample is a sample generated after extracting multimodal features based on the historical multimodal data sequence; The first training unit is used to input the historical multimodal data sample set into the pre-trained teacher model and the student model to be trained, respectively, and to train the student model based on the distillation objective function pre-deployed in the training computing framework to obtain the first model; wherein, the first model is the lightweight student model after training convergence.

[0073] Optionally, the root cause analysis device for abnormal video frames further includes: a second model building module for pre-building a second model; The second model construction module includes: a second acquisition unit, used to acquire a root cause labeling sample set; wherein the root cause labeling sample set includes multiple root cause labeling samples, the root cause labeling samples consist of abnormal images and root cause information of the abnormal images, the abnormal images are abnormal images identified and output by the first model after training, and the root cause information is obtained through manual labeling. The second training unit is used to construct and train a causal intervention and diagnosis network based on the root cause labeled sample set to obtain the second model; wherein, the second model is the root cause localization model after training convergence, and the second model locates the root cause of the fault based on the root cause labeled sample based on structured causal graph, intervention reasoning and counterfactual analysis.

[0074] Optionally, the first model building module includes: a difficulty coefficient determination unit, used to determine the sample training difficulty coefficient corresponding to each historical multimodal sample if the number of samples in the historical multimodal data sample set is less than the preset number of samples. The update unit is used to control the student model to train on the historical multimodal sample set in order of increasing difficulty according to the sample training difficulty coefficient, and to fine-tune the parameters of the student model based on the distillation objective function in order to update the first model.

[0075] Optionally, the building module includes: a multimodal feature determination unit, used to perform feature encoding on the multimodal data sequence based on a pre-built multimodal encoding module to obtain multimodal features to be processed; The multimodal feature representation determination unit is used to perform spatiotemporal feature alignment processing on each multimodal feature to be processed to obtain a multimodal feature representation.

[0076] Optionally, the root cause analysis device for abnormal video frames is also used to generate a work order to be processed based on the root cause analysis report of all abnormal video frames, and send the work order to be processed to the target processing platform.

[0077] The root cause analysis device for abnormal video frames provided in this embodiment of the invention can execute the root cause analysis method for abnormal video frames provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0078] Example 4 Figure 4A schematic diagram of the structure of an electronic device 10 according to an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0079] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0080] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0081] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as root cause analysis methods for abnormal video frames.

[0082] In some embodiments, the root cause analysis method for abnormal video frames may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the root cause analysis method for abnormal video frames described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the root cause analysis method for abnormal video frames by any other suitable means (e.g., by means of firmware).

[0083] Various implementations of the systems and techniques described above herein can be implemented in digital circuit systems, integrated circuits, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0084] Computer programs used for implementing the root cause analysis method for abnormal video frames of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0085] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0086] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0087] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0088] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0089] Example 5 This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the root cause analysis method for abnormal video frames as provided in any embodiment of this application.

[0090] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0091] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0092] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A root cause analysis method for abnormal video frames, characterized in that, include: A multimodal data sequence is acquired, and a multimodal feature representation corresponding to the multimodal data sequence is constructed; wherein, the multimodal data sequence includes a video frame sequence of the video to be detected, and a device status data sequence of the target device of the video to be detected; The multimodal feature representation is input into a pre-trained first model, and at least one abnormal video frame in the video to be detected is output based on the first model. The root cause analysis report is generated by performing root cause localization processing on each abnormal video frame based on the pre-trained second model.

2. The method according to claim 1, characterized in that, Before acquiring the multimodal data sequence and constructing the multimodal feature representation corresponding to the multimodal data sequence, the method further includes: A first model is pre-built; where, The pre-built first model includes: Obtain a historical multimodal data sample set under the target business scenario; wherein, the historical multimodal data sample set includes multiple historical multimodal positive samples and multiple historical multimodal negative samples, and each historical multimodal sample is a sample generated after extracting multimodal features based on the historical multimodal data sequence; The historical multimodal data sample set is input into the pre-trained teacher model and the student model to be trained, respectively. The student model is trained based on the distillation objective function pre-deployed in the training computing framework to obtain the first model; wherein, the first model is a lightweight student model after training convergence.

3. The method according to claim 2, characterized in that, Before acquiring the multimodal data sequence and constructing the multimodal feature representation corresponding to the multimodal data sequence, the method further includes: A second model is pre-built; where, The pre-built second model includes: Obtain a root cause labeling sample set; wherein, the root cause labeling sample set includes multiple root cause labeling samples, the root cause labeling samples consist of abnormal images and root cause information of the abnormal images, the abnormal images are abnormal images identified and output by the first model after training, and the root cause information is obtained through manual labeling; A causal intervention and diagnosis network is constructed and trained based on the root cause labeled sample set to obtain a second model; wherein, the second model is a root cause localization model after training convergence, and the second model performs fault root cause localization on the root cause labeled sample based on structured causal graph, intervention reasoning and counterfactual analysis.

4. The method according to claim 2, characterized in that, Also includes: If the number of samples in the historical multimodal data sample set is less than the preset number of samples, then the sample training difficulty coefficient corresponding to each historical multimodal sample is determined. Based on the training difficulty coefficient of the samples, the student model is trained on the historical multimodal sample set in order of increasing difficulty, and the parameters of the student model are fine-tuned based on the distillation objective function to update the first model.

5. The method according to claim 1, characterized in that, The construction of the multimodal feature representation corresponding to the multimodal data sequence includes: The multimodal data sequence is feature-encoded based on a pre-built multimodal coding module to obtain multimodal features to be processed; The multimodal features to be processed are aligned in time and space to obtain the multimodal feature representation.

6. The method according to claim 1, characterized in that, Also includes: Based on the root cause analysis report of all abnormal video frames, a work order to be processed is generated and sent to the target processing platform.

7. A root cause analysis device for abnormal video frames, characterized in that, include: A construction module is used to acquire a multimodal data sequence and construct a multimodal feature representation corresponding to the multimodal data sequence; wherein, the multimodal data sequence includes a video frame sequence of the video to be detected and a device status data sequence of the target device of the video to be detected; The detection module is used to input the multimodal feature representation into a pre-trained first model and output at least one abnormal video frame in the video to be detected based on the first model. The report generation module is used to perform root cause localization processing on each abnormal video frame based on a pre-trained second model and generate a root cause analysis report.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the root cause analysis method for abnormal video frames according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the root cause analysis method for abnormal video frames according to any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the root cause analysis method for abnormal video frames as described in any one of claims 1-6.