A method and system for monitoring video stream quality monitoring, repair and early warning report

Abstract

The application discloses a kind of monitoring video stream quality real-time monitoring, repair and early warning report generation method and system for complex environment of airport warehouse area.The special degradation sample data set of warehouse area is constructed, and the comparative learning of ambient light weight and perceptual similarity is carried out by introducing the pre-training module of no-reference image quality assessment;In the joint training stage, the multi-loss optimization function based on regression is used, combined with motion compensation coefficient and business detail mask, to realize the bidirectional supervision and collaborative convergence of repair network and evaluation network.The output repair image and MOS are output during real-time inference, and the early warning is triggered based on dynamic threshold.Through the multi-modal large language model driven by MOS, the inspection report containing degradation type, location and impact degree is automatically generated by combining the image before and after repair and the difference feature map, and hidden dangers such as smoke and flame are actively identified.The application fills the technical gap from passive recording to active quantitative evaluation and semantic early warning in warehouse monitoring.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, and in particular to a method and system for real-time monitoring, repair, and early warning of the quality of large-space surveillance videos. Background Technology

[0002] With the intelligent upgrading of the logistics system, airport warehouses, as the core transit and distribution center for air cargo, are facing increasingly prominent importance in their security monitoring systems. They urgently need to transform from "simple recording" to "precise identification, real-time repair, and intelligent early warning." Airport warehouses have unique physical attributes that significantly distinguish them from ordinary scenarios, mainly reflected in their ultra-large spatial scale, high-frequency mechanical operations (continuous movement of tractors, forklifts, and other equipment), and complex lighting environments around the clock (strong backlight, alternating dark and dark light, and drastic changes in day and night lighting). These unique characteristics bring numerous technical challenges to the real-time processing, quality assurance, and security early warning of surveillance videos. Traditional general-purpose surveillance video processing technologies are no longer sufficient to meet practical application needs.

[0003] In terms of video surveillance quality restoration, the movement paths of tractors, forklifts, and other operational targets within the warehouse area are highly nonlinear and random, easily resulting in severe unnatural texture blurring and image ghosting during long-distance imaging. Simultaneously, image distortion occurs during long-distance zooming of the camera, further exacerbating image quality degradation. Existing image deblurring algorithms are mostly designed based on linear motion assumptions, which cannot adapt to the nonlinear, high-speed movement characteristics of operational targets in the warehouse area. They struggle to effectively restore the edge details of moving targets, leading to the loss of crucial evidence information such as cargo labels and vehicle license plates, failing to provide reliable support for security traceability and business verification. Furthermore, traditional contrast enhancement techniques often have significant limitations in addressing the frequent alternation between strong backlighting areas and dark, low-light environments prevalent in warehouse areas. While increasing image brightness, they amplify image noise and fail to effectively extract visual features affected by extreme lighting conditions. Especially in identifying subtle anomalies such as smoke in the early stages of a fire or cargo spillage, existing technologies lack sufficient robustness and are prone to missed or false positives.

[0004] In terms of monitoring quality assessment and security early warning, the current airport storage area monitoring evaluation system relies excessively on manual review, which is inefficient and highly subjective. It lacks a quantitative correlation mechanism from the underlying image degradation mechanism to high-level business security risks, making it impossible to achieve automated and intelligent quality assessment and risk early warning. Although some deep learning-based image quality assessment methods have been applied to the monitoring field, their output is only a single numerical score, lacking intuitive semantic interpretation. They cannot establish a correlation between the degree of underlying image degradation and high-level security business needs (such as the availability of key detail recognition and the reliability of hazard identification), making it difficult to directly support automated inspection report generation and multi-dimensional risk early warning.

[0005] More critically, in existing technologies, image deblurring and restoration networks and image quality assessment networks often employ independent training modes. The lack of collaborative constraints and bidirectional supervision between the two leads to a disconnect between the restoration network's performance and the assessment network's quality judgment standards. This results in uncontrollable restoration quality and low fidelity in restoring key business details. This technological gap between surveillance video quality monitoring, automated restoration and restoration, and intelligent anomaly analysis directly restricts the real-time performance, accuracy, and reliability of airport warehouse security early warning systems. It fails to meet the high-standard security requirements of airport warehouses for cargo and operational safety, necessitating a surveillance video processing technology that can adapt to the unique environment of warehouses and achieve end-to-end collaborative optimization to overcome these technological bottlenecks. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a method for monitoring, repairing, and reporting the quality of surveillance video streams. This method solves the problems of uncontrollable repair quality and low restoration accuracy of key business details (such as license plates and cargo labels) caused by the independent training of traditional deblurring networks and quality assessment networks when facing severe image ghosting caused by the nonlinear high-speed movement of tractors and forklifts in ultra-large spatial scales, as well as the large-area loss of visual features caused by extreme lighting (strong backlight, dark light).

[0007] The present invention is achieved through the following technical solutions.

[0008] A method for monitoring, repairing, and reporting the quality of surveillance video streams includes: Construct a multi-type degradation sample dataset specifically for large-scale imaging degradation scenarios in reservoir areas; The no-reference image quality assessment network was pre-trained using the dataset. The pre-trained quality assessment network and the monitoring and repair network are jointly trained. During the joint training phase, the repair network and the quality assessment network are achieved through a loss function to achieve collaborative convergence and bidirectional feature supervision. The relative quality score is output based on the feature space distance to obtain the jointly trained network. Extract real-time monitoring video stream frames and input them into the jointly trained network to synchronously generate restored images with motion blur removed, and output the average subjective opinion score based on the relative quality score; The average subjective opinion score is converted into prior cue words, which, together with the image features of the repaired image and the original video stream frame, are input into a multimodal large language model to automatically generate an inspection and operation analysis report that includes the degradation type, the location of occurrence, and the quantified degree of impact.

[0009] Furthermore, the multi-type degradation sample dataset includes: The system collects real-time monitoring video stream data in all weather conditions, including nonlinear motion blur of the target during long-distance imaging, geometric distortion caused by zooming at the telephoto end, and degradation characteristics caused by alternating backlight and low light. The sampled images are then graded and labeled according to the intensity of motion blur gradient and the type of ambient light interference.

[0010] Furthermore, the pre-training of the no-reference image quality assessment network includes: An ambient light distribution weight is introduced for the reservoir area. The weight is dynamically determined by the ratio of the mean value of the real-time light intensity sensor in the reservoir area to the local contrast of the image. For the sampled images in the reservoir area, the perceptual similarity metric between different blurred versions of the image is calculated by combining the ambient light distribution weight. Based on this, a quality perceptual contrast loss is constructed to optimize the quality assessment network, so that it establishes a quality perception boundary for specific degradation patterns in the reservoir area.

[0011] Among them, the perceptual similarity metric is calculated based on the image feature extraction results. The formula is:

[0012] in, This represents the calculated perceptual similarity metric; and These represent two different degraded versions generated from the i-th baseline image in the collected reservoir area monitoring image dataset after being degraded to different degrees. The letter i represents the image sample number, and the letters j and k represent the corresponding degraded version numbers, respectively. Represents the entire image pixel space, symbol This represents summing over all pixels in the entire image pixel space, where the letter 'p' represents the image pixel space. The specific pixel coordinate index in the text; This represents the comparison of brightness and structural feature similarity between the two different degraded versions at pixel p, calculated based on the feature similarity index. This represents the dynamic ambient light distribution weight introduced to address the complex light and shadow environment of the reservoir area, which is dynamically determined by the ratio of the mean value of the real-time light intensity sensor in the reservoir area to the local contrast of the image. Indicates at pixel point The phase consistency metric extracted is used to characterize the dimensionless structural saliency features of local image areas.

[0013] Construct a quality-perceived contrast loss based on the perceived similarity metric. The formula is:

[0014] Where N is the total number of images, i.e. the total number of baseline image samples collected for reservoir monitoring; D is the number of blurred versions, i.e. the total number of degraded versions generated after each baseline image is processed by different degrees of motion blur gradient, backlighting or geometric distortion and other degradation modes; the single-sample loss calculated based on the perceptual similarity metric specifically represents the contrastive learning loss calculated by fragment sampling and feature extraction for the i-th baseline image in the j-th blurred version.

[0015] Furthermore, the step of achieving collaborative convergence and bidirectional feature supervision between the repair network and the quality assessment network through a loss function, and outputting a relative quality score based on the feature space distance, includes: A regression-based multi-loss optimization function is introduced to simultaneously extract deep distribution features of the repaired image and the standard clear image during the joint training phase, and the relative quality score is output by calculating the feature space distance. When calculating the relative quality score, a motion compensation coefficient for the reservoir operation target is introduced, which is determined based on the target motion vector magnitude and motion sensitivity factor.

[0016] Calculate the relative quality score The formula is:

[0017] in, The relative quality score, which represents the calculated output value in the range of [0,1], is used to quantify the relative sharpness and visual credibility of an image. The input image sample to be evaluated specifically refers to the library-restored image or the corresponding standard clear image output by the deblurring network during the joint training or monitoring phase. d(x) represents a preset scaling constant used to adjust the sensitivity of the relative quality score to changes in feature space distance; d(x) represents the feature space statistical distance. The magnitude of the motion vector of the vehicle operating in the reservoir area is extracted by optical flow or target tracking. It is a motion-sensitive factor.

[0018] Furthermore, the regression-based multi-loss optimization function includes: The deblurring network loss introduces a mask-based local attention loss, where the masks include warehouse cargo label masks and operating vehicle license plate masks, to enhance the restoration of key business details. The quality assessment network loss is evaluated by treating standard clear images as positive samples and repaired images as negative samples for discrimination training. The deblurring network loss and the quality assessment network loss achieve collaborative convergence through adversarial iteration.

[0019]

[0020] in, This represents the reference quality score output by the quality assessment network for the input standard-resolution image S, where S is a standard-resolution image sample used as the training target; D(B) represents the predicted quality score output by the quality assessment network for the restored image D(B) after processing by the deblurring network D, where B is the input original blurred monitoring image frame, and D(B) fully represents the sharpened restored image output by the deblurring network. This represents the introduced local attention loss function, used to specifically constrain and force enhancement of core security and business details in the image; This represents the preset weight adjustment coefficient, which is used to dynamically adjust the balance between local attention loss and overall image quality loss in the total loss function; A mask representing the labels on goods in the warehouse area, used to accurately define and extract key areas of the labels on goods in the monitoring footage; y represents the mask of the license plate of the work vehicle, used to accurately define and extract the key detail areas of the license plate; y represents the auxiliary guiding variable used in the joint training framework to guide the optimization direction of the deblurring network. It is the probability distribution of samples drawn from the training dataset during joint training, used to calculate the weighted average of the quality bias across the entire dataset; Calculate quality assessment network loss The formula is:

[0021] in, represents the probability distribution of standard image samples drawn from the training dataset during joint training, used to calculate the average loss on the standard sharp images; y represents the true quality label corresponding to the standard sharp images, used to provide supervision and guidance during joint training. This represents the predicted quality score output by the quality assessment network for the input standard-clarity image S; This represents the predicted quality score output by the quality assessment network for the restored image D(B) processed by the deblurring network D, where B is the input original blurred surveillance image frame, and D(B) fully represents the sharpened restored image output by the deblurring network. Further, the output of the average subjective opinion score based on the relative quality score includes: The relative quality score is converted into an average subjective opinion score according to the mapping relationship determined by the clearest benchmark score and the most ambiguous benchmark score defined by the jointly trained network.

[0022] The mapping relationship formula is:

[0023] in, It is the average subjective opinion score of the mapping output, using a 0-100 score system to conform to the intuitive subjective visual perception of security personnel regarding the credibility of the image; It is the theoretical maximum quality score limit preset by the system, corresponding to the maximum boundary value of the objective score output by the quality assessment network under an ideal non-destructive acquisition environment; It is the minimum quality score threshold set according to the specific reservoir area monitoring environment, used to characterize the critical point of video stream availability to ensure basic security monitoring accuracy; It is a monitoring image of the current frame that is captured during the real-time operation phase and repaired by a deblurring network.

[0024] Furthermore, the automatic generation of inspection and operation analysis reports, which includes the type of degradation, location of occurrence, and quantified impact, includes: The quantified average subjective opinion score is converted into natural language prompts that represent the severity of ambiguity and the level of business availability, and then combined with the preset baseline warning task description to form compound prompt words. The composite prompt word, along with the images before and after repair and their difference feature maps, are input into the multimodal large language model to output a structured report.

[0025] Furthermore, it also includes: setting dynamic thresholds based on the safety levels of different areas of the reservoir; when the relative quality score is lower than the dynamic threshold, performing downward nonlinear suppression and forcibly triggering an early warning. The multimodal large language model identifies whether there are translucent textures resembling smoke or abnormal bright areas with flame spectral characteristics in the repaired image. Correlation analysis is then performed based on the changing trend of the relative quality score to achieve automatic triggering and hierarchical output of disaster early warning.

[0026] Further, the step of converting the average subjective opinion score into prior cue words and inputting them, along with the image features of the repaired image and the original video stream frame, into the multimodal large language model includes: performing a difference operation on the repaired image and the original video stream frame to generate a difference feature map; and inputting the difference feature map, along with the image features of the repaired image, the original video stream frame, and the prior cue words, into the multimodal large language model.

[0027] On the other hand, the present invention also discloses a monitoring video stream quality monitoring, repair, and early warning reporting system, including: The video data acquisition module is used to extract monitoring video stream frames in real time and construct the multi-type degradation sample dataset. The quality pre-cognition module is used to pre-train the no-reference image quality assessment network to establish a quality cognition boundary for specific degradation patterns in the reservoir area. The feature mutual supervision inpainting module is used to jointly train the pre-trained quality assessment network and the monitoring inpainting network. It achieves bidirectional feature constraints and collaborative convergence between the image inpainting network and the quality assessment network through a loss function, and outputs a relative quality score based on the feature space distance. The real-time monitoring and restoration module is used to input real-time monitoring frames into the jointly trained network, synchronously output the restored image and average subjective opinion score, and make early warning trigger judgments based on the set dynamic safety threshold. The intelligent semantic analysis and early warning report module is used to convert the average subjective opinion score into prior prompt words, and drive a multimodal large language model by combining the image features before and after repair and the difference feature map, so as to automatically generate inspection reports and graded disaster early warnings.

[0028] Compared with the prior art, the present invention has the following beneficial effects: First, it overcomes the physical perception bottleneck of complex storage area environments: Addressing the unique characteristics of storage areas such as strong backlighting, long-distance monitoring, and high dynamic displacement, this invention combines an optimization strategy of ambient light weighting and motion compensation. This frees the model from the dependence on ideal linear fuzzy assumptions, establishing a unique quality perception boundary that highly matches the actual physical characteristics of airport storage areas. Second, it achieves business-level collaborative convergence of monitoring and repair: It innovatively proposes a regression multi-loss optimization function that integrates masks of key business details in the storage area. This solves the problem of detail loss in traditional general models when repairing small targets at a distance, ensuring that the output repaired image can effectively meet the actual security needs of cargo label recognition and license plate tracking. Third, it bridges the semantic gap between low-level vision and high-level security early warning: Addressing the pain point that existing monitoring can only passively record or output a single quality score, this invention, for the first time, combines objectively quantified MOS scores with image difference features as a priori cognitive driving force for a multimodal large language model. It not only achieves precise image quality evaluation, but also proactively identifies initial smoke and fire hazards in the reservoir area, and outputs intelligent early warning reports that combine quantitative objectivity with security guidance, filling the technological gap between reservoir area monitoring video restoration and automated safety inspection.

[0029] In summary, this invention, by deeply focusing on the unique characteristics of image quality degradation in the large-space, high-dynamic operating environment of airport storage areas, completely breaks the traditional technical pattern of isolated image restoration and quality assessment. This invention not only establishes cognitive boundaries for complex lighting and long-distance imaging degradation in storage areas through no-reference quality assessment pre-training, but also achieves the coordinated evolution of the monitoring and restoration networks through regression multi-loss optimization functions and motion vector compensation mechanisms. This ensures high fidelity of output images under high-speed, irregular movement environments of targets such as forklifts and tractors, providing a reliable physical basis for cargo label restoration and accurate license plate tracking.

[0030] More importantly, this invention successfully overcomes the technical challenge of aligning "lower-level degradation scores" with "higher-level risk semantics" in warehouse security early warning scenarios. By transforming objectively quantified MOS scores into the cognitive driving force of a multimodal large language model, this invention achieves a leap from low-level pixel-level repair to high-level initial smoke and fire identification, anomaly analysis, and inspection decision suggestions. This not only significantly improves the efficiency and accuracy of manual inspections and anomaly investigations in warehouse areas but also fills the technical gap in the transformation of airport monitoring equipment from "passive recording" to "active quantitative assessment and semantic early warning," which has significant engineering implications for ensuring the safety of airport logistics assets and improving the level of automated supervision in warehouse areas. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the system flow of the present invention; Figure 2 This is the schematic diagram of the system structure of the present invention; Figure 3 This is a structural diagram of the quality sensing module of the present invention; Figure 4 A diagram of the joint training framework based on RMOF. Detailed Implementation

[0032] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] This invention is applicable to security monitoring scenarios in intelligent warehouse areas with large spatial scales, such as airport cargo terminals, large logistics hubs, and customs bonded zones. Addressing the unique operational environment of warehouse areas, characterized by wide fields of view for telephoto lenses, frequent changes in lighting, and dense concentrations of highly dynamic targets such as tractors and forklifts, the system adopts an edge-cloud collaborative architecture. It primarily includes: an all-weather monitoring data acquisition module, a no-reference quality assessment pre-training module incorporating ambient light weights, a joint training module based on RMOF and service masks, a real-time repair and service threshold quantization scoring module, and a quantization-driven multimodal large-model early warning report generation module. The repair and real-time scoring modules are deployed on edge computing nodes at the monitoring poles in the warehouse area to meet microsecond-level latency requirements; the large-model inference module is deployed on a secure server within the airport's intranet to ensure absolute security of monitoring data and seamless service integration.

[0034] like Figures 1-4 As shown.

[0035] A method for monitoring, repairing, and reporting the quality of surveillance video streams includes the following steps: Construct a multi-type degradation sample dataset specifically for large-scale imaging degradation scenarios in reservoir areas; The construction of the multi-type degradation sample dataset includes: collecting all-weather real-time monitoring video stream data for the ultra-large spatial scale operation scenario of airport storage area, which includes nonlinear motion blur caused by high dynamic targets during long-distance imaging, geometric distortion caused by zoom at the telephoto end, and degradation features of frequent alternation between strong backlight and dark light; and classifying and labeling the sampled images according to the motion blur gradient intensity and the type of ambient light interference.

[0036] The no-reference image quality assessment network was pre-trained using the dataset. An ambient light distribution weight is introduced for the reservoir area. The weight is dynamically determined by the ratio of the mean value of the real-time light intensity sensor in the reservoir area to the local contrast of the image. For the sampled images in the reservoir area, the perceptual similarity metric between different blurred versions of the image is calculated by combining the ambient light distribution weight. Based on this, a quality perceptual contrast loss is constructed to optimize the quality assessment network, so that it establishes a quality perception boundary for specific degradation patterns in the reservoir area.

[0037] To enable the evaluation network to establish accurate quality perception in the extremely complex lighting environment of the reservoir area, this system introduces a pre-training module.

[0038] During the pre-training process for quality assessment, a weighted approach to the environmental light distribution in the reservoir area was introduced, taking into account the extreme light characteristics of the reservoir area. Calculate the perceptual similarity metric based on image feature extraction results. :

[0039] in, This represents the calculated perceptual similarity metric; and These represent two different degraded versions generated from the i-th baseline image in the collected reservoir area monitoring image dataset after being degraded to different degrees. The letter i represents the image sample number, and the letters j and k represent the corresponding degraded version numbers, respectively. Represents the entire image pixel space, symbol This represents summing over all pixels in the entire image pixel space, where the letter 'p' represents the image pixel space. The specific pixel coordinate index in the text; This represents the comparison of brightness and structural feature similarity between the two different degraded versions at pixel p, calculated based on the feature similarity index. This represents the dynamic ambient light distribution weight introduced to address the complex light and shadow environment of the reservoir area, which is dynamically determined by the ratio of the mean value of the real-time light intensity sensor in the reservoir area to the local contrast of the image. Indicates at pixel point The phase consistency metric extracted is used to characterize the dimensionless structural saliency features of local image areas.

[0040] Subsequently, the contrastive loss of the feature vectors is calculated, and the summations are used to obtain the overall quality-perceived contrastive loss. Used to optimize networks:

[0041] Where N is the total number of images, i.e. the total number of baseline image samples collected for reservoir monitoring; D is the number of blurred versions, i.e. the total number of degraded versions generated after each baseline image is processed by different degrees of motion blur gradient, backlighting or geometric distortion and other degradation modes; the single-sample loss calculated based on the perceptual similarity metric specifically represents the contrastive learning loss calculated by fragment sampling and feature extraction for the i-th baseline image in the j-th blurred version.

[0042] Through this pre-training phase, the network gains the ability to accurately identify image quality degradation at the lower levels, even under poor lighting conditions.

[0043] The pre-trained quality assessment network and the monitoring and repair network are jointly trained. During the joint training phase, the repair network and the quality assessment network are achieved through a loss function to achieve collaborative convergence and bidirectional feature supervision. The relative quality score is output based on the feature space distance to obtain the jointly trained network. Specifically, collaborative convergence and bidirectional feature supervision are achieved through a loss function, and a relative quality score is output based on the feature space distance, including: A regression-based multi-loss optimization function is introduced to simultaneously extract deep distribution features of the repaired image and the standard clear image during the joint training phase, and the relative quality score is output by calculating the feature space distance. When calculating the relative quality score, a motion compensation coefficient for the reservoir operation target is introduced, which is determined based on the target motion vector magnitude and motion sensitivity factor.

[0044] Multiple loss optimization functions based on regression include: The deblurring network loss introduces a mask-based local attention loss, where the masks include warehouse cargo label masks and operating vehicle license plate masks, to enhance the restoration of key business details. The quality assessment network loss is evaluated by treating standard clear images as positive samples and repaired images as negative samples for discrimination training. Cooperative convergence is achieved through adversarial optimization of the deblurring network loss and the quality assessment network loss.

[0045] This is the core module for achieving two-way collaborative optimization of surveillance image deblurring and quality assessment. Input: a blurred surveillance image. With corresponding clear standard image The blurred image is processed by a deblurring network to output the repaired result. Utilizing a pre-trained quality assessment network Obtain its characteristic mean With covariance Then the feature distance is calculated. To address the evaluation errors caused by the high dynamic nonlinear displacement of vehicles in the reservoir area, the relative quality score was calculated... At that time, a target motion compensation coefficient was introduced:

[0046] in, The relative quality score, which represents the calculated output value in the range of [0,1], is used to quantify the relative sharpness and visual credibility of an image. The input image sample to be evaluated specifically refers to the library-restored image or the corresponding standard clear image output by the deblurring network during the joint training or monitoring phase. d(x) represents a preset scaling constant used to adjust the sensitivity of the relative quality score to changes in feature space distance; d(x) represents the feature space statistical distance. The magnitude of the motion vector of the vehicle operating in the reservoir area is extracted by optical flow or target tracking. It is a motion-sensitive factor.

[0047] Simultaneously, a regression multiple loss optimization function (RMOF) is introduced to calculate the deblurring network loss separately. With quality assessment network loss To ensure that the restored images effectively meet the security needs of the storage area, The computation incorporates a mask-based local attention loss. :

[0048] in, This represents the reference quality score output by the quality assessment network for the input standard-resolution image S, where S is a standard-resolution image sample used as the training target; D(B) represents the predicted quality score output by the quality assessment network for the restored image D(B) after processing by the deblurring network D, where B is the input original blurred monitoring image frame, and D(B) fully represents the sharpened restored image output by the deblurring network. This represents the introduced local attention loss function, used to specifically constrain and force enhancement of core security and business details in the image; This represents the preset weight adjustment coefficient, which is used to dynamically adjust the balance between local attention loss and overall image quality loss in the total loss function; A mask representing the labels on goods in the warehouse area, used to accurately define and extract key areas of the labels on goods in the monitoring footage; y represents the mask of the license plate of the work vehicle, used to accurately define and extract the key detail areas of the license plate; y represents the auxiliary guiding variable used in the joint training framework to guide the optimization direction of the deblurring network. It is the probability distribution of samples drawn from the training dataset during joint training, used to calculate the weighted average of the quality bias across the entire dataset; Calculate quality assessment network loss The formula is:

[0049] in, represents the probability distribution of standard image samples drawn from the training dataset during joint training, used to calculate the average loss on the standard sharp images; y represents the true quality label corresponding to the standard sharp images, used to provide supervision and guidance during joint training. This represents the predicted quality score output by the quality assessment network for the input standard-clarity image S; The value represents the predicted quality score output by the quality assessment network for the restored image D(B) processed by the deblurring network D, where B is the original blurred surveillance image frame as input, and D(B) fully represents the sharpened restored image output by the deblurring network. Through this strong constraint, the restoration network is "forced" to prioritize the absolute clarity of core business details (license plates, labels) while restoring the overall image.

[0050] Extract real-time monitoring video stream frames and input them into the jointly trained network to synchronously generate restored images with motion blur removed, and output the average subjective opinion score based on the relative quality score; The process of outputting an average subjective opinion score based on the relative quality score includes: converting the relative quality score into an average subjective opinion score according to the mapping relationship determined by the clearest benchmark score and the most ambiguous benchmark score defined by the jointly trained network.

[0051] During the actual deployment phase in the reservoir area, real-time captured monitoring frames The image is input into a joint network. The network outputs a high-fidelity restored image in real time, eliminating non-linear motion blur, to help security personnel see details at the scene. Simultaneously, the system outputs an objective evaluation score, which is mapped to an average subjective opinion score (MOS) (range 0-100) through a geometric transformation.

[0052] in, It is the average subjective opinion score of the mapping output, using a 0-100 score system to conform to the intuitive subjective visual perception of security personnel regarding the credibility of the image; It is the theoretical maximum quality score limit preset by the system, corresponding to the maximum boundary value of the objective score output by the quality assessment network under an ideal non-destructive acquisition environment; It is the minimum quality score threshold set according to the specific reservoir area monitoring environment, used to characterize the critical point of video stream availability to ensure basic security monitoring accuracy; It is a monitoring image of the current frame that is captured during the real-time operation phase and repaired by a deblurring network.

[0053] Furthermore, dynamic thresholds are set based on the safety levels of different areas within the warehouse (such as hazardous materials areas and general cargo areas). .when When this happens, the system will perform downward nonlinear suppression on the MOS score, using this as a hard trigger condition to directly intercept and push the warning to the early warning module, preventing security misses caused by image quality marginalization.

[0054] The average subjective opinion score is converted into prior cue words, which, together with the image features of the repaired image and the original video stream frame, are input into a multimodal large language model to automatically generate an inspection and operation analysis report that includes the degradation type, the location of occurrence, and the quantified degree of impact.

[0055] Automatically generate inspection and operation analysis reports that include degradation type, location of occurrence, and quantified impact level, including: The quantified average subjective opinion score is converted into natural language prompts that represent the severity of ambiguity and the level of business availability, and then combined with the preset baseline warning task description to form compound prompt words. The composite prompt word, along with the images before and after repair and their difference feature maps, are input into the multimodal large language model to output a structured report.

[0056] To bridge the cognitive gap between visual pixel repair and high-level security early warning systems, this module utilizes a MOS (Mean Offset Syndrome) score-driven multimodal large language model (MLLM). The system converts suppressed or triggered MOS values ​​into objective reference prompts. , compared with the baseline early warning task description splicing:

[0057] Subsequently, the system synchronously inputs the compound prompt words, along with the images before and after the repair and their difference maps, into MLLM:

[0058] The multimodal large model not only reads numerical values ​​but also deeply analyzes the areas representing "blur elimination" or "hazard exposure" in the difference feature map. The model combines these features to infer and output a structured business report. For example: "Anomaly detected: The bottom area of ​​monitoring screen 3 in Zone C of the warehouse is significantly blurred in the original image due to high-speed motion and backlighting, with local details unrecognizable; after image restoration, a continuous semi-transparent diffuse structure appears in this area, exhibiting typical smoke diffusion characteristics; the current image quality score is 45 points, and the local structure is significantly enhanced after restoration. Combined with the difference area analysis, it is determined that this structure is real scene information obscured by blur; based on comprehensive inference, it is inferred that there is a potential fire risk in this area, and it is recommended to immediately issue an inspection work order and coordinate with the fire protection system for on-site verification." This method can also set dynamic thresholds according to the safety levels of different areas of the reservoir. When the relative quality score is lower than the dynamic threshold, downward nonlinear suppression is performed and an early warning is forcibly triggered. The multimodal large language model identifies whether there are translucent textures resembling smoke or abnormal bright areas with flame spectral characteristics in the repaired image. Correlation analysis is then performed based on the changing trend of the relative quality score to achieve automatic triggering and hierarchical output of disaster early warning.

[0059] This solution can be deployed in industrial-grade edge computing gateway devices, and interact with network hard disk recorders or network cameras through RTSP or GB28181 standard video streaming protocols, and use the computing power of embedded NPU or GPU to realize real-time processing and analysis of video streams.

[0060] The system's software algorithms are integrated into an industrial-grade edge computing gateway, directly connecting to NVRs (Network Video Recorders) or IPCs (Internet Protocol Cameras) in the airport hangar area via standard video streaming protocols such as RTSP / GB28181. Leveraging embedded domestic high-performance NPU or GPU operators, the "dynamic monitoring-intelligent repair-threshold scoring" process is completed in milliseconds. The generated multimodal inspection and early warning reports are pushed in real-time to the airport security command center's large screen and security personnel's handheld terminals (PDAs) via the airport's internal Internet of Things (IoT) and API interfaces, providing first-hand, accurate, and objective quantitative data support for emergency response and accountability in the hangar area.

[0061] To address the aforementioned problems, this invention also discloses a surveillance video stream quality monitoring, repair, and early warning reporting system, comprising: The video data acquisition module is used to extract monitoring video stream frames in real time and construct the multi-type degradation sample dataset. The quality pre-cognition module is used to pre-train the no-reference image quality assessment network to establish a quality cognition boundary for specific degradation patterns in the reservoir area. The feature mutual supervision inpainting module is used to jointly train the pre-trained quality assessment network and the monitoring inpainting network. It achieves bidirectional feature constraints and collaborative convergence between the image inpainting network and the quality assessment network through a loss function, and outputs a relative quality score based on the feature space distance. The real-time monitoring and restoration module is used to input real-time monitoring frames into the jointly trained network, synchronously output the restored image and average subjective opinion score, and make early warning trigger judgments based on the set dynamic safety threshold. The intelligent semantic analysis and early warning report module is used to convert the average subjective opinion score into prior prompt words, and drive a multimodal large language model by combining the image features before and after repair and the difference feature map, so as to automatically generate inspection reports and graded disaster early warnings.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring, repairing, and reporting the quality of surveillance video streams, characterized in that, include: Construct a multi-type degradation sample dataset specifically for large-scale imaging degradation scenarios in reservoir areas; The no-reference image quality assessment network was pre-trained using the dataset. The pre-trained quality assessment network and the monitoring and repair network are jointly trained. During the joint training phase, the repair network and the quality assessment network are achieved through a loss function to achieve collaborative convergence and bidirectional feature supervision. The relative quality score is output based on the feature space distance to obtain the jointly trained network. Extract real-time monitoring video stream frames and input them into the jointly trained network to synchronously generate restored images with motion blur removed, and output the average subjective opinion score based on the relative quality score; The average subjective opinion score is converted into prior cue words, which, together with the image features of the repaired image and the original video stream frame, are input into a multimodal large language model to automatically generate an inspection and operation analysis report that includes the degradation type, the location of occurrence, and the quantified degree of impact.

2. The method for monitoring, repairing, and reporting the quality of surveillance video streams according to claim 1, characterized in that, The multi-type degradation sample dataset includes: The system collects real-time monitoring video stream data around the clock, including nonlinear motion blur of the target during long-distance imaging, geometric distortion caused by zooming at the telephoto end, and degradation features caused by alternating backlight and low light conditions. The sampled images were then graded and labeled according to the intensity of motion blur gradient and the type of ambient light interference.

3. The method for monitoring, repairing, and reporting the quality of surveillance video streams according to claim 1, characterized in that, The pre-training of the no-reference image quality assessment network includes: An ambient light distribution weight is introduced for the reservoir area. The weight is dynamically determined by the ratio of the mean value of the real-time light intensity sensor in the reservoir area to the local contrast of the image. For the sampled images in the reservoir area, the perceptual similarity metric between different blurred versions of the image is calculated by combining the ambient light distribution weight. Based on this, a quality perceptual contrast loss is constructed to optimize the quality assessment network, so that it establishes a quality perception boundary for specific degradation patterns in the reservoir area. Among them, the perceptual similarity metric is calculated based on the image feature extraction results. The formula is: in, This represents the calculated perceptual similarity metric. and These represent two different degraded versions generated after the i-th baseline image in the collected reservoir area monitoring image dataset has undergone different degrees of degradation. The letter i represents the image sample number, and the letters j and k represent the corresponding degradation version numbers, respectively. Represents the entire image pixel space, symbol This represents summing over all pixels in the entire image pixel space, where the letter 'p' represents the image pixel space. The specific pixel coordinate index in the text; This represents the comparison of brightness and structural feature similarity between the two different degraded versions at pixel p, calculated based on the feature similarity index. This represents the dynamic ambient light distribution weight introduced to address the complex light and shadow environment of the reservoir area, which is dynamically determined by the ratio of the mean value of the real-time light intensity sensor in the reservoir area to the local contrast of the image. Indicates at pixel point The phase consistency metric extracted is used to characterize the dimensionless structural saliency features of local image areas. Construct a quality-perceived contrast loss based on the perceived similarity metric. The formula is: Where N is the total number of images, i.e. the total number of baseline image samples collected for reservoir monitoring; D is the number of blurred versions, i.e. the total number of degraded versions generated after each baseline image is processed by different degrees of motion blur gradient, backlighting or geometric distortion and other degradation modes; the single-sample loss calculated based on the perceptual similarity metric specifically represents the contrastive learning loss calculated by fragment sampling and feature extraction for the i-th baseline image in the j-th blurred version.

4. The method for monitoring, repairing, and reporting the quality of surveillance video streams according to claim 1, characterized in that, The method of achieving collaborative convergence and bidirectional feature supervision between the repair network and the quality assessment network through a loss function, and outputting a relative quality score based on the feature space distance, includes: A regression-based multi-loss optimization function is introduced to simultaneously extract deep distribution features of the repaired image and the standard clear image during the joint training phase, and the relative quality score is output by calculating the feature space distance; when calculating the relative quality score, a motion compensation coefficient for the reservoir operation target is introduced, which is determined based on the target motion vector magnitude and motion sensitivity factor; Calculate the relative quality score The formula is: in, The relative quality score, which represents the calculated output value in the range of [0,1], is used to quantify the relative sharpness and visual credibility of an image. The input image sample to be evaluated specifically refers to the library-restored image or the corresponding standard clear image output by the deblurring network during the joint training or monitoring phase. d(x) represents a preset scaling constant used to adjust the sensitivity of the relative quality score to changes in feature space distance; d(x) represents the feature space statistical distance. The magnitude of the motion vector of the vehicle operating in the reservoir area is extracted by optical flow or target tracking. It is a motion-sensitive factor.

5. The method for monitoring, repairing, and reporting the quality of surveillance video streams according to claim 4, characterized in that, The regression-based multi-loss optimization function includes: The deblurring network loss introduces a mask-based local attention loss, where the masks include warehouse cargo label masks and operating vehicle license plate masks, to enhance the restoration of key business details. The quality assessment network loss is evaluated by treating standard clear images as positive samples and repaired images as negative samples for discrimination training. The deblurring network loss and the quality assessment network loss achieve collaborative convergence through adversarial iteration; Calculate the loss of the deblurring network The formula is: in, This represents the reference quality score output by the quality assessment network for the input standard-resolution image S, where S is a standard-resolution image sample used as the training target; D(B) represents the predicted quality score output by the quality assessment network for the restored image D(B) after processing by the deblurring network D, where B is the input original blurred monitoring image frame, and D(B) fully represents the sharpened restored image output by the deblurring network. This represents the introduced local attention loss function, used to specifically constrain and force enhancement of core security and business details in the image; This represents the preset weight adjustment coefficient, which is used to dynamically adjust the balance between local attention loss and overall image quality loss in the total loss function; A mask representing the labels on goods in the warehouse area, used to accurately define and extract key areas of the labels on goods in the monitoring footage; y represents the mask of the license plate of the work vehicle, which is used to accurately define and extract the key detail areas of the license plate; y represents the auxiliary guiding variable used in the joint training framework to guide the optimization direction of the deblurring network. It is the probability distribution of samples drawn from the training dataset during joint training, used to calculate the weighted average of the quality bias across the entire dataset; Calculate quality assessment network loss The formula is: in, represents the probability distribution of standard image samples drawn from the training dataset during joint training, used to calculate the average loss on the standard sharp images; y represents the true quality label corresponding to the standard sharp images, used to provide supervision and guidance during joint training. This represents the predicted quality score output by the quality assessment network for the input standard-clarity image S; D(B) represents the predicted quality score output by the quality assessment network for the restored image D(B) processed by the deblurring network D, where B is the input original blurred monitoring image frame, and D(B) fully represents the sharpened restored image output by the deblurring network.

6. The method for monitoring, repairing, and reporting the quality of surveillance video streams according to claim 1, characterized in that, The average subjective opinion score output based on the relative quality score includes: The relative quality score is converted into an average subjective opinion score according to the mapping relationship determined by the clearest benchmark score and the most ambiguous benchmark score defined by the jointly trained network. The mapping relationship formula is: in, It is the average subjective opinion score of the mapping output, using a 0-100 score system to conform to the intuitive subjective visual perception of security personnel regarding the credibility of the image; It is the theoretical maximum quality score limit preset by the system, corresponding to the maximum boundary value of the objective score output by the quality assessment network under an ideal non-destructive acquisition environment; It is the minimum quality score threshold set according to the specific reservoir area monitoring environment, used to characterize the critical point of video stream availability to ensure basic security monitoring accuracy; It is a monitoring image of the current frame that is captured during the real-time operation phase and repaired by a deblurring network.

7. The method for monitoring, repairing, and reporting the quality of surveillance video streams according to claim 1, characterized in that, The automatically generated inspection and operation analysis report includes the type of degradation, location of occurrence, and quantified impact level, and includes: The quantified average subjective opinion score is converted into natural language prompts that represent the severity of ambiguity and the level of business availability, and then combined with the preset baseline warning task description to form compound prompt words. The composite prompt word, along with the images before and after repair and their difference feature maps, are input into the multimodal large language model to output a structured report.

8. The method for monitoring, repairing, and reporting the quality of surveillance video streams according to claim 1, characterized in that, Also includes: Dynamic thresholds are set according to the safety levels of different areas of the reservoir. When the relative quality score is lower than the dynamic threshold, downward nonlinear suppression is performed and an early warning is forcibly triggered. The multimodal large language model identifies whether there are translucent textures resembling smoke or abnormal bright areas with flame spectral characteristics in the repaired image. Correlation analysis is then performed based on the changing trend of the relative quality score to achieve automatic triggering and hierarchical output of disaster early warning.

9. A method for monitoring, repairing, and reporting the quality of surveillance video streams according to claim 1, characterized in that, The step of converting the average subjective opinion score into prior cue words, and inputting them together with the image features of the repaired image and the original video stream frame into a multimodal large language model, includes: The repaired image is compared with the original video stream frame by performing a difference operation to generate a difference feature map; The differential feature map, the repaired image, the image features of the original video stream frame, and the prior cue words are input into the multimodal large language model.

10. A system for monitoring, repairing, and reporting the quality of surveillance video streams, characterized in that, include: The video data acquisition module is used to extract monitoring video stream frames in real time and construct the multi-type degradation sample dataset. The quality pre-cognition module is used to pre-train the no-reference image quality assessment network to establish a quality cognition boundary for specific degradation patterns in the reservoir area. The feature mutual supervision inpainting module is used to jointly train the pre-trained quality assessment network and the monitoring inpainting network. It achieves bidirectional feature constraints and collaborative convergence between the image inpainting network and the quality assessment network through a loss function, and outputs a relative quality score based on the feature space distance. The real-time monitoring and restoration module is used to input real-time monitoring frames into the jointly trained network, synchronously output the restored image and average subjective opinion score, and make early warning trigger judgments based on the set dynamic safety threshold. The intelligent semantic analysis and early warning report module is used to convert the average subjective opinion score into prior prompt words, and drive a multimodal large language model by combining the image features before and after repair and the difference feature map to automatically generate inspection reports and graded disaster early warnings.

Images