A method and system for detecting abnormal operation of a network camera

Abstract

The present invention relates to the field of security monitoring technology, and in particular to a method and system for detecting abnormal operation of a network camera, including training a basic data set to form a large model, collecting normal and abnormal images, marking and preprocessing after block processing, training through multi-task learning of a CNN network structure model, using SSIM as a loss function and generating simulated abnormal data to enhance generalization ability, and evaluating model performance; obtaining key frames or fragments of a video stream and then using a large model to identify the result R0, reviewers regularly review, and output if abnormal, and then train the large model after accumulating certain results. The system includes modules such as data collection, model training, detection data acquisition, data block, abnormal detection, review and model optimization. The present invention improves detection efficiency and accuracy, reduces manual intervention, and effectively solves the problems of low video multi-channel detection performance, cumbersome threshold setting and low detection accuracy in the prior art, and is suitable for various security monitoring scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of security monitoring, and in particular to a method and system for detecting abnormal operation of a network camera. Background Art

[0002] In today's security monitoring field, cameras are the foundation and key components of security systems. Whether they work properly is directly related to the effectiveness and reliability of the entire security system. With the development of society, the scale of security systems continues to expand, the monitoring areas covered are becoming increasingly extensive, and the number of cameras has increased dramatically. While this development trend brings more comprehensive monitoring coverage, it also brings huge challenges to the management and anomaly detection of monitoring images.

[0003] Traditionally, the review of surveillance footage mainly relies on manual methods. However, faced with massive amounts of surveillance video data, manual review faces many difficult problems. On the one hand, the workload of manual review is growing exponentially, making it easy for staff to get tired during long-term continuous viewing of footage, which leads to lack of concentration and makes it difficult to ensure the accuracy and comprehensiveness of the review. On the other hand, even if a large amount of manpower is invested, it is impossible to conduct a 7*24-hour real-time review of all surveillance footage, which makes the security system have monitoring loopholes during some periods of time, unable to detect abnormal situations in time, and brings potential risks to security prevention.

[0004] In order to ease the pressure of manual review, the image and graphics detection technology based on OpenCV came into being and replaced manual image anomaly detection to a certain extent. This technology can identify a variety of common image anomalies, including signal loss, image freezing, camera occlusion, abnormal image brightness, image color cast, image blur, image noise or stripes, camera shaking, etc. However, this traditional recognition technology still has obvious limitations in practical applications.

[0005] In terms of performance, when capturing video frames and then comparing the front and back of the video frames, especially when dealing with the task of detecting anomalies in multiple surveillance video images, the computing resources consumed are huge, the detection efficiency is low, and it is difficult to meet the real-time requirements of large-scale security systems. This means that in actual applications, after an abnormal situation occurs, the system may not give an alarm in time, thus delaying the time to deal with the problem.

[0006] In addition, the OpenCV-based graphics detection technology needs to set different thresholds for different screen anomalies. Only when the abnormal quantization value reaches or exceeds the threshold can it be determined whether the video screen is normal. For a large security system, the environment and screen characteristics of each monitoring point may be different, which makes it extremely complicated and cumbersome to set a reasonable threshold for each monitoring channel. In the early stage of system construction, it takes a lot of manpower and time to set the threshold. In the subsequent use process, if the environment changes or new abnormal patterns appear, the threshold needs to be readjusted, which undoubtedly increases the difficulty and cost of system maintenance.

[0007] In summary, the existing manual review and OpenCV-based detection technology have many shortcomings. There is an urgent need for a new, more efficient and more intelligent network camera working abnormality detection method and system to adapt to the growing needs of the security industry and improve the reliability and intelligence level of the security system. Therefore, a network camera working abnormality detection method and system are proposed to address the above problems. Summary of the invention

[0008] The purpose of the present invention is to provide a method and system for detecting abnormal operation of a network camera to solve the problems raised in the above background technology.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] A method for detecting abnormal operation of a network camera comprises the following steps:

[0011] Step S1, training a basic large model formed by a basic data set;

[0012] Step S1-1, collecting normal monitoring images and abnormal monitoring images or video clips from the existing security system as data basis, and evenly cutting each image into blocks;

[0013] Step S1-2, putting the segmented data base into the basic large model for training, labeling each data base sample during training, if it is a data base with a normal picture, it is marked as a normal picture / video, and then the marked data base is preprocessed;

[0014] Step S1-3: After the data foundation is preprocessed, multi-task learning is performed through a network structure model based on a CNN convolutional neural network, and different types of anomalies are detected at the same time. Feature classification judgment is extracted based on the video frame, and the abnormal state is output and the abnormal type is identified; the model training uses the structural similarity index SSIM as the loss function, and uses synthetic abnormal data to generate simulated abnormal video frames to enhance the generalization ability;

[0015] Step S1-4: Comprehensively evaluate the model performance using accuracy, precision, recall, and F1-score;

[0016] Step S2, obtaining a video stream from the security system, taking out key frames or video clips from the video stream as data to be detected; dividing the video frames in the data to be detected into blocks as materials to be detected, putting the materials to be detected into the basic large model for identification, and obtaining a detection result R0;

[0017] Step S3: The reviewer regularly reviews the test result R0, and if the picture is abnormal, the abnormal picture is output;

[0018] Step S4: When the detection results R0 accumulate to a set number, they are used as new detection materials to train the basic large model;

[0019] Step S5: When the new detected material reaches the set threshold, step S3 and step S4 are executed regularly.

[0020] As a preferred solution, when each picture is evenly divided into blocks, a picture division method is determined to evenly divide each picture into rectangular blocks, where is a positive integer; calculate the size of each rectangular block, for a resolution of The number of pixels in each block in the horizontal direction is , the vertical direction is ; Starting from the upper left corner of the screen, divide the screen in sequence according to the calculated block size and number the divided blocks.

[0021] As a preferred solution, when preprocessing the labeled data base, the data base is processed by formula Normalization is performed, where Represents a pixel value in the original image dataset, represents the pixel value after normalization. is the mean of the image dataset, is the standard deviation of the image dataset, using the commonly used mean of the ImageNet dataset and standard deviation , the data is standardized to a mean of 0 and a standard deviation of 1.

[0022] As a preferred solution, the network structure model based on the CNN convolutional neural network includes multiple convolutional layers, pooling layers and fully connected layers. The convolutional layer performs convolution operations by sliding the convolution kernel on the image, which is expressed by the following formula:

[0023] ;

[0024] in, The output feature map of the convolutional layer is The pixel value of the position, is the convolution kernel weight, is the pixel value of the input image at the corresponding position, is the bias term, is the convolution kernel size, extracting local features of the video frame;

[0025] The pooling layer uses the following formula to perform dimensionality reduction:

[0026] ;in, is the pooling window area;

[0027] The fully connected layer maps the extracted features to different abnormal category spaces through the formula Calculate the output value, according to the output value Make classification judgments, among which, For the The fully connected layer output value corresponding to each category, is the weight, is the i-th element of the input feature vector, is the input feature vector dimension, It is a bias term. At the same time, it detects different types of anomalies, extracts features from video frames, classifies and judges them, outputs the abnormal state and identifies the abnormal type.

[0028] As a preferred solution, the model training uses the structural similarity index SSIM as the loss function. The loss function formula is as follows:

[0029] ;

[0030] in, and are two images to be compared, and is an image and The mean of and is an image and The variance of yes and The covariance of and is a small constant introduced to avoid the denominator being;

[0031] The model parameters are optimized by minimizing the loss function; synthetic abnormal data are used to generate simulated abnormal video frames to increase the model's generalization ability for abnormal patterns.

[0032] As a preferred solution, the comprehensive evaluation of model performance using accuracy, precision, recall, and F1-score includes the following steps:

[0033] The accuracy rate measures the proportion of correctly classified samples to the total number of samples and is calculated using the following method:

[0034] ;

[0035] in, is the true positive, that is, the number of samples correctly detected as abnormal, is the true negative, that is, the number of samples correctly detected as normal. is the number of false positives, i.e., normal samples are mistakenly detected as abnormal. is the number of false negatives, i.e., abnormal samples are mistakenly detected as normal;

[0036] Precision measures the proportion of samples that are actually abnormal among all samples predicted to be abnormal. It is calculated as follows:

[0037] ;

[0038] The recall rate measures the proportion of actual abnormal samples that are correctly detected and is calculated using the following method:

[0039] ;

[0040] F1-score is the harmonic mean of precision and recall, and is calculated as:

[0041] ; Determine whether the model meets the requirements based on the evaluation results. If not, adjust the model parameters or retrain.

[0042] A network camera working abnormality detection system, the system is applied to a network camera working abnormality detection method.

[0043] It can be seen from the technical solution provided by the present invention that the method and system for detecting abnormal operation of a network camera provided by the present invention have the following beneficial effects:

[0044] 1. Efficient detection capabilities

[0045] Based on the large model architecture and unique block detection strategy, the rapid processing of video images is achieved; in actual application scenarios, such as large enterprise parks or urban security monitoring systems, facing massive monitoring video data, the present invention can quickly block video frames and input them into the large model for recognition; compared with traditional detection methods, the detection speed is greatly improved, and the detection of multiple videos can be completed in a short time, ensuring that abnormal situations are discovered in time, greatly enhancing the response timeliness of the security system, and effectively responding to various emergencies;

[0046] The multi-task learning mechanism is adopted, and with the powerful parallel processing capability of CNN convolutional neural network, multiple types of anomalies can be detected at the same time. Multiple convolutional layers, pooling layers and fully connected layers in the model work together, making full use of hardware resources (such as GPU) to achieve parallel computing, greatly shortening the processing time. Whether it is common screen freezing and occlusion, or more complex brightness anomalies, color cast and other problems, they can be fully analyzed and judged in one detection process, which significantly improves the overall detection efficiency and enables the security system to process more monitoring information in a limited time.

[0047] 2. Accurate detection performance

[0048] The rich and diverse data foundation provides comprehensive learning materials for the model; by collecting a large number of normal and abnormal monitoring images and video clips covering various actual scenes (different lighting, scene content, abnormal types and degrees), the model can learn extensive and detailed feature patterns, thus having a strong generalization ability; in complex and changeable actual environments, such as outdoor monitoring under severe weather conditions or indoor scenes with complex lighting, the model of the present invention can accurately identify various abnormalities, effectively reduce missed detections and false detections, and provide reliable protection for security monitoring;

[0049] The use of synthetic abnormal data further expands the model's understanding of abnormal patterns; by simulating various possible abnormal situations, such as pixel-level modification, adding different noises, simulated occlusion, etc., the model is exposed to more potential abnormalities and enhances the ability to identify unknown abnormalities; at the same time, the closed-loop detection and optimization system ensures that the model continuously learns new abnormal features; as the system runs, new detection results are continuously fed back for model retraining, so that it can dynamically adapt to emerging abnormal patterns, continuously improve detection accuracy, accurately locate abnormal situations, and provide accurate basis for subsequent processing and decision-making;

[0050] 3. Reduce manual dependence

[0051] Achieved highly automated anomaly detection and output; after the system is running stably, there is no need for manual real-time monitoring of the monitoring screen, and it can automatically monitor the working status of the network camera and output abnormal results in a timely manner; this is particularly important in large-scale security systems, greatly saving labor costs, improving the efficiency of security work, and allowing security personnel to devote more energy to other key tasks, such as handling abnormal situations and formulating security strategies;

[0052] Innovatively, the manual threshold setting is replaced by model adaptive learning; the cumbersome and error-prone threshold setting work in traditional methods is effectively solved in the present invention; the large model automatically grasps the characteristic differences between normal and abnormal images by learning from massive data, and can accurately judge abnormalities without human intervention, reducing detection errors caused by human factors and improving the stability and reliability of the detection system; at the same time, with the continuous optimization of the model, the demand for manual review is gradually reduced, and ultimately efficient, intelligent, and autonomous abnormality detection is achieved, promoting the development of security monitoring technology to a higher level. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The present invention is a schematic diagram of the overall structure of a detection system for abnormal operation of a network camera. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0055] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0056] The embodiment of the present invention provides a method for detecting abnormal operation of a network camera, comprising the following steps:

[0057] Step S1, training a basic large model formed by a basic data set;

[0058] Step S1-1, collecting normal monitoring images and abnormal monitoring images or video clips from the existing security system as data basis, and evenly cutting each image into blocks;

[0059] When each picture is evenly divided into blocks, a picture division method is determined to divide each picture evenly into rectangular blocks, where is a positive integer; calculate the size of each rectangular block, for a resolution of The number of pixels in each block in the horizontal direction is , the vertical direction is ; Starting from the upper left corner of the picture, divide the picture in sequence according to the calculated block size, and number the divided blocks;

[0060] Step S1-2, putting the segmented data base into the basic large model for training, labeling each data base sample during training, if it is a data base with a normal picture, it is marked as a normal picture / video, and then the marked data base is preprocessed;

[0061] Among them, when preprocessing the labeled data foundation, the data foundation is processed by formula Normalization is performed, where is the mean of the image dataset, is the standard deviation of the image dataset, using the commonly used mean of the ImageNet dataset and standard deviation , standardize the data to a mean of 0 and a standard deviation of 1;

[0062] Step S1-3: After the data foundation is preprocessed, multi-task learning is performed through a network structure model based on a CNN convolutional neural network, and different types of anomalies are detected at the same time. Feature classification judgment is extracted based on the video frame, and the abnormal state is output and the abnormal type is identified; the model training uses the structural similarity index SSIM as the loss function, and uses synthetic abnormal data to generate simulated abnormal video frames to enhance the generalization ability;

[0063] Furthermore, the network structure model based on the CNN convolutional neural network includes multiple convolutional layers, pooling layers and fully connected layers. The convolutional layer performs convolution operations by sliding the convolution kernel on the image, which is expressed by the following formula:

[0064]

[0065] in, The output feature map of the convolutional layer is The pixel value of the position, is the convolution kernel weight, is the pixel value of the input image at the corresponding position, is the bias term, is the convolution kernel size, extracting local features of the video frame;

[0066] The pooling layer uses the following formula to perform dimensionality reduction:

[0067]

[0068] in, is the pooling window area;

[0069] The fully connected layer maps the extracted features to different abnormal category spaces through the formula Calculate the output value, according to the output value Make classification judgments, among which, is the output value of the fully connected layer corresponding to the i-th category, is the weight, is the i-th element of the input feature vector, is the input feature vector dimension, It is a bias term. At the same time, it detects different types of anomalies, extracts features from video frames, classifies and judges them, outputs the anomaly status and identifies the anomaly type;

[0070] Step S1-4: Comprehensively evaluate the model performance using accuracy, precision, recall, and F1-score;

[0071] Step S2, obtaining a video stream from the security system, taking out key frames or video clips from the video stream as data to be detected; dividing the video frames in the data to be detected into blocks as materials to be detected, putting the materials to be detected into the basic large model for identification, and obtaining a detection result R0;

[0072] Step S3: The reviewer regularly reviews the test result R0, and if the picture is abnormal, the abnormal picture is output;

[0073] Step S4: When the detection results R0 accumulate to a set number, they are used as new detection materials to train the basic large model;

[0074] Step S5: When the new detected material reaches the set threshold, step S3 and step S4 are executed regularly.

[0075] In this embodiment, the model training uses the structural similarity index SSIM as the loss function, and the loss function formula is as follows:

[0076]

[0077] in, and are two images to be compared, and is an image and The mean of and is an image and The variance of yes and The covariance of and is a small constant introduced to avoid the denominator being;

[0078] The model parameters are optimized by minimizing the loss function; synthetic abnormal data are used to generate simulated abnormal video frames to increase the model's generalization ability for abnormal patterns.

[0079] In this embodiment, the comprehensive evaluation of model performance using accuracy, precision, recall, and F1-score includes the following steps:

[0080] The accuracy rate measures the proportion of correctly classified samples to the total number of samples and is calculated using the following method:

[0081]

[0082] in, is the true positive, that is, the number of samples correctly detected as abnormal, is the true negative, that is, the number of samples correctly detected as normal. is the number of false positives, i.e., normal samples are mistakenly detected as abnormal. is the number of false negatives, i.e., abnormal samples are mistakenly detected as normal;

[0083] Precision measures the proportion of samples that are actually abnormal among all samples predicted to be abnormal. It is calculated as follows:

[0084]

[0085] The recall rate measures the proportion of actual abnormal samples that are correctly detected and is calculated using the following method:

[0086]

[0087] F1-score is the harmonic mean of precision and recall, and is calculated as:

[0088]

[0089] Determine whether the model meets the requirements based on the evaluation results. If not, adjust the model parameters or retrain.

[0090] like Figure 1 As shown, a network camera working abnormality detection system is applied to a network camera working abnormality detection method. The specific details of the detection system are as follows:

[0091] 1. System Architecture

[0092] The network camera abnormality detection system is mainly composed of data acquisition module, model training module, detection data acquisition module, data segmentation module, abnormality detection module, review module and model optimization module. The modules work together to achieve effective detection and processing of network camera abnormalities.

[0093] 2. Module Function

[0094] Data acquisition module:

[0095] Function: Responsible for collecting normal and abnormal monitoring images or video clips from the security system; there are various collection methods, and real-time or scheduled collection can be selected according to the system settings; real-time collection can obtain the latest monitoring data in a timely manner, which is suitable for scenarios with high real-time requirements; scheduled collection can collect data at predetermined time intervals (such as every hour, a specific period of time every day, etc.), so that the system can obtain a variety of data samples in different time periods, and it can also balance the use of system resources; the collection frequency can be flexibly set according to the scale, importance and system resource status of the security system; for example, for monitoring points in important areas, a higher collection frequency (such as once a minute) can be set to ensure timely detection of abnormalities; and for relatively minor areas, the collection frequency can be appropriately reduced (such as once every 10 minutes) to avoid excessive data transmission and storage pressure;

[0096] Data processing: The collected images are divided into blocks in the same way as in the subsequent detection and training stages, that is, the images are evenly divided into n×n rectangular blocks (n is a positive integer, determined according to the image resolution and detection accuracy requirements); the size of each rectangular block is calculated, and the divided blocks are numbered and sorted, and converted into a format suitable for subsequent processing (such as array or matrix form), and the relevant information of the image (such as acquisition time, camera number, monitoring area, etc.) and the number and location information of each block are recorded at the same time; for abnormal images, the abnormal type needs to be marked in detail and closely associated with the corresponding block data to provide an accurate data basis for subsequent model training and anomaly detection;

[0097] Model training module:

[0098] Function: Receive the annotated and preprocessed basic data provided by the data acquisition module, build a network structure model based on the CNN convolutional neural network, and perform multi-task learning training; this module uses the structural similarity index (SSIM) as the loss function, and continuously adjusts the model parameters so that the model can accurately classify and identify different types of anomalies; during the training process, it also uses synthetic abnormal data to generate simulated abnormal video frames to enhance the model's generalization ability for various abnormal patterns, so that it can adapt to abnormal situations in different scenarios; after the training is completed, the performance of the model is comprehensively evaluated, and the accuracy, precision, recall, F1-score and other indicators are used to measure the effectiveness and reliability of the model to determine whether the model meets the requirements of practical applications;

[0099] Model optimization: If the model evaluation results do not meet the requirements, the module can automatically adjust the model parameters according to the feedback of the evaluation indicators, such as adjusting the size and number of convolution kernels of the convolution layer, the size and step size of the pooling window of the pooling layer, and the number of neurons in the fully connected layer; or re-select the training algorithm, adjust the distribution of training data, and re-train the model until the model performance reaches the preset standard, forming an accurate and reliable basic large model, providing a solid foundation for subsequent anomaly detection;

[0100] Detection data acquisition module:

[0101] Function: Get the video stream from the security system and extract key frames or video clips as the data to be detected; provide a variety of flexible strategies for users to choose when obtaining key frames or video clips; selection based on time interval is a common method. Users can set the time interval according to actual needs (such as every 5 seconds, 10 seconds, etc.). The system evenly extracts key frames from the video stream according to the set time interval. This method can cover different time periods of the video to a certain extent and obtain representative picture samples; selection based on the degree of picture change is more intelligent, by calculating the pixel difference between adjacent frames or using advanced motion detection algorithms to determine the degree of picture change; when the picture changes greatly (such as scene switching, rapid movement of objects, etc.), select these frames as key frames, which can more effectively capture important information in the video, reduce data redundancy, and improve detection efficiency. It also helps to find abnormal situations caused by picture changes (such as sudden object occlusion, severe picture shaking, etc.);

[0102] Data chunking module:

[0103] Function: Block the video frames in the data to be detected provided by the detection data acquisition module. The block parameters (such as block size, number, etc.) can be flexibly configured according to the picture characteristics (such as resolution, picture content complexity, etc.) and detection requirements to ensure that the image data after block can accurately reflect the picture characteristics and facilitate model processing; the block mode is consistent with the picture block mode in the data acquisition module, that is, according to the predetermined The video frame is divided into multiple rectangular blocks, and the image data after the blocks are effectively organized and managed, such as caching operations, to improve data processing efficiency and reduce data reading time, so that the system can quickly provide the block-based materials to be detected to the anomaly detection module for identification; the video clips are not divided into blocks, but directly passed to the anomaly detection module, retaining the original structural information of the video clips, so that the model can judge the anomalies of the video clips as a whole, and adapt to the detection needs of video data of different types and durations;

[0104] Anomaly Detection Module:

[0105] Function: Receive the material to be detected (blocked video frames or unblocked video clips) processed by the data segmentation module, and input it into the pre-trained basic large model for identification; the model performs comprehensive and in-depth feature extraction and classification judgment on the input material to be detected based on the learned normal and abnormal picture feature patterns; for the video frames after segmentation, the model will perform independent feature analysis on each block, and comprehensively consider the results of all blocks to obtain an abnormal judgment on the entire video frame; for the video clips, the model will analyze the overall structure and dynamic changes to determine whether there are abnormalities; after a complex calculation and judgment process, the detection result R0 is finally obtained. The detection result R0 contains rich information, such as the abnormal probability of each block or video clip and possible abnormal type information (such as the probability of occlusion abnormality is 0.7, the probability of signal loss abnormality is 0.1, etc.), providing detailed and accurate basis for subsequent review and processing;

[0106] Review Module:

[0107] Function: The reviewer shall regularly review the detection result R0 output by the anomaly detection module; the review time interval can be flexibly set according to the importance of the security system and the frequency of anomalies. For important areas (such as key entrances and exits, core equipment areas, etc.), a shorter review time interval (such as review every half an hour or one hour) can be set to ensure timely detection and handling of anomalies; for general areas, the review time interval can be appropriately extended (such as review every two hours or four hours) to balance the review workload and the timeliness of anomaly detection; during the review process, the reviewer can comprehensively check the detection result R0 and the corresponding video screen Determine whether the image is truly abnormal; if the image is determined to be abnormal, the reviewer can record the abnormality type (such as specific occlusion position, signal loss degree, etc.), occurrence time (accurate to seconds), monitoring area and other detailed information in detail, and output the abnormal image information in time to notify relevant personnel to handle it; if the image is determined to be normal, continue to the next round of detection result R0 review; in addition, the review module also has the function of setting different review levels and permissions. According to the importance and security level of different monitoring areas, different reviewers are assigned corresponding permissions and tasks, so as to realize differentiated review of security areas of different importance and improve review efficiency and accuracy;

[0108] Model optimization module:

[0109] Function: When the detection results R0 accumulate to a certain extent (reaching the set number threshold, which can be dynamically determined based on the model training effect and system resource conditions), these detection results R0 are fed back to the model training module as new detection materials, triggering the model retraining process, further optimizing the model parameters and performance, and improving the large model's ability to identify new abnormal situations and accuracy; during the model optimization process, the module can automatically adjust the threshold for the accumulation of detection results R0 and the period of regular review and model optimization operations according to changes in model performance indicators (such as accuracy, recall rate, F1-score, etc.); for example, when the model performance indicators show a downward trend, Appropriately lower the accumulation threshold and speed up the frequency of model optimization; when the model performance is stable and good, appropriately increase the accumulation threshold to reduce unnecessary computing resource consumption; at the same time, during the model training process, the module can also automatically adjust the training parameters (such as learning rate, number of iterations, etc.) according to the characteristics of the new data to ensure the effectiveness and efficiency of model training; in addition, the model optimization module is also responsible for generating detailed operation logs, recording the time, results, and changes in model performance indicators of each review and model optimization operation, and other important information, to provide strong support for system maintenance and troubleshooting, so that technical personnel can timely understand the system operation status, find potential problems and make targeted improvements;

[0110] 3. System operation process

[0111] When the system is running, the data acquisition module first collects data from the security system and processes it in blocks, and then hands the processed data to the model training module for model training and evaluation; after the training is completed, the detection data acquisition module obtains the key frames or video clips in the video stream, and the data segmentation module segments them (the video clips are processed on demand), and the anomaly detection module uses the trained model to identify and output the detection result R0; the review module reviews it, and outputs the anomaly information if an anomaly is found. At the same time, the model optimization module triggers the retraining and optimization operations of the model in a timely manner according to the review results and accumulated data, continuously improving the performance and accuracy of the system, forming a closed-loop, continuously optimized network camera working anomaly detection system.

[0112] Example 1: Model training and initial evaluation

[0113] 1. Purpose of the Embodiment

[0114] This embodiment mainly focuses on the training process of the basic large model in the network camera working anomaly detection method, including data collection and preprocessing, model construction and training, and preliminary model performance evaluation, aiming to show how to gradually build a basic model with certain detection capabilities from raw data, laying the foundation for subsequent anomaly detection and model optimization;

[0115] 2. Implementation steps:

[0116] 1. Data collection and preprocessing

[0117] Data Collection:

[0118] From a security system with 50 monitoring points, 2,000 normal monitoring images in different monitoring scenarios at different time periods within two weeks were collected through network connection, as well as 1,200 abnormal monitoring images (150 signal loss, 200 image freezes, 250 occlusions, 150 abnormal brightness, 100 color casts, 200 blurs, 100 noises or stripes, and 50 jitters); the image resolutions covered two common specifications: 720P and 1080P;

[0119] For each picture, record the acquisition time, camera number, monitoring area and other related information in detail to facilitate subsequent data management and analysis;

[0120] Screen division and labeling:

[0121] Determine the image partitioning method. For a 1080P (1920×1080) resolution image, choose to evenly partition it into 4×4 rectangular blocks, that is, n=4; calculate the size of each rectangular block, the number of pixels in each block in the horizontal direction is 1920 / 4=480, and the number in the vertical direction is 1080 / 4=270; starting from the upper left corner of the image, partition the image in sequence according to the calculated block size, and number the partitioned blocks from left to right and from top to bottom from 1 to 16;

[0122] Each block of data is labeled. The blocks of normal images are marked as "normal", and the blocks of abnormal images are labeled in detail according to the specific abnormality type, such as "signal loss abnormality - upper left corner area of ​​the image (blocks 1-4)" (for the case where the signal loss abnormality mainly affects the four blocks in the upper left corner of the image), etc. At the same time, the corresponding relationship between the labeled information and the block data is established to ensure the accuracy and traceability of the data;

[0123] Data normalization:

[0124] Normalize all the labeled block data and pass the formula:

[0125]

[0126] Calculate, where Represents a pixel value in the original image dataset, represents the pixel value after normalization. is the mean of the image dataset, is the standard deviation of the image dataset, using the commonly used mean of the ImageNet dataset and standard deviation ; Take the pixel value of a block in a 1080P picture as an example. Assuming that the original RGB value of a pixel in the block is (100, 80, 60), calculate its normalized value:

[0127] For the red channel (In order to demonstrate the calculation process here, the entire data set will be batch normalized in actual applications);

[0128] Similarly, for the green channel and blue channel Perform similar calculations to obtain normalized pixel values, standardize the data to a mean of 0 and a standard deviation of 1, speed up the convergence of model training, and improve the generalization ability of the model;

[0129] 2. Model construction and training

[0130] Model architecture design:

[0131] Construct a network structure model based on CNN convolutional neural network, including 2 convolutional layers, 2 pooling layers and 2 fully connected layers;

[0132] Convolution layer: The convolution kernel size of the first convolution layer is 3×3, and the number of convolution kernels is 32; the convolution kernel size of the second convolution layer is 5×5, and the number of convolution kernels is 64; the convolution layer performs convolution operations by sliding the convolution kernel on the image block to extract local features of the video frame, such as edges, textures, and other information; for example, for an input image block (assuming it is a block data after preprocessing), the convolution calculation process is as follows: Let the convolution kernel weight ( and From 0 to 2 or 0 to 4, depending on the kernel size), the input image is divided into blocks at positions The pixel value is , bias term , then the convolutional layer output feature map is Pixel value at position:

[0133]

[0134] in, is the convolution kernel size;

[0135] Pooling layer: Maximum pooling is used, the pooling window size is 2×2, and the step size is 2; using the formula:

[0136]

[0137] Perform dimensionality reduction operations, where is the pooling window area; for example, for a 2×2 area in the input feature map (assuming its pixel values ​​are 2, 3, 4, and 5 respectively), after pooling, the pixel value at the corresponding position of the output feature map is 5 (the maximum value in the area); the role of the pooling layer is to reduce the amount of data and maintain the main features to prevent overfitting;

[0138] Fully connected layer: The first fully connected layer has 256 neurons, and the second fully connected layer has 9 neurons, corresponding to 8 abnormal types and 1 normal type; the fully connected layer maps the extracted features to different abnormal category spaces and calculates the output value using the following formula:

[0139]

[0140] (Here we assume that the feature vector dimension of the first fully connected layer output is 256, that is, ),in, For the The fully connected layer output value corresponding to each category, is the weight, is the first feature vector of the input elements, is the input feature vector dimension, is the bias term; according to the output value Perform classification and judgment, detect different types of anomalies at the same time, extract features based on video frames, classify and judge, output abnormal status and identify abnormal types;

[0141] Model training process

[0142] The model training uses the structural similarity index (SSIM) as the loss function:

[0143]

[0144] in, and are two images to be compared, and is an image and The mean of and is an image and The variance of yes and The covariance of and It is a small constant introduced to avoid the denominator being 0;

[0145] The normalized data set is divided into a training set and a validation set in a ratio of 80%:20%. During the training process, the learning rate is set to 0.001 and the number of iterations is set to 80. Synthetic abnormal data is used to generate simulated abnormal video frames, such as randomly adding salt and pepper noise (noise density is 0.05) to normal images, simulating partial area occlusion (randomly selecting a block area to occlude 50% of the area), etc., to increase the model's generalization ability for abnormal patterns. In each iteration, the SSIM value between the current model prediction result and the true label is calculated, and then the model's convolution kernel weight, fully connected layer weight, bias term and other parameters are adjusted according to the gradient descent algorithm to gradually reduce the SSIM value and make the model prediction result closer to the true label.

[0146] 3. Model performance evaluation

[0147] Evaluation indicator calculation:

[0148] After the model training is completed, the validation set is used to evaluate the model performance; the accuracy, precision, recall, F1-score and other indicators are calculated; assuming that in one validation, the number of samples correctly detected as abnormal (TP) is 60, the number of samples correctly detected as normal (TN) is 180, the number of normal samples incorrectly detected as abnormal (FP) is 8, and the number of abnormal samples incorrectly detected as normal (FN) is 12;

[0149] Accuracy:

[0150]

[0151] Accuracy:

[0152]

[0153] Recall:

[0154]

[0155] F1-score:

[0156]

[0157] Evaluation results analysis

[0158] According to the evaluation results, the accuracy of the current model has reached 0.923, indicating that the model can generally accurately determine the category of samples (normal or abnormal); the precision is 0.882, indicating that among the samples predicted by the model as abnormal, the proportion of actual abnormalities is high and the false alarm rate is relatively low; the recall rate is 0.833, which means that the proportion of actual abnormal samples that can be correctly detected by the model is acceptable, but there is still room for improvement, and some abnormal situations may not be discovered in time; the F1-score is 0.857, which takes into account the precision and recall rate, reflecting the performance of the model under unbalanced data sets (the number of normal samples and abnormal samples is quite different);

[0159] Compared with the preset performance requirements (such as accuracy reaching 0.95, F1-score reaching 0.9, etc.), although the model has achieved certain results, it still needs to be improved; analyze the factors that may affect the performance of the model, such as the size and diversity of the data set, the complexity of the model architecture, the setting of training parameters, etc.; consider further improving the model performance by increasing the amount of training data (especially adding some special scenarios or rare abnormal types of data), adjusting the model architecture (such as adding convolutional layers or adjusting the convolution kernel size), optimizing training parameters (such as adjusting the learning rate, number of iterations, etc.), etc., to provide a more reliable foundation for subsequent anomaly detection;

[0160] 3. Summary of Examples

[0161] Example 1 shows in detail the training process of the basic large model in the network camera working anomaly detection method, from data collection and preprocessing, model construction and training to model performance evaluation, forming a model with preliminary detection capabilities. By analyzing the evaluation results, the advantages and disadvantages of the model are clarified, providing a direction for subsequent improvement and optimization. In practical applications, the model training process can be further adjusted and improved according to specific needs and resource conditions to improve the accuracy and reliability of the model and better realize the detection of network camera working anomalies.

[0162] Example 2: Anomaly detection and continuous model optimization

[0163] 1. Purpose of the Embodiment

[0164] Based on the basic large model constructed in Example 1, this embodiment focuses on demonstrating the anomaly detection process of the network camera working anomaly detection system in actual operation, and how to optimize the model by continuously accumulating the detection results R0 to adapt to the complex and changeable actual situation in the security system and improve the accuracy and timeliness of the system's detection of abnormal situations;

[0165] 2. Implementation Steps

[0166] 1. Anomaly detection process

[0167] Get the data to be tested:

[0168] The video stream is obtained from the security system in real time, and key frames or video clips are taken from the video stream every 3 seconds as the data to be detected; the key frames are selected based on the degree of picture change and the time interval; by calculating the pixel difference between adjacent frames (such as calculating the sum of the RGB value differences of each pixel between two frames, setting the threshold to 8000, when the sum of the differences exceeds the threshold, it is considered that the picture has changed significantly), frames with large picture changes are selected first, such as frames with scene switching, rapid object movement, etc.; at the same time, in order to ensure the integrity of the data, some key frames are evenly selected as supplements according to the time interval; for example, within 3 seconds, if there is no frame with a picture change degree exceeding the threshold, the last frame within the 3 seconds is selected as the key frame; this can not only effectively capture important information in the video, but also ensure that representative data to be detected can be obtained in different situations;

[0169] Data segmentation and identification:

[0170] The video frames in the data to be detected are divided into blocks in the same way as in the training stage, that is, for video frames with a resolution of 1080P, they are divided into 4×4 rectangular blocks; the divided material to be detected (video frame block data) is placed in the basic large model trained in Example 1 for recognition; after recognition by the large model, a detection result R0 is obtained; the detection result R0 includes the abnormal probability of each block and possible abnormal type information, for example, the probability of a certain block being judged as an occlusion abnormality is 0.75, the probability of a signal loss abnormality is 0.05, etc.; the video clips are not divided into blocks, but are directly placed in the model for recognition. The model determines whether the video clip as a whole has an abnormality based on the learned feature pattern, such as whether the video clip has an overall brightness abnormality, continuous screen jitter, etc.;

[0171] Manual review and recording:

[0172] Arrange reviewers to review the test result R0 every 45 minutes; reviewers comprehensively judge whether the picture is truly abnormal by checking the test result R0 and the corresponding video picture; if the picture is judged to be abnormal, such as finding that a certain area in a certain monitoring picture is blocked and consistent with the test result R0, the reviewer records the abnormality type (such as the location of the blocked area, the degree of blockage, etc.), the time of occurrence (accurate to the second), the monitoring area (such as a production line in a factory workshop) and other detailed information in detail, and outputs the picture abnormality information to notify the relevant maintenance personnel to handle it; if the picture is judged to be normal, continue the next round of test result R0 review; for important monitoring areas (such as warehouse valuables storage areas, data center core computer rooms, etc.), set higher levels of review levels and permissions, shorten the review time interval to 30 minutes, and increase the number of reviewers to two people reviewing at the same time to ensure that abnormal situations can be discovered and handled in time to reduce security risks;

[0173] 2. Model Optimization Process

[0174] New data accumulation and judgment:

[0175] As the detection process continues, the detection results R0 continue to accumulate; when 300 detection results R0 are accumulated, the system automatically conducts a preliminary analysis of these results to determine whether model optimization is needed; the analysis content includes the distribution of abnormal types, the changing trend of model prediction accuracy, etc.; for example, if a new abnormal type that is not fully reflected in the training set (such as interference stripes of a certain pattern) is found, and its frequency of occurrence is gradually increasing, or the accuracy of the model for some common abnormal types shows a significant downward trend, the system determines that model optimization is needed;

[0176] Model retraining and parameter adjustment:

[0177] The accumulated 300 test results R0 are used as new test materials and put into the basic large model test program for retraining. During the training process, the training parameters are dynamically adjusted according to the characteristics of the new data. Since the new data may contain more complex anomalies, the learning rate is appropriately reduced to 0.0008 to adjust the model parameters more finely, and the number of iterations is increased to 100 times, so that the model can better learn the characteristic patterns in the new data. In addition, according to the distribution of anomaly types in the new data, the weights in the loss function are adjusted. For example, for newly appeared and frequent anomaly types, their weights in the loss function are appropriately increased to prompt the model to pay more attention to the detection accuracy of such anomalies.

[0178] Model performance evaluation and optimization effect verification:

[0179] After the model is retrained, a new test set (selected from the re-collected data that did not participate in the training) is used to evaluate the model's performance and recalculate indicators such as accuracy, precision, recall, and F1-score. Assume that after optimization, in the new test set, the number of samples correctly detected as abnormal (TP) is 70, the number of samples correctly detected as normal (TN) is 190, the number of normal samples incorrectly detected as abnormal (FP) is 5, and the number of abnormal samples incorrectly detected as normal (FN) is 10;

[0180] Accuracy:

[0181]

[0182] Accuracy:

[0183]

[0184] Recall:

[0185]

[0186] F1-score:

[0187]

[0188] Compared with the model performance indicators before optimization, it was found that the accuracy rate increased from 0.923 to 0.945, the precision rate increased from 0.882 to 0.933, the recall rate increased from 0.833 to 0.875, and the F1-score increased from 0.857 to 0.903, indicating that the model optimization has achieved good results and the model's ability to detect abnormal situations has been significantly improved;

[0189] Perform optimization operations regularly:

[0190] After multiple accumulations and model optimizations of new detection materials, the model performance tends to be stable, and the detection result R0 accuracy is high; at this time, the system sets a periodic review and model optimization operation cycle of once every two days; during the regular execution process, the model performance indicators such as accuracy and recall rate are continuously monitored; if the indicators fluctuate or show a downward trend, the execution cycle is adjusted to once a day in a timely manner, and the new data and model parameters are rechecked and targeted optimization is performed to ensure that the model always maintains a good performance state and achieves efficient and accurate detection of abnormalities in network camera operations;

[0191] 3. Summary of Examples

[0192] This embodiment demonstrates the complete process of the network camera working anomaly detection system in actual operation based on the basic large model constructed in Example 1, including two key parts: anomaly detection and continuous model optimization; through a reasonable strategy for acquiring data to be detected, a strict manual review mechanism and an effective model optimization method, the system can promptly detect and handle abnormal situations in the work of the network camera, while continuously improving the model's ability to detect various anomalies; as the system continues to run, the model constantly adapts to new abnormal situations, providing more reliable protection for security monitoring, effectively solving the problems existing in the prior art, and having high practical value and promotion significance; in actual applications, the parameters and strategies of each link can be further adjusted and optimized according to the specific needs and characteristics of different security systems to achieve the best detection effect.

[0193] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting abnormal operation of a network camera, characterized in that: The following steps are involved: Step S1, training a basic large model formed by a basic data set; Step S1-1, collecting normal monitoring images and abnormal monitoring images or video clips from the existing security system as data basis, and evenly cutting each image into blocks; Step S1-2, putting the segmented data base into the basic large model for training, labeling each data base sample during training, if it is a data base with a normal picture, it is marked as a normal picture / video, and then the marked data base is preprocessed; Step S1-3: After the data foundation is preprocessed, multi-task learning is performed through a network structure model based on a CNN convolutional neural network, and different types of anomalies are detected at the same time. Feature classification judgment is extracted based on the video frame, and the abnormal state is output and the abnormal type is identified; the model training uses the structural similarity index SSIM as the loss function, and uses synthetic abnormal data to generate simulated abnormal video frames to enhance the generalization ability; Step S1-4: Comprehensively evaluate the model performance using accuracy, precision, recall, and F1-score; Step S2, obtaining a video stream from the security system, taking out key frames or video clips from the video stream as data to be detected; dividing the video frames in the data to be detected into blocks as materials to be detected, putting the materials to be detected into the basic large model for identification, and obtaining a detection result R0; Step S3: The reviewer regularly reviews the test result R0, and if the picture is abnormal, the abnormal picture is output; Step S4: When the detection results R0 accumulate to a set number, they are used as new detection materials to train the basic large model; Step S5: When the new detected material reaches the set threshold, step S3 and step S4 are executed regularly; When evenly dividing each picture into blocks, determine the way to divide each picture into blocks. rectangular blocks, where is a positive integer; calculate the size of each rectangular block, for a resolution of The number of pixels in each block in the horizontal direction is , the vertical direction is ; Starting from the upper left corner of the picture, divide the picture in sequence according to the calculated block size, and number the divided blocks; When preprocessing the labeled data base, the data base is processed through the formula Normalization is performed, where Represents a pixel value in the original image dataset, represents the pixel value after normalization. is the mean of the image dataset, is the standard deviation of the image dataset, using the commonly used mean of the ImageNet dataset and standard deviation , standardize the data to a mean of 0 and a standard deviation of 1; The network structure model based on the CNN convolutional neural network includes multiple convolutional layers, pooling layers and fully connected layers. The convolutional layer performs convolution operations by sliding the convolution kernel on the image, which is expressed by the following formula: ； in, The output feature map of the convolutional layer is The pixel value of the position, is the convolution kernel weight, is the pixel value of the input image at the corresponding position, is the bias term, is the convolution kernel size, extracting local features of the video frame; The pooling layer uses the following formula to perform dimensionality reduction: ;in, is the pooling window area; The fully connected layer maps the extracted features to different abnormal category spaces through the formula Calculate the output value, according to the output value Make classification judgments, among which, For the The fully connected layer output value corresponding to each category, is the weight, is the i-th element of the input feature vector, is the input feature vector dimension, It is a bias term. At the same time, it detects different types of anomalies, extracts features from video frames, classifies and judges them, outputs the anomaly status and identifies the anomaly type; The model training uses the structural similarity index SSIM as the loss function. The loss function formula is as follows: ； in, and are two images to be compared, and is an image and The mean of and is an image and The variance of yes and The covariance of and is a small constant introduced to avoid the denominator being; The model parameters are optimized by minimizing the loss function; synthetic abnormal data are used to generate simulated abnormal video frames to increase the model's generalization ability for abnormal patterns.

2. The method for detecting abnormal operation of a network camera according to claim 1, characterized in that: The comprehensive evaluation of model performance using accuracy, precision, recall, and F1-score includes the following steps: The accuracy rate measures the proportion of correctly classified samples to the total number of samples and is calculated using the following method: ； in, is the true positive, that is, the number of samples correctly detected as abnormal, is the true negative, that is, the number of samples correctly detected as normal. is the number of false positives, i.e., normal samples are mistakenly detected as abnormal. is the number of false negatives, i.e., abnormal samples are mistakenly detected as normal; Precision measures the proportion of samples that are actually abnormal among all samples predicted to be abnormal. It is calculated as follows: ； The recall rate measures the proportion of actual abnormal samples that are correctly detected and is calculated using the following method: ； F1-score is the harmonic mean of precision and recall, and is calculated as: ; Determine whether the model meets the requirements based on the evaluation results. If not, adjust the model parameters or retrain.

3. A network camera abnormality detection system, characterized by: The system is applied to the method for detecting abnormal operation of a network camera as described in any one of claims 1-2.

Images