Intelligent fault monitoring method, device, equipment and system for security camera

Abstract

The invention discloses an intelligent fault monitoring method, device, equipment and system for a security camera, and relates to the related field of intelligent management of image communication equipment, and the method comprises the steps: carrying out the multi-mode data collection of the security camera; standardization processing and feature extraction are carried out, and multi-resolution image features, time sequence network features and hardware state features are generated; calling a pre-training generator to generate pseudo data simulating a normal state; inputting the multi-modal data collected in real time and the pseudo data into a pre-training discriminator for comparison, and outputting an abnormality determination result; and performing fault analysis according to the abnormality judgment result, and generating a fault monitoring result. The technical problems of low monitoring precision and poor reliability caused by the fact that an existing fault monitoring method for the security and protection camera cannot comprehensively cover the operation state of the camera are solved, and the technical effects of comprehensively and accurately monitoring the fault of the security and protection camera and improving the monitoring precision and reliability are achieved.

Description

Technical Field

[0001] The present application relates to the field of intelligent management of image communication equipment, and in particular to an intelligent fault monitoring method, device, equipment and system for security cameras. Background Art

[0002] With the enhancement of security awareness, the application of security systems is becoming more and more widespread. Among them, security cameras are important security monitoring equipment, and their stability and reliability are crucial. Security systems not only need to capture and record video data in real time, but also need to ensure the stability of network connections and the normal operation of hardware equipment. However, traditional methods mainly rely on single indicators such as the integrity and clarity of video data to judge the working status of the camera. This method cannot fully reflect the overall operating status of the camera, especially when the fault originates from the network or hardware, it is often not discovered in time, and there are fault omissions or false alarms.

[0003] In the current related technologies, the fault monitoring method used for security cameras cannot fully cover the operating status of the camera, resulting in technical problems such as low monitoring accuracy and poor reliability. Summary of the invention

[0004] The present application provides an intelligent fault monitoring method, device, equipment and system for security cameras. By collecting multimodal data, including video data, network status data and hardware status data, the multimodal data is standardized and feature extracted to generate multi-resolution image features, time series network features and hardware status features. A pre-trained generator is used to learn the distribution of multimodal data in a normal state, pseudo data simulating a normal state is generated, and the pseudo data is compared with the multimodal data collected in real time. The abnormal judgment result is output by a pre-trained discriminator, and the fault analysis is performed according to the abnormal judgment result to generate fault monitoring results and other technical means, thereby achieving the technical effect of realizing comprehensive and accurate fault monitoring of security cameras and improving monitoring accuracy and reliability.

[0005] The present application provides an intelligent fault monitoring method for a security camera, comprising: Multimodal data collection is performed on the security camera, and the multimodal data includes video data, network status data, and hardware status data; the multimodal data is standardized and feature extracted to generate multi-resolution image features, time series network features, and hardware status features; a pre-trained generator is called to generate pseudo data simulating a normal state, and the pre-trained generator is trained based on the multi-resolution image features, time series network features, and hardware status features, and is used to learn the distribution of multimodal data under a normal state; the multimodal data collected in real time and the pseudo data generated by the pre-trained generator are input into a pre-trained discriminator for comparison, and an abnormal judgment result is output; fault analysis is performed according to the abnormal judgment result to generate a fault monitoring result.

[0006] In a possible implementation, before calling the pre-trained generator to generate pseudo data simulating a normal state, the following processing is performed: Build a generator structure; collect historical normal operation data of security cameras to build a training data set; use the training data set to train and converge the generator structure to obtain the pre-trained generator.

[0007] In a possible implementation, the generator structure is constructed to perform the following processing: Input layer, the input layer is used to input latent variables, and the latent variables include conditional variables, and the conditional variables are current environmental information; convolution layer, the convolution layer includes an encoder part and a decoder part, the encoder gradually reduces the dimension of the input space through multiple convolution layers to extract high-level features, and the decoder gradually restores it to the output space through the deconvolution layer to generate an image consistent with the size of the input data; jump connection layer, used to pass the output of the encoder to the decoder; multi-resolution generation layer, using a pyramid network structure, so that the generator generates data at multiple scales, gradually from low resolution to high resolution; output layer, used to output images and time series data that match the data under normal operating conditions.

[0008] In a possible implementation, the generator structure is trained and converged using the training data set, and the following processing is performed: Initialize the network weights of the generator and the discriminator; alternately train the generator and the discriminator, and use real data and pseudo data generated by the generator to train the discriminator; optimize the generator through back propagation; optimize the generator and the discriminator through adversarial loss function, and perform iterative alternating training until the convergence target is reached.

[0009] In a possible implementation, the multimodal data collected in real time is compared with the pseudo data generated by the pre-training generator and input into the pre-training discriminator, and the following processing is performed: Build a discriminator structure, which includes a multi-branch discriminant module, a multi-scale convolution module, and a hierarchical attention module, wherein each branch discriminant module processes image features, network state features, and hardware state features respectively, the multi-scale convolution module is used to extract features in different receptive field ranges, and the hierarchical attention module is used to integrate features at different levels; obtain the pre-trained discriminator by alternately training the discriminator structure and the generator; calculate the confidence of the pre-trained discriminator for the multimodal data and pseudo data collected in real time, mark the data whose confidence reaches the abnormal threshold, and output the abnormal judgment result.

[0010] In a possible implementation, the output abnormality determination result performs the following processing: The historical normal operation data is divided into multiple subsets to construct multiple training subsets; the multiple training subsets are used to independently train generators, and the pseudo data of the independently trained generators are fused to perform ensemble learning of the multiple generators to obtain an ensemble learning generator; the ensemble learning generator is used to perform alternating training with a multi-branch discriminant module, wherein the perceptual features output by the discriminant module are used to optimize the generation capability of the generator, and based on the shared feature layer of the multi-scale convolution module, the basic perceptual features are shared between different discriminators, and the output results of multiple branch discriminant modules are integrated through a weighted fusion strategy to generate a final abnormal discrimination result.

[0011] In a possible implementation, the multimodal data is standardized and feature extracted to generate multi-resolution image features, time series network features, and hardware status features, and the following processing is performed: Adjust the resolution of video data and the size of image data, perform standardization processing, extract multi-resolution image features, edge features, and texture features of video data, and obtain multi-resolution image features; normalize the network status data, convert it to a unified scale, and extract time series features from the normalized network data to obtain time series network features; normalize the hardware status data, extract status features at different time points through a sliding window, and obtain time series hardware status features.

[0012] The present application also provides an intelligent fault monitoring device for a security camera, comprising: A multimodal data acquisition module, the multimodal data acquisition module is used to collect multimodal data for security cameras, and the multimodal data includes video data, network status data, and hardware status data; a feature extraction module, the feature extraction module is used to perform standardization processing and feature extraction on the multimodal data, and generate multi-resolution image features and time series network features, and hardware status features; a pseudo data generation module, the pseudo data generation module is used to call a pre-trained generator to generate pseudo data that simulates a normal state, the pre-trained generator is trained based on the multi-resolution image features and time series network features, and hardware status features, and is used to learn the distribution of multimodal data under a normal state; an abnormality determination module, the abnormality determination module is used to compare the multimodal data collected in real time with the pseudo data generated by the pre-trained generator input into a pre-trained discriminator, and output an abnormality determination result; a fault analysis module, the fault analysis module is used to perform fault analysis according to the abnormality determination result and generate a fault monitoring result.

[0013] The present application also provides an electronic device, including: A memory for storing executable instructions; The processor is used to implement an intelligent fault monitoring method for security cameras when executing the executable instructions stored in the memory.

[0014] The present application also provides a computer-readable storage medium, comprising: A computer program is stored thereon, and when the program is executed by a processor, an intelligent fault monitoring method for a security camera is implemented.

[0015] The intelligent fault monitoring method, device, equipment and system for security cameras proposed in this application first collect multimodal data of the security camera, and the multimodal data includes video data, network status data, and hardware status data. Then, the multimodal data is standardized and feature extracted to generate multi-resolution image features, time series network features, and hardware status features. Then, a pre-trained generator is called to generate pseudo data simulating a normal state. The pre-trained generator is trained based on multi-resolution image features, time series network features, and hardware status features, and is used to learn the distribution of multimodal data under a normal state. Then, the multimodal data collected in real time and the pseudo data generated by the pre-trained generator are input into the pre-trained discriminator for comparison, and an abnormal judgment result is output. Finally, fault analysis is performed based on the abnormal judgment result to generate a fault monitoring result, thereby achieving a technical effect of realizing comprehensive and accurate fault monitoring of security cameras and improving monitoring accuracy and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the accompanying drawings of the embodiment of the present invention will be briefly introduced below. A flow chart is used in the present application to illustrate the operations performed by the device according to the embodiment of the present application. It should be understood that the previous or following operations are not necessarily performed accurately in order. On the contrary, various steps can be processed in reverse order or simultaneously as needed. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0017] Figure 1 A flowchart of an intelligent fault monitoring method for a security camera provided in an embodiment of the present application.

[0018] Figure 2 A schematic diagram of the structure of an intelligent fault monitoring device for a security camera provided in an embodiment of the present application.

[0019] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0020] Explanation of the reference numerals: multimodal data acquisition module 10 , feature extraction module 20 , pseudo data generation module 30 , abnormality determination module 40 , fault analysis module 50 , input device 301 , processor 302 , memory 303 , output device 304 . DETAILED DESCRIPTION

[0021] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0022] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0023] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "including" and "having" and any variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field of this application. The terms used herein are for the purpose of describing the embodiments of the present application only.

[0024] The present application embodiment provides an intelligent fault monitoring method for a security camera, such as Figure 1 As shown, the method includes: Step S100, collect multimodal data from the security camera, and the multimodal data includes video data, network status data, and hardware status data. Specifically, start the security camera and turn on the data collection function, and the security camera begins to collect multiple types of data, including video data (real-time video stream captured by the security camera for monitoring and analysis), network status data (data reflecting the camera network connection status, such as network delay, packet loss rate, etc.) and hardware status data (data reflecting the camera hardware operating status, such as camera temperature, power supply voltage, etc.). The collected multimodal data from different sources or types are stored in a designated storage medium for subsequent processing and analysis.

[0025] Step S200, the multimodal data is standardized and feature extracted to generate multi-resolution image features, time series network features, and hardware status features. Specifically, the collected multimodal data is preprocessed, including adjusting the resolution of the video data, the size of the image data, etc., normalizing the network status data, and normalizing the hardware status data. Useful feature information is extracted from the standardized data, including multi-resolution image features (such as edge features, texture features), time series network features, and time series hardware status features. Among them, multi-resolution image features refer to image features extracted at different resolutions, which are used to reflect the information of the image at different scales. Time series network features are data features that reflect the changes of network status over time, such as fluctuations in network traffic, etc. Time series hardware status features are data features that reflect the changes of hardware operating status over time, such as fluctuations in temperature and power supply voltage, etc.

[0026] In a possible implementation, the multimodal data is standardized and feature extracted to generate multi-resolution image features and time series network features and hardware status features. Step S200 further includes step S210, adjusting the resolution and image data size of the video data, performing standardization, extracting multi-resolution image features, edge features, and texture features of the video data, and obtaining multi-resolution image features. Specifically, the video data stream is obtained from the security camera in real time, and the resolution of the video data (the number of pixels contained in a unit length in the image, which determines the clarity and detail of the image) is adjusted to meet different analysis requirements. For example, the resolution is reduced to reduce the amount of calculation, and the resolution is increased to obtain more detailed image information. The video frame image after the resolution is adjusted is cropped or scaled to a uniform size to ensure the consistency of subsequent feature extraction. The image data is standardized, such as denoising, to improve the accuracy of feature extraction. Using image processing algorithms, the features of the image at different resolutions are extracted, including edge features, texture features, etc. These features reflect the information of the image at different scales and are helpful for anomaly detection. The extracted multi-resolution image features are integrated to form a multi-resolution image feature set.

[0027] Step S220, normalize the network status data, convert it into a unified scale, and extract time series features from the normalized network data to obtain time series network features. Specifically, obtain network status data from the network module of the security camera, such as network delay, packet loss rate, etc. The network status data is converted into a unified scale to eliminate the dimensional differences between different data. Use the time series analysis algorithm to extract features from the network status data, and these features reflect the change of network status over time, such as trend, periodicity, etc. The extracted time series features are integrated to form a sequence network feature set. Step S230, normalize the hardware status data, extract the status features at different time points through a sliding window, and obtain the hardware status features of the time series. Specifically, obtain hardware status data from the hardware module of the security camera, such as temperature, power supply voltage, etc. The hardware status data is converted into a unified scale to eliminate the dimensional differences between different data. Use the sliding window technology to extract the features of the hardware status data at different time points, and these features reflect the change of the hardware status over time. The extracted time series features are integrated to form a hardware status feature set of the time series. This implementation method improves the accuracy and robustness of image feature extraction by adjusting the resolution of video data and the size of image data, performing standardization processing, and extracting multi-resolution image features. It captures the changing patterns of network status by normalizing network status data and extracting time series features, which helps to detect network faults in a timely manner. It reflects the changes in hardware status by normalizing hardware status data and extracting sliding window features, which helps to detect hardware faults in a timely manner, achieving the technical effect of obtaining comprehensive and accurate fault monitoring information.

[0028] Step S300, calling a pre-trained generator to generate pseudo data simulating a normal state, the pre-trained generator is trained based on the multi-resolution image features, time series network features, and hardware state features, and is used to learn the distribution of multimodal data in a normal state. Specifically, the pre-trained generator is loaded from a storage medium, and the pre-trained generator refers to a generator model that has been trained and converged. This model has learned the distribution characteristics of multimodal data in a normal state in the previous training stage, including multi-resolution image features, time series network features, and hardware state features, and can generate pseudo data that is very close to the normal state. These pseudo data are difficult to distinguish from real data in appearance and features, and are used as a benchmark for the normal state in subsequent comparative analysis. The pre-trained generator is used to transform the potential variables (variables that affect the output of the generator, also known as input noise) to generate pseudo data that is very close to the normal state, which is used to compare with the real-time collected data to identify potential faults or anomalies. The generated pseudo data is stored in a specified storage medium for subsequent use.

[0029] In a possible implementation, before calling the pre-trained generator to generate pseudo data simulating a normal state, the method further includes step S600, building a generator structure. Specifically, using a deep learning model architecture as the basis of the generator, the generator can generate seemingly real data samples from a latent space. According to the model architecture, the network layer of the generator is designed, including an input layer, a hidden layer, and an output layer, wherein the hidden layer is used to learn the complex features of the data. Determine the learning rate, batch size, number of iterations, and other hyperparameters of the network, which affect the training effect and convergence speed of the model. Step S700, collect the historical normal operation data of the security camera and construct a training data set. Specifically, collect video data, network status data, and hardware status data of the historical normal operation period from the security camera, remove noise, missing values, etc. in the data, and ensure data quality. Since the data here is data in a normal state, no additional labels are required, and the data only needs to be organized into a format suitable for model training. The cleaned data is divided into a training set, a validation set, and a test set for model training, validation, and testing. Step S800, use the training data set to train the generator structure to converge, and obtain the pre-trained generator. Specifically, the training data set is input into the generator, and the model parameters are adjusted through the back-propagation algorithm so that the data output by the generator is closer and closer to the real data. Use the loss function to measure the difference between the generator output and the real data, monitor the change of the loss function value, and when the loss value no longer decreases significantly or reaches the preset number of iterations, the model is considered to have converged. Save the trained generator model as a pre-trained generator for subsequent use. This implementation method trains the generator with historical normal operation data to obtain a pre-trained generator, which enables the model to adapt and detect faults faster when deployed in the actual environment, reducing the adaptation time of the model in the new environment.

[0030] In a possible implementation, the generator structure is constructed, and step S600 further includes step S610, an input layer, the input layer is used to input potential variables, the potential variables include conditional variables, and the conditional variables are current environmental information. Specifically, the input layer receives potential variables as inputs of the generator, and these potential variables are randomly sampled from a potential space, and they carry the randomness and diversity required to generate data. These potential variables also include conditional variables, specifically referring to current environmental information, such as lighting conditions, temperature, humidity, etc., which are essential for generating images and time series data that match the current environment. Step S620, a convolutional layer, the convolutional layer includes an encoder part and a decoder part, the encoder gradually reduces the dimension of the input space through multiple convolutional layers, extracts high-level features, and the decoder gradually restores to the output space through a deconvolution layer to generate an image of the same size as the input data. Specifically, the convolutional layer consists of an encoder and a decoder. The encoder part gradually extracts the features of the input data through multiple convolutional layers, while reducing the spatial dimension of the data. Each convolution layer contains multiple convolution kernels, which slide on the input data to calculate the feature values ​​of the local area. As the convolution layer goes deeper, the extracted features gradually transition from low-level to high-level. The decoder part gradually restores the spatial dimension of the data through the deconvolution layer, and generates an image of the same size as the input data. The deconvolution layer gradually enlarges the low-resolution feature map to the size of the original image through upsampling operations. Step S630, a jump connection layer, is used to pass the output of the encoder to the decoder. Specifically, the jump connection layer directly connects the output of the convolution layer in the encoder to the corresponding layer in the decoder. This connection allows the decoder to directly access the intermediate feature representation of the encoder when generating data, thereby retaining more detailed information. The jump connection layer helps to alleviate the gradient vanishing problem, accelerate the training process of the model, and improve the quality of generated data.

[0031] Step S640, a multi-resolution generation layer, using a pyramid network structure, enables the generator to generate data at multiple scales, gradually from low resolution to high resolution. Specifically, the multi-resolution generation layer adopts a pyramid network structure, which enables the generator to generate data at multiple scales. This layer gradually generates images and time series data from low resolution to high resolution by gradually increasing the resolution of the data. In the low-resolution stage, the generator first generates a rough framework of the image or time series data, and then gradually adds details and texture information as the resolution increases. This multi-resolution generation strategy helps to improve the fidelity and detail richness of the generated data. Step S650, an output layer, is used to output images and time series data that match the data under normal operating conditions. Specifically, the output layer is the last layer of the generator, which is used to generate images and time series data that match the data under normal operating conditions. These generated data are used to compare with the data collected in real time to detect abnormal conditions. The data format and size of the output layer match the potential variables and conditional variables received by the input layer to ensure that the generated data can be effectively compared and analyzed with the data collected in real time. This implementation method improves the generation quality and training efficiency of the generator through technical means such as multi-resolution generation, conditional variable integration, jump connection and latent variable guidance, thereby improving the intelligent fault monitoring capability of security cameras, enabling them to more accurately detect abnormal conditions and generate corresponding fault monitoring results.

[0032] In a possible implementation, the generator structure is trained and converged using the training data set, and step S800 further includes step S810, initializing the network weights of the generator and the discriminator. Specifically, the weights and biases of each network layer in the generator and the discriminator are initialized using random values ​​or specific initialization methods (such as He initialization, Xavier initialization, etc.), and these initialized weights and biases are the starting point of the subsequent training process. Step S820, the generator and the discriminator are alternately trained, and the discriminator is trained using real data and pseudo data generated by the generator. Specifically, the generator (a neural network for generating pseudo data similar to real data) and the discriminator (a neural network for distinguishing real data from pseudo data) are alternately trained. First, the discriminator is trained using real data (i.e., historical normal operation data of security cameras) so that it can distinguish between real data and pseudo data generated by the generator. Then, the generator is trained using feedback from the discriminator so that it can generate more realistic pseudo data that is more difficult to distinguish by the discriminator. This process is iterative, and each iteration includes one discriminator training and one generator training. Step S830, optimizing the generator by back propagation. Specifically, when training the generator, the back propagation algorithm is used to calculate the gradient of the generator parameters, and the weights and biases of the generator are updated according to these gradients. The back propagation algorithm guides the update direction of the parameters by calculating the derivative of the loss function (i.e., the difference between the pseudo data generated by the generator and the real data) with respect to the generator parameters. This process is iterative until the generator can generate sufficiently realistic pseudo data. Step S840, optimizing the generator and the discriminator by adversarial loss function, and training alternately by iteration until the convergence target is reached. Specifically, the adversarial loss function is used to optimize the generator and the discriminator at the same time. The adversarial loss function includes two parts: one part is used to measure the ability of the discriminator to distinguish between real data and pseudo data (i.e., the accuracy of the discriminator), and the other part is used to measure the similarity between the pseudo data generated by the generator and the real data (i.e., the fidelity of the generator). By iteratively alternating the training of the generator and the discriminator and minimizing the adversarial loss function, the ability of the generator to generate pseudo data is gradually improved, while the ability of the discriminator to distinguish between real data and pseudo data is improved. When the training reaches convergence, that is, when the generator can generate data that is almost indistinguishable from normal data, the generator model is saved as a pre-trained generator. This implementation gradually improves the generator's ability to generate pseudo data by alternately training the generator and the discriminator and optimizing their performance using an adversarial loss function, which enables the generator to learn the distribution characteristics of normal data and thus generate pseudo data similar to normal data.

[0033] Step S400, the multimodal data collected in real time and the pseudo data generated by the pre-trained generator are input into the pre-trained discriminator for comparison, and the abnormality determination result is output. Specifically, the pre-trained discriminator is loaded from the storage medium, and the pre-trained discriminator refers to a discriminator model that has been trained and converged, and can distinguish whether the input data is real data or generated data. The multimodal data collected in real time and the generated pseudo data are respectively input into the discriminator model, and the discriminator compares and analyzes the input data, and outputs the abnormality determination result, which is used to determine whether the security camera has an abnormality.

[0034] In a possible implementation, the multimodal data collected in real time is compared with the pseudo data generated by the pre-trained generator and input into the pre-trained discriminator for comparison. Step S400 further includes step S410, building a discriminator structure, the discriminator structure includes a multi-branch discriminant module, a multi-scale convolution module, and a hierarchical attention module, wherein each branch discriminant module processes image features, network state features, and hardware state features respectively, the multi-scale convolution module is used to extract features of different receptive field ranges, and the hierarchical attention module is used to integrate features of different levels. Specifically, the discriminator structure is designed to distinguish whether the input data is real data (i.e., multimodal data collected in real time by the security camera) or pseudo data generated by the generator. In the discriminator structure, multiple branches are designed, each branch specializing in processing one type of feature (such as image features, network state features, and hardware state features) to ensure that each feature is fully utilized and accurately analyzed. In each branch, a multi-scale convolution module is introduced to extract features of different receptive field ranges to capture detailed information and global information in the data and improve the accuracy of the discrimination. In the final stage of the discriminator, a hierarchical attention module is introduced to integrate features at different levels, fuse feature information at different scales and levels, and form a more comprehensive feature representation, thereby improving the robustness of the discrimination. Step S420, the pre-trained discriminator is obtained by alternately training the discriminator structure and the generator. Specifically, before the training starts, the network weights of the discriminator are initialized. During the training process, the generator and the discriminator are alternately trained, the generator generates pseudo data, and the discriminator attempts to distinguish between real data and pseudo data. The discriminator is trained using real data and pseudo data generated by the generator, and the network weights of the discriminator are optimized by the back propagation algorithm. Repeat the above process until the discriminator's discriminative ability reaches its limit, that is, it cannot effectively distinguish between true and false samples. At this time, the discriminator is considered to have been pre-trained. Step S430, calculate the confidence of the pre-trained discriminator for the multimodal data and pseudo data collected in real time, mark the data whose confidence reaches the abnormal threshold, and output the abnormal judgment result. Specifically, the multimodal data collected in real time and the pseudo data generated by the generator are simultaneously input into the pre-trained discriminator. The discriminator processes the input data and outputs the confidence of each data point. The confidence indicates the probability that the discriminator believes that the data point is real data. An abnormal threshold is set. When the confidence of a data point is lower than the threshold, the data point is considered abnormal. Data points with confidence lower than the abnormal threshold are marked and output as abnormal judgment results.

[0035] In this implementation, the discriminator aims to learn to distinguish between real data (i.e., security camera data in normal state) and pseudo data generated by the generator during training. The discriminator gradually improves its ability to distinguish between real data and pseudo data through continuous iterative learning. The goal of the generator is to generate pseudo data that is very close to the real data to deceive the discriminator. When the generator is trained well enough, it can generate highly realistic pseudo data whose distribution is very close to that of the real data. When the discriminator cannot clearly distinguish whether the input data is real data or pseudo data generated by the generator, it means that the distribution of the input data is very similar to that of the pseudo data generated by the generator. Since the generator is trained based on data in normal state, when the distribution of the input data is similar to that of the pseudo data generated by the generator, it can be inferred that the input data is also in normal state. Under normal operating conditions, the data of security cameras (such as video data, network status data, and hardware status data) have stable statistical characteristics. By learning these stable statistical characteristics, the generator can generate pseudo data that is very close to the real data. Therefore, when the discriminator cannot distinguish between input data and generated data, it is considered that the distribution of the input data is consistent with the distribution of the normal state, that is, the system is in a normal state. When the data from the security camera is abnormal, its distribution will deviate from the distribution of the normal state. Since the generator has only learned the distribution of the normal state, it cannot generate pseudo data similar to the abnormal state data. At this time, the discriminator can clearly distinguish between the input data and the pseudo data generated by the generator, thereby judging that the system is abnormal.

[0036] In a possible implementation, the output abnormality determination result, step S430 further includes step S431, dividing the historical normal operation data into multiple subsets, and constructing multiple training subsets. Specifically, the historical normal operation data is divided into multiple subsets, each subset contains a certain amount of data samples, and each subset reflects the data characteristics of the normal state. The generator is independently trained for each subset, that is, each subset is used to train a separate generator model, and these models will learn the distribution characteristics of the data in their respective subsets. Step S432, the generator is independently trained using the multiple training subsets, and the pseudo data of the independently trained generators are fused to perform ensemble learning of multiple generators to obtain an ensemble learning generator. Specifically, pseudo data is generated by each independently trained generator, and the outputs (i.e., pseudo data) of multiple generators are fused to perform ensemble learning, that is, the diversity of multiple generators is used to generate more realistic pseudo data to better simulate the data distribution of the normal state. Through the results of ensemble learning, the generation ability of the generator is further optimized so that it can generate pseudo data that is closer to real data. Step S433, the generator of the ensemble learning is used to perform alternating training with the multi-branch discriminant module, wherein the perceptual features output by the discriminant module are used to optimize the generator's generation capability, and based on the shared feature layer of the multi-scale convolution module, the basic perceptual features are shared between different discriminators, and the output results of multiple branch discriminant modules are integrated through a weighted fusion strategy to generate the final abnormal discrimination result. Specifically, the generator of the ensemble learning is used to perform alternating training with the multi-branch discriminant module. During the training process, the generator continuously generates pseudo data to deceive the discriminator, while the discriminator strives to distinguish between real data and pseudo data. The perceptual features output by the discriminant module (features extracted by the discriminator from the input data to distinguish between real data and pseudo data) are used to optimize the generator's generation capability, that is, the generator will continuously adjust its generation strategy according to the feedback of the discriminator to generate pseudo data that is more difficult to be distinguished by the discriminator. Based on the shared feature layer of the multi-scale convolution module, the basic perceptual features are shared between different discriminators, which helps to improve the generalization ability of the discriminator, so that it can more accurately identify different types of data features. The output results of multiple branch discrimination modules are integrated to generate the final anomaly discrimination result through a weighted fusion strategy, in which the weighted fusion strategy assigns weights according to the reliability or importance of different branch discrimination modules, thereby obtaining a more accurate anomaly discrimination result.

[0037] This implementation method can generate more realistic and diverse pseudo data that simulates normal conditions by independently training multiple generators and using the pseudo data of these generators for ensemble learning. As the training progresses, the generator continuously generates more realistic pseudo data, making it increasingly difficult for the discriminator to distinguish them. When the discriminator cannot clearly distinguish between real data and pseudo data (i.e., reaches a "Nash equilibrium" state), it is considered that the generator has successfully simulated the data distribution of the normal state. In practical applications, the trained generator is used to generate pseudo data, and these pseudo data are used as the benchmark of the normal state. When the multimodal data collected in real time is input into the discriminator, the discriminator will compare these data with the pseudo data and output an abnormal judgment result. If the multimodal data collected in real time differs greatly from the pseudo data (i.e., exceeds the fluctuation range of the normal state), the discriminator will mark it as abnormal data. Therefore, this implementation method uses technical means such as ensemble learning, alternating training and weighted fusion strategy to train the discriminator to the state of "indistinguishable" real data and pseudo data, and uses pseudo data as the benchmark for abnormal judgment, thereby realizing accurate monitoring and abnormal identification of real-time data.

[0038] Step S500, perform fault analysis according to the abnormal judgment result and generate a fault monitoring result. Specifically, the abnormal judgment result output by the pre-trained discriminator is analyzed to determine the type and location of the fault, and a fault monitoring report is generated according to the analysis result, including information such as the fault type, location, and occurrence time. The generated fault monitoring report is output to relevant personnel or systems for subsequent processing and maintenance. The embodiment of the present application collects multimodal data, including video data, network status data, and hardware status data, and performs standardization processing and feature extraction on the multimodal data to generate multi-resolution image features, time series network features, and hardware status features. The distribution of multimodal data in a normal state is learned by a pre-trained generator, pseudo data simulating a normal state is generated, and compared with the multimodal data collected in real time, the abnormal judgment result is output by a pre-trained discriminator, and the fault analysis is performed according to the abnormal judgment result, and the fault monitoring result and other technical means are generated to achieve comprehensive and accurate fault monitoring of security cameras and improve the monitoring accuracy and reliability.

[0039] In the above, refer to Figure 1 The present invention describes in detail an intelligent fault monitoring method for a security camera according to an embodiment of the present invention. Figure 2 An intelligent fault monitoring device for a security camera according to an embodiment of the present invention is described.

[0040] According to an embodiment of the present invention, an intelligent fault monitoring device for a security camera is used to solve the technical problem that the existing fault monitoring method for security cameras cannot fully cover the operating status of the camera, resulting in low monitoring accuracy and poor reliability, so as to achieve the technical effect of realizing comprehensive and accurate fault monitoring of security cameras and improving monitoring accuracy and reliability. An intelligent fault monitoring device for a security camera includes: a multimodal data acquisition module 10, a feature extraction module 20, a pseudo data generation module 30, an abnormality determination module 40, and a fault analysis module 50.

[0041] The multimodal data acquisition module 10 is used to collect multimodal data from security cameras, and the multimodal data includes video data, network status data, and hardware status data; the feature extraction module 20 is used to perform standardization processing and feature extraction on the multimodal data, and generate multi-resolution image features, time series network features, and hardware status features; the pseudo data generation module 30 is used to call a pre-trained generator to generate pseudo data that simulates a normal state, and the pre-trained generator is trained based on the multi-resolution image features, time series network features, and hardware status features, and is used to learn the distribution of multimodal data under a normal state; the abnormality judgment module 40 is used to compare the multimodal data collected in real time with the pseudo data generated by the pre-trained generator input into a pre-trained discriminator, and output an abnormality judgment result; the fault analysis module 50 is used to perform fault analysis based on the abnormality judgment result and generate a fault monitoring result.

[0042] Among them, before calling the pre-trained generator to generate pseudo data simulating a normal state, the device may further include: a generator structure building module for building a generator structure; a training data set building module for collecting historical normal operation data of security cameras and building a training data set; a generator pre-training module for using the training data set to train and converge the generator structure to obtain the pre-trained generator.

[0043] Among them, the generator structure building module can further include: an input layer, the input layer is used to input potential variables, the potential variables include conditional variables, and the conditional variables are current environmental information; a convolution layer, the convolution layer includes an encoder part and a decoder part, the encoder gradually reduces the dimension of the input space through multiple convolution layers to extract high-level features, and the decoder gradually restores it to the output space through a deconvolution layer to generate an image consistent with the input data size; a jump connection layer, used to pass the output of the encoder to the decoder; a multi-resolution generation layer, using a pyramid network structure, so that the generator generates data at multiple scales, gradually from low resolution to high resolution; an output layer, used to output images and time series data that match the data under normal operating conditions.

[0044] Among them, the generator structure is trained and converged using the training data set, and the generator pre-training module can further include: a network weight initialization unit is used to initialize the network weights of the generator and the discriminator; an alternating training unit is used to alternately train the generator and the discriminator, and the discriminator is trained using real data and pseudo data generated by the generator; a generator optimization unit is used to optimize the generator through back propagation; an iterative alternating training unit is used to optimize the generator and the discriminator through an adversarial loss function, and iterative alternating training is performed until the convergence target is reached.

[0045] The specific configuration of the abnormality determination module 40 will be described in detail below. As described above, the multimodal data collected in real time is compared with the pseudo data generated by the pre-trained generator input into the pre-trained discriminator, and the abnormality determination module 40 may further include: a discriminator structure building unit is used to build a discriminator structure, and the discriminator structure includes a multi-branch discriminant module, a multi-scale convolution module, and a hierarchical attention module, wherein each branch discriminant module processes image features, network state features, and hardware state features respectively, and the multi-scale convolution module is used to extract features of different receptive field ranges, and the hierarchical attention module is used to integrate features of different levels; the discriminator pre-training unit is used to obtain the pre-trained discriminator through alternating training of the discriminator structure and the generator; the confidence calculation unit is used to calculate the confidence of the pre-trained discriminator for the multimodal data and pseudo data collected in real time, mark the data whose confidence reaches the abnormal threshold, and output the abnormality determination result.

[0046] Among them, the output abnormal judgment result, the confidence calculation unit can further include: a data division subunit is used to divide the historical normal operation data into multiple subsets to construct multiple training subsets; an integrated learning subunit is used to use the multiple training subsets to independently train the generator, and fuse the pseudo data of the independently trained generator to perform integrated learning of multiple generators to obtain an integrated learning generator; the abnormal discrimination subunit is used to use the integrated learning generator to perform alternating training with the multi-branch discrimination module, wherein the perceptual features output by the discrimination module are used to optimize the generation ability of the generator, and based on the shared feature layer of the multi-scale convolution module, the basic perceptual features are shared between different discriminators, and the output results of multiple branch discrimination modules are integrated through a weighted fusion strategy to generate a final abnormal discrimination result.

[0047] The specific configuration of the feature extraction module 20 will be described in detail below. As described above, the multimodal data is standardized and feature extracted to generate multi-resolution image features and time series network features and hardware status features. The feature extraction module 20 may further include: a video data feature extraction unit for adjusting the resolution and image data size of the video data, performing standardization, extracting multi-resolution image features, edge features, and texture features of the video data, and obtaining multi-resolution image features; a network status data feature extraction unit for normalizing the network status data, converting it to a uniform scale, and performing time series feature extraction on the normalized network data to obtain time series network features; a hardware status data feature extraction unit for normalizing the hardware status data, extracting status features at different time points through a sliding window, and obtaining time series hardware status features.

[0048] An intelligent fault monitoring device for a security camera provided by an embodiment of the present invention can execute an intelligent fault monitoring method for a security camera provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0049] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules may be used and run on the user terminal and / or server, and the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0050] Based on the foregoing embodiments, the embodiments of the present application further provide an electronic device and a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of the electronic device, it can implement the method described in any of the foregoing embodiments.

[0051] Figure 3 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, showing a block diagram of an exemplary electronic device suitable for implementing an implementation of the present invention. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention. The electronic device is in the form of a general computing device, and its components may include but are not limited to an input device 301, a processor 302, a memory 303, and an output device 304. Among them, the processor 302 may be one or more; the memory 303 may include a computer-readable medium and at least one program product, and the program product has a set (at least one) of program modules, which are configured to perform the functions of each embodiment of the present application.

[0052] The memory 303 shown in the embodiment of the present invention may adopt any combination of one or more computer-readable media; the computer-readable storage medium may be, but is not limited to, an infrared, semiconductor system, device or component, or any combination of the above, for storing software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the intelligent fault monitoring method for a security camera in the embodiment of the present invention. The processor 302 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 303, thereby realizing the above-mentioned intelligent fault monitoring method for a security camera.

[0053] The above specific implementation manner does not constitute a limitation to the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiment and can still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. An intelligent fault monitoring method for a security camera, characterized in that: The method comprises: Collect multimodal data from security cameras, including video data, network status data, and hardware status data; Performing standardization and feature extraction on the multimodal data to generate multi-resolution image features, time series network features, and hardware status features; Calling a pre-trained generator to generate pseudo data simulating a normal state, wherein the pre-trained generator is trained based on the multi-resolution image features, time series network features, and hardware state features, and is used to learn the distribution of multimodal data in a normal state; Input the multimodal data collected in real time into the pre-trained discriminator for comparison with the pseudo data generated by the pre-trained generator, and output an abnormality determination result; Fault analysis is performed according to the abnormality determination result to generate a fault monitoring result.

2. The intelligent fault monitoring method for security cameras according to claim 1, characterized in that: The calling of the pre-training generator to generate pseudo data simulating a normal state also includes: Build the generator structure; Collect historical normal operation data of security cameras and build training data sets; The generator structure is trained and converged using the training data set to obtain the pre-trained generator.

3. The intelligent fault monitoring method for security cameras according to claim 2, characterized in that: The generator structure includes: An input layer, the input layer is used to input potential variables, the potential variables include conditional variables, and the conditional variables are current environment information; Convolutional layer, the convolutional layer includes an encoder part and a decoder part. The encoder gradually reduces the dimension of the input space through multiple convolutional layers to extract high-level features, and the decoder gradually restores it to the output space through deconvolution layers to generate an image with the same size as the input data; A skip connection layer, used to pass the output of the encoder to the decoder; The multi-resolution generation layer uses a pyramid network structure to enable the generator to generate data at multiple scales, gradually from low resolution to high resolution; The output layer is used to output images and time series data that match the data under normal operating conditions.

4. The intelligent fault monitoring method for security cameras according to claim 2, characterized in that: The generator structure is trained and converged using the training data set, including: Initialize the network weights of the generator and discriminator; The generator and the discriminator are trained alternately, and the discriminator is trained using real data and pseudo data generated by the generator; Optimizing the generator by back-propagation; The generator and discriminator are optimized and trained through adversarial loss functions, and the training is iteratively alternating until the convergence target is reached.

5. The intelligent fault monitoring method for security cameras according to claim 4, characterized in that: The multimodal data collected in real time is compared with the pseudo data generated by the pre-training generator and input into the pre-training discriminator, including: Build a discriminator structure, which includes a multi-branch discriminant module, a multi-scale convolution module, and a hierarchical attention module, wherein each branch discriminant module processes image features, network state features, and hardware state features respectively, the multi-scale convolution module is used to extract features of different receptive fields, and the hierarchical attention module is used to integrate features of different levels; The pre-trained discriminator is obtained by alternately training the discriminator structure and the generator; The confidence of the pre-trained discriminator for the multimodal data and pseudo data collected in real time is calculated, the data whose confidence reaches the abnormal threshold is marked, and the abnormal determination result is output.

6. The intelligent fault monitoring method for security cameras according to claim 5, characterized in that: The output abnormality determination result includes: Divide the historical normal operation data into multiple subsets and construct multiple training subsets; Using the multiple training subsets to independently train generators respectively, fusing the pseudo data of the independently trained generators, performing ensemble learning of the multiple generators, and obtaining an ensemble learning generator; The generator using the ensemble learning is trained alternately with the multi-branch discriminant module, wherein the perceptual features output by the discriminant module are used to optimize the generation capability of the generator, and basic perceptual features are shared between different discriminators based on the shared feature layer of the multi-scale convolution module. The output results of multiple branch discriminant modules are integrated through a weighted fusion strategy to generate the final abnormality discrimination result.

7. The intelligent fault monitoring method for security cameras according to claim 1, characterized in that: The multimodal data is standardized and feature extracted to generate multi-resolution image features, time series network features, and hardware status features, including: Adjust the resolution of video data and the size of image data, perform standardization processing, extract multi-resolution image features, edge features, and texture features of video data, and obtain multi-resolution image features; Normalize the network status data, convert it into a unified scale, and extract time series features from the normalized network data to obtain time series network features; The hardware status data is normalized, and the status features at different time points are extracted through a sliding window to obtain the hardware status features of the time series.

8. An intelligent fault monitoring device for a security camera, characterized in that: The device is used to implement the intelligent fault monitoring method for a security camera according to any one of claims 1 to 7, and the device comprises: A multimodal data acquisition module, wherein the multimodal data acquisition module is used to acquire multimodal data from a security camera, wherein the multimodal data includes video data, network status data, and hardware status data; A feature extraction module, which is used to perform standardization processing and feature extraction on the multimodal data to generate multi-resolution image features, time series network features, and hardware status features; A pseudo data generation module, the pseudo data generation module is used to call a pre-trained generator to generate pseudo data simulating a normal state, the pre-trained generator is trained based on the multi-resolution image features, time series network features, and hardware state features, and is used to learn the distribution of multimodal data in a normal state; An abnormality determination module, the abnormality determination module is used to input the multimodal data collected in real time into the pre-trained discriminator for comparison with the pseudo data generated by the pre-trained generator, and output an abnormality determination result; A fault analysis module is used to perform fault analysis according to the abnormality determination result and generate a fault monitoring result.

9. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; The processor is used to implement the intelligent fault monitoring method for a security camera as described in any one of claims 1 to 7 when executing the executable instructions stored in the memory.

10. An intelligent fault monitoring system for a security camera, characterized in that: include: The electronic device as claimed in claim 9.

Images