Transformer substation intrusion target detection and classification method based on event data constraint

By combining event data constraints in substation intrusion target detection, the problem of insufficient dynamic target recognition capabilities in the prior art is solved by combining deep neural networks with event data constraints, and a higher detection accuracy and classification accuracy are achieved, especially Robustness in under-light and rapid changes.

CN119068173APending Publication Date: 2024-12-03STATE GRID ZHEJIANG ELECTRIC POWER CO LTD HANGZHOU POWER SUPPLY CO
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411094388.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing substation intrusion target detection algorithm has limited dynamic target recognition and judgment capabilities under the influence of light changes, weather changes, occlusion and complex backgrounds, especially when detecting high-speed mobile targets, the feature recognition rate is low.

Method used

Using a deep neural network based on event data constraints, combining visible light images and event imaging data, the target detection network is formed through the event feature extraction module, visible light feature extraction module, event constraint attention calculation module and target classification module to improve dynamic intrusion. The detection accuracy and classification accuracy of the target reduce the impact on lighting and scene factors.

Benefits of technology

It significantly improves the detection accuracy and classification accuracy of dynamic intrusion targets of substations, especially in night and rapid changes, with high accuracy and robustness, and can more accurately identify and classify intrusion targets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068173A_ABST
    Figure CN119068173A_ABST
Patent Text Reader

Abstract

The invention provides a transformer substation intrusion target detection and classification method based on event data constraint, and belongs to the technical field of computer vision and artificial intelligence. An event imaging data constraint is added on the basis of a deep neural network of a visible light image to form an event data constraint target detection network, and the target detection network can extract imaging data features of a visible light image mode and an event imaging image mode at the same time. According to the invention, the detection precision and classification precision of the dynamic intrusion target can be improved; meanwhile, under the condition of insufficient illumination, the method also has the advantages of accuracy and robustness, and the influence of scene and illumination factors on the transformer substation intrusion dynamic target detection and classification method can be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and artificial intelligence, and provides a method for detecting electric sparks of substation equipment based on event data constraints. Background Art

[0002] At present, before the remote monitoring system (telemetry, telesignaling, remote control, and remote adjustment) was added to the monitoring system of the substation, the traditional substation mainly relied on manual 24-hour patrols to ensure the safety of the operating environment. With the popularization of remote monitoring systems, the application of video monitoring technology in power systems has become more extensive. Cameras are installed in high-risk equipment areas and areas where intrusion may occur in my country's substations for monitoring, effectively eliminating safety hazards caused by intrusions such as people, birds, mice, cats, dogs, and other wild animals, and timely warnings when intrusions occur, thereby avoiding accidents. However, this method still cannot completely replace manual patrols. The on-duty personnel need to observe the computer screen for a long time with the naked eye, which is easy to cause visual fatigue and negligence.

[0003] Based on the existing substation monitoring system, the detection of intrusion targets is realized based on target detection technology. This method can help the system automatically identify and detect intrusion targets and reduce the manual burden. The intrusion target detection algorithm for substation scenes can be roughly divided into two algorithms: those based on background modeling and those based on foreground target detection. Among them, the algorithm principle based on background modeling is to make a difference between the current frame image and the background model, and then compare and analyze all pixels of the image with the set threshold to obtain the moving foreground target; then use the foreground moving target segmentation to obtain the intrusion target to be detected; finally, the target type is determined based on the image features of the target area. This method is simple to implement and has high computational efficiency, but it is easily affected by many factors such as lighting changes, weather changes, occlusion, shadows, and complex backgrounds. Although the intrusion target detection algorithm based on foreground target features overcomes the limitations of application scenarios and expands the scope of application of target detection algorithms, its detection and recognition of dynamic target image features is limited by the prior modeling accuracy of target features, and the feature recognition rate for high-speed moving targets is low.

[0004] With the development of machine vision and deep learning technology, a large number of algorithms that use deep neural networks to achieve target detection have emerged, such as SPP-NET, R-CNN, Fast R-CNN, Faster R-CNN, YOLO, etc. These algorithms use data-driven supervised training methods to automatically extract the features of the target to be detected from a large amount of sample data to improve the accuracy and robustness of target detection. For example, the YOLO algorithm simplifies the target detection process by treating the target detection problem as a regression problem, and has become one of the mainstream target detection networks and is widely used in fields such as autonomous driving, robot vision, and real-time video analysis. However, most of the above-mentioned deep learning target detection networks are designed to process visible light images, infrared images, or depth sensing images. Due to the insufficient perception of the changing characteristics of dynamic targets by the above-mentioned data sources, the recognition and judgment of dynamic targets by existing machine vision and deep learning target detection algorithms are still severely limited by factors such as scenes and lighting. Summary of the invention

[0005] In view of the limitations of existing technical means for detecting intrusion targets in substation scenes, the present invention proposes a method for detecting and classifying intrusion targets in substations based on event data constraints. By adding event imaging data constraints to the deep neural network of visible light images to form an event data constrained target detection network, the target detection network can simultaneously extract imaging data features of both visible light image modality and event imaging image modality. The present invention can not only improve the detection accuracy and classification accuracy of dynamic intrusion targets; at the same time, it also has advantages in accuracy and robustness in the case of insufficient light, and can reduce the influence of scene and light factors on the detection and classification methods of dynamic intrusion targets in substations.

[0006] The technical solution of the present invention is as follows:

[0007] A method for detecting and classifying intrusion targets in a substation based on event data constraints comprises the following steps:

[0008] Step 1: Data collection: Select a set of event cameras and visible light cameras with synchronous collection functions to collect visible image data of intrusion targets in the substation scene. The collected data must cover the sample space of different categories of intrusion objects;

[0009] Step 2, data annotation: annotate the location of the intrusion target in the image collected in step 1; annotate the location and category of the intrusion target in the visible light image image by image, and also annotate the location and category of the intrusion target on the corresponding event image; the location is recorded by the coordinates of the corner points of the minimum circumscribed rectangular frame of the image area where the intrusion target is located, and the classification value is the category number corresponding to the intrusion target; the annotated intrusion target event-visible light image pairs are divided into N categories, each category contains M groups of data, thereby obtaining the substation intrusion target detection data set;

[0010] Step 3, target detection network construction: On the basis of the deep neural network of visible light images, event imaging data constraints are added to form a target detection network, that is, the target detection network is a deep neural network based on event data constraints; the target detection network consists of an event feature extraction module, a visible light feature extraction module, an event-constrained attention calculation module and a target classification module; first, an independent branch structure network is used to build the event feature extraction module, and the event image sequence is input into the event feature extraction module. The event feature extraction module calculates the time feature map, and the time feature map is used as the constraint vector; secondly, another independent branch structure network is used to build the visible light feature extraction module, and the visible light image sequence is input into the visible light feature module, so that the visible light feature module differentiates the visible light image sequence, and the visible light feature extraction module calculates the visible light feature map; then, the spatial attention and channel attention calculation architectures are used to build the event-constrained attention module, and the event constraint vector and the visible light feature map are used as the input of the event-constrained attention module to obtain the feature map after attention enhancement; finally, a multi-layer perceptron architecture network is used to build the target classification module, and the feature map after attention enhancement is input, and the target classification module calculates the classification probability value of the target;

[0011] Step 4: supervised training of the target detection network: use the classification data collected in step 2 to perform deep learning model training on the target detection network built in step 3. For the paired event-visible light image sequence of the intrusion target, set the intrusion target classification loss as the constraint tuning condition, and perform parameter training and tuning of the target detection network in a supervised training manner, that is, optimize the value of the neuron parameter of the target detection network by the gradient descent method; calculate the loss function value of the target detection network for the target classification result and the data annotation result through training, and when the loss function value decreases, the training parameter value is an effective parameter optimization result; after multiple rounds of iterative training, until the loss function value obtained by the target detection network training decreases by less than a given threshold, it is determined that the network training has converged; the decrease is the difference between the previous training and the next training, or until the number of training iterations reaches the set upper limit, it is determined that the target detection network training has converged, and the target detection network parameters at this time are the final training tuning results; the trained target detection network can quickly and accurately detect the pixel coordinates of the intrusion target in the image;

[0012] Step 5: Detect and classify dynamic intrusion targets: Input the paired event-visible light images of the intrusion targets into the trained target detection network to obtain the pixel coordinates and classification results of the intrusion targets.

[0013] In step 3, building a target detection network includes the following specific steps:

[0014] Step 3.1: Build an event feature extraction module: The event feature extraction module consists of 1 to 5 residual blocks; a convolution layer (Convolution), a deconvolution layer (deconvolution) and a normalization layer (D-Conv+BN layer) are a group, and each residual block is composed of multiple groups of such structures in turn; the event imaging data is input into the event feature extraction module to extract the event features of the dynamic target, and the event features are within the resolution of the event camera and the imaging viewing angle; based on the characteristics of the event imaging data, the event feature extraction module only extracts the features of the area where the brightness signal changes in the scene within the single-frame imaging time range of the event camera, and the factors causing the brightness signal to change include the change in the position of the dynamic target, the change in the light source, the change in the reflection and the change in the self-luminescence of the target;

[0015] Step 3.2: Build a visible light feature extraction module: The visible light feature extraction module uses a parameter-independent learnable feature pyramid network (FPN). First, the visible light image sequence is differentiated, and the differential result is used as the "foreground-background" prior modeling data; the visible light image sequence and the "foreground-background" prior modeling data are simultaneously input into the FPN network by channel splicing. The FPN network uses a variable-scale feature pooling operation to encode the visible light target features in the input data and output a visible light feature map. The resolution of the output visible light feature map is the same as the resolution of the event feature map in step 3.1; the FPN network uses a U-net encoder and decoder network structure; the encoder uses 1-5 groups of encoding units, each encoding unit is composed of multiple groups of network layers, each group of network layers includes a convolution layer, a Relu activation function layer, a pooling layer, a deconvolution and a normalization layer; deconvolution operations are used between each decoding unit for upsampling; the last layer of the decoder uses a residual connection method to perform channel splicing with the input features as the output visible light feature map;

[0016] Step 3.3: Build an event-constrained attention module: The event-constrained attention module adopts the Transformer neural network structure and the Transformer neural network computing idea. The event-constrained attention module is built by the input layer, linear transformation layer, scaled dot product attention layer, residual connection and normalization layer, and output layer. The event-constrained attention module uses event imaging data as the detection constraint of dynamic targets, and uses spatial attention and channel attention mechanisms to calculate the correlation between event imaging features and visible light imaging features. Finally, the event feature map is used as the constraint vector, and the high-quality intrusion target latent space features fused with the constraint vector and the visible light feature map are calculated through the event-constrained attention module.

[0017] Step 3.4: Build a target classification module: The target classification module is composed of a feature decoder unit and a multi-layer perceptron (MLP). The feature decoder unit is composed of multiple groups of network layers, each of which is composed of a convolutional layer, a pooling layer and an activation function; the multi-layer perceptron is composed of a 1-10 layer fully connected structure. The multi-layer perceptron is used to calculate the category of the hidden features of the intrusion target after being encoded by the attention mechanism, so as to obtain the multi-classification probability calculation results of the intrusion target, where the classification probability value with the largest value is the classification result of the intrusion target.

[0018] In step 4, the tuning condition is to minimize the target classification error as the tuning constraint condition. Minimizing the classification error means calculating the difference between the classification result of the labeled data and the classification result calculated by the target detection network. A small difference indicates that the classification result calculated by the network is more accurate, otherwise the classification result calculated by the network is more inaccurate.

[0019] An electronic device comprises: one or more processors; and a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the intrusion target detection and classification method described in steps 1 to 5 above.

[0020] A computer-readable storage medium stores program code, which can be called by a processor to execute the intrusion target detection and classification method described in steps 1 to 5 above.

[0021] The beneficial effects of the present invention are:

[0022] 1. Compared with the existing methods, the present invention introduces event imaging data to form an event data constrained target detection network based on a deep neural network, detects and classifies dynamic intrusion targets of substations based on the target detection network, and improves the detection efficiency and classification accuracy of dynamic intrusion targets of substations;

[0023] 2. Due to the high sensitivity of event imaging to brightness changes, the detection accuracy of the present invention for fast-moving targets is significantly improved.

[0024] 3. The present invention can process event imaging and visible light imaging data simultaneously. Therefore, the intrusion target detection and classification method also has advantages in accuracy and robustness at night and when the target changes rapidly. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of detection and classification of intrusion targets in substations;

[0026] Figure 2It is a deep neural network architecture for substation intrusion target detection. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0028] This embodiment provides a substation intrusion target detection and classification method based on event data constraints. The method adopts deep learning technology to build an event data constrained target detection deep neural network, namely, a target detection network, which can simultaneously process event camera images and visible light camera images.

[0029] A substation intrusion target detection and classification method based on event data constraints, the process is as follows Figure 1 As shown in the figure, the specific implementation steps of the substation intrusion target detection and classification method are as follows:

[0030] Step 1, data collection: Use one event camera and one visible light camera controlled by a driver, and arrange them based on the principle of minimizing the distance between the optical centers of the two cameras. When the distance between the left side of one camera and the right side of the other camera is 0, the distance between the optical centers of the two cameras is the minimum; the two cameras have a synchronous collection function, which is used to collect visible image data of intrusion targets in substation scenes. In order to improve the accuracy of the neural network calculation results, the collected data needs to cover the sample space of different categories of intrusion objects. For the same type of substation intrusion targets, synchronously collect and accumulate M groups (400000≥M≥400) of paired visible light-event image data of the same type of targets.

[0031] Step 2, data annotation: annotate the location of the intrusion target in the image collected in step 1; annotate the location and category of the intrusion target in the visible light image image by image, and also annotate the location and category of the intrusion target on the corresponding event image. The location is recorded by the coordinates of the corner points of the minimum outer rectangular frame of the image area where the intrusion target is located, and the classification value is the category number corresponding to the intrusion target; the annotated substation intrusion target pair event-visible light image is divided into N categories (1000≧N≧2), each category contains M (400000≧M≧400) groups of data, thereby obtaining the substation intrusion target detection data set;

[0032] Step 3, build a target detection network: on the basis of the deep neural network of visible light images, event imaging data constraints are added to form a target detection network, that is, the target detection network is a deep neural network based on event data constraints; the target detection network consists of an event feature extraction module, a visible light feature extraction module, an event-constrained attention calculation module and a target classification module; first, an independent branch structure network is used to build the event feature extraction module, and the event image sequence is input into the event feature extraction module, and the time feature map is calculated by the event feature extraction module, and the time feature map is used as the constraint vector; another independent branch structure network is used to build the visible light feature extraction module, and the visible light image sequence is input into the visible light feature module, so that the visible light feature module performs a difference on the visible light image sequence, and the visible light feature map is calculated by the visible light feature extraction module; the spatial attention and channel attention calculation architecture is used to build the event-constrained attention module, and the event constraint vector and the visible light feature map are used as the input of the event-constrained attention module to obtain the feature map after attention enhancement; finally, a multi-layer perceptron architecture network is used to build the target classification module, and the feature map after attention enhancement is input, and the target classification module calculates the classification probability value of the target;

[0033] Step 3.1: Build an event feature extraction module: The event feature extraction module consists of 1 to 5 residual blocks; each coding unit consists of multiple groups of network layers, each of which includes a convolution layer, a Relu activation function layer, a pooling layer, a deconvolution and a normalization layer; the event imaging data is input into the event feature extraction module to extract the event features of the dynamic target, and the event features are within the resolution of the event camera and the imaging viewing angle; based on the characteristics of the event imaging data, the event feature extraction module only extracts the features of the area where the brightness signal changes in the scene within the single-frame imaging time range of the event camera, and the factors causing the brightness signal to change include the change in the position of the dynamic target, the change in the light source, the change in the reflection and the change in the self-luminescence of the target;

[0034] The network structure of the event feature extraction module is shown in Table 1. The input of the event feature extraction module is set to a 32*32 event image block, and the output is a 256*256 event feature map. The output shape of the event feature extraction module should be the same as the output shape of the visible light image feature extraction module.

[0035] Step 3.2: Build a visible light feature extraction module: The visible light feature extraction module uses a parameter-independent learnable feature pyramid network (FPN). First, the visible light image sequence is differentiated, and the differential result is used as the "foreground-background" prior modeling data; the visible light image sequence and the "foreground-background" prior modeling data are simultaneously input into the FPN network by channel splicing. The FPN network uses a variable-scale feature pooling operation to encode the visible light target features in the input data and output a visible light feature map. The resolution of the output visible light feature map is the same as the resolution of the event feature map in step 3.1; the FPN network uses a U-net encoder and decoder network structure; the encoder uses 1-5 groups of encoding units, each encoding unit is composed of multiple groups of network layers, each group of network layers includes a convolution layer, a Relu activation function layer, a pooling layer, a deconvolution and a normalization layer; and the deconvolution operation is used between each decoding unit for upsampling; the last layer of the decoder uses a residual connection method to perform channel splicing with the input features as the output visible light feature map;

[0036] The visible light feature extraction module has basic U-shaped network characteristics, as shown in Table 2. The input of the visible light feature extraction module is set to a 256*256 visible light image block, and the output is a 256*256 visible light feature map. The output shape of the visible light feature extraction module is the same as the output shape of the event feature extraction module.

[0037] Step 3.3: Build an event-constrained attention module: The event-constrained attention module adopts the Transformer neural network structure and the Transformer neural network calculation idea. The event-constrained attention module is built by the input layer, linear transformation layer, scaled dot product attention layer, residual connection and normalization layer, and output layer. The event-constrained attention module uses event imaging data as the detection constraint of dynamic targets, and uses spatial attention and channel attention mechanisms to calculate the correlation between event imaging features and visible light imaging features. Finally, the event feature map is used as the constraint vector, and the high-quality intrusion target latent space features fused with the constraint vector and the visible light feature map are calculated through the Transformer neural network.

[0038] Since the event feature map and the visible light feature map have the same shape, the event feature map and the visible light feature map are concatenated in the channel dimension, that is, the concatenation operation is performed to form a feature map of 256*256*T, where T is the number of channels after concatenation.

[0039] The event-constrained attention network structure is shown in Table 3. The input layer consists of a 3*3 convolution layer, which accepts the concatenated feature map; the linear transformation layer contains three linear transformation modules, which are attention generation query (Q), key (K) and value (V). Each linear transformation module is implemented by a fully connected layer, and the ReLU or GELU activation function is used for nonlinear transformation. The number of input channels is T. i , the number of output channels is T s The scaled dot product attention layer is used to perform Transformer feature attention calculation. The scaled dot product attention layer is an independent module that accepts query (Q), key (K) and value (V) as input and outputs the weighted feature map W. The output of the scaled dot product attention layer is residually connected with the original input, and the normalization layer is applied to normalize the output after the residual connection. After the normalization layer, the feature map F is obtained.

[0040] Step 3.4: Build a target classification module: The target classification module is composed of a feature decoder unit and a multi-layer perceptron (MLP). The feature decoding unit is composed of 1-5 groups of network layers, each of which is composed of convolution, pooling and activation functions. The multi-layer perceptron is composed of 1-10 layers of fully connected structures. The multi-layer perceptron is used to calculate the category of the hidden features of the intrusion target after being encoded by the attention mechanism, so as to obtain the multi-classification probability calculation results of the intrusion target, among which the classification probability value with the largest value is the classification result of the intrusion target.

[0041] The input of the target classification module is the feature map output by the attention module after event constraints. A multi-layer perceptron is used to implement the classification calculation of the target. This embodiment uses a 2-layer fully connected layer to finally calculate the target probability of N classifications. Set the classification probability threshold Threshold_P. The classification probability threshold Threshold_P is adjustable. The range of Threshold_P is [0.0, 1.0]. A relatively small Threshold_P will increase the sensitivity of target detection and classification, and a relatively large Threshold_P will increase the accuracy of target classification. The target classification result probability greater than Threshold_P is the detected intrusion target category.

[0042] Step 4, supervised training of the target detection network: for the paired event-visible light image sequence of the intrusion target, the intrusion target classification loss is set as the constrained tuning condition, and the target detection network parameter training and tuning are performed in a supervised training manner, that is, the neuron parameter values ​​of the target detection network are optimized through the gradient descent method; specifically, the classification data collected in step 2 is used to perform deep learning model training on the target detection network built in step 3, and each training calculates the network's loss function value for the target classification result and the data labeling result. When the loss function value decreases, the parameter value of this training is an effective parameter optimization result. After multiple rounds of iterative training, until the loss function value obtained from the target detection network training decreases by less than a given threshold, it is determined that the network training has converged; the decrease is the difference between the previous training and the next training, or until the number of iterations of the target detection network training reaches the set upper limit, it is determined that the network training has converged, and the target detection network parameters at this time are the final network training tuning results;

[0043] The tuning condition is to minimize the target classification error as the tuning constraint. Minimizing the classification error means calculating the difference between the classification result of the labeled data and the classification result of the network calculation. A smaller difference means that the network calculation classification result is more accurate, otherwise the network calculation classification result is more inaccurate.

[0044] The target detection network built in step 3 is supervised and trained using the data set annotated in step 2. The target detection network training process is to input a set of paired event-visible light image inputs, calculate the position Po and classification probability Pc of the intrusion target through the target detection network, compare Po and Pc with the position and classification true value annotated in the data set, and calculate the loss function Loss of the training process:

[0045]

[0046] Among them, λ is a factor for adjusting position loss and classification loss, and the value range of λ is [0.0-100.0]. In this embodiment, λ=13.5. Changing λ will change the accuracy priority of target detection position and target classification. A larger λ value focuses on the accurate classification of the target, while a smaller λ improves the accuracy of target position detection. Si is a sample in the given training sample set S, and CE(·) is the cross entropy loss of classification. After multiple rounds of iterative training, the training is stopped until the change in the loss value Loss calculated by the network during the training process is less than the given threshold ε=0.02 or the number of iterations reaches the set upper limit. Save a set of network parameters with the smallest Loss value as the final training result of the neural network.

[0047] Step 5: Detect and classify the intrusion target: Input the paired event-visible light image of the intrusion target into the trained target detection network to obtain the pixel coordinates and classification results of the intrusion target. The trained target detection network can quickly and accurately detect the pixel coordinates of the intrusion target in the image.

[0048] First, the target detection network parameters saved by training in step 4 are used to initialize the target detection network built in step 3; then, a set of paired event-visible light image data to be detected is input; the target detection network is used to calculate the image pixel position and classification probability of the intrusion target; the classification probability range is [0.6-1.0], and the predicted position coordinates Po, classification number Ni and classification probability Pc are output as the final network output results.

[0049] Table 1 Network structure for event feature extraction

[0050] Layer number Network Type Input size Convolution Kernel Step Length Output size 1 Convolution 32*32 1*1 - 32*32 2 D-Conv+BN 32*32 - - 32*32 3 Convolution 64*64 1*1 - 64*64 4 D-Conv+BN 64*64 - - 64*64 5 Convolution 128*128 1*1 - 128*128 6 D-Conv+BN 128*128 - - 128*128 7 Convolution 256*256 1*1 - 256*256 8 D-Conv+BN 256*256 - - 256*256

[0051] Table 2. Network structure of visible light feature extraction

[0052] Layer number Network Type Input size Convolution Kernel Step Length Output size 1 Convolution 256*256 7*7 1 256*256 2 ReLU 256*256 - - 256*256 3 Max-pooling 256*256 3*3 2 128*128 4 Convolution 128*128 5*5 1 128*128 5 ReLU 128*128 - - 128*128 6 Max-pooling 128*128 3*3 2 64*64 7 Convolution 64*64 3*3 1 64*64 8 ReLU 64*64 - - 64*64 9 D-Conv+BN 128*128 3*3 1 128*128 10 Convolution 128*128 5*5 1 128*128 11 D-Conv+BN 256*256 7*7 1 256*256 12 Convolution 128*128 7*7 1 256*256

[0053] Table 3 Event-constrained attention network structure

[0054] Layer number Network Type Input size Output size 1 D-Conv 256*256 512*512 2 D-Conv 512*512 1024*1024 3 D-Conv 1024*1024 2048*2048 4 FullyConnected 2048*2048 1*2048 5 FullyConnected 1*2048 1*256 6 FullyConnected 1*256 1*256 7 FullyConnected 1*256 1*N

Claims

1. A method for detecting and classifying intrusion targets in a substation based on event data constraints, characterized by the following steps: Step 1: Data collection: Select a set of event cameras and visible light cameras with synchronous collection functions to collect visible image data of intrusion targets in the substation scene. The collected data must cover the sample space of different categories of intrusion objects; Step 2, data annotation: annotate the location of the intrusion target in the image collected in step 1; annotate the location and category of the intrusion target in the visible light image image by image, and also annotate the location and category of the intrusion target on the corresponding event image; the location is recorded by the coordinates of the corner points of the minimum circumscribed rectangular frame of the image area where the intrusion target is located, and the classification value is the category number corresponding to the intrusion target; The annotated intrusion target paired event-visible light image is divided into N categories, each category contains M groups of data, thus obtaining the substation intrusion target detection dataset; Step 3: Build a target detection network: Based on the deep neural network of visible light images, add event imaging data constraints to form a target detection network, that is, the target detection network is a deep neural network based on event data constraints; The target detection network consists of an event feature extraction module, a visible light feature extraction module, an event-constrained attention calculation module and a target classification module. First, an independent branch structure network is used to build the event feature extraction module, and the event image sequence is input into the event feature extraction module. The event feature extraction module calculates the time feature map, and the time feature map is used as the constraint vector. Secondly, another independent branch structure network is used to build the visible light feature extraction module, and the visible light image sequence is input into the visible light feature module, so that the visible light feature module differentiates the visible light image sequence, and the visible light feature map is calculated by the visible light feature extraction module. Then, the spatial attention and channel attention calculation architectures are used to build the event-constrained attention module, and the event constraint vector and the visible light feature map are used as the input of the event-constrained attention module to obtain the feature map after attention enhancement. Finally, a multi-layer perceptron architecture network is used to build the target classification module, and the feature map after attention enhancement is input, and the target classification module calculates the classification probability value of the target. Step 4: supervised training of the target detection network: use the classification data collected in step 2 to perform deep learning model training on the target detection network built in step 3. For the paired event-visible light image sequence of the intrusion target, set the intrusion target classification loss as the constraint tuning condition, and perform parameter training and tuning of the target detection network in a supervised training manner, that is, optimize the value of the neuron parameter of the target detection network by the gradient descent method; calculate the loss function value of the target detection network for the target classification result and the data labeling result through training, and when the loss function value decreases, the training parameter value is an effective parameter optimization result; after multiple rounds of iterative training, until the loss function value obtained by the target detection network training decreases by less than a given threshold, it is determined that the network training has converged; the decrease is the difference between the previous training and the next training, or until the number of training iterations reaches the set upper limit, it is determined that the target detection network training has converged, and the target detection network parameters at this time are the final training tuning results; The trained target detection network can quickly and accurately detect the pixel coordinates of the intruding target in the image; Step 5: Detect and classify dynamic intrusion targets: Input the paired event-visible light images of the intrusion targets into the trained target detection network to obtain the pixel coordinates and classification results of the intrusion targets.

2. According to claim 1, a method for detecting and classifying substation intrusion targets based on event data constraints is characterized by: In step 3, the specific steps of building the target detection network include the following: Step 3.1: Build an event feature extraction module: The event feature extraction module consists of 1 to 5 residual blocks; a convolution layer (Convolution), a deconvolution layer (deconvolution) and a normalization layer (D-Conv+BN layer) are a group, and each residual block is composed of multiple groups of such structures in turn; the event imaging data is input into the event feature extraction module to extract the event features of the dynamic target, and the event features are within the resolution of the event camera and the imaging viewing angle; based on the characteristics of the event imaging data, the event feature extraction module only extracts the features of the area where the brightness signal changes in the scene within the single-frame imaging time range of the event camera, and the factors causing the brightness signal to change include the change in the position of the dynamic target, the change in the light source, the change in the reflection and the change in the self-luminescence of the target; Step 3.2: Build a visible light feature extraction module: The visible light feature extraction module uses a parameter-independent learnable feature pyramid network (FPN). First, the visible light image sequence is differentiated, and the differential result is used as the "foreground-background" prior modeling data; the visible light image sequence and the "foreground-background" prior modeling data are simultaneously input into the FPN network by channel splicing. The FPN network uses a variable-scale feature pooling operation to encode the visible light target features in the input data and output a visible light feature map. The resolution of the output visible light feature map is the same as the resolution of the event feature map in step 3.1; the FPN network uses a U-net encoder and decoder network structure; the encoder uses 1-5 groups of encoding units, each encoding unit is composed of multiple groups of network layers, each group of network layers includes a convolution layer, a Relu activation function layer, a pooling layer, a deconvolution and a normalization layer; deconvolution operations are used between each decoding unit for upsampling; the last layer of the decoder uses a residual connection method to perform channel splicing with the input features as the output visible light feature map; Step 3.3: Build an event-constrained attention module: The event-constrained attention module adopts the Transformer neural network structure and the Transformer neural network computing idea. The event-constrained attention module is built by the input layer, linear transformation layer, scaled dot product attention layer, residual connection and normalization layer, and output layer. The event-constrained attention module uses event imaging data as the detection constraint of dynamic targets, and uses spatial attention and channel attention mechanisms to calculate the correlation between event imaging features and visible light imaging features. Finally, the event feature map is used as the constraint vector, and the high-quality intrusion target latent space features fused with the constraint vector and the visible light feature map are calculated through the event-constrained attention module. Step 3.4: Build a target classification module: The target classification module is composed of a feature decoder unit and a multi-layer perceptron (MLP). The feature decoder unit is composed of multiple groups of network layers, each of which is composed of a convolutional layer, a pooling layer and an activation function; the multi-layer perceptron is composed of a 1-10 layer fully connected structure. The multi-layer perceptron is used to calculate the category of the hidden features of the intrusion target after being encoded by the attention mechanism, so as to obtain the multi-classification probability calculation results of the intrusion target, where the classification probability value with the largest value is the classification result of the intrusion target.

3. According to claim 2, a method for detecting and classifying intrusion targets in a substation based on event data constraints is characterized by: In the step 3.2, the FPN network adopts a U-net encoder and decoder network structure; the encoder adopts 1-5 groups of encoding units, each encoding unit is composed of multiple groups of network layers, each group of network layers includes a convolution layer, a Relu activation function layer, a pooling layer, a deconvolution and a normalization layer; the deconvolution operation is used between the decoding units for upsampling; The last layer of the decoder uses a residual connection method to perform channel splicing with the input features as the output visible light feature map.

4. According to claim 1, a method for detecting and classifying substation intrusion targets based on event data constraints is characterized by: In step 4, the tuning condition is to minimize the target classification error as the tuning constraint condition. Minimizing the classification error means calculating the difference between the classification result of the labeled data and the classification result calculated by the target detection network. A small difference indicates that the classification result calculated by the network is more accurate, otherwise the classification result calculated by the network is more inaccurate.

5. An electronic device comprising one or more processors; and a memory; and one or more programs, characterized in that: The one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 4.

Citation Information

Cited By

  • Dual-mode target tracking method, system and equipment based on optical camera and event camera and medium

    CN119888269A