A method for identifying illegal video information based on a convolutional neural network model

Abstract

The application relates to the technical field of violation identification, in particular to a violation video information identification method based on a convolutional neural network model, which obtains a pretreated monitoring video of a power operation site; the pretreated video is divided into M segments, representative frames are extracted from the segments, and a representative frame sequence is obtained; the representative frame sequence is input into a residual convolution network, after the first convolution layer of each residual block, a channel attention module is used to calculate a channel weight vector, and then selective time shift is performed by taking the channel weight vector as a guide, and a frame feature vector is obtained after complete forward propagation; all the frame feature vectors are subjected to time sequence aggregation and multi-classification fusion, and a violation behavior identification result is output. The application fuses spatial features and time sequence information, realizes end-to-end automatic identification of power operation violation behaviors, and improves the timeliness and accuracy of violation detection.

Description

Technical Field

[0001] This application relates to the field of traffic violation recognition technology, and in particular to a method for recognizing traffic violation video information based on a convolutional neural network model. Background Technology

[0002] The scale of power industry engineering construction continues to expand, and production and maintenance tasks are becoming increasingly heavy. Power operation sites encompass various high-risk work scenarios, including working at heights and live-line operations, and involve densely populated personnel and complex operating procedures. Comprehensive and real-time monitoring of violations at work sites is essential to ensuring the personal safety of workers.

[0003] Currently, the power industry primarily relies on single-frame image detection methods for video recognition of violations at work sites. These methods use keyframe images extracted from the work video as input, employ convolutional neural networks to extract intra-frame spatial features, and output recognition results through classification or object detection. In engineering practice, monitoring videos are typically processed by extracting frames at a fixed frame rate, with each frame independently fed into the network for inference. The results from multiple frames are then subjected to voting or averaging post-processing to achieve the recognition of violations.

[0004] However, violations at power work sites are essentially dynamic processes, and their identification often relies on multiple consecutive frames rather than a single frame. Detection methods based on single-frame images can only extract static spatial features within a frame and cannot model the temporal information of personnel posture and limb movement trajectories between adjacent frames. This results in low accuracy in identifying violations that depend on the continuity of actions, such as crossing warning lines, the dynamic fastening of safety belts, and personnel falling to the ground. Consequently, these methods fail to meet the real-time monitoring requirements for high accuracy and low false negative rates at power work sites. Summary of the Invention

[0005] To address the technical problems of convolutional neural network detection methods based on single-frame images failing to model the temporal evolution of video actions and having low accuracy in recognizing dynamic violations, this application provides a violation video information recognition method based on a convolutional neural network model, which can improve the accuracy of violation detection.

[0006] This application provides a method for identifying illegal video information based on a convolutional neural network model. The method includes: preprocessing the monitoring video of a power operation site to obtain a preprocessed video; dividing the preprocessed video into M segments, extracting video frames from each segment to obtain a representative frame sequence; inputting the representative frame sequence into a residual convolutional network, calculating channel weight vectors on the current frame feature map through a channel attention module after the first convolutional layer of each residual block of the residual convolutional network, and performing selective temporal shift guided by the channel weight vectors, obtaining frame feature vectors after complete forward propagation through the residual convolutional network; performing temporal aggregation and fusion classification on the frame feature vectors of all representative frames, and outputting the illegal behavior identification result.

[0007] By first calculating the channel weight vector to filter discriminative channels after the first convolutional layer of each residual block in the residual convolutional network, and then performing selective temporal shift guided by the channel weight vector, spatial feature extraction and temporal information fusion are placed in the per-residual block processing flow of the same network. This allows the residual convolutional network to simultaneously capture intra-frame spatial features and cross-frame action evolution, thereby improving the accuracy of identifying dynamic violations.

[0008] Preferably, the preprocessing includes cropping and grayscale processing of the surveillance video to make the surveillance video meet the input size of the residual convolutional network.

[0009] Preferably, during the training phase of the residual convolutional network, after extracting video frames from each segment, the recognition method further includes performing at least one operation on the video frames: geometric transformation and pixel adjustment; the geometric transformation includes at least one of flipping, rotating, and scaling; the pixel adjustment includes at least one of brightness adjustment, contrast adjustment, and hue adjustment.

[0010] Preferably, the channel attention module sequentially includes a global average pooling layer, a one-dimensional convolutional layer, and a sigmoid activation function; the global average pooling layer is used to globally compress the feature map of the current frame to obtain a channel description vector; the one-dimensional convolutional layer is used to perform convolution operations on the channel description vector to obtain the original weights of each channel; the original weights are normalized by the sigmoid activation function to obtain a channel weight vector.

[0011] Preferably, after obtaining the channel weight vector of the kth residual block, the identification method further includes: passing the channel weight vector as historical weights to the channel attention module of the (k+1)th residual block; the channel weight vector of the (k+1)th residual block is obtained by weighted merging of two parts: one is the current weight calculated by the channel attention module in the (k+1)th residual block, and the other is the historical weight inherited from the kth residual block.

[0012] By passing the channel weight vector of the k-th residual block as historical weight to the (k+1)-th residual block, and merging it with the current weight calculated by the (k+1)-th residual block itself to obtain the final channel weight vector, the channel importance evaluation of deep residual blocks continues to inherit the discriminative prior established in the shallow layer. This suppresses the forgetting of attention in the network depth direction, ensures attention to low-level discriminative features such as the helmet outline, and improves the stability of deep network feature representation.

[0013] Preferably, the shift direction of each channel in the selective temporal shift is adaptively determined by the attention temporal gradient; for each residual block, the channel weight vectors calculated by the channel attention module for the current frame and the adjacent previous frame are subtracted to obtain the attention temporal gradient vector; for channels with positive attention temporal gradients, forward shift is performed; for channels with negative attention temporal gradients, backward shift is performed; for channels with absolute attention temporal gradient values ​​lower than a preset gradient threshold, they do not participate in the temporal shift.

[0014] The attention temporal gradient vector is obtained by calculating the difference between the channel weight vector of the current frame and the adjacent previous frame. The temporal shift direction of each channel is adaptively determined according to the positive or negative gradient. Channels with an absolute gradient value lower than a preset gradient threshold are excluded from shifting. This allows channels whose importance is increasing to learn from subsequent frame information and channels whose importance is decreasing to learn from historical frame information. At the same time, it avoids the introduction of invalid temporal noise by static channels, and achieves adaptive matching between the shift direction and the dynamic change law of the channels.

[0015] Preferably, the shift intensity of each channel is determined by the corresponding weight value in the channel weight vector; channels with weights lower than a preset weight threshold are judged as low-discriminative channels and do not participate in the timing shift; channels with weights not lower than the preset weight threshold have their shift intensity determined according to their respective weight ratios; the actual timing shift of each channel is jointly determined by the shift direction and the shift intensity.

[0016] Preferably, the shift intensity of each channel is scaled as a whole using the inter-frame difference degree; before performing the temporal shift, the distance between the flattened vectors of the feature maps of adjacent frames in the current video segment is calculated to obtain the inter-frame difference degree, and the shift intensity of each channel is scaled as a whole using the inter-frame difference degree as a scaling factor.

[0017] Before performing temporal shift, the distance between the flattened vectors of the feature maps of adjacent frames is calculated as the inter-frame difference. The shift intensity of all active channels is scaled as a whole using the inter-frame difference. When the video content is static, the shift intensity approaches zero to avoid introducing invalid disturbances. When the operator performs large-scale illegal actions, the shift intensity remains at a high level to fully explore the action evolution information of adjacent frames. This makes the overall amplitude of the temporal shift highly adaptive and matched with the dynamic degree of the video content, thereby reducing redundant calculations in static frames and enhancing the temporal feature extraction capability of dynamic key segments.

[0018] Preferably, the channel attention module and the selective temporal shift module are jointly embedded in each residual block of the residual convolutional network, located after the first convolutional layer and before the second convolutional layer.

[0019] Preferably, the fusion classification adopts a multi-classification approach. After performing temporal aggregation on the frame feature vectors of all video frames, the probability distribution of each violation behavior category is output through the classification layer, and the category with the highest probability is taken as the violation behavior recognition result.

[0020] Compared with the prior art, the technical solution of this application has the following beneficial technical effects: The preprocessed monitoring video is segmented and then fed into a residual convolutional network frame by frame. A channel attention module and a selective temporal shift module are introduced into each residual block, so that the network can extract the spatial features of each frame while fusing the temporal action information of adjacent frames. Finally, the violation behavior recognition results are output through temporal aggregation and multi-classification, realizing end-to-end automatic recognition of various violations at the power operation site, which significantly improves the timeliness and accuracy of violation detection. Attached Figure Description

[0021] Figure 1 This is a flowchart of a method for identifying traffic violation video information based on a convolutional neural network model, according to an embodiment of this application. Detailed Implementation

[0022] Before providing a detailed description of the embodiments of this application, the application scenarios of these embodiments will first be illustrated by way of example. The method provided in these embodiments can be applied to power construction work sites to identify violations at power construction work sites. Figure 1 This is a flowchart illustrating a method for identifying traffic violation video information based on a convolutional neural network model, according to an embodiment of this application. Figure 1 As shown, the method for identifying traffic violation video information based on a convolutional neural network model includes steps S101 to S104, which are described in detail below.

[0023] S101, acquire the monitoring video of the power operation site, preprocess the monitoring video to obtain the preprocessed video.

[0024] In one embodiment, the power work site encompasses various high-risk work scenarios, including working at heights and live-line operations. The site is characterized by a high density of workers and complex operational procedures, posing numerous risks such as not wearing safety helmets, not wearing safety belts while working at heights, and unauthorized crossing of warning lines. Image acquisition devices are deployed at fixed locations at the work site entrances and key work areas to continuously record the entire operation, acquiring surveillance video of the power work site.

[0025] To ensure that the surveillance video meets the input specifications of the subsequent residual convolutional network, it needs to undergo standardized preprocessing. This preprocessing includes cropping and grayscale conversion of the surveillance video to meet the input size and format requirements of the residual convolutional network.

[0026] The surveillance videos acquired by on-site image acquisition equipment typically have a resolution as high as 1920×1080 pixels, far exceeding the standard input size of residual convolutional networks. If input directly without cropping, it not only consumes a large amount of computational resources but also introduces a significant amount of edge background information irrelevant to violation identification. The cropping operation trims each frame of the surveillance video to the standard input size required by the residual convolutional network; for example, a center crop is performed with a target size of 224×224 pixels. After cropping and grayscale processing, each frame of the surveillance video is converted into a preprocessed image conforming to the input specifications of the residual convolutional network. All preprocessed images are arranged in their original chronological order to constitute the preprocessed video.

[0027] S102, the preprocessed video is divided into M segments, and video frames are extracted from each segment to obtain a representative frame sequence.

[0028] In one embodiment, violations at power operation sites are random. If only a fixed number of frames are extracted from the video as input, critical violations are easily missed due to sampling position deviations. To ensure uniform coverage of the entire video's temporal content under limited sample conditions, the preprocessed video is uniformly divided into M segments along the time axis. The value of M matches the average duration of the power operation scenario. For example, M can be 10, meaning a complete operation monitoring video is uniformly divided into 10 consecutive time periods. One frame is randomly sampled from each segment as the representative frame for that segment, resulting in the extraction of a total of M video frames. This ensures that representative frames are present in all time periods for subsequent processing, forming a representative frame sequence.

[0029] During the training phase of the residual convolutional network, in order to improve the generalization ability of the residual convolutional network under complex field conditions and avoid overfitting due to the single distribution of training data, data augmentation processing is required for the video frames. The data augmentation processing includes performing at least one operation of geometric transformation and pixel adjustment on the video frames; the geometric transformation includes at least one of flipping, rotating, and scaling; the pixel adjustment includes at least one of brightness adjustment, contrast adjustment, and hue adjustment.

[0030] Understandably, the installation angles of image acquisition equipment at power operation sites vary, and the lighting conditions at different construction sites differ significantly. Geometric transformations are used to simulate random fluctuations in the viewing angle of the image acquisition equipment. For example, video frames are horizontally flipped with a 50% probability, randomly rotated within a range of ±15 degrees, and randomly scaled within a ratio range of 0.8 to 1.2. Pixel adjustments are used to simulate lighting fluctuations such as direct sunlight and shadow occlusion at the site. For example, brightness is randomly adjusted within a range of ±30%, contrast is randomly adjusted within a range of 0.8 to 1.2 times, and hue is randomly adjusted within a range of ±10%. The above enhancement operations are applied independently and randomly to each video frame during the training phase, but not during the inference phase, to ensure the consistency of input during the testing phase. After data augmentation processing, M video frames correspond to M augmented frames. All augmented frames are arranged in their original chronological order to form an augmented frame sequence, which is used to train the residual convolutional network.

[0031] In this way, by uniformly segmenting the preprocessed video and sampling representative frames segment by segment, complete coverage of the temporal distribution of the entire video is ensured. At the same time, during the training phase, the diversity of training samples is effectively expanded through joint data augmentation of geometric transformation and pixel adjustment, so that the residual convolutional network can still maintain a stable violation recognition capability when facing complex perspective changes and lighting interference at the power operation site.

[0032] S103, Obtain the frame feature vector of each representative frame in the representative frame sequence: Input the representative frame sequence into the residual convolutional network. After the first convolutional layer of each residual block of the residual convolutional network, calculate the channel weight vector of the current frame feature map through the channel attention module. Then, perform selective temporal shift guided by the channel weight vector, so that the channel with the larger channel weight participates in the temporal shift to a greater extent. After the residual convolutional network completes the forward propagation, the frame feature vector is obtained.

[0033] In one embodiment, taking a high-altitude work scenario as an example, determining whether a worker is correctly fastening their safety belt requires not only observing the spatial morphological features of the safety belt in a single frame image, but also considering the continuous changes in the worker's posture in adjacent frames. Each frame in the representative frame sequence is input frame-by-frame into a residual convolutional network, which performs spatial feature extraction and cross-frame temporal information fusion, ultimately outputting a frame feature vector corresponding to each representative frame.

[0034] In this embodiment, the residual convolutional network is a ResNet-101 network; the channel attention module and the selective temporal shift module are jointly embedded in each residual block of the residual convolutional network, located after the first convolutional layer and before the second convolutional layer, so that each residual block extracts spatial features while fusing temporal features of adjacent frames guided by the discriminative channel.

[0035] It should be noted that the ResNet-101 network consists of one initial convolutional layer, one max-pooling layer, and four residual layers. These four residual layers contain 3, 4, 23, and 3 residual blocks respectively, for a total of 33 residual blocks. Each residual block consists of two convolutional layers and a cross-layer identity connection. This cross-layer identity connection ensures stable gradient propagation in deep networks, preventing gradient vanishing during training. The channel attention module and the selective temporal shift module are embedded after the first convolutional layer and before the second convolutional layer in each residual block. This allows each residual block to immediately perform channel importance evaluation and temporal information fusion on the extracted feature map after completing the first layer of spatial feature extraction, and then feed the fused feature map into the second convolutional layer, achieving deep coupling between spatial feature extraction and temporal modeling.

[0036] To select the channels with the highest discriminative value for violation identification from the current frame feature map, a channel attention module is introduced after the first convolutional layer of each residual block. The channel attention module sequentially includes a global average pooling layer, a one-dimensional convolutional layer, and a sigmoid activation function. The global average pooling layer is used to globally compress the current frame feature map to obtain channel description vectors. The one-dimensional convolutional layer is used to perform convolution operations on the channel description vectors to obtain the original weights of each channel. The original weights are then normalized by the sigmoid activation function to obtain the channel weight vector.

[0037] It should be noted that, assuming the spatial size of the current frame feature map is H×W and the number of channels is C, the global average pooling layer calculates the mean of the current frame feature map in the spatial dimension H×W, compressing each of the C channels into a scalar and outputting a channel description vector of length C. This completes the dimensionality reduction from two-dimensional spatial features to one-dimensional channel statistics, preserving the intensity information of the overall response of each channel. The one-dimensional convolutional layer performs convolution operations on the channel description vectors, capturing the local dependencies between adjacent channels and outputting the original weights of each of the C channels. The original weights of each channel are compressed to the interval between 0 and 1 by the sigmoid activation function to obtain the channel weight vector. Each element in the vector corresponds to the importance score of a channel; the closer the value is to 1, the more important the channel is for the violation detection of the current frame.

[0038] After calculating the channel weight vector of the current residual block, the weight vector is passed to the next residual block to achieve cross-block inheritance. After obtaining the channel weight vector of the k-th residual block, the identification method further includes: passing the channel weight vector as historical weights to the channel attention module of the (k+1)-th residual block; the channel weight vector of the (k+1)-th residual block is obtained by weighted merging of two parts: one is the current weight calculated by the channel attention module in the (k+1)-th residual block, and the other is the historical weight inherited from the k-th residual block, and the merging coefficient of the two parts is a preset hyperparameter; through the cross-block inheritance mechanism, the deep residual blocks integrate the discriminative priors established in the shallow layers when evaluating channel importance, thereby achieving attention consistency constraints in the network depth direction.

[0039] Specifically, let the channel weight vector finally output by the channel attention module of the k-th residual block be... The current weights calculated independently by the channel attention module of the (k+1)th residual block are: The merging coefficient is a preset hyperparameter. Then the final channel weight vector of the (k+1)th residual block Satisfying the relation: In the formula, The current weights are calculated independently by the channel attention module of the (k+1)th residual block. The historical weights inherited from the k-th residual block, The merging coefficient, ranging from 0 to 1, is determined through a grid search on the training set with a search step size of 0.1. The value corresponding to the highest violation recognition accuracy on the final validation set is used. For example, A value of 0.7 can be taken, meaning that the adaptive judgment of the current residual block accounts for 70% of the weight, and the prior judgment from the shallow layer accounts for 30% of the weight; for the first residual block, there is no previous historical weight, and the current weight calculated by itself is directly used as the final channel weight vector.

[0040] Understandably, shallow residual blocks are responsible for capturing low-level local features such as the helmet outline and seatbelt buckle texture, while deep residual blocks are responsible for integrating these features to form high-level semantic judgments. If the channel attention of each residual block is calculated independently and not passed on to each other, after the features are highly abstracted, the attention of deep blocks may shift to channels unrelated to violation judgment, weakening the continued attention to the key judgment clues already focused on in the shallow layer. By passing the channel importance judgments already established in the shallow layer to the deep layer in the form of historical weights, the deep residual blocks always maintain consistency with the judgment priors of the shallow layer when calculating channel attention, suppressing the forgetting of attention in the deep network and improving the stability of the frame feature vector in expressing the semantics of violation behavior.

[0041] After the channel attention module calculation is completed, selective temporal shifting is performed guided by the channel weight vector. First, the shifting direction of each channel is determined. The shifting direction of each channel in the selective temporal shifting is adaptively determined by the attention temporal gradient. For each residual block, the channel weight vectors calculated by the channel attention module for the current frame and the adjacent previous frame are subtracted to obtain the attention temporal gradient vector. For channels with positive attention temporal gradients, forward shifting is performed; for channels with negative attention temporal gradients, backward shifting is performed; for channels with absolute attention temporal gradient values ​​lower than a preset gradient threshold, they do not participate in temporal shifting.

[0042] Specifically, let the channel weight vector calculated by the channel attention module in the k-th residual block of the t-th frame in the representative frame sequence be... The channel weight vector corresponding to the (t-1)th frame of the adjacent previous frame is Then the attention time gradient vector Satisfying the relation: In the formula, Let be the channel weight vector output by the channel attention module in the k-th residual block of the t-th frame. Let be the channel weight vector corresponding to frame t-1. The c-th component This indicates the changing trend of the importance of the c-th channel between adjacent frames; a preset gradient threshold is used. The attention temporal gradient absolute value distribution of all residual blocks and all frame pairs on the validation set is statistically analyzed, and its 20th percentile is used to determine the distribution. For example, A value of 0.05 is acceptable. For The channel that is more discriminative in the current frame than in the previous frame is shifted forward, incorporating contextual information from subsequent frames; for For channels whose importance is decreasing compared to the previous frame, shift them backward and incorporate the historical context of the preceding frame; for The importance of the channel changes very little between adjacent frames, it does not participate in timing shift, and avoids introducing invalid timing noise.

[0043] Understandably, taking the violation of crossing the warning line as an example, the channel responsible for sensing leg movement features suddenly increases in importance in the key frame of the person crossing, and the attention temporal gradient is positive. This channel performs a forward shift, so that the current frame can focus on the position information of the person when landing from subsequent frames, which helps to fully capture the trajectory of the crossing action. On the other hand, the channel responsible for sensing static background texture changes smoothly in importance throughout the crossing process, and the absolute value of the gradient is lower than the preset gradient threshold. It does not participate in the temporal shift, thus avoiding the introduction of redundant background information unrelated to violation recognition into the frame feature vector.

[0044] Based on the determined shift direction, the shift intensity of each channel is further determined by the corresponding weight value in the channel weight vector; channels with weights lower than a preset weight threshold are judged as low-discriminative channels and do not participate in the timing shift; channels with weights not lower than the preset weight threshold have their shift intensity determined according to their respective weight ratios, with larger channel weights resulting in larger shift intensity; the actual timing shift of each channel is jointly determined by the shift direction and the shift intensity.

[0045] Specifically, preset weight threshold Take the average of all channel weight values ​​in the channel weight vector, for example, A value of 0.5 can be used. The final channel weight vector of the k-th residual block... Medium weight value The channel is classified as a low-discriminative channel and will not participate in timing shift; the weight value will be... The channels constitute the active channel set, denoted as The shift intensity of the c-th channel in the active channel set. Satisfying the relation: In the formula, Let be the weight value of the c-th channel in the k-th residual block. The maximum weight value in the active channel set. It is the basic shift ratio, which is the proportion of the feature dimension that the channel with the largest weight is allowed to participate in the temporal shift to the total feature dimension of that channel. The impact of different values ​​on recognition accuracy is determined, for example, by validating the effect on recognition accuracy on the training set. A value of 0.25 can be used, meaning the channel with the highest weight will participate in the temporal shift with a maximum of 25% of its feature dimensions. For each channel c in the active channel set, the top 25% of its feature dimensions will be used in the temporal shift. Each dimension is shifted by one time step along the aforementioned determined shift direction, where Let c be the number of feature dimensions. This indicates rounding down; the feature dimensions that were not shifted remain unchanged and are concatenated with the shifted dimensions to form the output features of the time-enhanced channel.

[0046] Understandably, in the helmet recognition scenario, the channel corresponding to the head area usually has a high channel weight and a large shift intensity, which can more fully exchange temporal information with adjacent frames and help distinguish the shape changes of the helmet under different perspectives when the worker briefly turns their head. On the other hand, the channel corresponding to the ground, sky and other background areas has a weight close to zero and is judged as a low discriminative channel. It is excluded from the temporal shift to avoid introducing static background changes that are not related to violation recognition into the frame feature vector. This reduces redundant calculations and avoids the feature vector being contaminated by noise.

[0047] In this embodiment, the shift intensity of each channel is scaled overall using the inter-frame difference. Before performing the temporal shift, the distance between the flattened vectors of the feature maps of adjacent frames within the current video segment is calculated to obtain the inter-frame difference. This inter-frame difference is then used as a scaling factor to scale the shift intensity of each channel overall. A larger inter-frame difference results in a larger overall shift intensity, and a smaller inter-frame difference results in a smaller overall shift intensity. The inter-frame difference can be calculated using L2 distance or cosine distance.

[0048] The inter-frame difference of all adjacent frame pairs within the current video segment is min-max normalized to obtain the normalized inter-frame difference. The shift intensity of all channels in the active channel set is scaled using the normalized inter-frame difference as a scaling factor, resulting in the final shift intensity. Satisfying the relation: In the formula, The final shift intensity of the c-th channel in the k-th residual block of the t-th frame. To normalize the inter-frame difference, The base shift strength is determined by the channel weights; for each channel c in the active channel set, with Alternative Perform a temporal shift operation on the feature dimension.

[0049] Understandably, when workers are stationary and waiting, the feature changes between adjacent frames are minimal, the normalized inter-frame difference is close to 0, and the final shift intensity of all channels is compressed to near zero, avoiding the introduction of invalid noise by performing large-scale temporal shifts when the video content is almost unchanged. When workers are performing actions with large amplitude, such as crossing fences or swinging their arms at heights, the feature differences between adjacent frames are significant, the normalized inter-frame difference is close to 1, the overall shift intensity remains at a high level, and rich motion evolution information between adjacent frames is fully obtained.

[0050] The channel attention module and the selective temporal shift module are executed sequentially in each residual block of ResNet-101, representing that each frame in the frame sequence goes through a complete forward propagation path. Finally, after being compressed by the global average pooling layer at the end of ResNet-101, the corresponding frame feature vector is output.

[0051] Thus, by introducing a channel attention module into each residual block to filter discriminative channels, and ensuring the consistency of attention between deep and shallow layers through a cross-block inheritance mechanism, the shift direction is adaptively determined by the attention temporal gradient, the shift intensity is determined proportionally by the channel weights, and the overall shift amplitude is dynamically scaled by the inter-frame difference. This allows the residual convolutional network to accurately integrate the temporal information of adjacent frames while extracting spatial features frame by frame, thereby improving the accuracy of identifying violations that require dynamic judgment, such as not wearing a safety belt while working at height and crossing a warning line.

[0052] S104 performs temporal aggregation and fusion classification on the frame feature vectors of all representative frames, and outputs the violation behavior recognition results.

[0053] In one embodiment, the M representative frames in the frame sequence correspond to M frame feature vectors, each frame feature vector independently describing the spatial features and temporal fusion information of its corresponding representative frame. To synthesize the information from the M frames and make an overall violation judgment on the entire surveillance video, temporal aggregation is first performed on the frame feature vectors of all video frames. The temporal aggregation adopts a dimension-wise mean aggregation method, taking the arithmetic mean of the M frame feature vectors dimension by dimension to obtain a video-level feature vector with the same dimension as the frame feature vectors.

[0054] The fusion classification adopts a multi-classification approach. After temporally aggregating the frame feature vectors of all video frames, the probability distribution of each violation category is output through the classification layer, and the category with the highest probability is taken as the violation identification result. The violation identification result includes at least one of the following categories: not wearing a safety helmet, not using a safety rope, crossing a hard fence, crossing or crawling over a warning line, smoking, placing oxygen cylinders or acetylene cylinders upside down, not wearing insulated gloves when testing electricity, and people falling to the ground.

[0055] The classification layer consists of a fully connected layer that receives video-level feature vectors as input and outputs a dimension equal to the number of violation categories. For example, when the identified categories cover the above 8 types of violations, the classification layer outputs an 8-dimensional original score vector. After the 8-dimensional original score vector is normalized by the Softmax function, the probability distribution of each violation category is obtained. Each dimension corresponds to the predicted probability of a type of violation. The sum of all 8 probabilities is 1, and the category with the highest probability is taken as the violation identification result output.

[0056] It should be noted that the residual convolutional network is trained end-to-end using the cross-entropy loss function on the labeled dataset of violations of power operation regulations. The training process of the residual convolutional network is a well-known technique in this field and will not be described in detail here.

[0057] Thus, by performing mean time-series aggregation on the feature vectors of M frames, the temporal distribution characteristics of the entire monitoring video are fully captured, avoiding recognition fluctuations caused by the randomness of key frame sampling positions; by fusing classification to obtain the probability distribution of each violation category, accurate identification of violations at the power operation site is achieved.

[0058] It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this patent application shall be determined by the appended claims.

Claims

1. A method for identifying traffic violation video information based on a convolutional neural network model, characterized in that, The identification method includes: Preprocessing is performed on the monitoring video of the power operation site to obtain preprocessed video; The preprocessed video is divided into M segments, and video frames are extracted from each segment to obtain a representative frame sequence; The representative frame sequence is input into the residual convolutional network. After the first convolutional layer of each residual block in the residual convolutional network, the channel weight vector is calculated on the current frame feature map through the channel attention module. Selective temporal shift is performed with the channel weight vector as a guide. After the residual convolutional network completes forward propagation, the frame feature vector is obtained. Perform temporal aggregation and fusion classification on the frame feature vectors of all representative frames, and output the violation behavior recognition results; In the selective temporal shift, the shift direction of each channel is adaptively determined by the attention temporal gradient. For each residual block, the channel weight vectors calculated by the channel attention module for the current frame and the adjacent previous frame are subtracted to obtain the attention temporal gradient vector. For channels with positive attention temporal gradients, forward shift is performed; for channels with negative attention temporal gradients, backward shift is performed; for channels with absolute attention temporal gradient values ​​lower than a preset gradient threshold, they do not participate in the temporal shift.

2. The method for identifying traffic violation video information based on a convolutional neural network model according to claim 1, characterized in that, The preprocessing includes cropping and grayscale processing of the surveillance video to make the surveillance video meet the input size of the residual convolutional network.

3. The method for identifying traffic violation video information based on a convolutional neural network model according to claim 1, characterized in that, During the training phase of the residual convolutional network, after extracting video frames from each segment, the recognition method further includes performing at least one operation on the video frames: geometric transformation and pixel adjustment; the geometric transformation includes at least one of flipping, rotating, and scaling; the pixel adjustment includes at least one of brightness adjustment, contrast adjustment, and hue adjustment.

4. The method for identifying traffic violation video information based on a convolutional neural network model according to claim 1, characterized in that, The channel attention module sequentially includes a global average pooling layer, a one-dimensional convolutional layer, and a sigmoid activation function. The global average pooling layer is used to globally compress the feature map of the current frame to obtain a channel description vector. The one-dimensional convolutional layer is used to perform convolution operations on the channel description vector to obtain the original weights of each channel. The original weights are normalized by the sigmoid activation function to obtain a channel weight vector.

5. The method for identifying traffic violation video information based on a convolutional neural network model according to claim 4, characterized in that, After obtaining the channel weight vector of the k-th residual block, the identification method further includes: The channel weight vector is passed as historical weight to the channel attention module of the (k+1)th residual block; the channel weight vector of the (k+1)th residual block is obtained by weighted merging of two parts: one is the current weight calculated by the channel attention module in the (k+1)th residual block, and the other is the historical weight inherited from the kth residual block.

6. The method for identifying traffic violation video information based on a convolutional neural network model according to claim 1, characterized in that, The shift intensity of each channel is determined by the corresponding weight value in the channel weight vector; channels with weights lower than a preset weight threshold are judged as low-discriminative channels and do not participate in timing shift; channels with weights not lower than the preset weight threshold have their shift intensity determined according to their respective weight ratios; the actual timing shift of each channel is jointly determined by the shift direction and the shift intensity.

7. The method for identifying traffic violation video information based on a convolutional neural network model according to claim 6, characterized in that, The shift intensity of each channel is scaled as a whole using the inter-frame difference. Before performing the temporal shift, the distance between the flattened vectors of the feature maps of adjacent frames in the current video segment is calculated to obtain the inter-frame difference, and the inter-frame difference is used as a scaling factor to scale the shift intensity of each channel as a whole.

8. The method for identifying traffic violation video information based on a convolutional neural network model according to claim 1, characterized in that, The channel attention module and the selective temporal shift module are jointly embedded in each residual block of the residual convolutional network, located after the first convolutional layer and before the second convolutional layer.

9. The method for identifying traffic violation video information based on a convolutional neural network model according to claim 1, characterized in that, The fusion classification adopts a multi-classification approach. After temporally aggregating the frame feature vectors of all video frames, the probability distribution of each violation category is output through the classification layer, and the category with the highest probability is taken as the violation recognition result.

Images