Power transmission work vehicle detection method and system based on frame sequence

Abstract

The invention provides a power transmission working vehicle detection method and system based on a frame sequence, and the method comprises the steps: carrying out the frame division of a target video related to a power transmission working scene, so as to form multiple frames of image data, carrying out the depth extraction, so as to obtain depth features, carrying out the time fusion of the depth features corresponding to each frame of image data, so as to obtain a fusion feature map, therefore, by combining the time sequence information of the multiple frames of images, the detection precision and robustness of the target under the conditions of shielding, motion blur and scale change are effectively improved, then target detection about the detected target is performed to obtain the candidate target frame and the confidence score corresponding to the candidate target frame, and the target detection accuracy is improved. According to the method, weighted average is carried out based on the confidence score to obtain a smoothing result score, and the smoothing result score is compared with a pre-obtained threshold to confirm a detection result of the detection target, so that dynamic confidence evaluation and adaptive threshold adjustment are carried out based on multi-frame information, single-frame false detection and missing detection can be remarkably reduced, and the detection stability is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle detection, and more particularly to a vehicle detection method based on a neural network, and more particularly to a power transmission working vehicle detection method and system based on a frame sequence. Background Art

[0002] Vehicle detection is often required in power transmission work scenarios. The presence of vehicles often affects power transmission operations. Therefore, a method is needed that can automatically monitor the power transmission work scenario and then automatically detect vehicles to determine whether there are redundant vehicles in the power transmission work scenario. Existing power transmission work vehicle detection methods are mainly based on single-frame image target detection technology. Such methods usually use convolutional neural networks (such as YOLO, Faster R-CNN, etc.) to extract features and detect targets from single-frame images. Although these methods perform well in static scenarios, in practical applications, especially in power transmission work environments, vehicle detection faces the following challenges:

[0003] 1. Occlusion and deformation: There are often obstructions such as equipment, trees, and other vehicles in power transmission work scenes, which may cause the vehicle to be partially or completely blocked, affecting the detection accuracy. At the same time, the vehicle may be deformed due to changes in angle or load, further increasing the difficulty of detection.

[0004] 2. Motion blur and scale change: In surveillance videos, factors such as the speed of the vehicle and the shaking of the camera may cause motion blur in the image. In addition, due to the different distances between the camera and the vehicle, the scale of the vehicle in the image changes greatly, resulting in unstable detection.

[0005] 3. Environmental interference: Different lighting conditions, weather changes (such as rain and snow), and increased background complexity will significantly affect the stability and accuracy of the detection model.

[0006] Traditional single-frame detection methods are difficult to cope with the dynamic changes in these complex scenes, resulting in frequent false detections and missed detections. Although some existing improvement schemes, such as motion detection based on optical flow or multi-frame detection based on target tracking, have alleviated the above problems to a certain extent, they still have defects such as poor detection stability and insufficient robustness.

[0007] Therefore, there is an urgent need for a frame sequence-based power transmission vehicle detection method and system that can improve detection accuracy and robustness, reduce single-frame false detection and missed detection, enhance detection stability, improve practicality, and expand the scope of application. Summary of the invention

[0008] In view of the above problems, the purpose of the present invention is to provide a method and system for detecting power transmission working vehicles based on frame sequences, so as to solve the technical problems that traditional single-frame detection methods are difficult to cope with the dynamic changes in these complex scenes, resulting in frequent false detections and missed detections, poor detection stability and insufficient robustness.

[0009] The present invention provides a method for detecting a power transmission vehicle based on a frame sequence, which comprises:

[0010] Dividing the target video about the power transmission working scene into frames to form multiple frames of image data, and preprocessing the image data of each frame to form a standard image;

[0011] Performing depth extraction on the standard image to obtain depth features, and performing time-series fusion on the depth features corresponding to each frame of image data to obtain a fusion feature map;

[0012] Based on the fused feature map, target detection is performed on the multiple frames of image data with respect to the detection target to obtain a candidate target frame and a confidence score corresponding to the candidate target frame;

[0013] A smoothing process is performed based on the confidence score to obtain a smoothed result score, and the smoothed result score is compared with a pre-acquired threshold to confirm the detection result of the detection target.

[0014] Optionally, the preprocessing includes denoising, normalization and resizing.

[0015] Optionally, performing depth extraction on the standard image to obtain depth features, and performing time-series fusion of the depth features corresponding to each frame of image data to obtain a fusion feature map, includes:

[0016] A convolutional neural network is used to extract deep features from each frame of the standard image to obtain a deep feature vector;

[0017] The depth feature vector corresponding to the continuous multi-frame image data is input into a preset long short-term memory network to obtain a depth feature map corresponding to the depth feature vector, and the depth feature map is temporally fused to form a fused feature map.

[0018] Optionally, the weight corresponding to each frame of the standard image is dynamically adjusted through the attention mechanism;

[0019] The depth feature maps corresponding to the weights are temporally fused according to the weights to form a fused feature map.

[0020] Optionally, the depth feature map includes spatial position features, shape features and texture information.

[0021] Optionally, performing smoothing based on the confidence score to obtain a smoothed result score includes:

[0022] Associating candidate target frames and confidence scores corresponding to the same detection target in multiple frames of image data, and using a dynamic confidence evaluation mechanism to update the confidence score of the detection target;

[0023] Taking a weighted average of the confidence scores generated by the last update according to the weights to generate a comprehensive confidence score;

[0024] The comprehensive confidence score is subjected to time series smoothing processing by a Kalman filter algorithm or a sliding average algorithm to obtain a smoothed result score.

[0025] Optionally, when prefetching thresholds:

[0026] According to the comprehensive confidence score, the preset comparison value is dynamically adjusted based on a preset adaptive threshold strategy, and the comparison value obtained by the last adjustment is used as the threshold.

[0027] Optionally, in the process of comparing the smoothed result score with a pre-acquired threshold value to confirm the detection result of the detection target,

[0028] If the smoothing result scores corresponding to each frame of the standard image are higher than the threshold, it is confirmed that the detection target exists.

[0029] The present invention also provides a power transmission vehicle detection system based on a frame sequence, which is used to implement the power transmission vehicle detection method based on a frame sequence as described above, and the system comprises:

[0030] A preprocessing unit, used for dividing the target video about the power transmission working scene into frames to form multiple frames of image data, and preprocessing the image data of each frame to form a standard image;

[0031] A feature extraction unit, used to perform depth extraction on the standard image to obtain depth features, and to perform time-sequential fusion of the depth features corresponding to each frame of image data to obtain a fused feature map;

[0032] A feature fusion unit, configured to perform target detection on a plurality of frames of image data with respect to a detection target based on the fused feature map to obtain a candidate target frame and a confidence score corresponding to the candidate target frame;

[0033] The detection judgment unit is used to perform weighted averaging based on the confidence scores to obtain a smoothing result score, and compare the smoothing result score with a pre-acquired threshold to confirm the detection result of the detection target.

[0034] Optionally, the feature extraction unit includes:

[0035] Convolutional neural network, used to extract deep features from each frame of the standard image to obtain a deep feature vector;

[0036] A long short-term memory network is used to obtain a deep feature map corresponding to the deep feature vector, and perform temporal fusion on the deep feature map to form a fused feature map.

[0037] It can be seen from the above technical scheme that the power transmission working vehicle detection method and system based on frame sequence provided by the present invention first divide the target video of the power transmission working scene into frames to form multiple frames of image data, and then perform depth extraction to obtain depth features respectively, and perform time-series fusion of the depth features corresponding to each frame of image data to obtain a fusion feature map, thereby effectively improving the detection accuracy and robustness of the target under occlusion, motion blur and scale change by combining the timing information of multiple frames of images, and then performing target detection on the detected target to obtain a candidate target frame and a confidence score corresponding to the candidate target frame, and performing weighted averaging based on the confidence score to obtain a smooth result score, and comparing the smooth result score with the pre-acquired threshold to confirm the detection result of the detected target, thereby performing dynamic confidence evaluation and adaptive threshold adjustment based on multiple frame information, which can significantly reduce single-frame false detection and missed detection, enhance the stability of detection, and adopt a lightweight convolutional neural network structure, the amount of calculation is relatively controllable, suitable for real-time monitoring, and has strong environmental adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] By referring to the following description in conjunction with the accompanying drawings, and with a more comprehensive understanding of the present invention, other objects and results of the present invention will become more clear and easy to understand. In the accompanying drawings:

[0039] Figure 1 A flowchart of a power transmission working vehicle detection method based on a frame sequence according to an embodiment of the present invention;

[0040] Figure 2 The figure is a logic block diagram of a power transmission working vehicle detection system based on frame sequence according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] Existing power transmission vehicle detection methods are mainly based on single-frame image target detection technology. Such methods usually use convolutional neural networks (such as YOLO, Faster R-CNN, etc.) to extract features and detect targets on single-frame images. Although these methods perform well in static scenes, in practical applications, especially in power transmission working environments, vehicle detection is susceptible to occlusion and deformation, and is easily affected by vehicle motion blur and scale changes, as well as environmental interference, which affects the stability and accuracy of the detection model, and the detection stability is poor and the robustness is insufficient.

[0042] In view of the above problems, the present invention provides a method and system for detecting power transmission working vehicles based on frame sequences. The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0043] In order to illustrate the frame sequence-based power transmission vehicle detection method and system provided by the present invention, Figure 1 , Figure 2 The embodiments of the present invention are exemplarily indicated.

[0044] The following description of exemplary embodiments is merely illustrative in nature and is not intended to limit the present invention and its application or use. Technologies and devices known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technologies and devices should be considered as part of the specification.

[0045] like Figure 1 As shown, the present invention provides a method for detecting a power transmission vehicle based on a frame sequence, comprising:

[0046] S1: dividing the target video about the power transmission working scene into frames to form multiple frames of image data, and preprocessing the image data of each frame to form a standard image;

[0047] S2: Performing depth extraction on the standard image to obtain depth features, and performing time-series fusion on the depth features corresponding to each frame of image data to obtain a fusion feature map;

[0048] S3: performing target detection on the multiple frames of image data about the detection target based on the fused feature map to obtain candidate target frames and confidence scores corresponding to the candidate target frames;

[0049] S4: performing smoothing processing based on the confidence score to obtain a smoothed result score, and comparing the smoothed result score with a pre-acquired threshold to confirm the detection result of the detection target.

[0050] Step S1 is a process of dividing an image into multiple single-frame image data, and then preprocessing each single-frame image data. More specifically, the transmission site is first monitored and photographed by a camera device, which can be distributed at multiple locations in the transmission site to comprehensively monitor the transmission site, and then the target video of the transmission work scene shot in the transmission work scene is divided into single frames to obtain continuous multiple frames of image data, and then each frame of image data is preprocessed, including denoising, normalization, and resizing operations, in preparation for subsequent feature extraction.

[0051] Step S2 is a process of performing depth extraction on the standard image to obtain depth features, and performing time-series fusion of the depth features corresponding to each frame of image data to obtain a fused feature map.

[0052] Among them, the depth extraction of the standard image is performed to obtain the depth feature, and the depth feature corresponding to each frame of image data is continuously fused to obtain the fusion feature map, including:

[0053] S21: Use a convolutional neural network to extract deep features from each frame of the standard image to obtain a deep feature vector;

[0054] S22: Input the depth feature vector corresponding to the continuous multi-frame image data into a preset long short-term memory network to obtain a depth feature map corresponding to the depth feature vector, and perform time-series fusion on the depth feature map to form a fused feature map.

[0055] The convolutional neural network described in step S21 can be trained using commonly used neural network architectures such as ResNet, YOLO, etc. until it reaches the training standard as a convolutional neural network used directly. The trained convolutional neural network can extract features from each frame of image data and extract a deep feature map of the vehicle target, which contains the spatial position features, shape features and texture information of the target.

[0056] When performing feature fusion in step S22, the depth features corresponding to each standard image in the continuous multi-frame image data are first input into the long short-term memory network. In this embodiment, the long short-term memory network can be an LSTM network to capture the dynamic changes of the target in the time dimension. The long short-term memory network (LSTM) performs temporal fusion of multi-frame features to generate a temporal feature map as a fusion feature map, which can reflect the change trend of the target in the time series. More specifically, in this embodiment, the input of the LSTM is the deep feature vector F of the image data of each single frame. t , the time series from time tn to time t is {F t-n ,F t-(n-1) ,...,F t-1 ,F t}, the output of LSTM can be expressed as the time series feature H t :

[0057] H t =LSTM(F t-n ,F t-(n-1) ,...,F t-1 ,F t ), the graph corresponding to the time series feature is the time series feature graph, which can be used as a fusion feature graph.

[0058] In addition, in this embodiment, an attention mechanism is also included, through which the weight corresponding to each frame of the standard image is dynamically adjusted; according to the weight, the deep feature map corresponding to the weight is temporally fused to form a fused feature map, thereby enhancing the targetedness of obtaining the fused feature map.

[0059] In this embodiment, step S4 is a process of performing smoothing processing based on the confidence score to obtain a smoothing result score, and comparing the smoothing result score with a pre-acquired threshold to confirm the detection result of the detection target; wherein,

[0060] Performing a smoothing process based on the confidence score to obtain a smoothed result score includes:

[0061] S411: Associating candidate target frames and confidence scores corresponding to the same detection target in multiple frames of image data, and updating the confidence score of the detection target using a dynamic confidence evaluation mechanism;

[0062] S412: performing weighted averaging on the confidence scores generated by the latest update according to the weights to generate a comprehensive confidence score;

[0063] S413: Perform time series smoothing processing on the comprehensive confidence score by using a Kalman filter algorithm or a sliding average algorithm to obtain a smoothed result score.

[0064] When prefetching thresholds:

[0065] According to the comprehensive confidence score, the preset comparison value is dynamically adjusted based on a preset adaptive threshold strategy, and the comparison value obtained by the last adjustment is used as the threshold.

[0066] It should be noted that the comparison value is actually a preset threshold value, but when selecting the threshold value, it is necessary to first select multiple values ​​for experimentation, and the selected multiple values ​​become the comparison value. The adaptive threshold strategy dynamically adjusts the comparison value here means that after the last comparison value is used as the threshold, the threshold is dynamically adjusted according to the performance of the target in consecutive frames. For example, if the target is detected in multiple frames and the confidence score is higher than the current threshold each time, the threshold can be lowered to make it easier to detect similar targets. If the target has ordinary false detection or missed detection in multiple frames, the threshold can be increased to reduce misjudgment; even when the threshold is used, the trial results are fed back, and the last used threshold is adjusted again as the comparison value. Statistical methods, such as standard deviation, can be used to analyze the fluctuation of the comprehensive confidence. For example, when the standard deviation of the comprehensive confidence exceeds a certain threshold, the detection threshold can be increased to avoid false detection under high fluctuations, and then the last comparison value is used as the threshold for the next use. In other words, the threshold is continuously updated based on historical detection results. If the detections in multiple consecutive frames are confirmed, the threshold can be gradually lowered; if a false detection occurs, the threshold can be increased.

[0067] In the process of comparing the smoothed result score with the pre-acquired threshold value to confirm the detection result of the detection target,

[0068] S421: If the smoothing result score corresponding to each frame of the standard image is higher than the threshold, it is confirmed that the detection target exists.

[0069] More specifically, in a specific embodiment, based on the time series characteristic graph H t Perform target detection to obtain the candidate target frame B for each frame t and its confidence score C t :

[0070] D t ={(B t ,C t )};

[0071] Get frame confidence: For the same target in each frame (the target in the candidate target box), extract its confidence score C t , C t-1 , C t-2 Similarly, the number of obtained values ​​may depend on the situation, for example, the current frame plus the frame before and after the current frame, or the two frames before and after the current frame, and then a weighted average is performed; when calculating the weighted average, the weighted average of the preset number of confidence scores is obtained as the comprehensive confidence score:

[0072]

[0073] Among them, W i It is the weight assigned to each frame, usually based on the time distance between the frame and the current frame. For example, the closer the frame is to the current moment, the greater its frame weight. In this way, the closer frame has a greater impact on the comprehensive score.

[0074] After obtaining the comprehensive confidence score, smoothing is performed. At this stage, smoothing is to reduce the instability caused by instantaneous noise or short-term occlusion. Smoothing can be achieved by taking a time-weighted average of the comprehensive confidence score. The specific steps are as follows:

[0075] First, we define the smoothing formula in a way similar to the exponentially weighted moving average:

[0076] C 平滑 (t) = α*C 综合 (t)+(1-α)*C 平滑 (t-1);

[0077] Where: C 平滑 (t) is the smoothing result at the current moment;

[0078] C 综合 (t) is the comprehensive confidence score at the current moment;

[0079] C 平滑 (t-1) is the smoothing result of the previous moment;

[0080] α is a smoothing factor (e.g. 0.1), which controls the degree of influence of current value and historical value, and can be determined according to the situation;

[0081] When performing the smoothing process, a starting value is initially set, such as C 平滑 (0) = C 综合 (0), and then in subsequent frames, the smoothing result is continuously updated.

[0082] In summary, the comprehensive confidence score C 综合 It is obtained by weighted averaging the confidence of multiple consecutive frames, reflecting the dynamic performance of the target in time series; the smoothing process is based on the comprehensive confidence score, using the current comprehensive score and the smoothing result of the previous moment for weighted combination to reduce the impact of noise and short-term fluctuations.

[0083] The comprehensive confidence score is then compared with the pre-acquired threshold to determine whether the target exists. When the comprehensive confidence score of the same target in multiple consecutive frames exceeds the threshold, the final detection result of the target is confirmed, so that it can accurately determine whether there are redundant vehicles that affect the power transmission work in the power transmission work site. If there are redundant vehicles, they can be reminded and arranged in time.

[0084] As described above, the power transmission vehicle detection method based on frame sequence provided by the present invention has the following features:

[0085] Beneficial effects:

[0086] 1. Improve detection accuracy and robustness: By combining the temporal information of multiple frames of images, the detection accuracy and robustness of targets under occlusion, motion blur and scale changes can be effectively improved;

[0087] 2. Dynamic confidence evaluation: Dynamic confidence evaluation and adaptive threshold adjustment based on multi-frame information can significantly reduce single-frame false detection and missed detection, and enhance detection stability;

[0088] 3. Strong real-time performance: The present invention adopts a lightweight convolutional neural network and LSTM structure, the amount of calculation is relatively controllable, and it is suitable for use in real-time monitoring systems;

[0089] 4. Wide scope of application: The present invention is applicable to the detection of power transmission vehicles under various complex working conditions and has strong environmental adaptability.

[0090] like Figure 2 As shown, the present invention also provides a power transmission vehicle detection system 100 based on a frame sequence, which implements the power transmission vehicle detection method based on a frame sequence as described above, and the system includes:

[0091] A preprocessing unit 110 is used to divide the target video about the power transmission working scene into frames to form multiple frames of image data, and preprocess the image data of each frame to form a standard image;

[0092] A feature extraction unit 120 is used to perform depth extraction on the standard image to obtain depth features, and to perform time-sequential fusion of the depth features corresponding to each frame of image data to obtain a fused feature map;

[0093] A feature fusion unit 130, configured to perform target detection on the multiple frames of image data with respect to the detection target based on the fused feature map to obtain a candidate target frame and a confidence score corresponding to the candidate target frame;

[0094] The detection judgment unit 140 is used to perform weighted averaging based on the confidence scores to obtain a smoothing result score, and compare the smoothing result score with a pre-acquired threshold to confirm the detection result of the detection target.

[0095] The feature extraction unit 120 includes:

[0096] A convolutional neural network 121 is used to extract deep features from each frame of the standard image to obtain a deep feature vector;

[0097] The long short-term memory network 122 is used to obtain a depth feature map corresponding to the depth feature vector, and perform temporal fusion on the depth feature map to form a fused feature map.

[0098] A more specific implementation of the power transmission working vehicle detection system based on frame sequence can be referred to the above-mentioned embodiment of the power transmission working vehicle detection method based on frame sequence, which will not be described in detail here.

[0099] As described above with reference to the accompanying drawings, the power transmission working vehicle detection method and system based on frame sequence proposed in the present invention are described by way of example. However, those skilled in the art should understand that various improvements can be made to the power transmission working vehicle detection method and system based on frame sequence proposed in the present invention without departing from the content of the present invention. Therefore, the protection scope of the present invention should be determined by the content of the attached claims.

Claims

1. A method for detecting power transmission vehicles based on frame sequences, characterized in that: include: Dividing the target video about the power transmission working scene into frames to form multiple frames of image data, and preprocessing the image data of each frame to form a standard image; Performing depth extraction on the standard image to obtain depth features, and performing time-series fusion on the depth features corresponding to each frame of image data to obtain a fusion feature map; Based on the fused feature map, target detection is performed on the multiple frames of image data with respect to the detection target to obtain a candidate target frame and a confidence score corresponding to the candidate target frame; A smoothing process is performed based on the confidence score to obtain a smoothed result score, and the smoothed result score is compared with a pre-acquired threshold to confirm the detection result of the detection target.

2. The power transmission vehicle detection method based on frame sequence according to claim 1, characterized in that: The preprocessing includes denoising, normalization and resizing.

3. The power transmission vehicle detection method based on frame sequence according to claim 1, characterized in that: The step of performing depth extraction on the standard image to obtain depth features, and performing time-series fusion of the depth features corresponding to each frame of image data to obtain a fusion feature map, includes: A convolutional neural network is used to extract deep features from each frame of the standard image to obtain a deep feature vector; The depth feature vector corresponding to the continuous multi-frame image data is input into a preset long short-term memory network to obtain a depth feature map corresponding to the depth feature vector, and the depth feature map is temporally fused to form a fused feature map.

4. The power transmission vehicle detection method based on frame sequence according to claim 3, characterized in that: Dynamically adjust the weight corresponding to each frame of the standard image through the attention mechanism; The depth feature maps corresponding to the weights are temporally fused according to the weights to form a fused feature map.

5. The power transmission vehicle detection method based on frame sequence according to claim 4, characterized in that: The depth feature map includes spatial position features, shape features and texture information.

6. The power transmission vehicle detection method based on frame sequence according to claim 4, characterized in that: The step of performing smoothing based on the confidence score to obtain a smoothed result score includes: Associating candidate target frames and confidence scores corresponding to the same detection target in multiple frames of image data, and using a dynamic confidence evaluation mechanism to update the confidence score of the detection target; Taking a weighted average of the confidence scores generated by the last update according to the weights to generate a comprehensive confidence score; The comprehensive confidence score is subjected to time series smoothing processing by a Kalman filter algorithm or a sliding average algorithm to obtain a smoothed result score.

7. The power transmission vehicle detection method based on frame sequence according to claim 6, characterized in that: When prefetching thresholds: According to the comprehensive confidence score, the preset comparison value is dynamically adjusted based on a preset adaptive threshold strategy, and the comparison value obtained by the last adjustment is used as the threshold.

8. The power transmission vehicle detection method based on frame sequence according to claim 7, characterized in that: In the process of comparing the smoothed result score with the pre-acquired threshold value to confirm the detection result of the detection target, If the smoothing result scores corresponding to each frame of the standard image are higher than the threshold, it is confirmed that the detection target exists.

9. A power transmission vehicle detection system based on frame sequence, characterized in that: For implementing the power transmission vehicle detection method based on frame sequence according to any one of claims 1 to 8, the system comprises: A preprocessing unit, used for dividing the target video about the power transmission working scene into frames to form multiple frames of image data, and preprocessing the image data of each frame to form a standard image; A feature extraction unit, used to perform depth extraction on the standard image to obtain depth features, and to perform time-sequential fusion of the depth features corresponding to each frame of image data to obtain a fused feature map; A feature fusion unit, configured to perform target detection on the multiple frames of image data with respect to the detection target based on the fused feature map to obtain a candidate target frame and a confidence score corresponding to the candidate target frame; The detection judgment unit is used to perform weighted averaging based on the confidence scores to obtain a smoothing result score, and compare the smoothing result score with a pre-acquired threshold to confirm the detection result of the detection target.

10. The power transmission vehicle detection system based on frame sequence according to claim 9, characterized in that: The feature extraction unit comprises: Convolutional neural network, used to extract deep features from each frame of the standard image to obtain a deep feature vector; A long short-term memory network is used to obtain a deep feature map corresponding to the deep feature vector, and to perform temporal fusion on the deep feature map to form a fused feature map.

Images