Multi-target tracking method and system for electric power high-altitude operation

Abstract

The invention discloses an electric power high-altitude operation multi-target tracking method. The method comprises the following steps: obtaining a detection frame of a high-altitude operation worker and a corresponding detection score through GN-Lite; judging the obtained detection score, and dividing the detection frame into a high-confidence detection frame and a low-confidence detection frame; predicting a previous frame tracking frame based on historical motion information of the high-altitude operation worker to obtain a prediction frame of a current frame; and performing target matching with the prediction frame according to the obtained high-confidence detection frame and low-confidence detection frame, and completing continuous multi-target tracking of the electric power high-altitude operation. The invention also discloses a system for realizing the multi-target tracking method for the electric power high-altitude operation. According to the method, optimization is carried out by utilizing a width optimization Kalman filter, camera motion compensation and a Gaussian smoothing generalized intersection-and-union comparison algorithm, so that the resource consumption of a detection network is reduced, and the tracking precision of the electric power overhead working personnel is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of high-altitude operations on electric power lines, and in particular relates to a multi-target tracking method and system for high-altitude operations on electric power lines. Background Art

[0002] Power aerial work refers to work performed at heights in power facilities such as transmission lines, substations, and towers. These tasks usually require operators to work at a great height above the ground, which poses a great risk to the safety of operators. In the process of power production and system maintenance, power companies generally face the problem of operators falling from heights, and need to carry out targeted risk prevention and control.

[0003] Therefore, in the current power production operations, the use of drone monitoring video images, combined with computer vision and image processing technology, to achieve real-time monitoring and tracking of power high-altitude operations is of great significance to the implementation of smart construction by power companies. Early methods mainly tracked operators based on methods such as Kalman filters. Although they had good real-time performance, they did not involve situations with poor visual conditions (such as weak lighting). With the development of deep learning technology, the method of using target detection network to assist tracking has significantly improved the tracking accuracy of operators. However, the power operation environment is complex, and the existing power high-altitude operation data set is small. The tracking algorithm has not been optimized for various operation scenarios. For some operating environments, the existing tracking methods are not effective and the tracking accuracy is not high. Summary of the invention

[0004] One of the purposes of the present invention is to provide a multi-target tracking method for electric power aerial work, which can effectively improve the accuracy of target tracking and reduce network calculation and resource consumption.

[0005] A second object of the present invention is to provide a system for implementing the multi-target tracking method for high-altitude power operations.

[0006] The present invention provides a multi-target tracking method for electric power aerial work, comprising the following steps:

[0007] S1. Obtain the detection frame of the high-altitude workers and the corresponding detection score;

[0008] S2. judging the detection score obtained in step S1, and dividing the detection frame into a high-confidence detection frame and a low-confidence detection frame;

[0009] S3. Predict the previous frame tracking frame based on the historical motion information of the high-altitude workers to obtain the predicted frame of the current frame;

[0010] S4. Perform target matching based on the high-confidence detection frame and low-confidence detection frame obtained in step S2 and the prediction frame obtained in step S3 to complete continuous multi-target tracking of power aerial operations.

[0011] Step S1 is specifically as follows: a GN-Lite target detection network is constructed based on the YOLO network, target detection is performed on high-altitude workers, and a detection frame and a corresponding detection score are obtained.

[0012] Step S2 is specifically as follows: a confidence threshold is preset, and if the detection score is greater than or equal to the confidence threshold, the corresponding detection frame is classified as a high-confidence detection frame; if the detection score is lower than the confidence threshold, the corresponding detection frame is classified as a low-confidence detection frame.

[0013] Step S3 is specifically as follows: based on the previous frame tracking frame of the historical motion information of the aerial workers, a width-optimized Kalman filter and camera motion compensation processing are used to obtain a predicted frame of the current frame.

[0014] The state vector and measurement vector of the width optimized Kalman filter are expressed using the following formula:

[0015]

[0016] Among them, x k The state vector of the Kalman filter is optimized for width; z k The measurement vector for the width-optimized Kalman filter; x c (k) is the horizontal coordinate of the center point of the tracking frame in the kth frame; y c (k) is the ordinate of the center point of the tracking frame of the kth frame; w(k) is the aspect ratio of the tracking frame of the kth frame; h(k) is the height of the tracking frame of the kth frame; x v (k) is the speed of change of the horizontal coordinate of the center point of the tracking frame in the kth frame; v (k) is the changing speed of the ordinate of the center point of the tracking frame in the kth frame; Track the changing speed of the frame aspect ratio for the kth frame; Track the changing speed of the box height for the kth frame; Measure the horizontal coordinate of the center point of the tracking box for the kth frame; Measure the vertical coordinate of the center point of the tracking box for the kth frame; z w (k) is the aspect ratio measured by the tracking box of the kth frame; z h (k) is the measured height of the tracking box in the kth frame;

[0017] According to the change of the state vector of the width optimized Kalman filter, the process noise covariance and measurement noise covariance matrices are modified and expressed by the following formula:

[0018]

[0019] Among them, Q K is the process noise covariance matrix; R K is the measurement noise covariance matrix; σ p ,σ v ,σ m is the noise factor; diag[] is the diagonal element of the extraction matrix; Estimated aspect ratio of the tracking frame updated at the k-1th moment; The height estimate of the tracking box updated at the k-1th moment;

[0020] The width optimized Kalman filter predicts the position of the tracking target and then fuses it with the observed position data of the corresponding tracking target to obtain the optimal position estimate of the target. k and the measurement vector z k The input to the width-optimized Kalman filter, and The output of the Kalman filter is optimized for width.

[0021] The camera motion compensation process is specifically as follows: first, the ORB algorithm is used to extract background key points other than the operator from the original image of the current frame and the image of the current frame detected by GN-Lite obtained in step S1, and then the sparse optical flow is used to match the extracted background key points of the current frame with the background key points of the previous frame to obtain matched key points; then, RANSAC is used to obtain the affine transformation matrix of the background motion, and the affine transformation matrix is ​​used to transform the prediction frame of the operator in the current frame obtained in the width optimized Kalman filter from the coordinate system of the previous frame to the coordinate system of the current frame, and the position information of the prediction frame in the current frame is corrected to obtain the updated and corrected prediction frame of the current frame;

[0022] The affine transformation matrix of background motion obtained using RANSAC is expressed using the following formula:

[0023]

[0024] in, is the affine matrix from the coordinate system of the k-1 frame to the coordinate system of the k frame, where the k frame is the current frame; M 2x2 for The matrices for the scaling and rotation parts of 2x1 for The matrix of the translation part; a 11 、a 12 、a 13 、a 21 、a 22 、a 23 is the element in the affine matrix;

[0025] The matrices for scaling and rotating from frame k-1 to frame k are expressed as follows:

[0026]

[0027] in, The matrix of the scaling and rotation part from the k-1 frame to the k frame; diag{} is the diagonal element of the extraction matrix; M is the affine matrix Matrices for scaling and rotating parts;

[0028] The matrix of the translation part from frame k-1 to frame k is expressed using the following formula:

[0029]

[0030] in, is the matrix of the translation part from frame k-1 to frame k;

[0031] The relationship between the predicted state vectors of the width-optimized Kalman filter of k frames before and after camera motion compensation is expressed using the following formula:

[0032]

[0033] in, Optimize the predicted state vector of the Kalman filter for the width of the first k frames for camera motion compensation; Optimize the predicted state vector of the Kalman filter for the width of k frames after camera motion compensation;

[0034] The prediction covariance matrix of the kth frame after updating and correction is calculated using the following formula:

[0035]

[0036] Among them, P k|k-1 is the prediction covariance matrix of the k-1th frame; P k ' |k-1 is the prediction covariance matrix of the kth frame after updating correction.

[0037] Step S4 is specifically as follows: using Gaussian smoothed generalized intersection-over-union ratio to measure the similarity between the detection frame and the tracking frame for target matching, including two matches; in the first match, the high-confidence detection frame obtained in step S2 is matched with the updated and corrected prediction frame of the kth frame obtained in step S3, and the unmatched detection frames and prediction frames are retained, and the unmatched detection frames are used to create new tracks, and the unmatched prediction frames enter the second match, and the matched frames generate the tracking frames of the kth frame; in the second match, the unmatched prediction frames in the first match are matched with the low-confidence detection frames obtained in step S2, and the unmatched frames are directly deleted, and the matched frames generate the tracking frames of the kth frame;

[0038] The Gaussian smoothing interpolation algorithm is used to interpolate and smooth the tracking frames of all the k-th frames generated last, so as to obtain a more continuous and stable target tracking result and complete the target tracking.

[0039] The Gaussian smoothed generalized intersection-over-union (IoU) is based on the generalized intersection-over-union (IoU) and uses a Gaussian linear interpolation algorithm to smooth the successfully matched frame; wherein the IoU between the detection frame and the prediction frame represents the spatial relationship between the two, and is calculated using the following formula:

[0040]

[0041] Among them, IoU is the intersection over union ratio between the detection box and the prediction box; B dt =(x dt ,y dt ,w dt ,h dt ) is the coordinate position of the detection box, where (x dt ,y dt ) is the coordinate of the upper left corner of the detection box, (w dt ,h dt ) is the width and height of the detection box; B KF =(x KF ,y KF ,w KF ,h KF ) is the coordinate position of the prediction box, where (x KF ,y KF ) is the coordinate of the upper left corner of the prediction box, (w KF ,h KF ) is the width and height of the prediction box; |·∩·| is the intersection of the prediction box and the detection box; |·∪·| is the union of the prediction box and the detection box;

[0042] The calculation formula of Gaussian smoothed generalized intersection-union distance is:

[0043]

[0044] Among them, GS-GIoU is the Gaussian smoothed generalized intersection-over-union distance; C is the minimum box area surrounding the detection box and the prediction box;

[0045] The Gaussian smoothing interpolation algorithm comprises the following steps:

[0046] 1) Input the target tracking result file in.txt, the interpolation interval interval and the Gaussian smoothing parameter tau;

[0047] 2) Initialize variables, including the ID id_pre of the previous target, the previous frame number f_pre, and the previous row data row_pre;

[0048] 3) Get the frame number f_curr and target ID id_curr of the current row. If id_curr = id_pre, and f_pre < f_curr < f_curr + interval, then for each frame in between, perform linear interpolation based on the data of the previous row and the current row. Otherwise, directly update the current target ID, the frame number of the current row, and the current row data as the values ​​of id_pre, f_pre, and row_pre.

[0049] 4) Traverse each line of data in the target tracking result file in.txt according to the processing of step 3);

[0050] 5) Sort the list ouput1 by frame number and target ID, and obtain all unique target IDs in the list;

[0051] 6) Traverse all unique target IDs obtained in step 5), select the tracking data of the current target ID, and assign it to tracks;

[0052] 7) Calculate the length scale len_scale according to the amount of tracking data in tracks obtained in step 6) and the Gaussian smoothing parameter tau input in step 1);

[0053] 8) Initialize the Gaussian process regressor gpr according to the length scale len_scale obtained in step 7);

[0054] 9) Use the Gaussian process regressor gpr to smooth the position, width and height of the target in each dimension of the tracks obtained in step 6) to obtain the result processed by the Gaussian smoothing interpolation algorithm.

[0055] The present invention also provides a system for implementing the multi-target tracking method for electric power aerial work, including a data acquisition module, a detection frame classification module, a prediction frame prediction module and a matching tracking result module;

[0056] The data acquisition module obtains the detection frame and corresponding detection score of the aerial workers through GN-Lite, and uploads the data to the detection frame classification module;

[0057] The detection frame classification module divides the detection frames into high-confidence detection frames and low-confidence detection frames according to the received data, and uploads the data to the matching tracking result module;

[0058] The prediction frame prediction module predicts the previous frame tracking frame based on the historical motion information of the aerial workers, obtains the prediction frame of the current frame, and uploads the data to the matching tracking result module;

[0059] The matching tracking result module matches the detection box with the prediction box based on the received data to complete the continuous multi-target tracking of power aerial operations.

[0060] The present invention discloses a method and system for realizing the multi-target tracking of the electric power aerial work, which utilizes the width optimized Kalman filter, camera motion compensation and Gaussian smoothed generalized intersection and comparison algorithm for optimization, reduces the resource consumption of the detection network, and improves the tracking accuracy of the electric power aerial work personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a schematic diagram of the process of the present invention;

[0062] Figure 2 It is a structural schematic diagram of the system of the present invention;

[0063] Figure 3 Schematic diagram of the user end display of the high-altitude work protection detection system based on video images in the embodiment, wherein Figure 3 a is a schematic diagram showing multi-target human body detection at the user end of the high-altitude operation protection detection system based on video images; Figure 3 b is a schematic diagram showing the display of protective equipment detection at the user end of the high-altitude operation protection detection system based on video images;

[0064] Figure 4 Schematic diagram of multi-target tracking results of the high-altitude work protection detection system based on video images in the embodiment, wherein Figure 4 a is a schematic diagram of multi-target human tracking detection results of the high-altitude work protection detection system based on video images; Figure 4 b is a schematic diagram of the protective equipment tracking and detection results of the high-altitude work protection detection system based on video images. DETAILED DESCRIPTION

[0065] The present invention provides a multi-target tracking method for electric power aerial work, comprising the following steps:

[0066] S1. Obtain the detection frame of the high-altitude workers and the corresponding detection score;

[0067] Step S1 is specifically as follows: a GN-Lite target detection network is constructed based on the YOLO network, target detection is performed on high-altitude workers, and a detection frame and a corresponding detection score are obtained.

[0068] S2. Judge the detection score obtained in step S1 and divide the detection frame into a high-confidence detection frame and a low-confidence detection frame, specifically:

[0069] A confidence threshold is preset. If the detection score is greater than or equal to the confidence threshold, the corresponding detection frame is classified as a high-confidence detection frame; if the detection score is lower than the confidence threshold, the corresponding detection frame is classified as a low-confidence detection frame.

[0070] S3. Based on the historical motion information of the aerial workers, the previous frame tracking frame is predicted to obtain the prediction frame of the current frame, specifically:

[0071] Based on the previous frame tracking frame of the historical motion information of the aerial workers, the width optimized Kalman filter and camera motion compensation processing are used to obtain the predicted frame of the current frame.

[0072] The state vector and measurement vector of the width optimized Kalman filter are expressed using the following formula:

[0073]

[0074] Among them, x k The state vector of the Kalman filter is optimized for width; z k The measurement vector for the width-optimized Kalman filter; x c (k) is the horizontal coordinate of the center point of the tracking frame in the kth frame; y c (k) is the ordinate of the center point of the tracking frame of the kth frame; w(k) is the aspect ratio of the tracking frame of the kth frame; h(k) is the height of the tracking frame of the kth frame; x v (k) is the speed of change of the horizontal coordinate of the center point of the tracking frame in the kth frame; v (k) is the changing speed of the ordinate of the center point of the tracking frame in the kth frame; Track the changing speed of the frame aspect ratio for the kth frame; Track the changing speed of the box height for the kth frame; Measure the horizontal coordinate of the center point of the tracking box for the kth frame; Measure the vertical coordinate of the center point of the tracking box for the kth frame; z w (k) is the aspect ratio measured by the tracking box of the kth frame; z h (k) is the measured height of the tracking box in the kth frame;

[0075] According to the change of the state vector of the width optimized Kalman filter, the process noise covariance and measurement noise covariance matrices are modified and expressed by the following formula:

[0076]

[0077] Among them, Q K is the process noise covariance matrix; R K is the measurement noise covariance matrix; σ p ,σ v ,σ mis the noise factor; diag[] is the diagonal element of the extraction matrix; Estimated aspect ratio of the tracking frame updated at the k-1th moment; The height estimate of the tracking box updated at the k-1th moment;

[0078] The width optimized Kalman filter predicts the position of the tracking target and then fuses it with the observed position data of the corresponding tracking target to obtain the optimal position estimate of the target. k and the measurement vector z k The input to the width-optimized Kalman filter, and The output of the Kalman filter is optimized for width.

[0079] The camera motion compensation process is specifically as follows: first, the ORB algorithm is used to extract background key points other than the operator from the original image of the current frame and the image of the current frame detected by GN-Lite obtained in step S1, and then the sparse optical flow is used to match the extracted background key points of the current frame with the background key points of the previous frame to obtain matched key points; then, RANSAC is used to obtain the affine transformation matrix of the background motion, and the affine transformation matrix is ​​used to transform the prediction frame of the operator in the current frame obtained in the width optimized Kalman filter from the coordinate system of the previous frame to the coordinate system of the current frame, and the position information of the prediction frame in the current frame is corrected to obtain the updated and corrected prediction frame of the current frame;

[0080] The affine transformation matrix of background motion obtained using RANSAC is expressed using the following formula:

[0081]

[0082] in, is the affine matrix from the coordinate system of the k-1 frame to the coordinate system of the k frame, where the k frame is the current frame; M 2x2 for The matrices for the scaling and rotation parts of 2x1 for The matrix of the translation part; a 11 、a 12 、a 13 、a 21 、a 22 、a 23 is the element in the affine matrix;

[0083] The matrices for scaling and rotating from frame k-1 to frame k are expressed as follows:

[0084]

[0085] in, The matrix of the scaling and rotation part from the k-1 frame to the k frame; diag{} is the diagonal element of the extraction matrix; M is the affine matrix Matrices for scaling and rotating parts;

[0086] The matrix of the translation part from frame k-1 to frame k is expressed using the following formula:

[0087]

[0088] in, is the matrix of the translation part from frame k-1 to frame k;

[0089] The relationship between the predicted state vectors of the width-optimized Kalman filter of k frames before and after camera motion compensation is expressed using the following formula:

[0090]

[0091] in, Optimize the predicted state vector of the Kalman filter for the width of the first k frames for camera motion compensation; Optimize the predicted state vector of the Kalman filter for the width of k frames after camera motion compensation;

[0092] The prediction covariance matrix of the kth frame after updating and correction is calculated using the following formula:

[0093]

[0094] Among them, P k|k-1 is the prediction covariance matrix of the k-1th frame; P k ' |k-1 is the prediction covariance matrix of the kth frame after updating correction.

[0095] S4. Target matching is performed based on the high confidence detection frame and the low confidence detection frame obtained in step S2 and the prediction frame obtained in step S3 to complete continuous multi-target tracking of power aerial work, specifically:

[0096] The similarity between the detection frame and the tracking frame is measured using Gaussian smoothed generalized intersection-over-union ratio to perform target matching, including two matches; in the first match, the high-confidence detection frame obtained in step S2 is matched with the updated and corrected prediction frame of the kth frame obtained in step S3, and the unmatched detection frames and prediction frames are retained, and a new track is created using the unmatched detection frames. The unmatched prediction frames enter the second match, and the matched frames generate the tracking frames of the kth frame; in the second match, the unmatched prediction frames in the first match are matched with the low-confidence detection frames obtained in step S2, and the unmatched frames are directly deleted, and the matched frames generate the tracking frames of the kth frame;

[0097] The Gaussian smoothing interpolation algorithm is used to interpolate and smooth the tracking frames of all the k-th frames generated last, so as to obtain a more continuous and stable target tracking result and complete the target tracking.

[0098] The Gaussian smoothed generalized intersection-over-union (IoU) is based on the generalized intersection-over-union (IoU) and uses a Gaussian linear interpolation algorithm to smooth the successfully matched frame; wherein the IoU between the detection frame and the prediction frame represents the spatial relationship between the two, and is calculated using the following formula:

[0099]

[0100] Among them, IoU is the intersection over union ratio between the detection box and the prediction box; B dt =(x dt ,y dt ,w dt ,h dt ) is the coordinate position of the detection box, where (x dt ,y dt ) is the coordinate of the upper left corner of the detection box, (w dt ,h dt ) is the width and height of the detection box; B KF =(x KF ,y KF ,w KF ,h KF ) is the coordinate position of the prediction box, where (x KF ,y KF ) is the coordinate of the upper left corner of the prediction box, (w KF ,h KF ) is the width and height of the prediction box; |·∩·| is the intersection of the prediction box and the detection box; |·∪·| is the union of the prediction box and the detection box;

[0101] The calculation formula of Gaussian smoothed generalized intersection-union distance is:

[0102]

[0103] Among them, GS-GIoU is the Gaussian smoothed generalized intersection-over-union distance; C is the minimum box area surrounding the detection box and the prediction box;

[0104] The Gaussian smoothing interpolation algorithm comprises the following steps:

[0105] 1) Input the target tracking result file in.txt, the interpolation interval interval and the Gaussian smoothing parameter tau;

[0106] 2) Initialize variables, including the ID id_pre of the previous target, the previous frame number f_pre, and the previous row data row_pre;

[0107] 3) Get the frame number f_curr and target ID id_curr of the current row. If id_curr = id_pre, and f_pre < f_curr < f_curr + interval, then for each frame in between, perform linear interpolation based on the data of the previous row and the current row. Otherwise, directly update the current target ID, the frame number of the current row, and the current row data as the values ​​of id_pre, f_pre, and row_pre.

[0108] 4) Traverse each line of data in the target tracking result file in.txt according to the processing of step 3);

[0109] 5) Sort the list ouput1 by frame number and target ID, and obtain all unique target IDs in the list;

[0110] 6) Traverse all unique target IDs obtained in step 5), select the tracking data of the current target ID, and assign it to tracks;

[0111] 7) Calculate the length scale len_scale according to the amount of tracking data in tracks obtained in step 6) and the Gaussian smoothing parameter tau input in step 1);

[0112] 8) Initialize the Gaussian process regressor gpr according to the length scale len_scale obtained in step 7);

[0113] 9) Use the Gaussian process regressor gpr to smooth the position, width and height of the target in each dimension of the tracks obtained in step 6) to obtain the result processed by the Gaussian smoothing interpolation algorithm.

[0114] The present invention also provides a system for implementing the multi-target tracking method for electric power aerial work, including a data acquisition module, a detection frame classification module, a prediction frame prediction module and a matching tracking result module;

[0115] The data acquisition module obtains the detection frame and corresponding detection score of the aerial workers through GN-Lite, and uploads the data to the detection frame classification module;

[0116] The detection frame classification module divides the detection frames into high-confidence detection frames and low-confidence detection frames according to the received data, and uploads the data to the matching tracking result module;

[0117] The prediction frame prediction module predicts the previous frame tracking frame based on the historical motion information of the aerial workers, obtains the prediction frame of the current frame, and uploads the data to the matching tracking result module;

[0118] The matching tracking result module matches the detection box with the prediction box based on the received data to complete the continuous multi-target tracking of power aerial operations.

[0119] The present invention is further described below with reference to an embodiment:

[0120] The method of the present invention is applied to a high-altitude operation protection detection system based on image video in actual use.

[0121] Figure 3 The present invention is a user-side display schematic diagram of the system implementing multi-target tracking and identification in high-altitude power construction using the method of the present invention.

[0122] Figure 4 This is a schematic diagram of the results of implementing high-altitude power construction multi-target tracking and identification in an image and video-based high-altitude work protection detection system using the method of the present invention.

Claims

1. A multi-target tracking method for electric power aerial work, characterized in that: The following steps are involved: S1. Obtain the detection frame of the high-altitude workers and the corresponding detection score; S2. judging the detection score obtained in step S1, and dividing the detection frame into a high-confidence detection frame and a low-confidence detection frame; S3. Predict the previous frame tracking frame based on the historical motion information of the high-altitude workers to obtain the predicted frame of the current frame; S4. Perform target matching based on the high-confidence detection frame and low-confidence detection frame obtained in step S2 and the prediction frame obtained in step S3 to complete continuous multi-target tracking of power aerial operations.

2. The multi-target tracking method for electric power aerial work according to claim 1 is characterized in that: Step S1 is specifically as follows: a GN-Lite target detection network is constructed based on the YOLO network, target detection is performed on high-altitude workers, and a detection frame and a corresponding detection score are obtained.

3. The multi-target tracking method for electric power aerial work according to claim 2 is characterized in that: Step S2 is specifically as follows: a confidence threshold is preset, and if the detection score is greater than or equal to the confidence threshold, the corresponding detection frame is classified as a high-confidence detection frame; if the detection score is lower than the confidence threshold, the corresponding detection frame is classified as a low-confidence detection frame.

4. The multi-target tracking method for electric power aerial work according to claim 3 is characterized in that: Step S3 is specifically as follows: based on the previous frame tracking frame of the historical motion information of the aerial workers, a width-optimized Kalman filter and camera motion compensation processing are used to obtain a predicted frame of the current frame.

5. The multi-target tracking method for electric power aerial work according to claim 4 is characterized in that: The state vector and measurement vector of the width optimized Kalman filter are expressed using the following formula: Among them, x k The state vector of the Kalman filter is optimized for width; z k The measurement vector for the width-optimized Kalman filter; x c (k) is the horizontal coordinate of the center point of the tracking frame in the kth frame; y c (k) is the ordinate of the center point of the tracking frame of the kth frame; w(k) is the aspect ratio of the tracking frame of the kth frame; h(k) is the height of the tracking frame of the kth frame; x v (k) is the speed of change of the horizontal coordinate of the center point of the tracking frame in the kth frame; v (k) is the changing speed of the ordinate of the center point of the tracking frame in the kth frame; Track the changing speed of the frame aspect ratio for the kth frame; Track the changing speed of the box height for the kth frame; Measure the horizontal coordinate of the center point of the tracking box for the kth frame; Measure the vertical coordinate of the center point of the tracking box for the kth frame; z w (k) is the aspect ratio measured by the tracking box of the kth frame; z h (k) is the measured height of the tracking box in the kth frame; According to the change of the state vector of the width optimized Kalman filter, the process noise covariance and measurement noise covariance matrices are modified and expressed using the following formula: Among them, Q K is the process noise covariance matrix; R K is the measurement noise covariance matrix; σ p ,σ v ,σ m is the noise factor; diag[] is the diagonal element of the extraction matrix; Estimated aspect ratio of the tracking frame updated at the k-1th moment; The height estimate of the tracking box updated at the k-1th moment; The width optimized Kalman filter predicts the position of the tracking target and then fuses it with the observed position data of the corresponding tracking target to obtain the optimal position estimate of the target. k and the measurement vector z k The input to the width-optimized Kalman filter, and Optimize the output of the Kalman filter for width.

6. The multi-target tracking method for electric power aerial work according to claim 5 is characterized in that: The camera motion compensation process is specifically as follows: first, the ORB algorithm is used to extract background key points other than the operator from the original image of the current frame and the image of the current frame detected by GN-Lite obtained in step S1, and then the sparse optical flow is used to match the extracted background key points of the current frame with the background key points of the previous frame to obtain matching key points; Then, RANSAC is used to obtain the affine transformation matrix of the background motion. The prediction frame of the operator in the current frame obtained by the width-optimized Kalman filter is converted from the coordinate system of the previous frame to the coordinate system of the current frame using the affine transformation matrix. The position information of the prediction frame in the current frame is corrected to obtain the updated and corrected prediction frame of the current frame. The affine transformation matrix of background motion obtained using RANSAC is expressed using the following formula: in, is the affine matrix from the coordinate system of the k-1 frame to the coordinate system of the k frame, where the k frame is the current frame; M 2x2 for The matrices for the scaling and rotation parts of 2x1 for The matrix of the translation part; a 11 、a 12 、a 13 、a 21 、a 22 、a 23 is the element in the affine matrix; The matrices for scaling and rotating from frame k-1 to frame k are expressed as follows: in, The matrix of the scaling and rotation part from the k-1 frame to the k frame; diag{} is the diagonal element of the extraction matrix; M is the affine matrix Matrices for scaling and rotating parts; The matrix of the translation part from frame k-1 to frame k is expressed using the following formula: in, is the matrix of the translation part from frame k-1 to frame k; The relationship between the predicted state vectors of the width-optimized Kalman filter of k frames before and after camera motion compensation is expressed using the following formula: in, Optimize the predicted state vector of the Kalman filter for the width of the first k frames for camera motion compensation; Optimize the predicted state vector of the Kalman filter for the width of k frames after camera motion compensation; The prediction covariance matrix of the kth frame after updating and correction is calculated using the following formula: Among them, P k|k-1 is the prediction covariance matrix of the k-1th frame; P k ' |k-1 is the prediction covariance matrix of the kth frame after updating correction.

7. The multi-target tracking method for electric power aerial work according to claim 6 is characterized in that: Step S4 is specifically as follows: using Gaussian smoothed generalized intersection-over-union ratio to measure the similarity between the detection frame and the tracking frame for target matching, including two matches; in the first match, the high-confidence detection frame obtained in step S2 is matched with the updated and corrected prediction frame of the kth frame obtained in step S3, and the unmatched detection frames and prediction frames are retained, and the unmatched detection frames are used to create new tracks, and the unmatched prediction frames enter the second match, and the matched frames generate the tracking frames of the kth frame; in the second match, the unmatched prediction frames in the first match are matched with the low-confidence detection frames obtained in step S2, and the unmatched frames are directly deleted, and the matched frames generate the tracking frames of the kth frame; The Gaussian smoothing interpolation algorithm is used to interpolate and smooth the tracking frames of all the k-th frames generated last, so as to obtain a more continuous and stable target tracking result and complete the target tracking.

8. The multi-target tracking method for electric power aerial work according to claim 7 is characterized in that: The Gaussian smoothed generalized intersection-over-union (IoU) is based on the generalized intersection-over-union (IoU) and uses a Gaussian linear interpolation algorithm to smooth the successfully matched frame; wherein the IoU between the detection frame and the prediction frame represents the spatial relationship between the two, and is calculated using the following formula: Among them, IoU is the intersection over union ratio between the detection box and the prediction box; B dt =(x dt ,y dt ,w dt ,h dt ) is the coordinate position of the detection box, where (x dt ,y dt ) is the coordinate of the upper left corner of the detection box, (w dt ,h dt ) is the width and height of the detection box; B KF =(x KF ,y KF ,w KF ,h KF ) is the coordinate position of the prediction box, where (x KF ,y KF ) is the coordinate of the upper left corner of the prediction box, (w KF ,h KF ) is the width and height of the prediction box; |·∩·| is the intersection of the prediction box and the detection box; |·∪·| is the union of the prediction box and the detection box; The calculation formula of Gaussian smoothed generalized intersection-union distance is: Among them, GS-GIoU is the Gaussian smoothed generalized intersection-over-union distance; C is the minimum box area surrounding the detection box and the prediction box.

9. The multi-target tracking method for electric power aerial work according to claim 8 is characterized in that: The Gaussian smoothing interpolation algorithm comprises the following steps: 1) Input the target tracking result file in.txt, the interpolation interval interval and the Gaussian smoothing parameter tau; 2) Initialize variables, including the ID id_pre of the previous target, the previous frame number f_pre, and the previous row data row_pre; 3) Get the frame number f_curr and target ID id_curr of the current row. If id_curr = id_pre, and f_pre < f_curr < f_curr + interval, then for each frame in between, perform linear interpolation based on the data of the previous row and the current row. Otherwise, directly update the current target ID, the frame number of the current row, and the current row data as the values ​​of id_pre, f_pre, and row_pre. 4) Traverse each line of data in the target tracking result file in.txt according to the processing of step 3); 5) Sort the list ouput1 by frame number and target ID, and obtain all unique target IDs in the list; 6) Traverse all unique target IDs obtained in step 5), select the tracking data of the current target ID, and assign it to tracks; 7) Calculate the length scale len_scale according to the amount of tracking data in tracks obtained in step 6) and the Gaussian smoothing parameter tau input in step 1); 8) Initialize the Gaussian process regressor gpr according to the length scale len_scale obtained in step 7); The tracking data of each dimension in tracks obtained in step 6) is smoothed using the Gaussian process regressor gpr to obtain the result processed by the Gaussian smoothing interpolation algorithm.

10. A system for implementing the multi-target tracking method for electric power aerial work according to any one of claims 1 to 9, characterized in that: It includes data acquisition module, detection frame classification module, prediction frame prediction module and matching tracking result module; The data acquisition module obtains the detection frame and corresponding detection score of the aerial workers through GN-Lite, and uploads the data to the detection frame classification module; The detection frame classification module divides the detection frames into high-confidence detection frames and low-confidence detection frames according to the received data, and uploads the data to the matching tracking result module; The prediction frame prediction module predicts the k-1 frame tracking frame based on the historical motion information of the aerial workers, obtains the prediction frame of the k-th frame, and uploads the data to the matching tracking result module; The matching tracking result module matches the detection box with the prediction box based on the received data to complete the continuous multi-target tracking of power aerial operations.

Images