UAV hotspot tracking method for long-distance crude oil pipeline inspection

Abstract

The present invention provides a hotspot tracking method of a drone for inspection of a long-distance crude oil pipeline. The hotspot tracking method of a drone for inspection of a long-distance crude oil pipeline comprises: step 1, collecting video images taken in real time by a camera gimbal carried by a drone as input data; step 2, describing the characteristics of the target object according to its motion state and brightness change; step 3, identifying the target object using an improved optical flow method; step 4, extracting the distribution probability of the hue in the search window; step 5, continuously updating the search window by iterating the coordinate information to complete the tracking of a given object. The hotspot tracking method of a drone for inspection of a long-distance crude oil pipeline aims at the inspection characteristics of the long-distance crude oil pipeline, solves the problems of background interference and target occlusion in the working environment, and thus greatly increases the difficulty of tracking.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle inspection, and in particular to a method for tracking hot spots of unmanned aerial vehicles used for inspection of long-distance crude oil pipelines. Background Art

[0002] With the rapid development of drone-related technologies in recent years, object recognition and tracking technologies combined with drones have been widely used in the fields of observation and reconnaissance, video surveillance, industrial inspection, etc., especially in the inspection of long-distance crude oil pipelines. Using drones as a platform to complete object tracking has the characteristics of low cost and strong timeliness of information collection, which has obvious advantages over other methods.

[0003] Many countries are currently conducting extensive research on object tracking technology.

[0004] At present, under the conditions of low-speed target tracking, simple motion trajectory and single background image, the target tracking algorithm and technology based on the UAV platform are relatively complete. However, in the actual working environment, it is very common for the target to be interfered by the background and blocked by the target, which greatly increases the difficulty of tracking. In order to accurately obtain various static and dynamic data in the detection and tracking algorithm of moving objects, the current mobile object detection and following algorithm generally has a large amount of calculation, so it is not easy to meet the real-time requirements of the system; on the other hand, for those simpler algorithms, although they meet the real-time requirements of the system, their low accuracy will have an unignorable impact on the processing and analysis of moving target parameters. For this reason, we have invented a new UAV hotspot tracking method for crude oil long-distance pipeline inspection, which solves the above technical problems. Summary of the invention

[0005] The purpose of the present invention is to provide a method for improving the identification and tracking of targets based on the images transmitted by the drone camera gimbal, so as to achieve the purpose of continuously tracking the target object, and to provide a drone hotspot tracking method for crude oil long-distance pipeline inspection.

[0006] The purpose of the present invention can be achieved through the following technical measures: a drone hotspot tracking method for crude oil long-distance pipeline inspection, the drone hotspot tracking method for crude oil long-distance pipeline inspection comprising: step 1, collecting video images taken in real time by a camera gimbal carried by the drone as input data; step 2, describing the characteristics of the target object according to its motion state and brightness change; step 3, identifying the target object using an improved optical flow method; step 4, extracting the distribution probability of hue in the search window; step 5, continuously updating the search window by iterating the coordinate information to complete the tracking of a given object.

[0007] The purpose of the present invention can also be achieved by the following technical measures:

[0008] In step 1, the three components of the pixel R(x,y), G(x,y), and B(x,y) are collected from the RGB information of the input data, which are processed in the color space and converted into HSV mode, namely H'(x,y), V'(x,y); H(x,y) and V(x,y) are obtained through Gaussian filtering.

[0009] In step 2, F i 、F i+1 、F i+2 The three consecutive frames are subjected to brightness extraction and differential processing to obtain the target area J(x, y) containing the object.

[0010] In step 2, build a pyramid model J based on J(x,y) L After some further transformations and iterations, the optical flow value is obtained, and the moving target window W(x, y, t) is finally calculated by the inter-frame difference method.

[0011] In step 4, the first step is to determine whether this operation is the first execution. If so, continue to execute according to the normal steps; if it is a re-search after the object disappears in the search window, the probability distribution P(n) of the hue in the moving target window W(x, y, t) is re-solved, and the obtained result is compared with the hue probability distribution information P(n) of the specified object stored previously. 0 (n) Make a comparison. If the comparison result meets the set threshold, the next step is to perform the second step calculation of step 4. If it does not meet the threshold, it means that the object obtained by this calculation is not the originally specified object. In this case, return to step 1 and start tracking again.

[0012] In step 4, the second step is to calculate the joint probability distribution between the LBP texture and chromaticity of the image in the object tracking window Further, the position D of the window center is solved by the obtained probability distribution value t and the area of ​​the window S t .

[0013] In step 4, the third step is to compare the current window area S t and the window area S of the previous step t-1 The obtained results are matched with the motion test window according to several different situations, and the set of differential operation vectors V = {v 1 ,v 2 ,...,v s}.

[0014] In step 4, the fourth step, the motion vector estimate v of the tracking window cThe improved tracking window centroid position D can be further obtained from the modulus probability distribution value of the motion vector t '=D t-1 +v c , where D t : The center of mass position of the current tracking window; D t-1 : The centroid position of the tracking window at the last moment; v c : velocity vector.

[0015] In step 4, the fifth step is to compare the hue values ​​of all pixels in the original current frame of the improved tracking window and the tracking window before the improvement with the hue values ​​of the corresponding pixels in the previous frame tracking window. and In the formula, ∑: cumulative sum; H st (x, y): Hue value of the pixel in the current frame; H st-1 (x, y): the hue value of the pixel in the previous frame; then sum each difference in the window and compare the sizes, and finally select the minimum value corresponding to the centroid position D t and window area S t To refresh.

[0016] In step 5, if it is determined that it meets the convergence requirements after the update, it means that the tracking task for the current frame has been completed. If the end tracking command is received next, the work is terminated. If the tracking is to be continued, the Kalman filter operation is added to predict the position of the object and determine the image processing area T = S + δ s Whether it has crossed the edge of the current processing window, where T is the image processing area, S is the window area, and δ is the position change vector. If it has not crossed the edge, the next frame of the image is used as input and after completing the color space conversion and filtering processing, it jumps to the second step of step 4 and re-executes the entire tracking process. If it has crossed the edge, the posture position of the drone or camera gimbal is adjusted in time according to the position solution result and the data provided by the control model to avoid losing the target object.

[0017] In step 5, if the update does not meet the convergence requirements, further compare whether it exceeds the set window update operation number limit. If it does not exceed the number limit, the centroid position D is adjusted according to the minimum value obtained this time. t and window area S tGo directly to the third step of step 4 to set the tracking window position again; when the limit is exceeded, it means that the tracking object has been lost. At this time, jump to step 5 to use Kalman filtering to predict the position of the object. If it has not exceeded the frame of the picture, it means that the possibility of the object being within the range of the picture is still relatively high. In this case, execute step 1 to re-identify the object; if it has reached the boundary, it means that the object has been completely lost and the mission cannot continue, so an error alarm is issued. At the same time, the drone tries to re-identify the target object according to the originally predicted position.

[0018] The hotspot tracking method of unmanned aerial vehicle used for inspection of crude oil long-distance pipelines in the present invention is based on the collected original video images, describes its features according to the object's motion state and brightness change, and then uses the improved optical flow method to identify the object. The distribution probability of the hue in the search window is extracted, and the search window is continuously updated by iteration of the coordinate information to complete the tracking of a given object. On this basis, the prediction of the object position and the adjustment optimization algorithm of the search window position are added to improve the robustness of the object tracking method. The local binary pattern LBP (Local Binary Patterns) texture and chromaticity data of the picture are combined to analyze the texture information of the local picture, thereby improving the tracking accuracy of objects in motion. The method is based on the image transmitted back by the drone camera gimbal, and improves the method of identifying and tracking the target to achieve the purpose of continuously tracking the target object. Corresponding optimization is performed for the three more common interference items: interference with complex environment images, occlusion during movement, and light changes. The present invention has the following innovative features:

[0019] 1. The traditional target recognition and tracking method has been improved to make it more adaptable to various disturbances such as complex background interference.

[0020] 2. The position of the moving target can be predicted, making tracking more accurate and flexible.

[0021] 3. The search can be quickly restarted after the target is lost. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The present invention is a flowchart of a specific embodiment of the UAV hotspot tracking method for crude oil long-distance pipeline inspection. DETAILED DESCRIPTION

[0023] In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings.

[0024] like Figure 1 As shown, Figure 1This is a flow chart of the hotspot tracking method of a drone for inspection of long-distance crude oil pipelines of the present invention. The hotspot tracking method of a drone for inspection of long-distance crude oil pipelines uses the picture taken in real time by the camera gimbal carried by the drone as input. After setting the object to be tracked, it is processed through several steps in the method, and the system outputs the precise coordinate points of the given tracking object in real time in the video image. What has been completed so far is the "recognition" part, and the next step is to complete the "tracking" process. In the tracking window containing the given object, a range is delineated with the object as the center according to an appropriate threshold. As the object moves, corresponding adjustments are made in time before the range is about to leave the tracking window, so that it is always in the center position, thereby completing the tracking of the moving object by the drone. The above two steps together form a complete identification and tracking method. Specifically comprising the following steps:

[0025] 1. The three components of the pixel R(x,y), G(x,y), and B(x,y) are collected from the RGB information of the input image, and processed in the color space and converted into the HSV mode, namely H'(x,y), V'(x,y). Then H(x,y) and V(x,y) are further identified through Gaussian filtering.

[0026] 2.F i 、F i+1 、F i+2 For three consecutive frames, perform brightness extraction and differential processing to obtain the target area J(x,y) containing the object. Build a pyramid model J based on J(x,y) L After some further transformations and iterations, the optical flow value is obtained, and the moving target window W(x, y, t) is finally calculated through subsequent calculations.

[0027] 3. Determine whether this is the first execution. If so, continue with the normal steps. If it is a re-search after the object disappears in the search window, re-solve the probability distribution P(n) of the hue in the moving target window W(x, y, t), and compare the result with the previously stored hue probability distribution information P(n) of the specified object. 0 (n) Make a comparison. If the comparison result meets the set threshold, the next step is to proceed to step (4). If it does not meet the threshold, it means that the object obtained by this operation is not the originally specified object. In this case, return to step (1) and start tracking again.

[0028] 4. Calculate the joint probability distribution between LBP texture and chromaticity of the image in the object tracking window Further, the position D of the window center is solved by the obtained probability distribution value t and the area of ​​the window S t .

[0029] 5. Compare the current window area S t and the window area S of the previous step t-1 The obtained results are matched with the motion test window according to several different situations, and the set of differential operation vectors V = {v 1 ,v 2 ,...,v s}.

[0030] 6. Tracking window motion vector estimate v c The improved tracking window centroid position D' can be obtained by the probability distribution value of the motion vector modulus. t =D t-1 +v c .

[0031] 7. The hue values ​​of all pixels in the original current frame of the improved tracking window and the tracking window before the improvement are respectively subtracted from the hue values ​​of the corresponding pixels in the previous frame tracking window, that is, the comparison and Then, we sum each difference in the window and compare the values, and finally select the minimum value corresponding to the centroid position D. t and window area S t To refresh.

[0032] 8. After refreshing, if it is judged to meet the convergence requirements, it means that the tracking task for the current frame has been completed. If the end tracking command is received next, the work is terminated. If the tracking is to be continued, the Kalman filter operation is added to predict the position of the object and determine the image processing area T = S + δ s Check whether the image has crossed the edge of the current processing window. If not, the next frame image is used as input and after completing the color space conversion and filtering processing, the process jumps to step (5) and the entire tracking process is re-executed. If the image has crossed the edge, the position of the drone or camera gimbal is adjusted in time according to the position solution result and the data provided by the control model to avoid losing the target object.

[0033] If the refresh judgment does not meet the convergence requirements, further compare whether it exceeds the set window update operation limit. If it does not exceed the limit, then according to the D obtained this time t With S tGo directly to step (6) to set the tracking window position again; if the limit is exceeded, it means that the tracking object has been lost. At this time, jump to step (8) to use Kalman filtering to predict the location of the object. If it has not exceeded the frame of the picture, it means that the object is likely to be within the range of the picture. In this case, execute step (1) to re-identify the object; if it has exceeded the limit, it means that the object has been completely lost and the mission cannot be continued, so an error alarm is issued. At the same time, the drone tries to re-identify the target object according to the originally predicted position.

[0034] The hotspot tracking method of drones used for inspection of long-distance crude oil pipelines in the present invention adopts an adaptive improvement algorithm to reduce the adverse effects of the background in the static window, and can effectively improve the accuracy of object tracking. It has good adaptability to several common complex interferences such as background image interference, objects temporarily blocked by obstacles, and changes in illumination, and can basically achieve full recognition and tracking of specific objects. In view of the inspection characteristics of long-distance crude oil pipelines, it solves the problems of background interference and target occlusion in the working environment, thereby greatly increasing the difficulty of tracking.

[0035] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions recorded in the aforementioned embodiments or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0036] Except for the technical features described in the specification, all other technical features are known technologies to those skilled in the art.

Claims

1. UAV hotspot tracking method for long-distance crude oil pipeline inspection, It is characterized in that include: Step 1: collect video images taken in real time by the camera gimbal on the drone as input data; Step 2: According to the target object's motion state and brightness change, Describe its characteristics; Step 3, using the improved optical flow method to identify the target object; Step 4, extracting the distribution probability of the hue in the search window; Step 5, continuously updating the search window by iterating the coordinate information to complete the tracking of the target object; Step 4 specifically includes: the first step of determining whether this operation is the first execution, and if so, continuing the execution according to normal steps; If the object is searched again after it disappears in the search window, the probability distribution P(n) of the hue in the moving target window W(x, y, t) is solved again, and the result is compared with the hue probability distribution information P(n) of the specified object stored previously. 0 (n) Perform a comparison. If the comparison result meets the set threshold, the second step of calculation in step 4 is performed. If the threshold is not met, it means that the object obtained by this calculation is not the originally specified object, and the tracking is restarted in step 1; The second step is to calculate the joint probability distribution between the LBP texture and chromaticity of the image in the object tracking window. The window centroid position D is solved by the obtained probability distribution value t and window area S t ; The third step is to compare the current window area S t and the window area S of the previous step t-1 The obtained results are matched with the motion test window according to several different situations, and the set of differential operation vectors V = {v 1 ,v 2 ,...,v s }; Step 4: Track the estimated motion vector v of the window c The improved tracking window centroid position D is obtained by the modulus probability distribution value of the motion vector t '=D t-1 +v c , where D t-1 : The centroid position of the tracking window at the last moment; v c : velocity vector; The fifth step is to compare the hue values ​​of all pixels in the original current frame of the improved tracking window and the tracking window before improvement with the hue values ​​of the corresponding pixels in the previous frame tracking window. and In the formula, ∑: cumulative sum; The hue value of the pixel in the current frame; The hue value of the pixel in the previous frame; sum each difference in the window and then compare the sizes, and finally select the minimum value of the corresponding situation for the centroid position D t and window area S t To refresh.

2. The drone hotspot tracking method for crude oil long-distance pipeline inspection according to claim 1, It is characterized in that In step 1, the three components of the pixel R(x,y), G(x,y), and B(x,y) are collected from the RGB information of the input data, which are processed in the color space and converted into HSV mode, namely H'(x,y), V'(x,y); H(x,y) and V(x,y) are obtained through Gaussian filtering.

3. The drone hotspot tracking method for crude oil long-distance pipeline inspection according to claim 1, It is characterized in that In step 2, F i 、F i+1 、F i+2 The three consecutive frames are subjected to brightness extraction and differential processing to obtain the target area J(x, y) containing the object.

4. The drone hotspot tracking method for crude oil long-distance pipeline inspection according to claim 3, It is characterized in that In step 2, build a pyramid model J based on J(x,y) L , after transformation and iteration, the optical flow value is obtained, and the moving target window W(x, y, t) is finally calculated by the inter-frame difference method.

5. The drone hotspot tracking method for crude oil long-distance pipeline inspection according to claim 1, It is characterized in that In step 5, if it is determined that the convergence requirements are met after the update, it means that the tracking task for the current frame has been completed. If the end tracking command is received, the work is terminated. If the tracking is to be continued, the Kalman filter operation is added to predict the position of the object and determine the image processing area T = S t +δ s Whether it has crossed the edge of the current processing window, where T is the image processing area; δ s : Position change vector; if it is not crossed, the next frame image is taken as input and after completing the color space conversion and filtering processing, jump to the second step of step 4 and re-execute the entire tracking process; If the edge is crossed, the posture position of the drone or camera gimbal will be adjusted in time according to the position solution results and the data provided by the control model to avoid losing the target object.

6. The drone hotspot tracking method for crude oil long-distance pipeline inspection according to claim 1, It is characterized in that If the update does not meet the convergence requirements, compare whether it exceeds the set window update operation number limit. If it does not exceed the number limit, the centroid position D is adjusted according to the minimum value obtained this time. t and window area S t Go directly to the third step of step 4 to set the tracking window position again; when the number limit is exceeded, it means that the tracking object has been lost, jump to step 5 to use Kalman filtering to predict the location of the object. If it does not cross the border of the picture, it means that the object is likely to be within the range of the picture, and execute step 1 to re-identify the object; if it crosses the border of the picture, it means that the object is completely lost, the mission cannot continue, and an error alarm is issued. At the same time, the drone tries to re-identify the target object according to the originally predicted position.

Images