Safe obstacle avoidance method and system for unmanned aerial vehicle road inspection

Abstract

The invention relates to the technical field of image data processing, in particular to a safe obstacle avoidance method and system for unmanned aerial vehicle road inspection, and the method comprises the steps: obtaining feature angular points in a video image at the current moment, determining the final static possibility of each feature angular point, and screening out dynamic feature angular points, and then determining an obstacle avoidance threat factor of each dynamic feature angular point to perform enhancement operation on the video image at the current moment, thereby segmenting an obstacle region in the enhanced video image at each moment, and planning an optimal path for avoiding an obstacle for the unmanned aerial vehicle. According to the invention, through adaptive enhancement of the video image, the accuracy of obstacle identification is guaranteed, and the safety of unmanned aerial vehicle road inspection obstacle avoidance is further guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a safety obstacle avoidance method and system for UAV road inspection. Background Art

[0002] The safety obstacle avoidance method and system for UAV road inspection are key technologies to ensure that the UAV can effectively avoid obstacles and guarantee flight safety when performing road inspection tasks. During the inspection process, the UAV obtains real-time image information on the traveling path by carrying a high-definition camera or other image acquisition devices. Through image processing technologies, such as deep learning algorithms, obstacles in the image are detected and classified to determine the position and type of the obstacles. Using path planning algorithms (such as the rapidly-exploring random tree algorithm), an optimal or feasible path for the UAV to avoid obstacles is planned to avoid collisions.

[0003] Existing problems: Due to the vibration generated by the UAV during flight and the influence of wind, the collected video images may be blurred or jittery, and under complex external light conditions, details in the shadow and highlight areas of the video images may be lost. This may lead to inaccurate recognition of obstacles in the video images, thereby reducing the safety of obstacle avoidance during UAV road inspection. Summary of the Invention

[0004] The present invention provides a safety obstacle avoidance method and system for UAV road inspection to solve the existing problems.

[0005] The safety obstacle avoidance method and system for UAV road inspection of the present invention adopt the following technical solutions: An embodiment of the present invention provides a safety obstacle avoidance method for UAV road inspection, the method comprising the following steps: During the UAV road inspection process, obtain the flight speed, flight direction, shooting direction of the camera on the UAV, and video image at each moment; Obtain a plurality of feature corner points in the video image at the current moment; within a period when the flight speed of the UAV and the shooting direction of the camera on the UAV are simultaneously similar, determine the motion direction invariance of each feature corner point according to the reverse motion trajectory of each feature corner point in the video image, and then combine the change characteristics of the flight speed of the UAV to determine the final static possibility of each feature corner point; Filter out dynamic feature corner points according to the magnitude of the final static possibility of each feature corner point; determine the obstacle avoidance threat factor of each dynamic feature corner point according to the reverse motion trajectory of each dynamic feature corner point in the video image; perform an enhancement operation on the video image at the current moment according to the obstacle avoidance threat factor to obtain the enhanced video image at the current moment; Use a segmentation neural network to segment the obstacle regions in the enhanced video images at each moment; based on the obstacle regions in the enhanced video images at all moments, plan an optimal path for the drone to avoid obstacles.

[0006] Further, the determination of the motion direction invariance of each feature corner point includes the following specific steps: Determine the direction consistency between any two moments based on the difference in the flight direction of the drone and the difference in the shooting direction of the camera on the drone between the two moments; Denote the current moment as the t-th moment, and sequentially obtain the direction consistency between the t-th moment and the (t - 1)-th moment, the direction consistency between the t-th moment and the (t - 2)-th moment, until the direction consistency between the t-th moment and the (t - n)-th moment is greater than the preset angle threshold for the first time. Denote the time period formed from the (t - n - 1)-th moment to the t-th moment as the target time period; Within the target time period, starting from each feature corner point in the video image of the current moment, use the reverse optical flow method to obtain the matching points in the video images of other moments in reverse chronological order, one moment at a time, until the matching fails, and obtain the reverse sequence of the motion trajectory points of each feature corner point; In the reverse sequence of the motion trajectory points of the x-th feature corner point, use the Euclidean distance and direction from the position coordinates of the second element to the position coordinates of the first element to form the motion vector of the first element, calculate the cosine similarity of the motion vectors of any two elements, and take the average value of the cosine similarities of the motion vectors of all any two elements as the motion direction invariance of the x-th feature corner point.

[0007] Further, the determination of the direction consistency between any two moments includes the following specific steps: For any two moments, obtain the minimum angle between the flight directions of the drone, and then obtain the minimum angle between the shooting directions of the cameras on the drone. Take the average value of the minimum angle between the flight directions and the minimum angle between the shooting directions as the direction consistency between any two moments.

[0008] Further, the determination of the final static possibility of each feature corner point includes the following specific steps: In the reverse sequence of the motion trajectory points of the x-th feature corner point, sequentially obtain the flight speeds of the drone at the corresponding moments of all elements to form a reverse flight speed sequence, and then sequentially obtain the corresponding moments of all elements to form a reverse time sequence; Sequentially obtain the magnitudes of the motion vectors of all elements in the reverse sequence of the motion trajectory points of the x-th feature corner point to form a magnitude sequence, calculate the Pearson correlation coefficient between the magnitude sequence and the reverse flight speed sequence, and denote it as the motion speed consistency of the x-th feature corner point. Denote the product of the motion speed consistency and the motion direction invariance of the x-th characteristic corner point as the initial static possibility; In the reverse time sequence, obtain the inverse proportional value of the mean value of the direction consistencies between all arbitrary two moments, and denote it as the UAV flight invariance; Denote the normalized value of the product of the UAV flight invariance of the x-th characteristic corner point and the initial static possibility as the final static possibility.

[0009] Furthermore, the specific steps for screening out the dynamic characteristic corner points are as follows: Denote the characteristic corner points with the final static possibility less than or equal to the preset static-dynamic threshold as the dynamic characteristic corner points.

[0010] Furthermore, the specific steps for determining the obstacle avoidance threat factor of each dynamic characteristic corner point are as follows: In the video image at the current moment, use the included angle value of the motion vectors of any two dynamic characteristic corner points as the clustering distance, and use the K-means clustering algorithm to perform clustering operations on all dynamic characteristic corner points to obtain several clustering clusters; Calculate the ratio of the number of dynamic characteristic corner points in the b-th clustering cluster to the number of dynamic characteristic corner points in the video image at the current moment, then calculate the sum vector of the motion vectors of all dynamic characteristic corner points in the b-th clustering cluster, and obtain the inverse proportional value of the included angle value between the sum vector and the motion vector of the y-th dynamic characteristic corner point in the b-th clustering cluster. Denote the product of the ratio of the number of dynamic characteristic corner points and the inverse proportional value as the simultaneous motion unity of the y-th dynamic characteristic corner point in the b-th clustering cluster; Determine the motion trajectory variability of each dynamic characteristic corner point according to the included angle value of the motion vectors of adjacent elements in the reverse sequence of the motion trajectory points of each dynamic characteristic corner point; Obtain the normalized value of the ratio of the motion trajectory variability and the motion unity of the z-th dynamic characteristic corner point, and denote the sum value of the normalized value and the preset constant as the obstacle avoidance threat factor of the z-th dynamic characteristic corner point.

[0011] Furthermore, the specific steps for determining the motion trajectory variability of each dynamic characteristic corner point according to the included angle value of the motion vectors of adjacent elements in the reverse sequence of the motion trajectory points of each dynamic characteristic corner point are as follows: In the reverse sequence of the motion trajectory points of the z-th dynamic characteristic corner point, sequentially obtain the included angle values of the motion vectors of adjacent two elements to form an included angle value sequence. In the included angle value sequence, denote the included angle values greater than the preset included angle threshold as 1, and denote the included angle values less than or equal to the preset included angle threshold as 0 to form a 01 sequence; The ratio of the number of 1s in the 01 sequence to the length of the 01 sequence is denoted as the first ratio, the ratio of the maximum length of consecutive 1s in the 01 sequence to the length of the 01 sequence is denoted as the second ratio, and the mean value of all the included angle values in the included angle value sequence divided by the product of the first ratio and the second ratio is denoted as the motion trajectory variability of the z-th dynamic feature corner point.

[0012] Further, the obtaining of the enhanced video image at the current moment includes the following specific steps: In the video image at the current moment, taking each feature corner point as the initial clustering center, using the Euclidean distance between the position coordinates of pixel points as the clustering distance, and using the K-means clustering algorithm to perform only one clustering assignment on all pixel points to obtain a connected region corresponding to each feature corner point; The feature corner points with the final static possibility greater than the preset static-dynamic threshold are denoted as static feature corner points; Set the obstacle avoidance threat factor of each static feature corner point to a preset constant; Taking the obstacle avoidance threat factor of each feature corner point as the obstacle avoidance threat of the connected region corresponding to each feature corner point; the feature corner points are divided into dynamic feature corner points and static feature corner points; Determine the contrast limiting threshold of each connected region according to the obstacle avoidance threat of each connected region and the gray level feature of the pixel points in the connected region; Perform an enhancement operation on the video image at the current moment using the contrast-limited histogram equalization algorithm according to the contrast limiting threshold of each connected region to obtain the enhanced video image at the current moment.

[0013] Further, the determining of the contrast limiting threshold of each connected region according to the obstacle avoidance threat of each connected region and the gray level feature of the pixel points in the connected region includes the following specific steps: Calculate the mean value of the gray level gradient values of all pixel points in the w-th connected region as the clarity; Denote the normalized value of the ratio of the obstacle avoidance threat of the w-th connected region to the clarity as the limiting threshold coefficient of the w-th connected region; Obtain the gray level histogram of the w-th connected region, and denote the mean value of the number of pixel points at all gray levels in the gray level histogram as the standard limiting threshold; Denote the product of the standard limiting threshold of the w-th connected region and the limiting threshold coefficient as the contrast limiting threshold of the w-th connected region.

[0014] The present invention also provides a safety obstacle avoidance system for UAV road inspection, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the computer program stored in the memory to implement the steps of the aforementioned safety obstacle avoidance method for UAV road inspection.

[0015] The beneficial effects of the technical solution of the present invention are as follows: In the embodiment of the present invention, the feature corner points in the video image at the current moment are obtained, and the final static possibility of each feature corner point is determined to screen out the dynamic feature corner points, thereby distinguishing the dynamic feature corner points in the video image to ensure better enhancement of the area where the important dynamic feature corner points are located. The obstacle avoidance threat factor of each dynamic feature corner point is determined, and thus, by analyzing the obstacle avoidance threat caused by the unpredictability of the dynamic feature corner points, the enhancement effect of the area where the dynamic feature corner points are located is adapted. An enhancement operation is performed on the video image at the current moment, that is, adaptive enhancement is performed according to the obstacle avoidance threat and clarity of the connected region where each feature corner point is located to obtain an enhanced video image, so as to segment the obstacle regions in the enhanced video image at each moment and plan an optimal path for the UAV to avoid obstacles. Thus, through the adaptive enhancement of the video image, the present invention ensures the accuracy of obstacle recognition and further ensures the safety of UAV road inspection obstacle avoidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for describing the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0017] Figure 1 It is a flowchart of the steps of a safety obstacle avoidance method for UAV road inspection according to the present invention; Figure 2 It is a schematic diagram of UAV road inspection. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following will, in conjunction with the accompanying drawings and preferred embodiments, describe in detail the specific implementation manners, structures, features, and effects of a safety obstacle avoidance method and system for UAV road inspection according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this invention belongs.

[0020] The following specifically describes the specific solutions of a safety obstacle avoidance method and system for UAV road inspection provided by the present invention in conjunction with the accompanying drawings.

[0021] Please refer to Figure 1 , which shows a flowchart of the steps of a safety obstacle avoidance method for UAV road inspection provided by an embodiment of the present invention. The method includes the following steps: Step S001: During the UAV road inspection process, obtain the flight speed, flight direction, shooting direction of the camera on the UAV, and video image at each moment.

[0022] During the UAV road inspection process, collect the flight speed, flight direction (direction in the three-dimensional space of altitude, longitude, and latitude), shooting direction of the camera on the UAV (direction in the three-dimensional space of altitude, longitude, and latitude), and video image at each moment.

[0023] It should be noted that: In this embodiment, the collection frequency is 30 Hertz (Hz), and this is used as an example for description. Among them, through sensors carried on the UAV, such as gyroscopes, accelerometers, and magnetometers, the attitude information of the UAV can be provided. This information is used to determine the current flight direction of the UAV and the pointing direction of the camera. Then, through the Global Positioning System (GPS), the geographical location information of the UAV is provided, and the flight speed of the UAV can be calculated through continuous position data points. The camera on the UAV is responsible for capturing real-time video images. A schematic diagram of UAV road inspection is shown in Figure 2 as shown.

[0024] Step S002: Obtain several feature corner points in the video image at the current moment; during the time period when the flight speed of the UAV and the shooting direction of the camera on the UAV are simultaneously similar, determine the motion direction invariance of each feature corner point according to the reverse motion trajectory of each feature corner point in the video image, and then combine the change characteristics of the flight speed of the UAV to determine the final static possibility of each feature corner point.

[0025] It should be noted that: During the safe obstacle avoidance process of drone road inspection, significantly different degrees of attention need to be given to the dynamic and static regions in the video images. Dynamic obstacles (such as vehicles, pedestrians, other aircraft, and birds, etc.) have unpredictable motion characteristics and require higher real-time performance and response speed. While the positions of static obstacles (such as buildings and poles) are fixed, and obstacle avoidance can be processed through pre-planning. Therefore, when the quality of a certain video image collected is poor, for static obstacles, information in the video images with better quality at other times can be used for analysis. For dynamic obstacles, the detailed information in the video images needs to be improved as much as possible to ensure the timeliness and safety of obstacle avoidance. Since when the flight direction of the drone and the shooting direction of the camera on the drone remain unchanged, in the continuously captured video frames, the relative motion vectors of static objects in consecutive frames are usually consistent. This is because the motion of static objects relative to the drone is mainly caused by the motion of the drone itself. Therefore, first obtain consecutive frames with unchanged flight direction and shooting direction.

[0026] For any two moments, obtain the minimum angle between the flight directions of the drone, and then obtain the minimum angle between the shooting directions of the cameras on the drone. Use the average value of the minimum angle between the flight directions and the minimum angle between the shooting directions as the direction consistency between these two moments.

[0027] Denote the current moment as the t-th moment. Sequentially obtain the direction consistency between the t-th moment and the (t - 1)-th moment, the direction consistency between the t-th moment and the (t - 2)-th moment, until the first time the direction consistency between the t-th moment and the (t - n)-th moment is greater than the preset angle threshold. Denote the time period formed from the (t - n - 1)-th moment to the t-th moment as the target time period.

[0028] It should be noted that: In this embodiment, the preset angle threshold is 3 degrees, and this is used as an example for description. And by appropriately increasing the preset angle threshold, the duration of the target time period can be ensured to meet subsequent analysis. For the video images within the first 3 minutes after the drone starts flying, the histogram equalization algorithm is directly used for enhancement processing. Among them, the histogram equalization algorithm is a well-known technology, and the specific method is not introduced here.

[0029] Use the FAST corner detection algorithm to obtain several feature corners in the video image at the current moment.

[0030] Within the target time period, starting from each feature corner point in the video image at the current moment, using the reverse optical flow method, in reverse chronological order, obtain the matching points in the video images at other moments one by one until the matching fails, and obtain the reverse sequence of the motion trajectory points of each feature corner point. For example: {the m-th feature corner point in the video image at the t-th moment, the matching point of the m-th feature corner point in the video image at the t-1-th moment, the matching point of the m-th feature corner point in the video image at the t-2-th moment, the matching point of the m-th feature corner point in the video image at the t-3-th moment}.

[0031] It should be noted that: The FAST (Features from Accelerated Segment Test) corner detection algorithm and the reverse optical flow method are both well-known technologies, and the specific methods will not be introduced here. Among them, starting from each feature corner point in the video image at the current moment, in reverse chronological order, perform matching frame by frame in the target time period until the matching fails, that is, the feature corner point goes out of the picture, and obtain the reverse motion trajectory of each feature corner point. Therefore, the duration of the reverse motion trajectory may not be equal to the target time period. Since the feature corner point refers to the point with obvious changes in the local area of the image, usually corresponding to the edge of the object, the corner, or the position with significant texture changes, it is a key attention position during obstacle avoidance. In the video sequence, by tracking the motion trajectory of the feature corner point, the motion parameters of the object or the motion trajectory of the camera can be estimated. If the number of matching points of the feature corner point is less than 3, it means that this feature corner point just appears in the picture, and this feature corner point can be not analyzed, that is, delete this feature corner point.

[0032] In the reverse sequence of the motion trajectory points of the x-th feature corner point in the video image at the current moment, the Euclidean distance and direction from the position coordinates of the second element (matching point) to the position coordinates of the first element (feature corner point) form the motion vector of the first element.

[0033] It should be noted that: In this embodiment, taking the vertex at the lower left corner of the video image at each moment as the origin, with the horizontal right direction as the positive x-axis, and the vertical upward direction as the positive y-axis, a plane rectangular coordinate system is constructed. In the plane rectangular coordinate system, the position coordinates of each pixel point in the video image are obtained. And, let the motion vector of the last element be the motion vector of the penultimate element. Above, the motion vector of the first element is the motion vector of the x-th feature corner point in the video image at the current moment.

[0034] In the reverse sequence of the motion trajectory points of the x-th feature corner point in the video image at the current moment, sequentially obtain the flight speeds of the drone at the corresponding moments of all elements, and form a reverse flight speed sequence. Then sequentially obtain the corresponding moments of all elements, and form a reverse time sequence.

[0035] In the reverse sequence of the motion trajectory points of the x-th feature corner point in the video image at the current moment, calculate the cosine similarity of the motion vectors of any two elements, and take the mean of the cosine similarities of the motion vectors of all any two elements as the motion direction invariance of the x-th feature corner point.

[0036] Among them, the calculation of the cosine similarity is a well-known technology, and the specific method will not be introduced here. The cosine similarity only measures the directional relationship between vectors and does not consider the magnitude of the vectors. The greater the cosine similarity, the more consistent the vector directions.

[0037] It should be noted that: when the flight direction of the UAV and the shooting direction of the camera on the UAV remain unchanged, the motion vector of a static object should remain unchanged, while the motion vector direction of a dynamic object is jointly determined by the flight direction of the UAV and its own motion direction, that is, when its own motion direction changes, it will cause the motion vector direction to change. Therefore, the greater the motion direction invariance, the more likely the feature corner point is a static point. However, during the target time period, when the motion direction of the dynamic object also remains unchanged, the motion direction invariance is also relatively large. Therefore, further analysis is needed. Since the magnitude of the motion vector of a static object is only affected by the flight speed of the UAV, while the magnitude of the motion vector of a dynamic object is jointly affected by the flight speed of the UAV and its own motion speed.

[0038] For the x-th feature corner point in the video image at the current moment, sequentially obtain the magnitudes of the motion vectors of all elements in the reverse sequence of the motion trajectory points to form a magnitude sequence, and calculate the Pearson correlation coefficient between the magnitude sequence and the reverse flight speed sequence, denoted as the motion speed consistency of the x-th feature corner point.

[0039] Among them, the Pearson correlation coefficient is a well-known technology, and the specific method will not be introduced here. The greater the Pearson correlation coefficient, the stronger the linear relationship between the two sequences, the more likely the feature corner point is a static point, and when calculating the Pearson correlation coefficient, there is no need to unify the dimension.

[0040] For the x-th feature corner point in the video image at the current moment, denote the product of the motion speed consistency and the motion direction invariance as the initial static possibility. That is, use the motion speed consistency as the adjustment value of the motion direction invariance.

[0041] For the x-th feature corner point in the video image at the current moment, in the reverse time sequence, obtain the inverse proportional value of the mean of the direction consistencies between all any two moments and denote it as the UAV flight invariance.

[0042] It should be noted that: the smaller the mean value of the direction consistency between any two moments in the reverse time series, the greater the flight invariance of the UAV, and the more credible the above analysis based on the unchanged flight direction of the UAV and the shooting direction of the camera on the UAV. In this embodiment, represents the inverse proportional value of the mean value of the above direction consistency, is a linear normalization function, which is used to normalize the data value to between 0 and 1.

[0043] For the x-th feature corner point in the video image of the current moment, the normalized value of the product of the flight invariance of the UAV and the initial static possibility is denoted as the final static possibility.

[0044] Wherein, represents the normalized value of the above product, which is normalized to between 0 and 1.

[0045] In the above manner, the final static possibility of each feature corner point in the video image of the current moment can be obtained.

[0046] Step S003: Screen out the dynamic feature corner points according to the size of the final static possibility of each feature corner point; determine the obstacle avoidance threat factor of each dynamic feature corner point according to the reverse motion trajectory of each dynamic feature corner point in the video image; perform an enhancement operation on the video image of the current moment according to the obstacle avoidance threat factor to obtain the enhanced video image of the current moment.

[0047] The preset static-dynamic threshold is 0.7, and this is used as an example for description.

[0048] In the video image of the current moment, the feature corner points with the final static possibility greater than the preset static-dynamic threshold are denoted as static feature corner points, and the feature corner points with the final static possibility less than or equal to the preset static-dynamic threshold are denoted as dynamic feature corner points.

[0049] It should be noted that: for the dynamic feature corner points in the video image of the current moment, it is further necessary to analyze the predictability of the motion trajectory of the dynamic feature corner points. The worse the predictability of the motion trajectory, the greater the impact on safe obstacle avoidance, and the more necessary it is to enhance the image quality of the area where the dynamic feature corner points are located. For example: the motion trajectories of vehicles and pedestrians that abide by traffic rules move along the road in a certain direction, which are traceable and have strong predictability, while the motion trajectories of vehicles and pedestrians that do not abide by traffic rules are untraceable and have weak predictability. Also, a normal flying flock of birds usually flies in a unified direction, while a panicked and flying flock of birds has a chaotic flight direction.

[0050] According to the motion vectors of each dynamic feature corner point in the video image at the current moment, using the included angle value between the motion vectors of any two dynamic feature corner points as the clustering distance, and applying the K-means clustering algorithm to perform clustering operations on all dynamic feature corner points to obtain several clustering clusters.

[0051] Among them, the K-means clustering algorithm is a well-known technology, and the specific method will not be introduced here.

[0052] Calculate the ratio of the number of dynamic feature corner points in the b-th clustering cluster to the number of dynamic feature corner points in the video image at the current moment, and then calculate the sum vector of the motion vectors of all dynamic feature corner points in the b-th clustering cluster, and obtain the included angle value between the sum vector and the motion vector of the y-th dynamic feature corner point in the b-th clustering cluster Take the inverse proportional value, and record the product of the ratio and the inverse proportional value as the simultaneous motion uniformity of the y-th dynamic feature corner point in the b-th clustering cluster.

[0053] It should be noted that: the maximum included angle value between two vectors is 180 degrees. Therefore, in this embodiment, is used as the included angle value of the inverse proportional value. Among them, the larger the ratio, the more similar the motion directions of a large number of dynamic feature corner points in the video image at the current moment, and the better the predictability of the movement of the dynamic feature corner points in this clustering cluster. The smaller the included angle value , the more consistent the motion direction of the y-th dynamic feature corner point in the clustering cluster with the overall motion direction (the direction of the sum vector) of the clustering cluster. Therefore, the ratio is used as the adjustment value of the inverse proportional value of the included angle value to obtain the simultaneous motion uniformity.

[0054] In the above manner, the simultaneous motion uniformity of each dynamic feature corner point in the video image at the current moment can be obtained.

[0055] The preset included angle threshold is 5 degrees, and this will be used as an example for description.

[0056] In the reverse sequence of the motion trajectory points of the z-th dynamic feature corner point in the video image at the current moment, sequentially obtain the included angle values between adjacent two elements' motion vectors to form an included angle value sequence. In the included angle value sequence, record the included angle values greater than the preset included angle threshold as 1, and record the included angle values less than or equal to the preset included angle threshold as 0 to form a 01 sequence.

[0057] Record the ratio of the number of 1s in the 01 sequence to the length of the 01 sequence as the first ratio, record the ratio of the maximum continuous length of 1s in the 01 sequence to the length of the 01 sequence as the second ratio, obtain the mean value of all included angle values in the included angle value sequence, and record the product of the mean value and the first ratio and the second ratio as the motion trajectory variability of the z-th dynamic feature corner point in the video image at the current moment.

[0058] It should be noted that the larger the mean of all angle values in the angle value sequence is, the greater the change in the direction of movement over time is. The number of 1s in the 01 sequence, that is, the number of large mutations in the direction of movement, and the maximum length of consecutive 1s in the 01 sequence (such as: the maximum length of consecutive 1s in 01110011110 is 4) reflect the continuity of large mutations in the direction of movement. The larger the two are, the more drastic the change in the direction of movement over time is.

[0059] The preset constant is 1, and this is used as an example for description.

[0060] The normalized value of the ratio of the motion trajectory variability to the motion uniformity of the zth dynamic feature corner point in the video image at the current moment is obtained, and the sum of the normalized value and a preset constant is recorded as the obstacle avoidance threat factor of the zth dynamic feature corner point in the video image at the current moment.

[0061] It should be noted that: In this embodiment, the The linear normalization function is used to normalize the ratio of the above motion trajectory variability to motion uniformity to between 0 and 1. The smaller the motion uniformity, the more unique its motion direction is, that is, the more difficult it is to predict the motion direction, the greater the obstacle avoidance threat; the greater the motion trajectory variability, the more variable the motion trajectory, the more difficult it is to predict, and the greater the obstacle avoidance threat.

[0062] According to the above method, the obstacle avoidance threat factor of each dynamic feature corner point in the video image at the current moment can be obtained.

[0063] The obstacle avoidance threat factor of each static feature corner point in the video image at the current moment is set to a preset constant, thereby obtaining the obstacle avoidance threat factor of each feature corner point.

[0064] In the video image at the current moment, each feature corner point is taken as the initial cluster center, and the Euclidean distance between the position coordinates of the pixel points is taken as the cluster distance. The K-means clustering algorithm is used to perform cluster assignment on all pixels only once, and each initial cluster center corresponds to a connected area, that is, each feature corner point corresponds to a connected area, and there is only one feature corner point in each connected area.

[0065] Thus, the video image at the current moment is divided into multiple connected areas, each of which corresponds to a feature corner point.

[0066] It should be noted that the K-means clustering algorithm is a well-known technology. The single-time clustering assignment is as follows: for each pixel in the video image at the current moment, calculate its Euclidean distance from all initial clustering centers, and assign the pixel to the nearest initial clustering center. In this way, each initial clustering center will correspond to a region, that is, the region composed of all pixels assigned to this center. And because the clustering process will gather pixel points that are close to each other in space, continuous regions will be formed in the image, that is, one initial clustering center corresponds to a connected region.

[0067] In the video image at the current moment, use the obstacle avoidance threat factor of each feature corner point as the obstacle avoidance threat of the connected region where each feature corner point is located.

[0068] Calculate the mean value of the gray-scale gradient values of all pixel points in the w-th connected region in the video image at the current moment as the clarity of the w-th connected region.

[0069] Denote the normalized value of the ratio of the obstacle avoidance threat to the clarity of the w-th connected region as the restricted threshold coefficient of the w-th connected region.

[0070] It should be noted that the greater the obstacle avoidance threat and the smaller the clarity, the more this connected region needs to improve the contrast to highlight the detailed information. In this embodiment, use a linear normalization function to normalize the ratio of the above-mentioned obstacle avoidance threat to the clarity to between 0.8 and 1.2.

[0071] Obtain the gray-scale histogram of the w-th connected region, and denote the mean value of the number of pixel points at all gray levels in the gray-scale histogram as the standard restricted threshold.

[0072] Denote the product of the standard restricted threshold of the w-th connected region and the restricted threshold coefficient as the contrast restricted threshold of the w-th connected region.

[0073] It should be noted that the video image at the current moment is grayscale processed, and the Canny edge detection algorithm is used to obtain the gray-scale gradient value of each pixel point in the video image at the current moment. Among them, the grayscale processing of the image, the acquisition of the gray-scale histogram, and the Canny edge detection algorithm are all well-known technologies, and the specific methods will not be introduced here.

[0074] In the video image at the current moment, according to the contrast restricted threshold of each connected region, use the contrast limited histogram equalization algorithm to perform enhancement operations on the video image at the current moment to obtain the enhanced video image at the current moment.

[0075] It should be noted that the contrast-limited histogram equalization algorithm is a well-known technology, and the specific method will not be introduced here. In the algorithm, the size of the contrast-limited threshold has a significant impact on the effect of image enhancement. A larger contrast-limited threshold can significantly improve the contrast of the image and make the image details more prominent. The enhancement operation of the contrast-limited histogram equalization algorithm on the video image at the current moment is as follows: First, divide the video image at the current moment into several connected regions, obtain the gray-level histogram of each connected region, use the contrast-limited threshold of each connected region to clip the gray-level histogram, perform histogram equalization processing on the clipped gray-level histogram of each connected region, and then use bilinear interpolation to eliminate the block effect and output the enhanced video image.

[0076] In the above manner, a real-time enhanced video image is obtained.

[0077] Step S004: Use a segmentation neural network to segment the obstacle regions in the enhanced video image at each moment; based on the obstacle regions in the enhanced video images at all moments, plan an optimal path for the drone to avoid obstacles.

[0078] In the embodiment of the present invention, a segmentation neural network is used to identify and segment the obstacle regions and background regions in the enhanced video image.

[0079] The relevant content of the segmentation neural network is as follows: The segmentation neural network used in this embodiment is the Mask R-CNN neural network; the dataset used is the enhanced video image dataset. Among them, Mask R-CNN is a well-known technology, and the specific method will not be introduced here. The Chinese full name of Mask R-CNN is "Mask Region-based Convolutional Neural Network", and the English full name is "Mask Region-based Convolutional Neural Network".

[0080] The pixel points to be segmented are divided into 2 categories. The label annotation process for the training set corresponding labels is as follows: For the single-channel semantic label, the pixel points at the corresponding positions belonging to the background region are labeled as 0, and those belonging to the obstacle region are labeled as 1.

[0081] The task of the network is classification, so the loss function used is the cross-entropy loss function.

[0082] The obstacle regions and background regions in the enhanced video image at each moment are obtained through the segmentation neural network. This process is a well-known technology, and the specific method will not be introduced here.

[0083] Based on the obstacle regions in the enhanced video images at all moments, use the RRT path planning algorithm to plan an optimal path for the drone to avoid obstacles.

[0084] Among them, the obstacle avoidance method based on the RRT (Rapidly-Exploring Random Tree) algorithm can dynamically generate a new flight path according to the position of obstacles, thereby avoiding collisions. This is a well-known technology, and the specific method will not be introduced here.

[0085] The present invention also provides a safety obstacle avoidance system for unmanned aerial vehicle road inspection, including a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the computer program stored in the memory to implement the steps of the aforementioned safety obstacle avoidance method for unmanned aerial vehicle road inspection.

[0086] Thus, the present invention is completed.

[0087] In summary, in the embodiments of the present invention, feature corner points in the video image at the current moment are obtained, the final static possibility of each feature corner point is determined to screen out dynamic feature corner points, and then the obstacle avoidance threat factor of each dynamic feature corner point is determined to perform an enhancement operation on the video image at the current moment, so as to segment the obstacle areas in the enhanced video image at each moment and plan an optimal path for the unmanned aerial vehicle to avoid obstacles. The present invention adaptively enhances the video image to ensure the accuracy of obstacle recognition, and further ensures the safety of unmanned aerial vehicle road inspection obstacle avoidance.

[0088] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A safety obstacle avoidance method for drone road inspection, characterized in that, The method includes the following steps: During the process of the drone's road inspection, obtain the flight speed, flight direction of the drone, the shooting direction of the camera on the drone, and the video image at each moment; Obtain several feature corner points in the video image at the current moment; within the time period when the flight speed of the drone and the shooting direction of the camera on the drone are simultaneously similar, determine the motion direction invariance of each feature corner point according to the reverse motion trajectory of each feature corner point in the video image, and then combine the change characteristics of the flight speed of the drone to determine the final static possibility of each feature corner point; Filter out the dynamic feature corner points according to the magnitude of the final static possibility of each feature corner point; determine the obstacle avoidance threat factor of each dynamic feature corner point according to the reverse motion trajectory of each dynamic feature corner point in the video image; perform an enhancement operation on the video image at the current moment according to the obstacle avoidance threat factor to obtain the enhanced video image at the current moment; Use a segmentation neural network to segment the obstacle areas in the enhanced video image at each moment; plan an optimal path for the drone to avoid obstacles according to the obstacle areas in the enhanced video images at all moments.

2. The safety obstacle avoidance method for drone road inspection according to claim 1, wherein The specific steps included in determining the motion direction invariance of each feature corner point are as follows: Determine the direction consistency between any two moments according to the difference in the flight direction of the drone and the difference in the shooting direction of the camera on the drone between any two moments; Denote the current moment as the t-th moment, and sequentially obtain the direction consistency between the t-th moment and the (t - 1)-th moment, the direction consistency between the t-th moment and the (t - 2)-th moment, until the direction consistency between the t-th moment and the (t - n)-th moment is greater than the preset angle threshold for the first time. Denote the time period formed from the (t - n - 1)-th moment to the t-th moment as the target time period; Within the target time period, starting from each feature corner point in the video image at the current moment, use the reverse optical flow method to obtain the matching points in the video images at other moments in reverse chronological order, until the matching fails, and obtain the reverse sequence of the motion trajectory points of each feature corner point; In the reverse sequence of the motion trajectory points of the x-th feature corner point, use the Euclidean distance and direction from the position coordinates of the second element to the position coordinates of the first element to form the motion vector of the first element, calculate the cosine similarity of the motion vectors of any two elements, and take the mean of the cosine similarities of the motion vectors of all any two elements as the motion direction invariance of the x-th feature corner point.

3. The safety obstacle avoidance method for UAV road inspection according to claim 2, characterized in that, The specific steps included in determining the direction consistency between any two moments are as follows: For any two moments, obtain the minimum included angle between the flight directions of the drone, and then obtain the minimum included angle between the shooting directions of the cameras on the drone. Take the mean of the minimum included angle between the flight directions and the minimum included angle between the shooting directions as the direction consistency between any two moments.

4. The safety obstacle avoidance method for drone road inspection according to claim 2, characterized in that, The specific steps included in determining the final static possibility of each feature corner point are as follows: In the reverse sequence of the movement trajectory points of the x-th characteristic corner point, successively obtain the flight speeds of the UAV at the corresponding moments of all elements, form a reverse flight speed sequence, and then successively obtain the corresponding moments of all elements to form a reverse time sequence; Successively obtain the magnitudes of the motion vectors of all elements in the reverse sequence of the movement trajectory points of the x-th characteristic corner point, form a magnitude sequence, and calculate the Pearson correlation coefficient between the magnitude sequence and the reverse flight speed sequence, which is denoted as the motion speed consistency of the x-th characteristic corner point; Denote the product of the motion speed consistency of the x-th characteristic corner point and the motion direction invariance as the initial static possibility; In the reverse time sequence, obtain the inverse proportional value of the mean of the direction consistencies between any two moments, which is denoted as the UAV flight invariance; Denote the normalized value of the product of the UAV flight invariance of the x-th characteristic corner point and the initial static possibility as the final static possibility.

5. The safety obstacle avoidance method for drone road inspection according to claim 1, wherein, The steps for screening out dynamic characteristic corner points are as follows: Denote the characteristic corner points with the final static possibility less than or equal to the preset static-dynamic threshold as dynamic characteristic corner points.

6. The safety obstacle avoidance method for drone road inspection according to claim 2, wherein, The steps for determining the obstacle avoidance threat factor of each dynamic characteristic corner point are as follows: In the video image at the current moment, use the included angle value of the motion vectors of any two dynamic characteristic corner points as the clustering distance, and use the K-means clustering algorithm to perform clustering operations on all dynamic characteristic corner points to obtain several clustering clusters; Calculate the ratio of the number of dynamic characteristic corner points in the b-th clustering cluster to the number of dynamic characteristic corner points in the video image at the current moment, and then calculate the sum vector of the motion vectors of all dynamic characteristic corner points in the b-th clustering cluster. Obtain the inverse proportional value of the included angle value between the sum vector and the motion vector of the y-th dynamic characteristic corner point in the b-th clustering cluster. Denote the product of the ratio of the number of dynamic characteristic corner points and the inverse proportional value as the simultaneous motion unity of the y-th dynamic characteristic corner point in the b-th clustering cluster; Determine the movement trajectory variability of each dynamic characteristic corner point according to the included angle value of the motion vectors of adjacent elements in the reverse sequence of the movement trajectory points of each dynamic characteristic corner point; Obtain the normalized value of the ratio of the movement trajectory variability and the motion unity of the z-th dynamic characteristic corner point, and denote the sum value of the normalized value and the preset constant as the obstacle avoidance threat factor of the z-th dynamic characteristic corner point.

7. The safety obstacle avoidance method for drone road inspection according to claim 6, characterized in that, The steps for determining the movement trajectory variability of each dynamic characteristic corner point according to the included angle value of the motion vectors of adjacent elements in the reverse sequence of the movement trajectory points of each dynamic characteristic corner point are as follows: In the reverse sequence of the movement trajectory points of the z-th dynamic characteristic corner point, successively obtain the included angle values of adjacent two elements to form an included angle value sequence. In the included angle value sequence, denote the included angle values greater than the preset included angle threshold as 1, and denote the included angle values less than or equal to the preset included angle threshold as 0 to form a 01 sequence; The ratio of the number of 1s in the 01 sequence to the length of the 01 sequence is denoted as the first ratio, the ratio of the maximum length of consecutive 1s in the 01 sequence to the length of the 01 sequence is denoted as the second ratio, and the mean value of all the included angle values in the included angle value sequence divided by the product of the first ratio and the second ratio is denoted as the motion trajectory variability of the z-th dynamic feature corner point.

8. The safety obstacle avoidance method for drone road inspection according to claim 1, characterized in that, The obtaining of the enhanced video image at the current moment includes the following specific steps: In the video image at the current moment, taking each feature corner point as the initial clustering center, using the Euclidean distance between the position coordinates of the pixel points as the clustering distance, and using the K-means clustering algorithm to perform only one clustering assignment on all the pixel points to obtain a connected region corresponding to each feature corner point; The feature corner points with the final static possibility greater than the preset static-dynamic threshold are denoted as static feature corner points; Set the obstacle avoidance threat factor of each static feature corner point to a preset constant; Taking the obstacle avoidance threat factor of each feature corner point as the obstacle avoidance threat of the connected region corresponding to each feature corner point; the feature corner points are divided into dynamic feature corner points and static feature corner points; Determine the contrast limit threshold of each connected region according to the obstacle avoidance threat of each connected region and the gray level features of the pixel points in the connected region; According to the contrast limit threshold of each connected region, use the contrast limited histogram equalization algorithm to perform an enhancement operation on the video image at the current moment to obtain the enhanced video image at the current moment.

9. The safety obstacle avoidance method for drone road inspection according to claim 8, wherein, The determining of the contrast limit threshold of each connected region according to the obstacle avoidance threat of each connected region and the gray level features of the pixel points in the connected region includes the following specific steps: Calculate the mean value of the gray level gradient values of all the pixel points in the w-th connected region as the sharpness; The normalized value of the ratio of the obstacle avoidance threat of the w-th connected region to the sharpness is denoted as the limit threshold coefficient of the w-th connected region; Obtain the gray level histogram of the w-th connected region, and denote the mean value of the number of pixel points at all gray levels in the gray level histogram as the standard limit threshold; Denote the product of the standard limit threshold of the w-th connected region and the limit threshold coefficient as the contrast limit threshold of the w-th connected region.

10. A safety obstacle avoidance system for drone road inspection, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the computer program is executed by a processor, it implements the steps of a method for safe obstacle avoidance in drone road inspection as described in any one of claims 1-9.

Images