A safe obstacle avoidance method and system for drone road inspection

Abstract

The present invention relates to the field of image data processing technology, and more specifically to a safe obstacle avoidance method and system for drone road inspections. The method comprises: obtaining characteristic corner points in a video image at the current moment, determining the final static probability of each characteristic corner point to screen out dynamic characteristic corner points, and then determining the obstacle avoidance threat factor of each dynamic characteristic corner point to enhance the video image at the current moment, thereby segmenting the obstacle area in the enhanced video image at each moment and planning an optimal obstacle avoidance path for the drone. The present invention ensures the accuracy of obstacle recognition and the safety of drone road inspection obstacle avoidance by adaptively enhancing the video image.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a safe obstacle avoidance method and system for unmanned aerial vehicle (UAV) road inspection. Background Art

[0002] A safe obstacle avoidance method and system for drone road inspections is a key technology for ensuring drones can effectively avoid obstacles and maintain flight safety during road inspection missions. During inspections, drones use high-definition cameras or other image acquisition devices to capture real-time image information along their path. Image processing techniques, such as deep learning algorithms, detect and classify obstacles in these images, determining their location and type. Path planning algorithms, such as the fast random tree algorithm, are then used to plan an optimal or feasible path for the drone, avoiding obstacles and preventing collisions.

[0003] Existing Problem: Due to the vibrations generated by drones during flight and the influence of wind, the captured video images may be blurred or jittery. In addition, under complex external lighting conditions, details in shadows and highlights of the video images may be lost. This may lead to inaccurate obstacle recognition in the video images, thereby reducing the safety of obstacle avoidance during drone road inspections. Summary of the Invention

[0004] The present invention provides a safe obstacle avoidance method and system for unmanned aerial vehicle (UAV) road inspection to solve existing problems.

[0005] The present invention provides a safe obstacle avoidance method and system for drone road inspections using the following technical solutions:

[0006] An embodiment of the present invention provides a safe obstacle avoidance method for drone road inspection, the method comprising the following steps:

[0007] During the drone road inspection process, obtain the drone's flight speed, flight direction, shooting direction of the drone's camera, and video images at each moment;

[0008] Acquire several characteristic corner points in the current video image; during a period when the drone's flight speed and the camera's shooting direction are similar, determine the invariance of the motion direction of each characteristic corner point based on its reverse motion trajectory in the video image. Combined with the changing characteristics of the drone's flight speed, determine the final static probability of each characteristic corner point;

[0009] Screening out dynamic feature corner points based on the size of the final static probability of each feature corner point; determining the obstacle avoidance threat factor of each dynamic feature corner point based on the reverse motion trajectory of each dynamic feature corner point in the video image; and performing an enhancement operation on the video image at the current moment based on the obstacle avoidance threat factor to obtain an enhanced video image at the current moment;

[0010] A segmentation neural network is used to segment the obstacle area in the enhanced video image at each moment; based on the obstacle area in the enhanced video image at all moments, an optimal path to avoid obstacles is planned for the drone.

[0011] Furthermore, the specific steps of determining the motion direction invariance of each feature corner point include the following:

[0012] Determining the directional consistency between any two moments based on the difference in the flight direction of the drone and the difference in the shooting direction of the camera on the drone;

[0013] The current moment is recorded as the tth moment, and the direction consistency between the tth moment and the t-1th moment, and the direction consistency between the tth moment and the t-2th moment are obtained in sequence, until the direction consistency between the tth moment and the tnth moment is greater than the preset angle threshold for the first time. The period from the tn-1th moment to the tth moment is recorded as the target period;

[0014] During the target period, starting from each feature corner point in the video image at the current moment, the inverse optical flow method is used to obtain matching points in the video image at other moments in reverse time order until the matching fails, and the reverse sequence of motion trajectory points of each feature corner point is obtained;

[0015] In the reverse sequence of the motion trajectory points of the x-th feature corner point, the Euclidean distance and direction from the position coordinates of the second element to the position coordinates of the first element are used to form the motion vector of the first element. The cosine similarity of the motion vectors of any two elements is calculated, and the average of the cosine similarities of the motion vectors of all any two elements is used as the motion direction invariance of the x-th feature corner point.

[0016] Furthermore, the determining of the direction consistency between the arbitrary two moments includes the following specific steps:

[0017] For any two moments, obtain the minimum angle between the flight directions of the drone, and then obtain the minimum angle between the shooting directions of the cameras on the drone, and use the average of the minimum angle between the flight directions and the minimum angle between the shooting directions as the direction consistency between the any two moments.

[0018] Furthermore, the determination of the final static probability of each feature corner point includes the following specific steps:

[0019] In the reverse sequence of the motion trajectory points of the x-th feature corner point, the flight speed of the drone at the corresponding moments of all elements is obtained in sequence to form a reverse flight speed sequence. Then, the moments corresponding to all elements are obtained in sequence to form a reverse moment sequence.

[0020] Obtain the modulus lengths of the motion vectors of all elements in the reverse sequence of the motion trajectory points of the x-th feature corner point in sequence to form a modulus length sequence. Calculate the Pearson correlation coefficient between the modulus length sequence and the reverse flight speed sequence, which is recorded as the motion speed consistency of the x-th feature corner point.

[0021] The product of the consistency of motion speed and the invariance of motion direction of the x-th feature corner point is recorded as the initial static possibility;

[0022] In the reverse time series, the inverse proportional value of the mean of the directional consistency between any two moments is obtained, which is recorded as the UAV flight invariance;

[0023] The normalized value of the product of the UAV flight invariance of the x-th feature corner point and the initial static likelihood is recorded as the final static likelihood.

[0024] Furthermore, the specific steps of screening out dynamic feature corner points include the following:

[0025] The feature corner points whose final static probability is less than or equal to the preset static / dynamic threshold are recorded as dynamic feature corner points.

[0026] Furthermore, the determination of the obstacle avoidance threat factor of each dynamic feature corner point includes the following specific steps:

[0027] In the current video image, the angle between the motion vectors of any two dynamic feature corner points is used as the clustering distance. The K-means clustering algorithm is used to cluster all dynamic feature corner points to obtain several clusters.

[0028] Calculate the ratio of the number of dynamic feature corner points in the bth cluster to the number of dynamic feature corner points in the video image at the current moment, then calculate the sum vector of the motion vectors of all dynamic feature corner points in the bth cluster, obtain the inverse proportional value of the angle between the sum vector and the motion vector of the yth dynamic feature corner point in the bth cluster, and multiply the product of the ratio of the number of dynamic feature corner points and the inverse proportional value as the simultaneous motion uniformity of the yth dynamic feature corner point in the bth cluster;

[0029] Determine the variability of the motion trajectory of each dynamic feature corner point according to the angle value of the motion vectors of adjacent elements in the reverse sequence of the motion trajectory points of each dynamic feature corner point;

[0030] Obtain a normalized value of the ratio of the motion trajectory variability to the motion uniformity of the zth dynamic feature corner point, and record the sum of the normalized value and a preset constant as the obstacle avoidance threat factor of the zth dynamic feature corner point.

[0031] Furthermore, the method of determining the variability of the motion trajectory of each dynamic feature corner point according to the angle value of the motion vectors of adjacent elements in the reverse sequence of the motion trajectory points of each dynamic feature corner point includes the following specific steps:

[0032] In the reverse sequence of the motion trajectory points of the zth dynamic feature corner point, the angle values ​​of the motion vectors of two adjacent elements are obtained in sequence to form an angle value sequence. In the angle value sequence, the angle value greater than the preset angle threshold is recorded as 1, and the angle value less than or equal to the preset angle threshold is recorded as 0, forming a 01 sequence;

[0033] The ratio of the number of 1s in the 01 sequence to the length of the 01 sequence is recorded as the first ratio, the ratio of the maximum length of consecutive 1s in the 01 sequence to the length of the 01 sequence is recorded as the second ratio, and the product of the mean of all angle values ​​in the angle value sequence, the first ratio, and the second ratio is recorded as the motion trajectory variability of the zth dynamic feature corner point.

[0034] Furthermore, the obtaining of the enhanced video image at the current moment includes the following specific steps:

[0035] In the current video image, each feature corner point is used as the initial cluster center, and the Euclidean distance between the position coordinates of the pixel points is used as the cluster distance. The K-means clustering algorithm is used to perform cluster assignment on all pixels only once to obtain a connected area corresponding to each feature corner point.

[0036] The feature corner points whose final static probability is greater than the preset static / dynamic threshold are recorded as static feature corner points;

[0037] The obstacle avoidance threat factor of each static feature corner point is set to a preset constant;

[0038] The obstacle avoidance threat factor of each feature corner point is used as the obstacle avoidance threat of the connected area corresponding to each feature corner point; the feature corner points are divided into dynamic feature corner points and static feature corner points;

[0039] Determine the contrast limit threshold of each connected region based on the obstacle avoidance threat of each connected region and the grayscale characteristics of the pixels within the connected region;

[0040] According to the contrast limiting threshold of each connected area, a contrast limiting histogram equalization algorithm is used to perform an enhancement operation on the video image at the current moment to obtain an enhanced video image at the current moment.

[0041] Furthermore, the step of determining the contrast limit threshold of each connected region based on the obstacle avoidance threat of each connected region and the grayscale characteristics of the pixels in the connected region includes the following specific steps:

[0042] Calculate the mean of the grayscale gradient values ​​of all pixels in the w-th connected area as the clarity;

[0043] The normalized value of the ratio of the obstacle avoidance threat to the clarity of the w-th connected region is recorded as the restriction threshold coefficient of the w-th connected region;

[0044] Get the grayscale histogram of the w-th connected area, and record the mean of the number of pixels at all gray levels in the grayscale histogram as the standard limit threshold;

[0045] The product of the standard limit threshold of the w-th connected area and the limit threshold coefficient is recorded as the contrast limit threshold of the w-th connected area.

[0046] The present invention also proposes a safety obstacle avoidance system for drone road inspection, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. The processor executes the computer program stored in the memory to implement the steps of the aforementioned safety obstacle avoidance method for drone road inspection.

[0047] The beneficial effects of the technical solution of the present invention are:

[0048] In an embodiment of the present invention, characteristic corner points in the video image at the current moment are obtained, and the final static probability of each characteristic corner point is determined to screen out dynamic characteristic corner points, thereby distinguishing the dynamic characteristic corner points in the video image, and ensuring that the areas where important dynamic characteristic corner points are located are subsequently enhanced. The obstacle avoidance threat factor of each dynamic characteristic corner point is determined, and thereby, by analyzing the obstacle avoidance threat caused by the unpredictability of the dynamic characteristic corner points, the enhancement effect of the area where the dynamic characteristic corner points are located is adapted. The video image at the current moment is enhanced, that is, adaptive enhancement is performed based on the obstacle avoidance threat and clarity of the connected area where each characteristic corner point is located, and an enhanced video image is obtained, thereby segmenting the obstacle area in the enhanced video image at each moment, and planning an optimal path for the drone to avoid obstacles. Thus, the present invention ensures the accuracy of obstacle recognition by adaptively enhancing the video image, thereby ensuring the safety of drone road inspection and obstacle avoidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 This is a flowchart of the steps of a safe obstacle avoidance method for drone road inspection in the present invention;

[0051] Figure 2 Schematic diagram of drone road inspection. DETAILED DESCRIPTION

[0052] To further illustrate the technical means and effectiveness of the present invention in achieving its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effectiveness of a safe obstacle avoidance method and system for drone road inspections proposed in accordance with the present invention. In the following description, references to different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0053] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0054] The following describes in detail a specific solution of a safe obstacle avoidance method and system for drone road inspection provided by the present invention with reference to the accompanying drawings.

[0055] See also Figure 1 , which shows a flowchart of a safe obstacle avoidance method for drone road inspection provided by one embodiment of the present invention, the method comprising the following steps:

[0056] Step S001: During the drone road inspection process, obtain the drone's flight speed, flight direction, shooting direction of the camera on the drone, and video images at each moment.

[0057] During the drone road inspection process, the drone's flight speed, flight direction (the direction in the three-dimensional space of altitude, longitude and latitude), the shooting direction of the drone's camera (the direction in the three-dimensional space of altitude, longitude and latitude), and video images are collected at each moment.

[0058] It should be noted that: in this embodiment, the acquisition frequency is 30 Hz, which is described as an example. Among them, the drone's onboard sensors, such as gyroscopes, accelerometers, and magnetometers, can provide the drone's attitude information. This information is used to determine the drone's current flight direction and the direction of the camera. The global positioning system (GPS) then provides the drone's geographic location information, and the drone's flight speed can be calculated through continuous location data points. The camera on the drone is responsible for capturing real-time video images. Schematic diagram of drone road inspection, as shown Figure 2 shown.

[0059] Step S002: Acquire several feature corner points in the video image at the current moment; within the time period when the UAV's flight speed and the shooting direction of the camera on the UAV are similar, determine the invariance of the motion direction of each feature corner point based on the reverse motion trajectory of each feature corner point in the video image, and then determine the final static possibility of each feature corner point in combination with the change characteristics of the UAV's flight speed.

[0060] It should be noted that during safe obstacle avoidance during drone road inspections, significantly different attention needs to be paid to dynamic and static areas within the video image. Dynamic obstacles (such as vehicles, pedestrians, other aircraft, and birds) exhibit unpredictable motion and require higher real-time response speeds. Static obstacles (such as buildings and towers) are fixed in position and can be avoided through pre-planned avoidance. Therefore, when the quality of a captured video image is poor, static obstacles can be analyzed using information from higher-quality video images at other times. However, for dynamic obstacles, the video image details must be maximized to ensure timely and safe obstacle avoidance. When the drone's flight direction and camera's shooting direction remain unchanged, the relative motion vectors of static objects in consecutive frames of captured video are typically consistent. This is because the motion of static objects relative to the drone is primarily caused by the drone's own motion. Therefore, consecutive frames with unchanged flight and shooting directions are first acquired.

[0061] For any two moments, the minimum angle between the flight directions of the drone is obtained, and then the minimum angle between the shooting directions of the cameras on the drone is obtained. The average of the minimum angle between the flight directions and the minimum angle between the shooting directions is taken as the direction consistency between the two moments.

[0062] The current moment is recorded as the tth moment, and the direction consistency between the tth moment and the t-1th moment, and the direction consistency between the tth moment and the t-2th moment are obtained in sequence, until the direction consistency between the tth moment and the tnth moment is greater than the preset angle threshold for the first time. The time period from the tn-1th moment to the tth moment is recorded as the target time period.

[0063] It should be noted that in this embodiment, the preset angle threshold is 3 degrees, which is used as an example. The preset angle threshold can be appropriately increased to ensure the duration of the target period to meet the requirements of subsequent analysis. The video images within the first 3 minutes of the drone's flight are directly enhanced using the histogram equalization algorithm. The histogram equalization algorithm is a well-known technique, and its specific method will not be described here.

[0064] Use the FAST corner detection algorithm to obtain several feature corner points in the video image at the current moment.

[0065] During the target period, starting from each feature corner point in the video image at the current moment, the reverse optical flow method is used to obtain the matching points in the video images at other moments one by one in reverse chronological order until the matching fails, and the reverse sequence of motion trajectory points of each feature corner point is obtained, for example: {the mth feature corner point in the video image at the tth moment, the matching point of the mth feature corner point in the video image at the t-1th moment, the matching point of the mth feature corner point in the video image at the t-2th moment, and the matching point of the mth feature corner point in the video image at the t-3th moment}.

[0066] It should be noted that the FAST (Features from Accelerated Segment Test) corner detection algorithm and the inverse optical flow method are both well-known technologies, and their specific methods are not described here. Starting with each feature corner point in the video image at the current moment, the algorithm matches each frame in reverse chronological order within the target time period until a match fails, indicating that the feature corner point has left the frame. The inverse motion trajectory of each feature corner point is then obtained, and the duration of the inverse motion trajectory may not be equal to the target time period. Feature corner points are points with significant changes in local areas of the image, typically corresponding to edges, corners, or locations with significant texture changes. These are key locations for obstacle avoidance. By tracking the motion trajectory of feature corner points in a video sequence, object motion parameters or the camera's motion trajectory can be estimated. If the number of matching points for a feature corner point is less than three, it indicates that the feature corner point has just appeared in the frame and can be omitted from analysis and deleted.

[0067] In the reverse sequence of the motion trajectory points of the x-th feature corner point in the video image at the current moment, the Euclidean distance and direction from the position coordinates of the second element (matching point) to the position coordinates of the first element (feature corner point) constitute the motion vector of the first element.

[0068] It should be noted that in this embodiment, a rectangular coordinate system is constructed with the vertex at the lower left corner of the video image at each moment as the origin, the horizontal axis pointing rightward as the horizontal axis, and the vertical axis pointing upward as the positive vertical axis. The position coordinates of each pixel in the video image are obtained in this rectangular coordinate system. Furthermore, the motion vector of the last element is set to be the motion vector of the second-to-last element. As mentioned above, the motion vector of the first element is the motion vector of the xth characteristic corner point in the video image at the current moment.

[0069] In the reverse sequence of the motion trajectory points of the xth feature corner point in the video image at the current moment, the flight speed of the drone at the corresponding moment of each element is obtained in sequence to form a reverse flight speed sequence. Then, the time corresponding to each element is obtained in sequence to form a reverse time sequence.

[0070] In the reverse sequence of the motion trajectory points of the x-th feature corner point in the video image at the current moment, the cosine similarity of the motion vectors of any two elements is calculated, and the average of the cosine similarities of the motion vectors of all any two elements is used as the motion direction invariance of the x-th feature corner point.

[0071] The calculation of cosine similarity is well-known, and the specific method will not be described here. Cosine similarity only measures the directional relationship between vectors, regardless of their magnitude. The larger the cosine similarity, the more consistent the vector directions.

[0072] It should be noted that, while the drone's flight direction and the camera's shooting direction remain unchanged, the motion vector of a static object should remain unchanged. However, the direction of a dynamic object's motion vector is determined by both the drone's flight direction and its own motion direction. This means that a change in its own motion direction will also cause the motion vector's direction to change. Therefore, the greater the motion direction invariance, the more likely the feature corner point is a static point. However, if the motion direction of a dynamic object remains unchanged during the target period, the motion direction invariance is also greater. Therefore, further analysis is required, as the modulus of a static object's motion vector is affected only by the drone's flight speed, while the modulus of a dynamic object's motion vector is affected by both the drone's flight speed and its own motion speed.

[0073] For the x-th feature corner point in the video image at the current moment, the modulus lengths of the motion vectors of all elements in the reverse sequence of the motion trajectory points are obtained in sequence to form a modulus length sequence. The Pearson correlation coefficient between the modulus length sequence and the reverse flight speed sequence is calculated and recorded as the motion speed consistency of the x-th feature corner point.

[0074] The Pearson correlation coefficient is a well-known technique, and its specific method is not described here. A larger Pearson correlation coefficient indicates a stronger linear relationship between the two sequences, and a more likely static corner point. Calculating the Pearson correlation coefficient does not require uniform dimensions.

[0075] For the xth feature corner point in the current video image, the product of motion speed consistency and motion direction invariance is recorded as the initial static probability. That is, motion speed consistency is used as the adjustment value of motion direction invariance.

[0076] For the x-th feature corner point in the video image at the current moment, in the reverse time sequence, obtain the mean of the directional consistency between all two arbitrary moments The inverse proportional value is recorded as the UAV flight invariance.

[0077] It should be noted that the smaller the mean value of the direction consistency between any two moments in the reverse time sequence, the greater the invariance of the drone's flight, and the more reliable the above analysis based on the fact that the drone's flight direction and the shooting direction of the drone's camera remain unchanged. The mean of the above directional consistency Inverse proportional value, It is a linear normalization function used to normalize data values ​​to between 0 and 1.

[0078] For the x-th feature corner point in the video image at the current moment, multiply the drone flight invariance by the initial static likelihood The normalized value of is recorded as the final static possibility.

[0079] Among them, Represents the above product The normalized value is normalized to between 0 and 1.

[0080] In the above manner, the final static likelihood of each feature corner point in the video image at the current moment can be obtained.

[0081] Step S003: Filter out dynamic feature corner points based on the size of the final static probability of each feature corner point; determine the obstacle avoidance threat factor of each dynamic feature corner point based on the reverse motion trajectory of each dynamic feature corner point in the video image; and enhance the video image at the current moment based on the obstacle avoidance threat factor to obtain an enhanced video image at the current moment.

[0082] The default motion threshold is 0.7, which is used as an example for description.

[0083] In the video image at the current moment, the feature corner points whose final static probability is greater than the preset motion threshold are recorded as static feature corner points, and the feature corner points whose final static probability is less than or equal to the preset motion threshold are recorded as dynamic feature corner points.

[0084] It should be noted that for the dynamic feature corner points in the video image at the current moment, the predictability of their motion trajectories needs to be further analyzed. The worse the predictability of the motion trajectory, the greater the impact on safe obstacle avoidance, and the more important it is to enhance the image quality of the area where the dynamic feature corner points are located. For example, the motion trajectories of vehicles and pedestrians that comply with traffic regulations, moving along the road in a certain direction, are traceable and highly predictable, while the motion trajectories of vehicles and pedestrians that do not comply with traffic regulations are untraceable and less predictable. Furthermore, normally flying flocks of birds usually fly in a unified direction, while frightened flocks of birds fly in a chaotic manner.

[0085] According to the motion vector of each dynamic feature corner point in the video image at the current moment, the angle between the motion vectors of any two dynamic feature corner points is used as the clustering distance. The K-means clustering algorithm is used to cluster all dynamic feature corner points to obtain several clusters.

[0086] Among them, the K-means clustering algorithm is a well-known technology, and the specific method will not be introduced here.

[0087] Calculate the ratio of the number of dynamic feature corner points in the bth cluster to the number of dynamic feature corner points in the video image at the current moment, then calculate the sum vector of the motion vectors of all dynamic feature corner points in the bth cluster, and obtain the angle between the sum vector and the motion vector of the yth dynamic feature corner point in the bth cluster. The product of the ratio and the inverse proportional value is recorded as the simultaneous motion unity of the y-th dynamic feature corner point in the b-th cluster.

[0088] It should be noted that the maximum angle between the two vectors is 180 degrees. As the angle value The inverse proportional value of . Among them, the larger the ratio, the similar the movement direction of a large number of dynamic feature corners in the video image at the current moment, the better the predictability of the movement of the dynamic feature corners in the cluster, and the smaller the angle value. The smaller the value, the more consistent the motion direction of the yth dynamic feature corner point in the cluster is with the overall motion direction of the cluster (and the direction of the vector), and thus the ratio is used as the angle value. The adjustment value is inversely proportional to the value of , to obtain the uniformity of simultaneous movement.

[0089] According to the above method, the simultaneous motion uniformity of each dynamic feature corner point in the video image at the current moment can be obtained.

[0090] The preset angle threshold is 5 degrees, which is used as an example for description.

[0091] In the reverse sequence of the motion trajectory points of the zth dynamic feature corner point in the video image at the current moment, the angle values ​​of the motion vectors of two adjacent elements are obtained in sequence to form an angle value sequence. In the angle value sequence, the angle value greater than the preset angle threshold is recorded as 1, and the angle value less than or equal to the preset angle threshold is recorded as 0, forming a 01 sequence.

[0092] The ratio of the number of 1s in the 01 sequence to the length of the 01 sequence is recorded as the first ratio, and the ratio of the maximum length of consecutive 1s in the 01 sequence to the length of the 01 sequence is recorded as the second ratio. The mean of all angle values ​​in the angle value sequence is obtained, and the product of the mean, the first ratio, and the second ratio is recorded as the motion trajectory variability of the zth dynamic feature corner point in the video image at the current moment.

[0093] What needs to be explained is that the larger the mean of all angle values ​​in the angle value sequence is, the greater the change in the direction of movement over time. The number of 1s in the 01 sequence, that is, the number of large mutations in the direction of movement, and the maximum length of consecutive 1s in the 01 sequence (for example, the maximum length of consecutive 1s in 01110011110 is 4), reflect the continuity of large mutations in the direction of movement. The larger the two are, the more drastic the change in the direction of movement over time.

[0094] The preset constant is 1, and this is used as an example for description.

[0095] Obtain a normalized value of the ratio of the motion trajectory variability to the motion uniformity of the zth dynamic feature corner point in the video image at the current moment, and record the sum of the normalized value and a preset constant as the obstacle avoidance threat factor of the zth dynamic feature corner point in the video image at the current moment.

[0096] It should be noted that: In this embodiment, the A linear normalization function is used to normalize the ratio of the motion trajectory variability to the motion uniformity to a value between 0 and 1. The smaller the motion uniformity, the more unique the motion direction, that is, the more unpredictable the motion direction, and the greater the obstacle avoidance threat. The greater the motion trajectory variability, the more variable and unpredictable the motion trajectory, and the greater the obstacle avoidance threat.

[0097] According to the above method, the obstacle avoidance threat factor of each dynamic feature corner point in the video image at the current moment can be obtained.

[0098] The obstacle avoidance threat factor of each static feature corner point in the video image at the current moment is set to a preset constant, thereby obtaining the obstacle avoidance threat factor of each feature corner point.

[0099] In the video image at the current moment, each feature corner point is used as the initial cluster center, and the Euclidean distance between the position coordinates of the pixel points is used as the cluster distance. The K-means clustering algorithm is used to perform cluster assignment on all pixels only once, and each initial cluster center corresponds to a connected area, that is, each feature corner point corresponds to a connected area, and there is only one feature corner point in each connected area.

[0100] Thus, the video image at the current moment is divided into multiple connected areas, each of which corresponds to a feature corner point.

[0101] It should be noted that the K-means clustering algorithm is a well-known technique. Single cluster assignment involves calculating the Euclidean distance between each pixel in the current video image and all initial cluster centers and assigning the pixel to the nearest initial cluster center. In this way, each initial cluster center corresponds to a region consisting of all pixels assigned to that center. Furthermore, because the clustering process groups spatially close pixels together, continuous regions are formed in the image. In other words, each initial cluster center corresponds to a connected region.

[0102] In the video image at the current moment, the obstacle avoidance threat factor of each feature corner point is used as the obstacle avoidance threat of the connected area where each feature corner point is located.

[0103] Calculate the mean of the grayscale gradient values ​​of all pixels in the w-th connected region in the video image at the current moment as the clarity of the w-th connected region.

[0104] The normalized value of the ratio of the obstacle avoidance threat to the clarity of the w-th connected region is recorded as the restriction threshold coefficient of the w-th connected region.

[0105] It should be noted that: the greater the obstacle avoidance threat, the lower the clarity, the more the connected area needs to improve the contrast and highlight the details. A linear normalization function is used to normalize the ratio of the obstacle avoidance threat to the clarity to a value between 0.8 and 1.2.

[0106] Get the grayscale histogram of the w-th connected area, and record the mean of the number of pixels at all gray levels in the grayscale histogram as the standard limit threshold.

[0107] The product of the standard limit threshold of the w-th connected area and the limit threshold coefficient is recorded as the contrast limit threshold of the w-th connected area.

[0108] What needs to be explained is that the video image at the current moment is grayscaled, and the Canny edge detection algorithm is used to obtain the grayscale gradient value of each pixel in the video image at the current moment. Among them, the grayscale processing of the image, the acquisition of the grayscale histogram, and the Canny edge detection algorithm are all well-known technologies, and the specific methods are not introduced here.

[0109] In the video image at the current moment, according to the contrast limiting threshold of each connected area, a contrast limiting histogram equalization algorithm is used to perform an enhancement operation on the video image at the current moment to obtain an enhanced video image at the current moment.

[0110] It should be noted that the contrast-limited histogram equalization algorithm is a well-known technology and the specific method will not be introduced here. In the algorithm, the size of the contrast limiting threshold has a significant impact on the effect of image enhancement. A larger contrast limiting threshold can significantly improve the contrast of the image and make the image details more prominent. The contrast-limited histogram equalization algorithm enhances the video image at the current moment as follows: first, the video image at the current moment is divided into several connected areas, the grayscale histogram of each connected area is obtained, and the grayscale histogram is cropped using the contrast limiting threshold of each connected area. The cropped grayscale histogram of each connected area is subjected to histogram equalization processing, and then bilinear interpolation is performed to eliminate the block effect and output the enhanced video image.

[0111] According to the above method, a real-time enhanced video image is obtained.

[0112] Step S004: Use a segmentation neural network to segment the obstacle area in the enhanced video image at each moment; based on the obstacle area in the enhanced video image at all moments, plan an optimal path for the drone to avoid obstacles.

[0113] The embodiment of the present invention adopts a segmentation neural network to identify obstacle areas and background areas in segmentation-enhanced video images.

[0114] The relevant content of the segmentation neural network is as follows:

[0115] The segmentation neural network used in this example is the Mask R-CNN neural network, and the dataset used is an enhanced video image dataset. Mask R-CNN is a well-known technique, and the specific method is not described here. Mask R-CNN stands for "Mask Region-based Convolutional Neural Network" in Chinese and "Mask Region-based Convolutional Neural Network" in English.

[0116] The pixels that need to be segmented are divided into two categories, that is, the labeling process of the training set is: single-channel semantic label, the corresponding position pixel belongs to the background area is labeled as 0, and belongs to the obstacle area is labeled as 1.

[0117] The task of the network is classification, so the loss function used is the cross entropy loss function.

[0118] The obstacle area and background area in the enhanced video image at each moment are obtained by segmenting the neural network. This process is a well-known technology and the specific method will not be introduced here.

[0119] According to the obstacle areas in the enhanced video images at all times, the RRT path planning algorithm is used to plan an optimal path for the drone to avoid obstacles.

[0120] Among them, the obstacle avoidance method based on the RRT (Rapid Random Tree) algorithm can dynamically generate a new flight path according to the position of the obstacle, thereby avoiding collision. This is a well-known technology and the specific method will not be introduced here.

[0121] The present invention also provides a safety obstacle avoidance system for drone road inspection, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. The processor executes the computer program stored in the memory to implement the steps of the aforementioned safety obstacle avoidance method for drone road inspection.

[0122] So far, the present invention is completed.

[0123] In summary, in an embodiment of the present invention, characteristic corner points are obtained from the video image at the current moment, and the final static probability of each characteristic corner point is determined to screen out dynamic characteristic corner points. The obstacle avoidance threat factor of each dynamic characteristic corner point is then determined to enhance the video image at the current moment, thereby segmenting the obstacle area in the enhanced video image at each moment and planning an optimal obstacle-avoiding path for the drone. The present invention ensures the accuracy of obstacle recognition and, in turn, the safety of drone road inspection and obstacle avoidance by adaptively enhancing the video image.

[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A safe obstacle avoidance method for drone road inspection, characterized in that: The method comprises the following steps: During the drone road inspection process, obtain the drone's flight speed, flight direction, shooting direction of the drone's camera, and video images at each moment; Acquire several characteristic corner points in the current video image; during a period when the drone's flight direction is similar to the camera's shooting direction, determine the invariance of the motion direction of each characteristic corner point based on its reverse motion trajectory in the video image. Combined with the changing characteristics of the drone's flight speed, determine the final static probability of each characteristic corner point; Screening out dynamic feature corner points based on the size of the final static probability of each feature corner point; determining the obstacle avoidance threat factor of each dynamic feature corner point based on the reverse motion trajectory of each dynamic feature corner point in the video image; and performing an enhancement operation on the video image at the current moment based on the obstacle avoidance threat factor to obtain an enhanced video image at the current moment; Use a segmentation neural network to segment the obstacle area in the enhanced video image at each moment; based on the obstacle area in the enhanced video image at all moments, plan an optimal path for the drone to avoid obstacles; The specific steps of determining the motion direction invariance of each feature corner point are as follows: Determining the directional consistency between any two moments based on a difference in the flight direction of the drone and a difference in the shooting direction of a camera on the drone at the two moments; The current moment is recorded as the tth moment, and the direction consistency between the tth moment and the t-1th moment, and the direction consistency between the tth moment and the t-2th moment are obtained in sequence, until the direction consistency between the tth moment and the tnth moment is greater than the preset angle threshold for the first time. The period from the tn-1th moment to the tth moment is recorded as the target period; During the target period, starting from each feature corner point in the video image at the current moment, the inverse optical flow method is used to obtain matching points in the video image at other moments in reverse time order until the matching fails, and the reverse sequence of motion trajectory points of each feature corner point is obtained; In the reverse sequence of the motion trajectory points of the x-th feature corner point, the Euclidean distance and direction from the position coordinates of the second element to the position coordinates of the first element are used to form the motion vector of the first element. The cosine similarity of the motion vectors of any two elements is calculated, and the average of the cosine similarities of the motion vectors of all any two elements is used as the motion direction invariance of the x-th feature corner point.

2. A safety obstacle avoidance method for drone road inspection according to claim 1, characterized in that: The specific steps of determining the direction consistency between the two arbitrary moments are as follows: For any two moments, obtain the minimum angle between the flight directions of the drone, and then obtain the minimum angle between the shooting directions of the cameras on the drone, and use the average of the minimum angle between the flight directions and the minimum angle between the shooting directions as the direction consistency between the any two moments.

3. A safety obstacle avoidance method for drone road inspection according to claim 1, characterized in that: The specific steps of determining the final static probability of each feature corner point are as follows: In the reverse sequence of the motion trajectory points of the x-th feature corner point, the flight speed of the drone at the corresponding moments of all elements is obtained in sequence to form a reverse flight speed sequence. Then, the moments corresponding to all elements are obtained in sequence to form a reverse moment sequence. Obtain the modulus lengths of the motion vectors of all elements in the reverse sequence of the motion trajectory points of the x-th feature corner point in sequence to form a modulus length sequence. Calculate the Pearson correlation coefficient between the modulus length sequence and the reverse flight speed sequence, which is recorded as the motion speed consistency of the x-th feature corner point. The product of the consistency of motion speed and the invariance of motion direction of the x-th feature corner point is recorded as the initial static possibility; In the reverse time series, the inverse proportional value of the mean of the directional consistency between any two moments is obtained, which is recorded as the UAV flight invariance; The normalized value of the product of the UAV flight invariance of the x-th feature corner point and the initial static likelihood is recorded as the final static likelihood.

4. A safety obstacle avoidance method for drone road inspection according to claim 1, characterized in that: The specific steps of screening out dynamic feature corner points are as follows: The feature corner points whose final static probability is less than or equal to the preset static / dynamic threshold are recorded as dynamic feature corner points.

5. The safety obstacle avoidance method for drone road inspection according to claim 1, characterized in that: The specific steps of determining the obstacle avoidance threat factor of each dynamic feature corner point are as follows: In the current video image, the angle between the motion vectors of any two dynamic feature corner points is used as the clustering distance. The K-means clustering algorithm is used to cluster all dynamic feature corner points to obtain several clusters. Calculate the ratio of the number of dynamic feature corner points in the bth cluster to the number of dynamic feature corner points in the video image at the current moment, then calculate the sum vector of the motion vectors of all dynamic feature corner points in the bth cluster, obtain the inverse proportional value of the angle between the sum vector and the motion vector of the yth dynamic feature corner point in the bth cluster, and multiply the product of the ratio of the number of dynamic feature corner points and the inverse proportional value as the simultaneous motion uniformity of the yth dynamic feature corner point in the bth cluster; Determine the variability of the motion trajectory of each dynamic feature corner point according to the angle value of the motion vectors of adjacent elements in the reverse sequence of the motion trajectory points of each dynamic feature corner point; Obtain a normalized value of the ratio of the motion trajectory variability to the motion uniformity of the zth dynamic feature corner point, and record the sum of the normalized value and a preset constant as the obstacle avoidance threat factor of the zth dynamic feature corner point.

6. A safety obstacle avoidance method for drone road inspection according to claim 5, characterized in that: The method of determining the variability of the motion trajectory of each dynamic feature corner point according to the angle value of the motion vectors of adjacent elements in the reverse sequence of the motion trajectory points of each dynamic feature corner point includes the following specific steps: In the reverse sequence of the motion trajectory points of the zth dynamic feature corner point, the angle values ​​of the motion vectors of two adjacent elements are obtained in sequence to form an angle value sequence. In the angle value sequence, the angle value greater than the preset angle threshold is recorded as 1, and the angle value less than or equal to the preset angle threshold is recorded as 0, forming a 01 sequence; The ratio of the number of 1s in the 01 sequence to the length of the 01 sequence is recorded as the first ratio, the ratio of the maximum length of consecutive 1s in the 01 sequence to the length of the 01 sequence is recorded as the second ratio, and the product of the mean of all angle values ​​in the angle value sequence, the first ratio, and the second ratio is recorded as the motion trajectory variability of the zth dynamic feature corner point.

7. A safety obstacle avoidance method for drone road inspection according to claim 1, characterized in that: The specific steps of obtaining the enhanced video image at the current moment are as follows: In the current video image, each feature corner point is used as the initial cluster center, and the Euclidean distance between the position coordinates of the pixel points is used as the cluster distance. The K-means clustering algorithm is used to perform cluster assignment on all pixels only once to obtain a connected area corresponding to each feature corner point. The feature corner points whose final static probability is greater than the preset static / dynamic threshold are recorded as static feature corner points; The obstacle avoidance threat factor of each static feature corner point is set to a preset constant; The obstacle avoidance threat factor of each feature corner point is used as the obstacle avoidance threat of the connected area corresponding to each feature corner point; the feature corner points are divided into dynamic feature corner points and static feature corner points; Determine the contrast limit threshold of each connected region based on the obstacle avoidance threat of each connected region and the grayscale characteristics of the pixels within the connected region; According to the contrast limiting threshold of each connected area, a contrast limiting histogram equalization algorithm is used to perform an enhancement operation on the video image at the current moment to obtain an enhanced video image at the current moment.

8. A safe obstacle avoidance method for drone road inspection according to claim 7, characterized in that: The specific steps of determining the contrast limit threshold of each connected region based on the obstacle avoidance threat of each connected region and the grayscale characteristics of the pixels within the connected region are as follows: Calculate the mean of the grayscale gradient values ​​of all pixels in the w-th connected area as the clarity; The normalized value of the ratio of the obstacle avoidance threat to the clarity of the w-th connected region is recorded as the restriction threshold coefficient of the w-th connected region; Get the grayscale histogram of the w-th connected area, and record the mean of the number of pixels at all gray levels in the grayscale histogram as the standard limit threshold; The product of the standard limit threshold of the w-th connected area and the limit threshold coefficient is recorded as the contrast limit threshold of the w-th connected area.

9. A safety obstacle avoidance system for drone road inspection, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is executed by a processor, the steps of a safe obstacle avoidance method for drone road inspection as described in any one of claims 1 to 8 are implemented.

Images