UAV track detection and height estimation method based on image segmentation and medium

Abstract

The invention discloses an unmanned aerial vehicle track barrier detection and height estimation method and medium based on image segmentation. The unmanned platform is used to install a monocular visible light camera and a small computing device to detect and identify the track barrier target; the visible light camera realizes the shooting of the visible light image of the target area and outputs the visible light video stream; the small computing device realizes the reception of the visible light camera video and runs the target key point detection algorithm and the height estimation algorithm; the high-speed target key point detection algorithm realizes the real-time detection of the target key point in the video stream, outputs the pixel coordinates of the target key point, and simultaneously outputs the pixel length, pixel width and pixel position of the target rectangular boundary box; the height estimation algorithm calculates the distance between the monocular visible light camera and the top surface of the track barrier according to the pixel coordinates of the target key point. The invention realizes the rough distance measurement of the monocular camera and realizes the hovering aiming of the unmanned platform within the specified height range at a relatively low cost.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection and recognition, and in particular to a method and medium for detecting and estimating the track barrier of an unmanned aerial vehicle based on image segmentation. Background Art

[0002] As a typical anti-landing obstacle, rail forts are usually deployed in clusters on the beach to hinder and delay landing. Therefore, "obstacle removal" has become an inevitable problem in landing operations. For the rail fort target cluster, the obstacle removal effect of the obstacle removal bomb is directly related to the target search success rate and key point aiming accuracy.

[0003] With the development of computer technology and the widespread application of computer vision, the research on real-time detection of targets using computer image and video processing technology has made great progress. This allows the obstacle removal platform to choose to use an optical camera to shoot images from the air to the ground and search for rail fort targets in the image. For detecting fixed targets in images, the most typical method is a feature-based detection algorithm. This method first extracts features from different candidate areas in the image to form a feature space; then a classifier is constructed in the feature space for classification. The image features selected by this method are very important. The optional features can be natural features such as the shape, size, texture, color, amplitude, etc. of the object, or artificially designed statistical features such as histograms, HOG, HAAR, SIFT, etc. The advantages of using the above traditional detection methods to detect rail forts are fast speed and good real-time performance; the disadvantage is poor robustness. Factors such as weather, sunshine, and background changes will cause the detection accuracy to decrease; in addition, this type of method cannot give accurate key point coordinates and cannot achieve height estimation.

[0004] In recent years, the accuracy of target detection methods based on deep learning has surpassed that of traditional methods. The essential reason is that based on a large amount of labeled data, as the number of layers of the neural network increases, it can extract better features than those designed manually, thereby greatly improving the performance of the algorithm. However, the deep neural network detection model has high requirements for the hardware platform and requires a GPU computing platform or a dedicated neural network chip. Therefore, it is difficult to deploy such algorithms on existing computing platforms, and replacing high-performance computing platforms will also increase the cost of the system.

[0005] Domestic and foreign technical analysis:

[0006] At present, there are limited public documents, patents and reports on rail bar detection and identification technology at home and abroad, and no relevant foreign literature has been retrieved. The domestic method for detecting and identifying rail bar is mainly to identify the target through image processing method.

[0007] The domestic patent "Railway fortification identification method and device based on aerial images" proposes to use histogram equalization processing, segment the color image after histogram equalization processing, convert the color image into a binary image, and obtain the pixel point set of the railway fortification in the color image; cluster the pixel points to obtain the image center point of the pixel point set of each railway fortification in the color image; determine the railway fortification closest to the image center point as the target railway fortification through a preset distance algorithm. This method identifies the target railway fortification through a series of image transformation and clustering operations, and can only calculate the position of the railway fortification, but cannot give the pixel coordinates of the key points on the surface of the railway fortification.

[0008] In the field of visual measurement, distance information is generally obtained through binocular camera calibration, binocular matching and other steps to calculate depth information. Summary of the invention

[0009] The invention proposes a method for detecting and estimating the height of a UAV rail barrier based on image segmentation, which can realize the rapid detection, identification and height estimation of the rail parts of the rail barrier by an unmanned obstacle-breaking or strike platform, improve the aiming accuracy of key target points and calculate the distance between platform targets, so as to achieve optimal strike.

[0010] To achieve the above object, the present invention adopts the following technical solutions:

[0011] A method for detecting and estimating the height of a UAV track based on image segmentation is applied to an unmanned platform, based on a monocular visible light camera and a small computing device communication connection, wherein:

[0012] The unmanned platform is used to install a monocular visible light camera and a small computing device to detect and identify rail fort targets;

[0013] The visible light camera is used to capture visible light images of the target area and output visible light video streams;

[0014] The small computing device receives the visible light camera video and runs the target key point detection algorithm and the height estimation algorithm;

[0015] The high-speed target key point detection algorithm realizes real-time detection of target key points in the video stream, outputs the pixel coordinates of the target key points, and simultaneously outputs the pixel length, pixel width and pixel position of the target rectangular bounding box;

[0016] The height estimation algorithm calculates the distance between the monocular visible light camera and the top surface of the rail fortress according to the pixel coordinates of the target key points, that is, the height data of the unmanned system from the top surface of the rail fortress.

[0017] Furthermore, the high-speed target key point detection algorithm specifically includes the following steps:

[0018] After reading a frame of image in the video stream, it first enters the preprocessing module, which uses median filtering and Gaussian filtering to reduce the noise in the image and slightly smooth the edges, and then performs color space conversion to enhance the pixels where the rusty rails are located, which is convenient for subsequent processing;

[0019] The frame image frame after denoising and enhancement enters the feature point detection module, and outputs the pixel coordinates of two feature points kp1 and kp2; the specific process is: frame is binarized to obtain thresh_image; then thresh_image is processed by mathematical morphology to eliminate the interference of pixels far away from the rails, and the pixels inside the rails are connected to obtain a new temp_image; temp_image obtains the convex set pixel points of the candidate convex hull through contour search as candidate feature points feature_points;

[0020] The temp_image image contains two straight lines with a certain angle, which are the two long sides of the railroad. Using this feature, find the pixel coordinates (x crs ,y crs );

[0021] The candidate feature point set feature_points, combined with the pixel coordinates of cross_point output by the line detection module, selects the best feature points kp1 and kp2 as key points, with coordinates (x1, y1) and (x2, y2) respectively;

[0022] Using the Euclidean distance formula through (x1, y1) and (x2, y2) Find the pixel distance d between two key points pix ;

[0023] After setting the distance on the perpendicular bisector of the line connecting the key points kp1 and kp2, the coordinates (x3, y3) of the center point kp3 on the upper surface of the rail bar are calculated;

[0024] Finally, the post-processing module is used to output the pixel coordinates of the three key points and the estimated height values.

[0025] Furthermore, the height estimation algorithm specifically includes:

[0026] The height is estimated using the pinhole imaging model. The height estimation formula is as follows:

[0027]

[0028] Where h is the height, track_width is the track width, f is the focal length of the lens, a is the pixel size of the visible light COMS sensor, and d is pixis the pixel distance between two key points.

[0029] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0030] It can be seen from the above technical scheme that the UAV track barrier detection and height estimation method based on image segmentation of the present invention, on the basis of existing unmanned obstacle breaking or attacking platforms such as UAVs, adds a monocular visible light camera, a small computing device, a high-speed target key point detection algorithm based on image segmentation and a height estimation algorithm, so as to realize the rapid detection and identification of the position and key points of the track barrier target by the UAV platform, complete the height estimation of the UAV from the upper surface of the track barrier, realize the rough ranging of the monocular camera, and realize the hovering and aiming of the unmanned platform within the specified height range at a relatively low cost.

[0031] Compared with the prior art, the present invention has the significant advantage that it constructs a target key point detection algorithm and a height estimation algorithm based on image segmentation. The key point detection algorithm can not only detect the position of the target rail in real time, but also detect two key points on the upper surface of the rail and calculate the pixel coordinates of the midpoint of the upper surface of the rail; based on the key point coordinates, only a monocular visible light camera is needed to achieve the distance measurement of the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the composition and data processing of the device of the present invention;

[0033] Figure 2 Flowchart of high-speed rail key point recognition and height estimation based on image segmentation;

[0034] Figure 3 Key point detection and height estimation algorithm flow based on image segmentation;

[0035] Figure 4 Key points and height estimation results of rail fortress;

[0036] Figure 5 Schematic diagram of height estimation by pinhole imaging;

[0037] Figure 6 Schematic diagram of key point 3 (midpoint). DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0039] like Figure 1 As shown, the UAV track barrier detection and height estimation method based on image segmentation described in this embodiment, based on existing unmanned obstacle-breaking or strike platforms such as UAVs, adds a monocular visible light camera, a small computing device, a high-speed target key point detection algorithm based on image segmentation, and a height estimation algorithm, so as to realize the UAV platform to quickly detect and identify the position and key points of the track barrier target, complete the height estimation of the UAV from the upper surface of the track barrier, realize the rough ranging of the monocular camera, and realize the hovering and aiming of the unmanned platform within the specified height range at a relatively low cost.

[0040] The monocular camera is used to obtain visible light images of the target and background in real time;

[0041] The small computing device is used to receive real-time images captured by the monocular camera through the CSI video interface, run the target key point detection algorithm and the height estimation algorithm, and connect with the UAV flight control system;

[0042] The high-speed target key point detection algorithm based on image segmentation processes the real-time image, detects the rail part of the rail bar from the image and calculates the two corner points where the root of the rail bar intersects with the upper surface of the rail bar as the pixel coordinates of the key points;

[0043] The key point recognition algorithm adopts an image segmentation algorithm including preprocessing, edge detection, binarization, contour processing, connected domain analysis, feature point extraction, straight line detection and analysis;

[0044] The key point recognition algorithm can simultaneously perform multiple tasks of rail track area detection and target key detection;

[0045] The key point recognition algorithm can run in real time on the selected small computing device with a frame rate ≥ 25PFS (frames / s).

[0046] Among them, the height estimation algorithm based on the monocular camera does not require binocular cameras, laser radars, ultrasonic and other ranging hardware devices to provide additional information; the height estimation algorithm uses the rail width data specified in the rail fort preparation standard, and obtains the height estimation data by converting the ratio of the rail pixel width in the image to the physical width specified in the standard; the height estimation algorithm can calculate the height of the camera from the top surface of the rail fort target in real time. Specifically:

[0047] Under the condition of known track physical width and pixel width, the proportional relationship between image coordinates and physical coordinates and the distance between the camera sensor plane and the top surface of the target are estimated by using proportional transformation. The actual area of ​​the top surface of the track fort can be calculated by using national standards and other conditions; the pixel coordinates of the four corner points of the top surface of the track fort can be provided by using a high-speed target key point detection algorithm based on image segmentation;

[0048] The height estimation algorithm uses the pixel coordinates of two key points on the top surface of the rail fort to calculate the physical size corresponding to the pixel spacing; the height estimation algorithm uses the pinhole imaging principle. The focal length image of the optical system and the pixel size of the CMOS sensor are known. The pixel distance of the two key points is calculated after image segmentation, and the distance between the camera sensor plane and the upper surface of the rail fort can be estimated; according to the pixel coordinates of the two key points and the corresponding physical dimensions, the pixel coordinates of the midpoint of the upper surface of the rail fort can be calculated, so the algorithm can output the pixel coordinates of three key points; the camera is installed on the unmanned system, and the distance between the camera sensor and the target can be used as the height of the unmanned system from the target for use by the flight control system.

[0049] The following are detailed descriptions:

[0050] like Figure 2 As shown in the figure, the high-speed rail key point detection algorithm based on image segmentation:

[0051] The algorithm includes a preprocessing module, a feature point detection module, a straight line detection module, a key point extraction module, a key point distance calculation module, a center point calculation module, a height estimation module and a post-processing module.

[0052] 1.1 Preprocessing module

[0053] After the algorithm reads a frame of image in the video stream, it first enters the preprocessing module. The preprocessing module reduces the noise in the image through median filtering and Gaussian filtering, slightly smoothes the edges, and then performs color space conversion to enhance the pixels where the rusty rails are located, which is convenient for subsequent processing.

[0054] 1.2 Feature Point Detection Module

[0055] The frame image frame after denoising and enhancement enters the feature point detection module, and outputs the pixel coordinates of two feature points kp1 and kp2. The specific process is: binarize the frame to obtain thresh_image; then the thresh_image is processed by mathematical morphology to eliminate the interference of pixels far away from the rails, and the pixels inside the rails are connected to obtain a new temp_image; the temp_image is searched by contour to obtain the convex set pixel points of the candidate convex hull as candidate feature points feature_points.

[0056] 1.3 Linear detection module

[0057] The temp_image image contains two straight lines with a certain angle, which are the two long sides of the railroad. Using this feature, we can find the pixel coordinates (x crs ,y crs ).

[0058] 1.4 Key Point Extraction Module

[0059] The candidate feature point set feature_points, combined with the pixel coordinates of cross_point output by the line detection module, can screen out the best feature points kp1 and kp2 as key points, with coordinates (x1, y1) and (x2, y2) respectively.

[0060] 1.5 Key point distance calculation module

[0061] Using the Euclidean distance formula through (x1, y1) and (x2, y2) The pixel distance d between two key points can be calculated pix .

[0062] 1.6 Center point calculation module

[0063] After setting the distance on the perpendicular bisector of the line connecting the key points kp1 and kp2, the coordinates (x3, y3) of the center point kp3 on the upper surface of the rail bar can be calculated.

[0064] 1.7 Post-processing module

[0065] The post-processing module is used to output the pixel coordinates of three key points and the estimated height values.

[0066] 2 Height estimation algorithm, such as Figure 3 As shown,

[0067] The height is estimated using the pinhole imaging model. The height estimation formula is as follows:

[0068]

[0069] Where h is the height, track_width is the track width, f is the focal length of the lens, a is the pixel size of the visible light COMS sensor, and d is pix is the pixel distance between two key points.

[0070] The pseudo code is as follows:

[0071]

[0072]

[0073] Combination Figures 4 to 6It can be seen that the UAV track barrier detection and height estimation method based on image segmentation in the embodiment of the present invention, on the basis of existing unmanned obstacle breaking or attacking platforms such as UAVs, adds a monocular visible light camera, a small computing device, a high-speed target key point detection algorithm based on image segmentation, and a height estimation algorithm, so as to realize the rapid detection and identification of the position and key points of the track barrier target by the UAV platform, complete the estimation of the height of the UAV from the upper surface of the track barrier, realize the rough ranging of the monocular camera, and realize the hovering and aiming of the unmanned platform within the specified height range at a relatively low cost.

[0074] Compared with the prior art, the present invention constructs a target key point detection algorithm and a height estimation algorithm based on image segmentation. The key point detection algorithm can not only detect the position of the target rail in real time, but also detect two key points on the upper surface of the rail and calculate the pixel coordinates of the midpoint of the upper surface of the rail; based on the key point coordinates, only a monocular visible light camera is needed to achieve the distance measurement of the target.

[0075] In yet another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of any of the above methods.

[0076] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any of the above methods.

[0077] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the steps of any method in the above embodiments.

[0078] It is understandable that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the above method.

[0079] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0080] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0081] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting and estimating the height of a UAV track based on image segmentation, applied to an unmanned platform, based on a monocular visible light camera and a small computing device communication connection, characterized in that: The unmanned platform is used to install a monocular visible light camera and a small computing device to detect and identify rail fort targets; The visible light camera is used to capture visible light images of the target area and output visible light video streams; The small computing device receives the visible light camera video and runs the target key point detection algorithm and the height estimation algorithm; The target key point detection algorithm realizes real-time detection of target key points in the video stream, outputs the pixel coordinates of the target key points, and simultaneously outputs the pixel length, pixel width and pixel position of the target rectangular bounding box; The height estimation algorithm calculates the distance between the monocular visible light camera and the top surface of the rail fortress according to the pixel coordinates of the target key points, that is, the height data of the unmanned system from the top surface of the rail fortress; The target key point detection algorithm specifically includes the following steps: After reading a frame of image in the video stream, it first enters the preprocessing module, which uses median filtering and Gaussian filtering to reduce the noise in the image and slightly smooth the edges, and then performs color space conversion to enhance the pixels where the rusty rails are located, which is convenient for subsequent processing; The frame image after denoising and enhancement enters the feature point detection module and outputs 2 feature points and The specific process is as follows: binarize the frame to obtain thresh_image; then thresh_image eliminates the interference of pixels far away from the rails through mathematical morphology processing, and connects the pixels inside the rails to obtain a new temp_image; temp_image obtains the convex set pixel points of the candidate convex hull through contour search as candidate feature points feature_points; The temp_image image contains two straight lines with a certain angle, which are the two long sides of the railroad. Using this feature, find the pixel coordinates of the intersection of these two long sides, cross_point ; The candidate feature point set feature_points, combined with the pixel coordinates of cross_point output by the line detection module, selects the best feature points kp1 and kp2 as key points, with coordinates respectively and ; pass and Using the Euclidean distance formula , find the pixel distance between two key points ; After setting the distance on the perpendicular bisector of the line connecting the key points kp1 and kp2, the coordinates of the center point kp3 on the upper surface of the rail bar are calculated. ; Finally, the post-processing module is used to output the pixel coordinates of the three key points and the estimated height values; The height estimation algorithm specifically includes: The height is estimated using the pinhole imaging model. The height estimation formula is as follows: In the formula, is the height, is the track width, is the focal length of the lens, is the pixel size of the visible light COMS sensor, is the pixel distance between two key points.

2. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to claim 1 .

Images