System and method for extracting surface markers for aircraft navigation

Abstract

A method comprising capturing a color image of a landing point including a landing surface marker with a vehicle vision sensor; converting the color image to a grayscale image; and performing multi-scale binarization to detect multiple edges of the grayscale image and produce a binary image. The method determines the contours of the edges of the binary image having closed shapes, detects the closed shapes of the contours of the edges having four corners, and verifies whether the four-sided candidate contours are valid as potential landing surface markers. If more than one contour is associated with a valid ID in a surface marker library, the contour within the minimum window size is selected. If multiple contours with the same window size can be associated with a valid ID, the average of the corresponding angles of the multiple contours is calculated. The method then performs corner refinement of the valid quadrilateral candidate contours identified as potential landing surface markers.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Indian Provisional Patent Application No. 202311064467 of the same title filed on September 26, 2023, the entire contents of which are incorporated herein by reference. Background Art

[0003] Unmanned Aerial System (UAS) applications in the National Airspace (NAS) include delivery, sampling, surveillance, and passenger transportation. UAS flights can be divided into various segments or phases, including takeoff, ascent to cruise, cruise, descent to landing, and landing. The cruise segment may include a mission segment in which the UAS hovers or performs other tasks. UAS landing sites include assigned landing zones, such as vertical lift airports on the ground, on top of a building, or on a mobile vehicle (e.g., a truck or ship).

[0004] UAS navigation systems must meet stringent requirements to enable autonomous landings at sites such as vertical lift airports to support these applications. These navigation systems must operate in both global navigation satellite system (GNSS) available and GNSS denied environments. In addition, some UAS operators require vertical lift airports to be unpowered or provide minimal power, so the ground or surface infrastructure supporting the navigation system cannot include powered features or GNSS ground stations. In the following, the term "surface" will refer to vertical lift airports located on the ground, or on top of a building, or on a mobile vehicle. Therefore, UAS navigation systems require sensors that replace GNSS to help navigation systems in landing areas that rely on unpowered features.

[0005] A camera mounted on a UAS is one such sensor that can be used in combination with Aruco codes on the surface to aid the UAS navigation system during the landing phase. Aruco codes are not powered and can be located within the final approach and takeoff (FATO) area of ​​a vertiport. These codes provide the onboard camera with known surface features to calculate measurements for the navigation system. Summary of the invention

[0006] A method, the method comprising capturing a color image of a landing site including one or more landing surface markings with a visual sensor; converting the color image to a grayscale image; and performing multi-scale binarization to detect multiple edges of the grayscale image and generate multiple binary images with different sizes based on the grayscale image. Multiple windows of different sizes are used to detect edges in the grayscale image, and within a given window, an adaptive threshold is used to slide the window across the grayscale image to generate a complete edge map for the given window. The method also includes determining the contours of the edges of the binary image with closed shapes; and detecting any closed shape of the contours of the edges with four corners by a process including the following steps: selecting the contours of the edges that exceed a user-selected perimeter size threshold; retaining the selected contours with four sides; determining the minimum angular separation of the contours with four sides; eliminating any contours whose distance to the edge of the corresponding binary image is within a user-selected threshold; and retaining any remaining contours as four-sided candidate contours. The method also includes verifying whether the four-sided candidate contour is valid as a potential landing surface marker by a process including the following steps: warping each four-sided candidate contour into a fixed-size square contour, testing for sufficient variance of the image intensity values ​​by calculating the standard deviation of the image intensity values, and if the standard deviation of the image intensity values ​​is above a user-selected threshold, converting the fixed-size square contour into a binary image to determine a candidate bit pattern within the contour. If the encoded bit error corresponding to the boundary of each candidate bit pattern is below the user-selected threshold, the candidate bit pattern is accepted for further processing. The method determines whether the bits of the candidate bit pattern other than its boundary bits match any standard bit pattern within a surface marker library for a given landing point at a vertical lift airport, and if there is a match within the user-selected bit error threshold, the candidate bit pattern is accepted as a valid ID. If more than one contour is associated with a valid ID within the surface marker library, the method selects the contour within the smallest window size and ignores all other contours. If multiple contours within the same window size can be associated with a valid ID within the surface marker library, the method calculates an average of the corresponding angles of the multiple contours. The method also performs corner refinement of valid four-sided candidate contours identified as potential landing surface markers by a process that includes moving the corners of each four-sided candidate contour to a sharpest transition point using local gradients. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Features of the present invention will become apparent to those skilled in the art through the following description with reference to the accompanying drawings. Understanding that the drawings depict only typical embodiments and are not therefore to be considered limiting of the scope of the invention, the invention will be described with additional specificity and detail through the use of the accompanying drawings, in which:

[0008] Figure 1A and Figure 1Bis a flow chart of a method for extracting surface markings for use in navigation of a vehicle according to an example implementation;

[0009] Figure 2 is a block diagram of a system for extracting surface markers for use in navigation of a vehicle according to an exemplary embodiment;

[0010] Figure 3A and Figure 3B is a flow chart of a method for extracting surface markers according to an exemplary embodiment, illustrating hierarchical function calls for performing the method;

[0011] 4A to 4D Depicted by Figure 3A Example images and outputs produced by exemplary process operations in the method for extracting surface markers;

[0012] Figure 5 Depicted by Figure 3A Example images and outputs produced by another exemplary process operation in a method for extracting surface markers;

[0013] FIG. 6A to FIG. 6C Depicted by Figure 3A and Figure 3B Example images and outputs produced by another exemplary process operation in a method for extracting surface markers; and

[0014] Figure 7 Depicted from Figure 3A and Figure 3B Example images of various exemplary operations in a method for extracting surface markers. DETAILED DESCRIPTION

[0015] In the following specific embodiments, the embodiments are fully described to enable those skilled in the art to practice the present invention. It should be understood that other embodiments may be utilized without departing from the scope of the present invention. Therefore, the following detailed description should not be considered to have a limiting meaning.

[0016] This article describes a system and method for extracting surface landmarks for aircraft navigation during takeoff and landing. The system and method can be used in various flying vehicles, such as UAS, unmanned aerial vehicles (UAV), urban air mobility (UAM) vehicles, next generation autonomous vehicles, or the like.

[0017] The present method uses a technique to robustly extract surface markers (such as Aruco codes) from vehicle camera images as the flying vehicle descends during a landing segment, where the surface markers are extracted without the use of additional sensors (such as an inertial measurement unit (IMU) or other auxiliary sensors. In addition, the present technique can potentially be used for vision-based sensors that certify UAS landings at vertiports, as the integrity of the Aruco codes extracted from the images can be ensured.

[0018] The Aruco code is a binary square fiducial marker that includes a black border and an internal binary matrix (white squares and an internal binary matrix of black squares or bits) for identifying the marker. A single known marker position can help the navigation system uniquely determine the position of the camera, and therefore the vehicle to which the camera is attached, relative to the marker. If the position of the marker is known within the final approach and takeoff (FATO) area, the navigation system can then determine the position of the camera relative to the FATO area.

[0019] There are several standard non-machine learning techniques to extract corners and edges of Aruco codes from camera images. These standard techniques include six general steps: Step 1: Convert the camera image to a grayscale image; Step 2: Use one or more windows of different sizes to find shape edges in the grayscale image using standard edge detection techniques; Step 3: Use edges to find contours; Step 4: Use multiple threshold tests to trim contours; Step 5: Formulate potential Aruco codes based on the remaining contours; Step 6: Finally refine the Aruco code (e.g., 5×5, 6×6, or 7×7 bits) and compare it to a library of Aruco codes for a given vertical lift airport; Step 6a: If the minimum number of bits of the extracted Aruco code matches one of the Aruco code bits in the library defined by a user-specified threshold, the extracted Aruco code is considered to match the Aruco code in the library of the FATO region. The limitation of this approach is that it depends on the technique used to find shape edges in the grayscale image.

[0020] In the case of not relying on different sensors or measurements from previous moments, the present method enhances the robustness of extracting Aruco codes from images. For example, in step 2 above, instead of using standard techniques to identify shape edges from grayscale images, multiple techniques and scales (window sizes) are used to find shape edges in images. The entire contents of steps 3 and 4 above are completed using multiple techniques and window sizes. At the end of step 4, the remaining contours from each of the techniques and window sizes are compared with each other. For the contours considered in step 5 above, two or more contours tested by thresholds from different techniques and window sizes must exist in the same pixel area of ​​the camera image. Using multiple techniques and window sizes to identify contours in a single image at one measurement moment provides different ways to interpret information from the same image. By requiring confirmation from multiple techniques or window sizes to identify contours or regions of interest, the robustness of extracting Aruco codes from images is increased.

[0021] In one embodiment of the present method, a color input image is converted to a grayscale image. The threshold of the window is used to find edges by sliding windows of multiple sizes on the grayscale image, and an adaptive average threshold is performed to produce a binary image. The method then finds contours in the binary image, where contours are edges that form closed shapes. The contours are then pruned because small contours do not contain enough information to determine the identity of the marker, and contours that cannot be restricted to four-sided polygons are removed. Once a four-sided polygon is found, it needs to be large enough to support recognition. For example, the minimum side of the polygon should be greater than the number of pixels selected by the user, such as 10 pixels, because this eliminates the square that appears squeezed due to the perspective viewpoint. In addition, contours close to the edge of the image are eliminated because the corners of the contours should have a sufficient pixel distance from the edge of the image. For example, a user-selected pixel edge can be used, such as 30 pixels from the edge of the image. Pixels near the image boundary usually have some residual distortion, and using contours with corners in these areas can lead to erroneous results.

[0022] Next, during landmark recognition, each square candidate contour from the previous step is warped into a user-selected pixel grid, such as a 28×28 grid, and binarized. If the standard deviation of the contour's interior pixels is greater than a user-selected threshold, landmark recognition is performed. Bits are encoded using the majority of the user-selected interior pixels (such as the interior 4 pixels in each contour). The interior pixels are selected as image distortions, and interpolation may make the exterior pixels noisy. If a majority of the interior pixels (such as three or more interior pixels) are white in the threshold image, they are encoded as 1, otherwise they are encoded as 0. If, for example, less than a user-selected threshold (such as 33%) of the boundary pixels are error-free (black), the method proceeds further.

[0023] Thereafter, the bit pattern of the candidate contour is matched with each known code pattern in the Aruco code library of the vertical lift airport. The nearest matching ID from the Aruco code library is found based on the number of bits of the match that exceeds the threshold selected by the user. For example, for the 5x5 bit pattern of the contour to be declared as a valid ID, at least 24 of the 25 bits must match for robust tag association. If the bit pattern from multiple contour window sizes is associated with the same ID in the Aruco code library, only those contours with the smallest window size are retained, resulting in more accurate marking edges, especially at higher vehicle altitudes. If the bit pattern from a contour window size is associated with the same ID in the Aruco code library multiple times, the contour with the largest perimeter is retained or the average value of the associated contour is used. Contours with bit patterns that do not match the code pattern in the Aruco code library of the vertical lift airport are rejected (no spatial relationship is used). Additional tests are also performed on the contour. First, if the contour has an average edge pixel length greater than a user-selected threshold (e.g., 50 pixels), the contour is retained. Second, if a contour has an average edge length less than a user-selected threshold (e.g., 20 pixels), the contour is rejected. For contours with an average edge length between two user-selected thresholds (e.g., user-selected thresholds of 20 and 50 pixels), the contour with the largest perimeter or largest size is retained. Contour corner refinement is performed by moving the corners to the sharpest transition points using local gradients.

[0024] In another embodiment, the method determines whether the detected ID within the contour is in a list of valid IDs within the Aruco code library for a given landing point at the vertical lift airport. If the detected ID is in the list of valid IDs, the contour with the detected ID is considered valid for further processing. If the detected ID is not in the list of valid IDs, the contour with the detected ID is ignored and the method proceeds to the next contour with the detected ID. If the valid detected ID has a contour size above a first user-selected threshold, the contour with the valid detected ID is accepted for further processing. If the valid detected ID has a contour size below a second user-selected threshold, the contour with the valid detected ID is rejected. For any contour with a valid detected ID having a contour size between the first user-selected threshold and the second user-selected threshold, the contour with the valid detected ID of the largest size is accepted and all other valid IDs are rejected.

[0025] Additionally, when one or more nearby landing sites have common landmarks, the combination of profiles with valid detected IDs may be used to determine the correct landing site at a vertical takeoff and landing airport with multiple landing sites to avoid landing site ambiguity.

[0026] Additional details regarding the present method and system are described below and with reference to the accompanying figures.

[0027] Figure 1A and Figure 1B A flow chart of a method 100 for extracting surface markers for aircraft navigation according to an exemplary embodiment is depicted. First, as Figure 1A As shown, the method 100 captures a color image of a landing site including one or more landing surface markings, such as using a visual sensor on a vehicle (box 110). The method 100 converts the color image into a grayscale image, such as by using a standard conversion technique (box 112). The method 100 then performs multi-scale binarization to detect multiple edges of the grayscale image, and generates multiple binary images with different sizes based on the grayscale image (box 114). This can be done by using multiple windows of different sizes to detect edges in the grayscale image (box 116); and within a given window, sliding the window across the grayscale image using an adaptive threshold to generate a complete edge map for the given window (box 118).

[0028] Next, the method 100 determines the contours of the edges of the binary image having closed shapes (box 120). The method 100 detects any closed shapes of the contours of the edges having four corners (box 122) by a process including the following steps: selecting contours of the edges exceeding a user-selected perimeter size threshold (box 124); retaining the selected contours having four sides (box 126); determining the minimum angular separation of the contours having four sides (box 128); eliminating any contours whose distance to the edge of the corresponding binary image is within a user-selected threshold (box 130); and retaining any remaining contours as four-sided candidate contours (box 132).

[0029] like Figure 1B As shown, the method 100 then verifies whether the four-sided candidate contour is valid as a potential landing surface marker (box 134) by a process including the following steps: warping each four-sided candidate contour into a fixed-size square contour, testing for sufficient variance in the image intensity values ​​by calculating the standard deviation of the image intensity values ​​(box 136); if the standard deviation of the image intensity values ​​is above a user-selected threshold, converting the fixed-size square contour to a binary image to determine a candidate bit pattern within the contour (box 138); if the encoded bit error corresponding to the boundary of each candidate bit pattern is below a user-selected threshold, accepting the candidate bit pattern for further processing (box 140); and determining whether bits of the candidate bit pattern other than its boundary bits match any standard bit pattern within a library of surface markers for a given landing site at the vertiport, and if there is a match within a user-selected bit error threshold, accepting the candidate bit pattern as a valid ID (box 142).

[0030] If the method 100 determines that more than one contour is associated with a valid ID in the Aruco code library, the method 100 selects the contour with the smallest window size and ignores all other contours (box 144). If the method 100 determines that multiple contours with the same window size can be associated with a valid ID in the Aruco code library, the method 100 calculates the average of the corresponding angles of the multiple contours (box 146). The method 100 then performs corner refinement of the valid four-sided candidate contours identified as potential landing surface markers (box 148). This can be accomplished by using a local gradient to move the corners of each four-sided candidate contour to the sharpest transition point (box 150).

[0031] Figure 2 2 is a block diagram of a system 200 for extracting surface markers for use in navigation of a vehicle 202, according to an exemplary embodiment. The vehicle 202 may be a UAM vehicle, a UAS, a UAV, or the like. The vehicle 202 has at least one visual sensor 204 mounted thereon, such as one or more cameras, which operate to capture color images of a surface area. The vehicle 202 includes at least one processor 206 operatively coupled to the visual sensor 204. The processor 206 hosts a collection of program modules that are operable to extract landing site surface marker information for use in navigation of the vehicle 202.

[0032] The processor 206 includes program instructions executable by the processor 206 to execute a method 210 for executing a high-level software function call, which will be described in further detail below, and which implements the current method for extracting surface markers. Initially, the method 210 begins with a main function module 212 calling a detection marker function module 214, which provides the top-level function call for surface marker detection (e.g., defined in aruco.cpp available from the free online library OpenCV). All subsequent functions for marker detection are called from the detection marker function module 214. The main function module 212 provides a color input image to the detection marker function module 214, which converts the input image into a grayscale image.

[0033] The detect tag candidate function module 216 is the first function called from the detect tag function module 214, and receives the grayscale image from the detect tag function module 214. The detect tag candidate function module 216 returns the contours and corresponding angles that meet the user-selected criteria about the number of corners and contour length, which are possible candidates for tags. For example, a multi-scale binarization process can be used to detect multiple potential contour boundaries or edges of a grayscale image. The process produces multiple different edges within the same grayscale image. This can be done in the following ways: using multiple windows of different sizes to detect edges in the grayscale image, and within a given window, using an adaptive threshold to produce a complete edge map of the image using a selected window when the window slides across the grayscale image. Thereafter, the process identifies the contours of the edges of the binary image with closed shapes, detects closed shapes of contours with four corners, and keeps those contours as candidate contours that pass the threshold test selected by the user.

[0034] Identification mark candidate function module 218 obtains the outline and angle detected from detection mark candidate function module 216, and checks the validity of possible mark valid ID (for example, Aruco code mark) in Aruco code library. Identification mark candidate function module 218 returns those outlines and angles corresponding to valid ID, and also returns corresponding ID number. Since different window sizes can be used to detect marks, this can result in multiple outlines with the same mark ID. Although most of the nearby outlines are filtered out in detection mark candidate function module 216, some of these outlines still exist and are not filtered out. Therefore, filtering detection mark function module 220 is used to filter out multiple detections for the same mark, and retains the mark detected using the minimum window size for a given ID. In some cases, the detected angle may have some errors and is not aligned with the actual mark angle. Corner refinement function module 222 (for example, CornerSubPix available from free online library OpenCV) can be used to refine the detected angle and align it as closely as possible with the actual mark angle.

[0035] Once a marker with a valid identification (ID) is detected by the above functions, a profile angle (in meters) function module 224 is used to map the angle detected for each valid ID to a corresponding 3D coordinate in the marker coordinates located at the center of the marker, such as the North East Earth (NED) system. For example, each angle can be associated with a corresponding position from the center of the marker in the NED system. A map angle to ground truth (GT) function module 226 then combines all valid IDs and corresponding 3D coordinates into two separate vectors, one of which corresponds to pixel coordinates and the other corresponds to 3D coordinates in the locally defined NED system. This function is used to pass the angle and corresponding GT to the pose estimation algorithm. An estimate kinematics single marker function module 228 takes the pixel coordinates and 3D coordinate vectors from the mapped angles to the GT function module 226 to determine the camera kinematics relative to the valid ID. These kinematics include the position and angular orientation of the camera relative to the valid ID. Depending on the method selected by the user to estimate the relative camera kinematics relative to the effective ID and therefore relative to the surface, the estimate kinematics single marker function module 228 can use solvePnP (or solvePnPRansac) (available from the free online repository OpenCV). When the extraction of the surface marker information including the relative camera kinematics estimate is completed, the information is sent to the onboard vehicle navigation system 232 to help provide guidance used during takeoff or landing of the vehicle 202.

[0036] Various aspects of the method are described in further detail below.

[0037] Figure 3A and Figure 3B 300, which shows the details of the hierarchical function calls of the present surface marker detection method. The main function 302 provides the color input image to the detect marker function 304, which is the top-level function call for surface marker detection. The detect marker function 304 is called from within the main function 302 and calls other sub-functions to return the detected corners, contours and surface marker IDs to the main function 302.

[0038] The detect candidates function 306 is the first function called from the detect markers function 304. The detect candidates function 306 returns contours and corresponding corners that meet user-selected criteria regarding the number of corners and contour length, which identify possible candidates for surface markers. The detect candidates function 306 receives a grayscale image converted from a color input image and calls the detect initial candidates function 308 with various user-selected parameters as input.

[0039] In the first subprocess, the detect initial candidates function 308 checks the size of the input candidate against a user selected threshold and calls the detect initial candidates parallel function 310 when the condition is met. The detect initial candidates parallel function 310 uses OpenCV parallel calls (using parallel_for_(Range(0, nScales)) from the free online library OpenCV to execute the function simultaneously with different user selected window scales (e.g., 3, 5, 7, 9, 11, 13, 15 for a maximum window size of 15(nScales)) to speed up processing. The detect initial candidates parallel function 310 for each window size calls a threshold function 312 on the grayscale image with the selected window scale and an adaptive, user selected threshold value, which is nominally set to a default value (e.g., 7 is the default value of the free online library OpenCV). The threshold function 312 calls the OpenCV adaptiveThreshold function with the grayscale image, the window size, and the adaptive user selected threshold constant. The threshold function 312 returns a binary thresholded image of the selected window size that meets the user selected threshold. These images will be referred to as thresholded images hereinafter.

[0040] The thresholded image using windows of different sizes from the threshold function 312 serves as the input to the find marker contour function 314 for finding valid contours and corners. The find marker contour function 314 obtains the threshold image and calculates the minimum contour length and the maximum contour length. The find marker contour function 314 then calls the find contour function 316 (from the free online library OpenCV) for detecting contours within the thresholded image. Once the contours are detected, each contour is checked against the threshold selected by the user, and the contour length within the threshold selected by the user is passed to the following function for further processing.

[0041] All contours that meet the above user selection criteria are passed to approxPolyDP function 318 (from the idle online library OpenCV), which obtains the detected contours and tunable, user-selected precision parameters. Based on the precision parameters, approxPolyDP function 318 attempts to fit a polygon with the number of edges selected by the user to each contour. Since this method is only interested in 4-sided polygons with four corners (because they can be potential candidates for valid IDs in the Aruco code library), this method checks polygons approximated using four corners. After meeting the user-selected criteria, the corners are checked for the minimum pixel interval selected by the user between any two corners. If the contour meets the threshold test, all four corners of the contour are checked against the minimum distance selected by the user to the boundary pixels of the image. If the corners of the contour are within the minimum distance to the boundary pixel threshold and close to the image boundary, the detected contour corners will have more errors than the detected contours that exceed the minimum distance to the boundary threshold. If a contour satisfies all of the above user selected threshold tests by the find labeled contours function 314 and the detect initial candidates parallel function 310, it is returned as a possible contour candidate (valid polygon) to the detect initial candidates function 308. The window size of all contour candidates is also returned.

[0042] 4A to 4D Various example images and outputs resulting from exemplary operation of the first sub-process under the detect candidates function 306 are shown. Note that all outputs shown here are for a single window.

[0043] Figure 4A 410 is an example of a grayscale image 410 converted from a color input image. Grayscale image 410 shows vertical landing and take-off locations 412 with various Aruco codes 414, such as at a landing point at a vertical lift airport for a UAS. Grayscale image 410 is passed to detection candidate function 306. As described above, detection candidate function 306 calls detection initial candidate function 308, which in turn calls detection initial candidate parallel function 310, which provides a parallel implementation for contour detection under different window sizes. Detection initial candidate parallel function 310 simultaneously calls threshold function 312 and find marker contour function 314 at different window sizes to accelerate execution. Threshold function 312 calls OpenCV adaptiveThreshold internally, and converts grayscale image to binary image using threshold value selected by user based on window size selected by user. Binary image thresholded for window size selected by user is output to find marker contour function 314. Figure 4B is an example of a binary threshold image 420 based on the grayscale image 410 and output from the threshold function 312 .

[0044] The find labeled contours function 314 calls the find contours function 316 and the approxPolyDP function 318 to determine valid contours. The find contours function 316 determines all possible contours in the binary thresholded image. Figure 4C An example of an output image 430 having contours detected based on a binary threshold image 420 and output from the find contours function 316 is depicted. Each detected contour is checked for a user-selected minimum perimeter pixel length and a user-selected maximum perimeter pixel length, and contours that meet these user-selected thresholds are passed to the approxPolyDP function 318, which attempts to fit a closed polygon (contour) to the identified points based on the user-selected approximation accuracy parameter passed to it. Once the approximated polygons with corners are returned by the approxPolyDP function 318, only those polygons with four corners are considered. Figure 4D is an example depiction of a set of detected polygons 440 having four corners based on the output image 430 output from the approxPolyDP function 318. A magnified view of the detected polygons 440 is shown in FIG. Figure 4D 442 are shown.

[0045] Back to Figure 3A , in the second subprocess, the detect initial candidate function 308 outputs a valid contour (candidate) with four corners to the reorder candidate corner function 320, which uses three corners (e.g., 0, 1, 2) to calculate the cross product (e.g., between 1 to 0 and 0 to 2) to determine whether the order of the corners is a clockwise sequence or a counterclockwise sequence. If the order is not clockwise, the corners (e.g., 1 and 3) are swapped to reorder the corners. Thereafter, the filter too close candidate function 322 receives the candidate contours with corners in the clockwise direction. A nested loop is executed, including an outer loop for iterating on the size of the corners (the total number of polygons with four corners), and an inner loop that starts from the next available polygon with respect to the polygon selected in the outer loop until all available polygons have been evaluated. The minimum contour length is found between two polygons (one corresponding to the outer loop and one corresponding to the inner loop). The distance of each corner of a polygon to every other corner of the other polygon is found, and then the average distance is obtained by dividing the distance by 4. If the contours are nearby, the distance will be very small and expected to belong to the same contour.

[0046] This distance is then compared to a user-selected minimum distance threshold between contours in the pixel, which is calculated using minMarkerDistanceRate as follows: (minimum contour length among the contours under consideration) * minMarkerDistanceRate (adjustable parameter selected by the user, such as 0.05). If the contours are nearby (within the minimum distance threshold selected by the user), the distance will be greater than the average distance between them previously calculated. If this happens, this contour pair and the window level of the two contours are stored for further processing to remove duplicate candidates. After finding all contour sets that are close, filtering is performed based on the window size at which the contours are detected. This function iterates over nearby contours and checks the pairs detected as being nearby. If two nearby contours are detected at different window levels, the contours and corners detected at the smaller window size are retained. In the case of detecting two contours with the same window size, the average value of the angles corresponding to the contour pair under consideration is retained. Each different contour and corner retained is returned to the detection candidate function 306.

[0047] Figure 5 Various example images and outputs produced by exemplary operation of the second subprocess under the detect candidate function 306 are shown. (Based on the detected polygon 440, Figure 4D ) depicts an example of a counterclockwise polygon 502 having counterclockwise angles, and is shown as a clockwise polygon 504 after angle reordering. Additionally, an example of a polygon 506 having multiple outlines for the same boundary and a polygon 508 having a single outline for the same boundary are also shown (based on the detected polygon 440, Figure 4D ).

[0048] Back to Figure 3A , detect candidates function 306 outputs different contours and corners to identify candidates function 326, which returns valid marker IDs and corresponding corners and candidates. Identify candidates function 326 calls an OpenCV parallel function, namely, identify candidates parallel function 328. After returning from identify candidates parallel function 328, the following items are returned to the calling function: valid IDs and corresponding contours; rejected contours that do not correspond to valid contours; and the window size where the contours are detected.

[0049] The candidate identification parallel function 328 further calls the candidate identification function 330 in parallel. The candidate identification function 330 further calls the following functions in sequence: extract bit function 332, obtain boundary error function 334, identification function 336 and rotation function 338 ( Figure 3B ).

[0050] The extract bits function 332 determines the marker size with borders (e.g., when markerSizeWithBorders = dictionary->markerSize + 2*markerBorderBits); and determines the margin (e.g., pixels around the distorted image that need to be ignored when decoding to account for the inconsistency of pixel values ​​around the border for better decoding using cellMarginRate), and determines the cell size as cellMarginRate*cellSize. The extract bits function 332 calls Figure 3B The getPerspectiveTransform function 342 (OpenCV) shown in is used to calculate the perspective transform between the contour image and a square image (e.g., 28 pixels by 28 pixels) determined by the value perspectiveRemovePixelPerCell*markerSizeWithBorders. The transform calculated from the getPerspectiveTransform function 342 is used to warp the given contour image into a square image by calling the warpPerspective function 344 (OpenCV). For a given warped image, the mean and standard deviation of the interior area of ​​the warped image are calculated (i.e., ignoring perspectiveRemovePixelPerCell / 2 pixels on both sides of the warped image as boundary pixels). If the calculated standard deviation is greater than a user-selected minimum threshold minOtsuStdDev and the mean is greater than a user-selected threshold, such as 127 (i.e., there are more white pixels than black pixels within the contour and the contour is not entirely white pixels), then the warped image is eligible for further processing. The extract bits function 332 then calls a threshold function 346 (OpenCV) where the distorted image is thresholded to obtain a binary image which is output as a binary representation of the thresholded image at 348. Then, a group of pixels is taken (i.e., perspectiveRemovePixelPerCell in x and y) and the bit value corresponding to that block is calculated based on the majority of black pixels or white pixels (e.g., 4×4 pixels), assigning 0 when the majority of pixels are black and 1 otherwise.

[0051] FIG. 6A to FIG. 6C Various example images and outputs resulting from example operations under the extract bits function 332 are shown. Fig. 6A A bit pattern 610 corresponding to a threshold image (marker bits) is depicted. Figure 6B is a silhouette image 620 (eg, a 28×28 pixel image) after warping perspective using the bit pattern 610 . Figure 6C is a binary threshold image 630 generated from the contour image 620 .

[0052] Next, the boundary error is checked against the maximum boundary threshold selected by the user. This is calculated by the extract bits function 332 calling the get boundary error function 334, which counts the number of non-zero bits along the border and returns the total count as the boundary error. It is then determined whether the boundary error value is less than the maximum boundary threshold calculated at 335 by comparing the boundary error value with the maximum boundary threshold calculated above. If the boundary error value is below the maximum boundary threshold, it is passed to the identify function 336, because these bits will contain the actual bit encoding corresponding to the marker. If the boundary error value is not below the threshold, the method returns to the identify a candidate function 330 for further processing.

[0053] The recognition function 336 is an OpenCV implementation for decoding the bits and comparing them to the bits of the decoded surface markers in the Aruco code library.

[0054] The recognition function 336 checks the detected bit pattern against all markers in the Aruco code library to find a match and returns the matching ID (the marker ID) and the rotation associated with the ID (e.g., rotating the upper left corner of the ID in a given contour in a clockwise direction). At 337 ( Figure 3B ) determines whether the detected bit pattern represents a valid ID. If not, the method returns to the identify a candidate function 330 for further processing. If the detected bit pattern does represent a valid ID, it is passed to the rotate function 338.

[0055] The rotate function 338 rotates the corners using the given rotation value to ensure that the first corner corresponds to the upper left corner of the unrotated marker and the remaining corners are in a clockwise direction. The rotate function 338 returns the detected marker ID, corner, corresponding contour, and the window level where the marker ID was detected to identify a candidate function 330. The identify candidate parallel function 328 then returns the following information to the identify candidate function 326: valid IDs and corresponding contours; rejected contours that do not correspond to valid contours; and the window size where the contour was detected.

[0056] Filter detection tag self function 350 ( Figure 3A) receives various data inputs from the recognition candidate function 326. These data inputs include the window size at the corner, ID, outline and detection tag ID. The purpose of filtering the detection tag itself function 350 is to remove the extracted outline whose ID corresponds to the same ID, because some outlines are not removed during the candidate function 322 that is too close to filter. Nested loops iterate on the number of outlines (same as the number of detected IDs or different angle sets). If two outlines have the same ID, the outline detected with a lower window size is retained (because the outline detected at a smaller window level will be closer to the actual edge of the mark) and the other outline is rejected. If two outlines with the same ID are detected with the same window size, there are two methods to select the most appropriate angle. The first method takes the average value of the outline angle. The second method takes the angle corresponding to the maximum outline size. Filtering the detection tag itself function 350 returns a unique ID and the corresponding outline and angle of a given ID.

[0057] For any ID detected, the following process occurs. At 352, it is determined whether the detected ID is valid for a given layout (i.e., the detected ID is one of the IDs defined for the given layout). If the detected ID does not exist in the layout, even if it is a valid id for the given dictionary, the detected ID is rejected and the ID, corners and contours corresponding to the ID are deleted, and no further processing is performed on the ID. If the detected ID is a valid ID for the given layout and passes the above test, the minimum edge length for the given contour is found, and if the minimum edge length is above the defined threshold (at 354), it is accepted as a valid candidate for further processing. The valid candidates (corners) are then sent to the mark sub-pixel parallel function 356.

[0058] If the minimum edge length is below the defined minimum threshold, the ID, corner, and contour are removed and no further processing is performed. In the event that the number of minimum edge pixels is between these thresholds and the mark length is below the maximum length corresponding to any of the detected candidates (at 358), the ID is rejected; otherwise, the ID is retained. If the ID passes all of the above tests and is still valid, the corner corresponding to the ID is sent for refinement by calling the mark sub-pixel parallel function 356.

[0059] The Mark Subpixel Parallel Function 356 is a parallel implementation for angle refinement to provide a better estimate of the angle, and is called in parallel for all detected angles set for each ID. The Mark Subpixel Parallel Function 356 in turn calls the Angle Subpix Function 360 (OpenCV implementation) on different detected angles for different IDs. The Angle Subpixel Function 360 operates to return a more refined and accurate angle for a given ID to the Mark Subpixel Parallel Function 356.

[0060] Figure 7 Various example images from exemplary operation under the detect tag function 304 are shown. These images are images with various Aruco codes 414 (see Figure 4A ). A first example image 710 shows identified markers (ID and rotated ID). Due to multiple contours, there are multiple IDs for the same marker. A second example image 720 shows filtered detected markers, where there is a single ID for a given marker (because multiple contours are filtered out). A third example image 730 shows identified markers before corner refinement, where a zoomed-in view of one of the markers is shown at 732 (scaled ID before refinement - blue shift). A fourth example image 740 shows identified markers after corner refinement, where a zoomed-in view of one of the markers is shown at 742 (scaled ID after refinement - green shift). After corner refinement, the corners are better aligned with the edges of the markers than before corner refinement.

[0061] The computer or processor used in the present system and method can be implemented using software, firmware, hardware, or any suitable combination thereof known to those skilled in the art. These can be supplemented or incorporated by specially designed application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). The computer or processor may also include functions implemented using software programs, firmware, or other computer readable instructions to perform various processing tasks, calculations, and control functions used in the present system and method.

[0062] The method can be implemented by computer executable instructions (such as program modules or components) executed by at least one processor. Typically, program modules include routines, programs, objects, data components, data structures, algorithms, etc. that perform specific tasks or implement specific abstract data types.

[0063] Instructions for performing various process tasks, calculations, and generation of other data used in the operation of the methods described herein may be implemented in software, firmware, or other computer-readable instructions or processor-readable instructions. These instructions are typically stored on any suitable computer program product that includes a computer-readable medium for storing computer-readable instructions or data structures. Such a computer-readable medium may be any available medium that can be accessed by a general or special-purpose computer or processor or any programmable logic device.

[0064] Suitable processor-readable media may include storage devices or memory media, such as magnetic media or optical media. For example, the storage or memory media may include conventional hard disks, optical disks, DVDs, Blu-ray disks, or other optical storage media; volatile or non-volatile media, such as random access memory (RAM); read-only memory (ROM), electrically erasable programmable ROM (EEPROM), flash memory, etc.; or any other medium that can be used to carry or store desired program code in the form of computer-executable instructions or data structures.

[0065] Example Implementation

[0066] Embodiment 1 includes a method, the method comprising: capturing a color image of a landing site including one or more landing surface markers with a visual sensor on a vehicle; converting the color image to a grayscale image; performing multi-scale binarization to detect multiple edges of the grayscale image, and generating multiple binary images with different sizes based on the grayscale image; wherein multiple windows of different sizes are used to detect edges in the grayscale image, and within a given window, an adaptive threshold is used to slide the window across the grayscale image to generate a complete edge map for the given window; determining the contours of the edges of the binary image having closed shapes; detecting any closed shape of the contours of the edges having four corners by a process including the following steps: selecting contours of edges exceeding a user-selected perimeter size threshold; retaining the selected contours with four sides; determining the minimum angular spacing of the contours with four sides; eliminating any contours whose distance to the edge of the corresponding binary image is within a user-selected threshold; and retaining any remaining contours as four-sided candidate contours; verifying whether the four-sided candidate contours are valid as potential landing surface markers by a process including the following steps: warping each four-sided candidate contour into a square contour of a fixed size, by calculating The method of claim 1, wherein the method comprises: calculating a standard deviation of the image intensity values ​​to test for sufficient variance of the image intensity values, and if the standard deviation of the image intensity values ​​is above a user-selected threshold, converting a fixed-size square contour into a binary image to determine candidate bit patterns within the contour; wherein if the encoded bit error corresponding to the boundary of each candidate bit pattern is below the user-selected threshold, the candidate bit pattern is accepted for further processing; and determining whether bits of the candidate bit pattern other than its boundary bits match any standard bit pattern within a surface marker library for a given landing point at a vertical lift airport, and if a match exists within the user-selected bit error threshold, the candidate bit pattern is accepted as a valid ID; wherein if more than one contour is associated with a valid ID within the surface marker library, the contour within the smallest window size is selected and all other contours are ignored; wherein if multiple contours of the same window size can be associated with a valid ID within the surface marker library, the corresponding angles of the multiple contours are averaged; and performing corner refinement of valid four-sided candidate contours identified as potential landing surface markers by a process including the following steps: moving the corners of each four-sided candidate contour to the sharpest transition point using a local gradient.

[0067] Embodiment 2 includes the method according to embodiment 1, which further includes: determining whether the ID detected within the contour is in the list of valid IDs in the surface marker library for a given landing point; if the detected ID is in the list of valid IDs, the contour with the detected ID is considered valid for further processing; if the detected ID is not in the list of valid IDs, ignoring the contour with the detected ID and proceeding to the next contour with the detected ID; wherein if the valid detected ID has a contour size higher than a first user selection threshold, the contour with the valid detected ID is accepted for further processing; wherein if the valid detected ID has a contour size lower than a second user selection threshold, the contour with the valid detected ID is rejected; wherein for any contour with a valid detected ID having a contour size between the first user selection threshold and the second user selection threshold, the contour with the valid detected ID of the largest size is accepted and all other valid IDs are rejected.

[0068] Embodiment 3 includes a method according to embodiment 2, wherein a combination of contours with valid detected IDs may be used to determine the correct landing point at a vertical lift airport with multiple landing points to avoid landing point ambiguity when one or more landing points have a common marker.

[0069] Embodiment 4 comprises a method according to any one of Embodiments 1 to 3, wherein the vehicle comprises an unmanned aerial system (UAS) vehicle, an unmanned aerial vehicle (UAV), or an urban air mobility (UAM) vehicle.

[0070] Embodiment 5 includes the method of any one of Embodiments 1 to 4, wherein the one or more landing surface markings include one or more Aruco codes.

[0071] Embodiment 6 includes a system comprising: at least one machine vision sensor mounted on a vehicle; at least one processor operatively coupled to the at least one vision sensor; wherein the at least one processor includes program instructions executable by the at least one processor to perform a method comprising the following steps: capturing a color image of a landing site including one or more landing surface markings with the at least one vision sensor; converting the color image to a grayscale image; performing multi-scale binarization to detect multiple edges of the grayscale image, and generating multiple binary images with different sizes based on the grayscale image; wherein multiple windows of different sizes are used to detect edges in the grayscale image, and within a given window, an adaptive threshold is used to slide the window across the grayscale image to generate a complete edge map for the given window; determining contours of edges of the binary image having closed shapes; detecting any closed shape of contours of edges having four corners by a process comprising the following steps: selecting contours of edges exceeding a user-selected perimeter size threshold; retaining the selected contours having four sides; determining a minimum angular separation of contours having four sides; eliminating any contours whose distances from the edges of the corresponding binary image are within a user-selected threshold; and retaining any remaining contours as four-side candidate contours; validating the four-side candidates by a process comprising the following steps whether the contour is valid as a potential landing surface marker: warping each four-sided candidate contour into a fixed-size square contour, testing for sufficient variance of the image intensity values ​​by computing a standard deviation of the image intensity values, and if the standard deviation of the image intensity values ​​is above a user-selected threshold, converting the fixed-size square contour into a binary image to determine candidate bit patterns within the contour; wherein if the encoded bit error corresponding to the boundary of each candidate bit pattern is below the user-selected threshold, accepting the candidate bit pattern for further processing; and determining whether bits of the candidate bit pattern other than its boundary bits match a surface marker for a given landing site at the vertiport any standard bit pattern in the library, and if there is a match within a user-selected bit error threshold, the candidate bit pattern is accepted as a valid ID; wherein if more than one contour is associated with a valid ID in the surface marker library, the contour within the smallest window size is selected and all other contours are ignored; wherein if multiple contours of the same window size can be associated with a valid ID in the surface marker library, the average of the corresponding angles of the multiple contours is calculated; and corner refinement of valid four-sided candidate contours identified as potential landing surface markers is performed by a process including the following steps: using local gradients to move the corners of each four-sided candidate contour to the sharpest transition point.

[0072] Embodiment 7 includes a system according to embodiment 6, wherein at least one processor also includes program instructions executable by at least one processor to further perform a method including the following steps: determining whether the detected ID within the contour is in a list of valid IDs within a surface marker library for a given landing point; if the detected ID is in the list of valid IDs, the contour with the detected ID is considered valid for further processing; if the detected ID is not in the list of valid IDs, ignoring the contour with the detected ID and proceeding to the next contour with the detected ID; wherein if the valid detected ID has a contour size above a first user selection threshold, the contour with the valid detected ID is accepted for further processing; wherein if the valid detected ID has a contour size below a second user selection threshold, the contour with the valid detected ID is rejected; wherein for any contour with a valid detected ID having a contour size between the first user selection threshold and the second user selection threshold, the contour with the valid detected ID of the largest size is accepted and all other valid IDs are rejected.

[0073] Embodiment 8 includes a system according to any one of Embodiments 6 to 7, wherein the vehicle includes an unmanned aerial system (UAS) vehicle, an unmanned aerial vehicle (UAV), or an urban air mobility (UAM) vehicle.

[0074] Embodiment 9 includes the system of any one of Embodiments 6 to 8, wherein the one or more landing surface markings include one or more Aruco codes

[0075] Embodiment 10 includes a system comprising: at least one machine vision sensor mounted on a vehicle; at least one processor, the at least one processor being located on the vehicle and operatively coupled to the at least one vision sensor; and a navigation system, the navigation system being located on the vehicle and operatively coupled to the at least one processor; wherein the at least one processor hosts a collection of program modules, the collection of program modules being operable to extract landing site surface marking information used in navigation of the vehicle, the program modules comprising: a marker detection function module, the marker detection function module being operative to receive an input image captured by the at least one vision sensor and convert the input image into a grayscale image; a marker detection candidate function module, the marker detection candidate function module being called from the marker detection function module and being operative to receive the grayscale image from the marker detection function module, wherein the marker detection candidate function module is operable to extract landing site surface marking information used in navigation of the vehicle, the program modules comprising: a marker detection function module, the marker detection function module being operative to receive an input image captured by the at least one vision sensor and convert the input image into a grayscale image; a marker detection candidate function module, the marker detection candidate function module being operative to receive the grayscale image from the marker detection function module, The module operates to determine contours and corresponding corners that meet user-selected criteria regarding the number of corners and contour length from a grayscale image to detect potential candidates for a marker; an identify marker candidate function module operates to receive contours and corners detected from the detect marker candidate function module, check their validity against potential marker valid IDs within a surface marker library, return those contours and corners corresponding to the valid IDs, and return the corresponding ID numbers; a filter detect marker function module operates to filter out multiple detections for the same marker and retain markers detected with a minimum window size for a given ID; and a corner refinement function module for refining the detected corners by aligning the detected corners as closely as possible with the actual marker corners; wherein the extracted landing point surface landmark information is sent to a navigation system for further processing to provide guidance for use during takeoff or landing of a vehicle.

[0076] Embodiment 11 includes the system according to embodiment 10, wherein the detect marker candidate function module is further operative to perform multi-scale binarization to detect multiple edges of the grayscale image and generate multiple different edges within the same grayscale image.

[0077] Embodiment 12 comprises the system according to embodiment 11, wherein multiple windows of different sizes are used to detect edges in a grayscale image, and within a given window, an adaptive threshold is used to slide the window across the grayscale image to produce a complete edge map of the grayscale image using the window.

[0078] Embodiment 13 includes a system according to any one of Embodiments 11 to 12, wherein the marker candidate detection function module also operates to: determine the contours of the edges of a binary image having closed shapes; detect any closed shapes having contours of edges with four corners by processing including the following steps: select contours of edges that exceed a user-selected perimeter size threshold; retain the selected contours with four sides; determine the minimum angular separation of the contours with four sides; eliminate any contours whose distance from the edge of the corresponding binary image is within a user-selected threshold; and retain any remaining contours as four-sided candidate contours.

[0079] Embodiment 14 includes a system according to embodiment 13, wherein the identification landmark candidate function module is further operative to verify whether the four-sided candidate contour is valid as a potential landing surface landmark by a process comprising the following steps: warping each four-sided candidate contour into a fixed-size square contour, testing for sufficient variance of the image intensity values ​​by calculating the standard deviation of the image intensity values, and if the standard deviation of the image intensity values ​​is above a user-selected threshold, converting the fixed-size square contour to a binary image to determine a candidate bit pattern within the contour; wherein if the encoded bit error corresponding to the boundary of each candidate bit pattern is below the user-selected threshold, the candidate bit pattern is accepted for further processing; and determining whether bits of the candidate bit pattern other than its boundary bits match any standard bit pattern within a library of surface landmarks for a given landing site at the vertical lift airport, and if there is a match within a user-selected bit error threshold selection, the candidate bit pattern is accepted as a valid ID.

[0080] Embodiment 15 includes the system of embodiment 14, wherein if more than one contour is associated with a valid ID in the surface marker library, the filter detected marker function module is further operative to select contours within a minimum window size and ignore all other contours.

[0081] Embodiment 16 includes a system according to any one of embodiments 14 to 15, wherein if multiple contours within the same window size can be associated with valid IDs in the surface marker library, the filter detection mark function module is also operative to calculate an average of corresponding angles of the multiple contours.

[0082] Embodiment 17 includes a system according to any one of embodiments 14 to 16, wherein the corner refinement function module is also operative to perform corner refinement of valid four-sided candidate contours identified as potential landing surface markers by moving the corners of each four-sided candidate contour to a sharpest transition point using a local gradient.

[0083] Embodiment 18 includes the system of any one of Embodiments 14 to 17, wherein the potential landing surface markings include one or more Aruco codes.

[0084] Embodiment 19 comprises a system according to any one of Embodiments 10 to 18, wherein the vehicle comprises an unmanned aerial system (UAS) vehicle, an unmanned aerial vehicle (UAV), or an urban air mobility (UAM) vehicle.

[0085] In light of the foregoing, it should be understood that although specific embodiments are described herein for the purpose of illustration, various modifications may be made without departing from the scope of the present disclosure. Therefore, the embodiments described are to be considered in all respects as merely illustrative and not restrictive. In addition, all changes within the meaning and scope of the equivalents of the claims are intended to be encompassed within their scope.

Claims

1. A method, comprising: capturing a color image of the landing site including one or more landing surface markings using a vision sensor on the vehicle; Converting the color image into a grayscale image; Performing multi-scale binarization to detect multiple edges of the grayscale image and generating multiple binary images with different sizes based on the grayscale image; wherein a plurality of windows of different sizes are used to detect edges in the grayscale image, and within a given window, an adaptive threshold is used to slide the window across the grayscale image to generate a complete edge map for the given window; Determine the contours of edges of a binary image having a closed shape; Any closed shape with an outline of edges at four corners is detected by a process comprising the following steps: Selecting contours of edges that exceed a user-selected perimeter size threshold; Keep the selected contour with four edges; Determine the minimum angular separation of a contour having four sides; eliminating any contours whose distance from an edge of the corresponding binary image is within a user-selected threshold; and Keep any remaining contours as four-edge candidate contours; The validity of the four-sided candidate contour as a potential landing surface marker is verified by a process comprising the following steps: warping each four-sided candidate contour into a fixed-size square contour, testing for sufficient variance of the image intensity values ​​by computing a standard deviation of the image intensity values, and converting the fixed-size square contour into a binary image to determine candidate bit patterns within the contour if the standard deviation of the image intensity values ​​is above a user-selected threshold; wherein if the encoded bit error corresponding to the boundary of each candidate bit pattern is below a user-selected threshold, the candidate bit pattern is accepted for further processing; and determining whether bits of the candidate bit pattern other than its boundary bits match any standard bit pattern within a library of surface signatures for a given landing site at the vertiport, and accepting the candidate bit pattern as a valid ID if there is a match within a user-selected bit error threshold; wherein if more than one contour is associated with a valid ID in the surface marker library, the contour within the smallest window size is selected and all other contours are ignored; wherein if multiple contours within the same window size can be associated with valid IDs in the surface marker library, then the average of the corresponding angles of the multiple contours is calculated; and Corner refinement of valid four-sided candidate contours identified as potential landing surface markers is performed by a process comprising the following steps: Use local gradients to move the corners of each four-sided candidate contour to the sharpest transition point.

2. The method according to claim 1, further comprising: determining whether the ID detected within the contour is in a list of valid IDs within the surface marker library for the given landing point; If the detected ID is in the list of valid IDs, the contour with the detected ID is considered valid for further processing; If the detected ID is not in the list of valid IDs, ignoring the contour with the detected ID and continuing to the next contour with the detected ID; wherein if a valid detected ID has a contour size above a first user-selected threshold, the contour having said valid detected ID is accepted for further processing; wherein if a valid detected ID has a contour size below a second user-selected threshold, rejecting the contour having the valid detected ID; Wherein for any contours having valid detected IDs with contour sizes between the first user selection threshold and the second user selection threshold, the contour with the valid detected ID having the largest size is accepted and all other valid IDs are rejected.

3. A system, comprising: at least one vision sensor mounted on the vehicle; at least one processor located on the vehicle and operatively coupled to the at least one visual sensor; and a navigation system located on the vehicle and operatively coupled to the at least one processor; wherein the at least one processor hosts a set of program modules operative to extract landing site surface landmark information for use in navigation of the vehicle, the program modules comprising: a detection mark function module, the detection mark function module being operative to receive an input image captured by the at least one visual sensor and convert the input image into a grayscale image; a detect-marker-candidates functional module, the detect-marker-candidates functional module being called from the detect-marker-functional module and operative to receive the grayscale image from the detect-marker-functional module, wherein the detect-marker-candidates functional module is operative to determine contours and corresponding corners from the grayscale image that satisfy user-selected criteria regarding the number of corners and contour length to detect potential candidates for a marker; an Identify Marker Candidates function module, the Identify Marker Candidates function module operative to receive the contours and corners detected from the Detect Marker Candidates function module, check their validity against potential marker valid IDs within a surface marker library, return those contours and corners corresponding to the valid IDs, and return the corresponding ID numbers; a filter detection tag function module that operates to filter out multiple detections for the same tag and retain the tags detected with the smallest window size for a given ID; and An angle refinement function module operates to refine the detected angle by aligning the detected angle with the actual marked angle as closely as possible; wherein the extracted landing point surface landmark information is sent to the navigation system for further processing to provide guidance used during takeoff or landing of the vehicle.

Images