Method and apparatus for multi-layered integrity verification of vision-based guidance

Abstract

A method and apparatus for multi-layer integrity verification of vision-based vehicle guidance are disclosed. [Solution] An image sensor carried by a vehicle includes an image sensor for identifying light sources of a ground-based pattern display. In this case, the spatial arrangement of various distances between the light sources defines the light pattern array. An exemplary system further includes at least one processor circuit for performing multilayer integrity verification corresponding to the light pattern array based on the identified light sources for vehicle guidance via the light pattern array.

Description

[Technical Field]

[0001] Related applications

[0001] This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 744,056, filed on 10 January 2025. U.S. Provisional Patent Application No. 63 / 744,056 is incorporated herein by reference in its entirety. This specification claims priority to U.S. Provisional Patent Application No. 63 / 744,056.

[0002]

[0002] This disclosure relates more broadly to vehicle guidance, and more particularly to methods and apparatus for multilayer integrity verification of vision-based vehicle guidance. [Background technology]

[0003]

[0003] Aircraft such as vertical take-off and landing (VTOL) aircraft may require guidance for landing. Autonomous aircraft carrying passengers in particular may require precise position and attitude determination to comply with regulatory approvals. To meet regulatory approvals, several known systems employ ground-based augmentation systems (GBAS) and ILS (Instrument Landing Systems). However, known systems utilizing GBAS may be subject to jamming / spoofing, may require radio frequency (RF) spectrum allocation, may require complex hardware and software to function, which may limit their adoption at smaller airports. Thus, known GBAS implementations may require relatively complex and expensive equipment, devices, and / or components, which may also increase the weight of the aircraft. ILS also requires relatively complex and expensive ground infrastructure, including localizers and directional glideslope antennas for each approach path. [Overview of the project]

[0004]

[0004] An exemplary system includes an image sensor carried by a vehicle for identifying light sources of a ground-based pattern display. In this case, the spatial arrangement of various distances between the light sources defines the light pattern array. The exemplary system further includes at least one processor circuit for performing multilayer integrity verification corresponding to the light pattern array based on the identified light sources for vehicle guidance via the light pattern array.

[0005]

[0005] An exemplary apparatus for verifying data corresponding to a ground-based light pattern array for vehicle guidance includes an interface circuit communicatively coupled to an image sensor supported by the vehicle, machine-readable instructions, and a programmable circuit which, based on the output from the image sensor, identifies a light source of the ground-based light pattern array, the arrangement of the light sources defines the light pattern array, verifies data corresponding to the light pattern array based on the identified light source via multilayer integrity verification, and enables vehicle guidance via the light pattern array in accordance with the data corresponding to the verified light pattern array.

[0006]

[0006] An exemplary non-transient machine-readable medium includes a machine-readable instruction causing at least one processor to perform at least: identify a light source for a pattern display that defines a pattern arrangement based on the output from an image sensor of the vehicle; verify data corresponding to the pattern arrangement in a multilayer integrity verification based on the identified light source; and determine the position of the vehicle relative to the pattern arrangement in accordance with the verified data.

[0007]

[0007] An exemplary method includes identifying light sources of a ground-based light pattern array based on output from an image sensor carried by a vehicle. In this case, the spatial arrangement of the light sources defines the light pattern array. The method further includes verifying data corresponding to the light pattern array through multilayer integrity verification, and enabling vehicle guidance via the light pattern array in accordance with the data corresponding to the verified light pattern array.

[0008]

[0008] An exemplary method includes identifying the light sources of a ground-based light pattern array based on the output from an image sensor carried by the vehicle. In this case, the spatial arrangement of the light sources defines the light pattern array. The method further includes verifying the data corresponding to the light pattern array through multilayer integrity verification, and enabling vehicle guidance via the light pattern array in accordance with the data corresponding to the verified light pattern array. [Brief explanation of the drawing]

[0009] [Figure 1]

[0009] An exemplary vision-based guidance system as taught in this disclosure. [Figure 2A]

[0010] Figure 1 is a schematic diagram of an exemplary vision-based analytics architecture that can be implemented in the exemplary vision-based guidance system. [Figure 2B]

[0011] This is an exemplary process flow of a verification process that may be implemented in multiple embodiments of this disclosure. [Figure 3]

[0012] The present disclosure shows an apparatus having an alternative exemplary pattern display as taught herein. [Figure 4A]

[0013] This specification describes several exemplary embodiments of data analysis that can be implemented in the various embodiments disclosed herein. [Figure 4B] This specification describes several exemplary embodiments of data analysis that can be implemented in the various embodiments disclosed herein. [Figure 4C] Shows exemplary aspects of data analysis that can be implemented in the multiple embodiments disclosed herein. [Figure 5A]

[0014] Shows exemplary aspects of image preprocessing that can be implemented in the multiple embodiments disclosed herein. [Figure 5B] Shows exemplary aspects of image preprocessing that can be implemented in the multiple embodiments disclosed herein. [Figure 5C] Shows exemplary aspects of image preprocessing that can be implemented in the multiple embodiments disclosed herein. [Figure 5D] Shows exemplary aspects of image preprocessing that can be implemented in the multiple embodiments disclosed herein. [Figure 5E] Shows exemplary aspects of image preprocessing that can be implemented in the multiple embodiments disclosed herein. [Figure 6A]

[0015] Shows exemplary aspects of color removal and image thresholding that can be implemented in the multiple embodiments disclosed herein. [Figure 6B] [[ID=​​​​​​​​​​​​​​​​Figure 8 is a flowchart representing exemplary machine-readable instructions and / or exemplary actions that can be implemented, instantiated, and / or performed by exemplary programmable circuitry for implementing the vision-based guidance analysis system. [Figure 10] Figure 8 is a flowchart representing exemplary machine-readable instructions and / or exemplary actions that can be implemented, instantiated, and / or performed by exemplary programmable circuitry for implementing the vision-based guidance analysis system. [Figure 11]

[0019] This is a block diagram of an exemplary platform including programmable circuitry built to execute, instantiate, and / or perform the machine-readable instructions of Figures 9 and 10, and / or perform exemplary operations, for implementing the vision-based guidance analysis system of Figure 8. [Figure 12]

[0020] Figure 11 is a block diagram of an exemplary implementation of a programmable circuit. [Figure 13]

[0021] This is a block diagram of another exemplary implementation of the programmable circuit shown in Figure 11. [Modes for carrying out the invention]

[0010]

[0022] Generally, the same reference numeral is used throughout the (one or more) drawings and accompanying specifications to refer to the same or similar parts. The drawings are not necessarily to scale. Instead, the thickness of layers or areas may be enlarged in the drawings. Layers and areas are depicted with clear lines and boundaries in the drawings, but some or all of such lines and / or boundaries may be idealized. In reality, boundaries and / or lines may be unobservable, blended, and / or irregular.

[0011]

[0023] Methods and apparatus for multi-layer integrity verification of vision-based vehicle guidance are disclosed. Current solutions require pilot involvement in the takeoff and landing phases. Autonomous solutions can utilize machine learning (ML) / artificial intelligence (AI) based pattern recognition, which may be difficult to certify. Concerns regarding ML / AI solutions in particular may arise based on whether they are solution deterministic and whether they can be tested (e.g., for certification purposes). Currently, autonomous solutions using alternative technologies are only acceptable for the purpose of assisting approaches up to a certain minimum altitude, after which vertical takeoff and landing to the ground must be taken over by a pilot or operator. Furthermore, inertial landing systems (ILS), which are relatively slow to develop and are expected to be available at major airports, are unlikely to be available at vertiports due to cost and real estate reasons. Moreover, there are limitations to the ability to reuse the same RF spectrum across multiple vertiports that are relatively close to each other.

[0012]

[0024] Multiple embodiments disclosed herein utilize multi-layer integrity verification, thereby enabling the use of vision-based navigation / guidance for critical in-flight civilian applications requiring precise takeoffs and landings, as well as for GPS-denial navigation. Thus, multiple embodiments disclosed herein can meet the requirements of Category CAT III C, which corresponds to fully autonomous landings without pilot intervention. Currently, CAT III C requirements are considered to require a relatively high level of precision for approach and landing systems in aircraft.

[0013]

[0025] The embodiments disclosed herein offer numerous advantages. Unlike most navigation systems coupled to Global Navigation Satellite Systems (GNSS) or GPS receivers, the embodiments disclosed herein do not require information from external sensors. Furthermore, the image processing methods of the embodiments disclosed herein may be applicable to a wide range of day and night lighting conditions, visible and infrared (IR) or near-infrared (NIR) spectral cameras, and so on. Therefore, the embodiments disclosed herein may be verifiable. Moreover, the embodiments disclosed herein can be at least partially implemented in image sensor / camera hardware. The embodiments disclosed herein can be used in conjunction with AI / ML implementations.

[0014]

[0026] Multiple embodiments disclosed herein utilize visible, IR, or NIR light patterns (e.g., light patterns, light pattern arrays, light pattern displays, etc.) with light sources (e.g., target, vertiport lights, etc.) positioned along the periphery of a vehicle, such as an aircraft (e.g., a manned autonomous aircraft), to guide its movement. The light sources may be arranged as a pattern with irregular spacing (e.g., different edges of the light pattern have different spacings and / or distances between each light source). Multiple embodiments disclosed herein utilize multilayer verification (e.g., a multilayer verification process) along with integrity monitoring to ensure that the vehicle can be properly / accurately guided based on the light pattern. To perform multilayer verification, multiple embodiments disclosed herein may utilize clutter removal along with shape dimension / pattern / template matching. Furthermore, data from image sensors (e.g., solutions based on line-of-sight data corresponding to the image sensor) may be verified and matched with inertial data for integrity monitoring.

[0015]

[0027] According to some embodiments disclosed herein, image processing such as spatial filtering is performed to highlight the aforementioned light sources. In some embodiments, color-based removal (e.g., four-color-based removal) of data from an image sensor is performed. In some such embodiments, RGB content filtering is performed on RGB-NIR images. In some embodiments, image thresholding is performed to remove portions of the scene whose intensity is below a threshold. Further or alternatively, clustering and centroid calculation of data from an image sensor are performed. In some embodiments, various colors (e.g., alternating colors, patterned sequences of various colors, etc.) are used in the light pattern to make the light more distinguishable and easier to detect / detect. According to some embodiments disclosed herein, a combination of RGB and NIR light sources is used in the pattern. Further or alternatively, one of the light sources emits light in a first emission spectrum, while the other light source emits light in a second emission spectrum different from the first emission spectrum, thereby increasing the visibility of the light pattern.

[0016]

[0028] Figure 1 shows an exemplary vision-based guidance system 100 as taught in this disclosure. In the shown view of Figure 1, an exemplary aircraft 101 is shown being guided by a pattern display (e.g., a ground-based pattern display, a pattern array of light sources, etc.) 102. In this embodiment, the pattern display 102 is implemented as a landing pad (e.g., a vehicle pad, a landing / deployment pad, a vertiport, etc.) for the aircraft 101 to land. In this embodiment, the aircraft 101 includes a fuselage 103, a rear section 106, and a propeller assembly 110. Furthermore, the aircraft 101 includes at least one sensor 112, which in this embodiment is implemented as a camera, a light detection device, and / or an image sensor, as well as a flight controller 114. In this embodiment, (one or more) sensors 112 define the field of view 120 of the aircraft 101.

[0017]

[0029] During operation, the aircraft 101 is implemented to be switchable between hover mode (e.g., hovering mode, landing mode, etc.) and cruising mode (e.g., primary flight mode, cruising mode, etc.) by moving, rotating, and / or orienting the propeller assembly 110. To position the exemplary aircraft 101 in hover mode or cruising mode, the propeller assembly 110 (and / or the aerodynamic body coupled thereto) may rotate relative to the fuselage 103. The aircraft 101 may be a vertical take-off and landing (VTOL) aircraft, or any other aircraft, not limited to fixed-wing aircraft, short take-off and landing (STOL) aircraft, rotary-wing aircraft, aircraft with fan arrays, etc. Furthermore, several embodiments of the present disclosure may be implemented in other types of manned or unmanned vehicles, not limited to hovercraft, ships, submarines, spacecraft, ground-based vehicles, etc.

[0018]

[0030] To enable the aircraft 101 to be guided by an exemplary pattern display 102 (for example, for landing on the pattern display 102), the pattern display 102 includes a specific arrangement and / or spacing of light sources 134 (e.g., lights, light emitters, power supply light sources, light-emitting diodes (LEDs), etc.) detectable by (one or more) sensors 112 for guiding the aircraft 101. In particular, the aircraft 101 verifies data corresponding to the pattern display 102 in order to guide the aircraft 101 using the pattern display 102. The pattern display 102 in the illustrated embodiment includes a landing pad area 130 and a perimeter 132. The perimeter 132 in turn includes the aforementioned light sources 134. The exemplary perimeter 132 surrounds and extends around the landing pad area 130 and is generally quadrangular in shape (e.g., rectangular and / or square shapes, etc.). However, the perimeter 132 may be any other suitable shape and / or shape dimension profile (e.g., elliptical, triangular, hexagonal, oval, polygonal, etc.). As will be described in more detail below in relation to Figures 2 to 13, several embodiments disclosed herein enable cost-effective guidance of an aircraft 101 by a pattern display 102 (e.g., for taking off and landing from the surface of the pattern display 102, for moving toward or away from the pattern display 102). As stated above, the perimeter 132 includes light sources 134 (e.g., four light sources 134 on each side / edge of the perimeter 132). The light sources 134 are spaced apart from one another along the perimeter 132 (e.g., equally spaced, irregularly spaced, non-uniformly spaced, etc.). In other words, the light sources 134 are arranged as a pattern (e.g., spatial pattern, intrinsic spatial pattern, pattern array, optical pattern array, etc.) (e.g., spatially arranged, irregularly arranged, arranged along the perimeter, etc.). The light sources 134 may be the same or different types from one another (e.g., a combination of visible and IR / NIR light sources).

[0019]

[0031] To define the pattern and / or spatial arrangement of the light sources 134, several embodiments of the present disclosure utilize various spacings between the light sources 134 on each side / edge. Furthermore, at least two of the light sources 134 may have different optical properties from each other, thereby defining a recognizable pattern (e.g., a two-dimensional pattern, a spatial pattern, etc.). In particular, two of the light sources 134 may emit different signal / light types (e.g., visible / RGB and IR / NIR, etc.). In this embodiment, visual detection of the aforementioned light sources 134 based on the sensor output of (one or more) sensors 112 enables the flight controller 114 to determine, identify, and / or recognize the pattern and thus calculate the relative distance / position / attitude / orientation of the aircraft 101 moving toward or away from the pattern display 102. In particular, the flight controller 114 utilizes multiple line-of-sight (LOS) vectors for each of the light sources 134 to determine the position and attitude of the aircraft 101 relative to the pattern display 102 and / or the landing pad area 130.

[0020]

[0032] In some embodiments, one or more sensors 112 include and / or utilize filters (e.g., polarizing filters, optical spectral filters, etc.) to facilitate detection of light sources 134 on the periphery 132. Furthermore or alternatively, the light emitted from the light sources 134 on the periphery 132 is shifted in frequency and / or color from one another (e.g., shifted toward the blue region of the optical frequency spectrum) to facilitate detection of them by one or more sensors 112. In some such embodiments, the filters are fine-tuned and / or adjusted based on the characteristics of the light sources 134 (e.g., the filters are shifted in frequency to facilitate detection of light sources from relatively long distances and / or through adverse weather conditions).

[0021]

[0033] Figure 2A is a schematic diagram of an exemplary vision-based analytics architecture 200 that may be implemented in the exemplary vision-based guidance system 100 of Figure 1. In the embodiment shown in Figure 2A, the exemplary vision-based analytics architecture 200 includes at least one camera 202, an inertial measurement unit (IMU) 206, a GNSS receiver (e.g., a GPS receiver) 208, a hardware interface 210 (implemented in this embodiment as an FPGA and / or circuit), camera interface software 212, GNSS interface software 214, IMU interface software 216, image processing circuitry 218, altitude / position resolution processing 220, an initializer 222, and an all-source position navigation and timing (ASPNT) navigator 224 as an implementation of a vision landing system (ViTALS). In some embodiments, the vision-based analytics architecture 200 includes a GNSS / GPS receiver 230 and / or a real-time motion (RTK) receiver 232 and / or is coupled to them for communication. One of those receivers may be used for supplemental / backup information.

[0022]

[0034] During operation, the output from camera 202 (e.g., measured output, detected output, etc.) is utilized by camera interface software 212. The data corresponding to this output is then utilized by image processing circuit 218. Furthermore, the IMU 206 and GNSS / GPS receiver 208 provide output to the guidance / navigation board / circuit 210, which in turn provides and / or transmits data to the GNSS / GPS interface software 214 and the IMU interface software 216. In the embodiment shown in Figure 2A, the image processing circuit 218 utilizes the data / output from camera 202 for attitude / position resolution processing 220 (e.g., for camera-based resolution). Then, using the ASPNT navigator 224, according to the teachings of this disclosure, the solution from attitude / position resolution processing 220 corresponding to the image sensor is matched with inertial data from IMU 206 and / or IMU interface software 216 to verify the solution used in vehicle guidance. In some embodiments, the attitude / position resolution process 220 utilizes the least squares method to determine the position and / or attitude. In this embodiment, the ASPNT navigator provides updates to the initializer 222.

[0023]

[0035] According to several embodiments disclosed herein, integrity monitoring is performed using multiple layers. For example, the first layer may correspond to clutter removal (rejection). Alternatively, spatial filtering-based removal is performed.

[0024]

[0036] In this embodiment, the second layer corresponds to shape- and dimension-based candidate screening. In particular, the detection candidates may be defined.

[0025]

[0037] According to several embodiments disclosed herein, the third layer may correspond to a shape-dimension-based candidate orientation / position solution or an equivalent affine transformation. In particular, the solution may be verified by an internal consistency check.

[0026]

[0038] In this embodiment, the fourth layer corresponds to camera-based solution validation. In some embodiments, redundancy of light sources (e.g., four light sources are required to determine their location), and more than four light sources, may be used to cross-validate the measurements. Thus, to validate the camera-based solution, a database is used for the locations of the light sources and for the transformation of the locations to camera frames, using the camera's orientation and position (e.g., determined using four light sources). The predicted locations of the light sources detected by the camera are then matched against the measured locations of the light sources. If the predicted and measured light source locations match, the light sources are consistent. In contrast, if the predicted and measured light source locations are inconsistent, it may be considered that there has been an error in identifying a candidate solution, or that the light source pattern / array is incorrect.

[0027]

[0039] Figure 2B shows an exemplary process flow 240 of an exemplary multilayer verification process that may be implemented in several embodiments of the present disclosure. According to several embodiments disclosed herein, a first layer 242 may correspond to clutter rejection / removal, a second layer 244 may correspond to shape-dimension-based candidate screening / integrity protection, a third layer 246 may correspond to template matching with integrity monitoring, and a fourth layer 248 may correspond to the inertial navigation system (INS) level. In this embodiment, the fourth layer 248 corresponds to performing an integrity check on the INS / ASPNT solution by utilizing Kalman filter residues. In some embodiments, color-based removal (e.g., four-color-based removal) is performed during the first layer 242. Furthermore, spatial filtering-based removal may be performed.

[0028]

[0040] According to several embodiments disclosed herein, the first layer 242 may correspond to clutter removal via RGB-NIR filtering. Windowing and / or selection of regions of interest may be performed. Furthermore, windowing and spatial filtering of the selected regions of interest may be performed.

[0029]

[0041] In this embodiment, the second layer 244 includes candidate screening / integrity protection. For example, the second layer 244 includes candidate detection / identification. According to some embodiments disclosed herein, Desargues' theorem is used to form detection candidates. Further or alternatively, line formation, line segment ratio, etc., are performed for candidate detection. In some embodiments, projection invariants are identified.

[0030]

[0042] According to several embodiments disclosed herein, one or more of the following approaches may be used in the second layer 244: (i) candidate screening based on triple-formed triangles and Desargues' theorem, (ii) candidate screening and line segment ratio matching based on quadruple-formed lines, (iii) candidate screening based on corners formed by two lines, or (iv) candidate screening based on two parallel lines. Therefore, if more approaches are selected for use, the likelihood of clutter being present in the next layer will be lower. Conversely, if fewer approaches are used, there will be more candidates in the next layer. Thus, the decision may be a trade-off between the throughput usage of the two layers. This may be determined by the processing quality of the first layer 242. In particular, in most cases, the first layer 242 may leave less cluttering based on clutter removal. Therefore, in some embodiments, it may be advantageous to perform only (i) candidate screening based on triple-formed triangles and Desargues' theorem.

[0031]

[0043] In the exemplary embodiment shown in Figure 2B, the third layer 246 corresponds to template matching. In particular, the third layer 246 corresponds to the formation and verification of candidate pose / position solutions (or equivalent affine transformations) by internal consistency checks. According to several embodiments disclosed herein, a minimum of three candidate lights are used to identify a candidate pose / position solution. This may be equivalent to identifying an affine transformation. In some such embodiments, k is 4 or greater to ensure good accuracy of the candidate solutions. Furthermore, the remaining nk candidate lights can be used to check internal consistency in the following sense: the expected position of each light in the vertiport within the camera field of view (FOV) can be predicted by using the candidate pose / position solutions (or equivalent affine transformations) generated using k candidate lights and a vertiport light pattern database. As a result, the predicted position can be checked against the measured position as verification. According to several embodiments disclosed herein, an increase in the number of lights predicted and matched corresponds to a higher degree of completeness.

[0032]

[0044] In some embodiments, a Random Sample Consensus (RANSAC) analysis can be performed. In other words, a random proposal and check process can be carried out. Furthermore, for example, an estimation of affine transformations can be identified. In some embodiments, autonomous completeness monitoring (AIM) can be performed as follows: the process can return to the RANSAC analysis for convergence as needed. In some embodiments, the number of candidate / proposed solutions is reduced (e.g., reduced to one or two solutions). Furthermore, the fixation of position / altitude can be updated. In some embodiments, light measurements with identifiers are identified.

[0033]

[0045] According to several embodiments disclosed herein, in the fourth layer 248, the camera-based attitude / position solution is validated by comparing it with a solution propagated using inertial navigation algorithms and inertial measurement unit (IMU) data from a validated navigation solution. This may correspond to results using sensor measurements prior to the All Source PNT solution (ASPNT).

[0034]

[0046] According to several embodiments disclosed herein, two solutions are compared: (a) camera-based attitude / position solutions that have passed integrity tests on the first, second, and third layers 242, 244, and 246, and (b) attitude / position solutions propagated using inertial navigation algorithms and inertial measurement unit (IMU) data from validated navigation solutions (e.g., results using previous sensor measurements). Using the covariance matrix predicted by the Kalman filter and the covariance matrix identified at the end of the third layer 246, a threshold for the difference in position and attitude between the two solutions can be determined. Thus, if the difference exceeds the threshold, the camera-based solution is rejected. In particular, the reason why the inertially propagated ASPNT solution is retained at three different IMUs is, for example, to ensure the integrity of the inertial data.

[0035]

[0047] According to several embodiments disclosed herein, integrity checks may be performed on Kalman filters corresponding to INS data. The multilayer verification in Figure 2B is only one embodiment, and any other suitable number and / or type of layers may be implemented instead.

[0036]

[0048] Figure 3 shows an apparatus 300 having alternative exemplary pattern displays 302, 304 as taught in this disclosure. In this embodiment, the pattern displays 302, 304 are shown supported by a stand / mount 306. According to several embodiments disclosed herein, the pattern displays 302, 304 may be used separately from each other or in parallel. According to several embodiments disclosed herein, the pattern display 302 generally exhibits a rectangular shape having a grid of light sources arranged inside. An exemplary pattern display 302 may be uniquely identified based on the spacing of the light sources in its defined grid (e.g., different spacings between them). In this embodiment, the pattern display 302 corresponds to an NIR light emitter.

[0037]

[0049] In some embodiments, a pattern display 304 having a different shape from the pattern display 302 is used. An exemplary pattern display 304 may output visible light / RGB, be generally elliptical and / or circular in shape, and may include light sources arranged along its radiating portion. In some embodiments, the light sources of the pattern display 304 are spaced apart to define a unique pattern. For example, the light sources of the pattern display 304 are placed at different radial distances from each other, thereby defining its unique pattern. Several embodiments disclosed herein are used as guides for or as part of a landing zone.

[0038]

[0050] Figures 4A–4C illustrate several exemplary aspects of data analysis that may be implemented in several embodiments disclosed herein. Referring to Figure 4A, an exemplary spatial pattern / distribution of light sources 400 of a vertiport is shown. In the embodiment shown in Figure 4A, the spatial pattern / distribution of light sources includes light sources spaced along the periphery with defined intervals between them. In particular, each edge / side has four light sources in this embodiment, and the light sources in each side / edge have different spacing than those in the other sides / edges. In other words, the spacing between light sources is different / irregular in each side / edge in this embodiment. However, any other suitable number and spacing arrangement of light sources may be implemented instead.

[0039]

[0051] Figure 4B shows Graph 410, which illustrates the single-line cross ratio against the separation distance of the light sources. In particular, the depiction in Figure 4B corresponds to the separation between the light sources. In this particular embodiment, the separation is approximately 11 meters (m).

[0040]

[0052] Figure 4C shows Graph 420, which displays the identified and / or determined coordinates of the light source, shown as a coordinate grid. In particular, the depiction in Figure 4C corresponds to the investigated / realized light pattern. As can be seen in Figure 4C, the investigated / realized distance is relatively close to that shown in Figure 4A.

[0041]

[0053] Figures 5A to 5E illustrate several exemplary aspects of image preprocessing that may be implemented in several embodiments disclosed herein. According to several embodiments disclosed herein, a target (e.g., a vertiport light / light source) is enhanced against the rest of the image by performing image transformations such as image convolution for detection enhancement. For example, these transformations are selected to enhance the visibility of target features, such as their edges, Gaussian distribution shapes, or circular contours.

[0042]

[0054] According to several embodiments disclosed herein, spatial filters are image processing tools. These image processing tools analyze and manipulate the local neighboring pixel intensity to achieve a specific effect. The function of spatial filters may be to enhance, suppress, or extract features from an image, making them essential for tasks such as edge detection, noise reduction, and feature enhancement. For this purpose, filters may be selected based on the nature of the image and the desired result. For example, smoothing filters such as Gaussian blur are used to reduce noise, while edge detection filters such as Gaussian Laplacian (LoG) or Gaussian difference (DoG) emphasize boundaries and areas of intensity changes. Morphological filters such as top-hat transforms can emphasize relatively small structures, while gray-level transforms can adjust brightness and contrast to increase visualization. According to several embodiments disclosed herein, the selection of filters depends on the specific features to be enhanced or suppressed, the scale of interest, and the resolution, as well as the trade-off between noise suppression and computational efficiency.

[0043]

[0055] Referring to Figure 5A, according to several embodiments disclosed herein, the Gaussian Laplacian (LoG) may be advantageous for finding either edges or “blobs” in an image. The embodiment shown in Figure 5A consists of a Laplacian and a Gaussian filter. In particular, the Gaussian filter is an approximation to a Gaussian function. For example, a Gaussian filter may be employed to blur an image to reduce image noise and mitigate the effects of abrupt changes in intensity between pixels. To overcome the noise sensitivity of the Laplacian, the Gaussian filtering step may be performed, for example, before finding the zero crossing of the second derivative as a result of the Gaussian Laplacian.

[0044]

[0056] The left side of Figure 5A shows the one-dimensional (1D) intensity value distribution or intensity function, represented as F(x). The middle portion of Figure 5A shows the first derivative of the intensity function. Its equivalent when applied to a two-dimensional (2D) image (x,y) is the gradient function. The right portion of Figure 5A shows the second derivative of the intensity function, which is equivalent to the Laplacian operator in 2D. Therefore, the following expressions may be used in several embodiments disclosed herein. TIFF2026121293000002.tif11170

[0045]

[0057] In some embodiments, the light / light source of a vertiport may appear as a relatively large object in a shorter range of the image and as a relatively small object in a longer range of the image, so image convolution can be performed in two steps using a combination of kernels: (i) a standard kernel σ=1.01 (based on light and camera information) as one embodiment, and (ii) a dynamically calculated kernel based on the details of the convolutional image statistics. In step (ii), the function for calculating the size of the kernel for the second convolution is based on the maximum intensity of the image after convolution with σ=1.01, expressed, for example, as follows: σ² = k * max(I(σ¹)) This two-step convolution allows for the dynamic application of the appropriate kernel, where k is a weighting coefficient based on the camera's dynamic range.

[0046]

[0058] According to several embodiments disclosed herein, a LoG kernel can be utilized. In particular, the Laplacian is an isotropic measure used to compute the second derivative of an image and can therefore provide a quantification of the rate of change of the first derivative of the image, thereby making it suitable for edge detection. Thus, edges appear as peaks in the first derivative, while zero crossings in the second derivative of the image indicate potential edge locations. As shown in the middle / center of Figure 5A, if the peak of ∇I(x,y) exceeds a certain threshold, the peak can be classified as an edge. In the right portion of Figure 5A, the peak of ∇I(x,y) exceeds ∇2 We can observe how it transforms into a zero intersection of I(x,y).

[0047]

[0059] Referring to Figures 5B to 5E, according to several embodiments disclosed herein, there may be two options for Laplacian filtering. The first option involves (i) a positive Laplacian that identifies the outer edges in the image. This feature may be advantageous in vertical detection for highlighting the circular contours of light and then performing circular detection. The second option is (ii) a negative Laplacian that identifies the inner edges in the image. This may be useful for detecting pseudolights / stars in vertiports when the pixel intensity in the image is relatively high (e.g., light sources / stars are represented as clusters). They may be filtered in a simple manner, for example, using thresholding.

[0048]

[0060] Figure 5B shows an example of a LoG kernel with σ=2 and size 13x13 that uses a negative Laplacian.

[0049]

[0061] Figure 5C shows an example of a LoG kernel with σ=2 and size 13x13 that uses a positive Laplacian.

[0050]

[0062] Figure 5D shows the result of convolving the kernel in Figure 5A using an image of the sample's vertiport (with enhanced light clumps), while Figure 5E shows the result of convolving the kernel in Figure 5C using an image of the sample's vertiport (enhanced circular outline of the pseudostellar).

[0051]

[0063] Figures 6A to 6C illustrate several exemplary embodiments of color removal and image thresholding that may be implemented in several embodiments disclosed herein. Referring to Figure 6A, a color-based removal method is used to discard any pixels from a vertiport that are inconsistent with the expected characteristics of the NIR light source. In this embodiment, the purpose of the function is to remove any pixels (where the RGB content exceeds a threshold). The threshold is set so that the vertiport light pattern is emitted primarily within the NIR band. For example, each pixel is mapped from a location 2 kilometers (km) away to a 7x7 m area on the ground. This causes the vertiport light to merge with the background. If any of the R, G, or B channels exceeds this threshold, their corresponding locations in the NIR channel may be masked with, for example, a zero value. Furthermore, the vertiport light appears as a bright object in the NIR channel, and therefore a second clutter removal step is implemented to discard areas in the NIR image that fall below a given threshold. This can be expressed, for example, as follows: max_r_thresh, max_g_thresh, max_b_thresh-red,green,blue image layer thresholds min_nir_thresh - NIR image threshold For each pixel i,j in Image I perform: If I(i,j)>max_r_thresh OR G(i,j)>max_g_thresh OR B(i,j)>max_b_thresh then: filtered_nir(i,j)=0 Else: If NIR(i,j) > min_nir_thresh: filtered_nir(i,j)=NIR(i,j) Else: filtered_nir(i,j)=0 return filtered_nir

[0052]

[0064] Figure 6A shows an example of RGB content filtering for an RGB-NIR image. Specifically, it shows: (1) extraction of the R, G, and B channels; (2) the result of thresholding operations on each channel. The thresholds max_r_thresh, max_g_thresh, and max_b_thresh are set to 100; and (3) the result of an AND operation on the thresholded channels (left), and this mark is multiplied by the NIR input (center), resulting in the NIR image filtered based on RGB content (right). Figure 6A also shows (4) creation of mask_nir (left) using a min_nir_thresh value of 160, and multiplying it by the filtered nir_rgb (center), resulting in the filtered_nir image (right).

[0053]

[0065] Figure 6B shows an example of RGB-NIR clutter removal on real-world image data. It shows (a) the RGB input image, (b) the NIR channel of the same image, (c) a mask based on RGB content above a threshold of 75, (d) filtered_nir_rgb, i.e., the result of applying mask (c) to the NIR channel, and (e) the filtered_nir image obtained after thresholding of (d).

[0054]

[0066] Figure 6C corresponds to image thresholding. According to several embodiments disclosed herein, image thresholding can be employed to remove portions of a scene in a view that have an intensity lower than a threshold. After the input image from the camera is transformed by image convolution to enhance the target against the background, the thresholding step can remove most of the background. The following shows an exemplary process of image thresholding. Threshold_image(image,threshold) For any pixel i,j in image If image(i,j)>threshold: thresholded_image(i,j)=1 Else: thresholded_image(i,j)=0 Return thresholded_image

[0055]

[0067] According to several embodiments disclosed herein, an image transformed using a special filter may be used as the input image. Subsequently, in one embodiment, a threshold may be calculated using image statistics such as mean deviation and standard deviation. m = Σ j H Σ i V (I(i,j)) / (H×V), th=(max(I(i,j))+m) / 2+3σ I

[0056]

[0068] Figure 6C shows (a) the convolutional LogG image before thresholding and (b) the resulting threshold image. Thus, Figure 6C shows one embodiment of the thresholding function using actual flight image data for vertiport configuration. Since the input LogG image is obtained from a negative Laplacian, the threshold described above was set in this embodiment.

[0057]

[0069] Figures 7A and 7B show exemplary clustering and centroid calculations that can be implemented in several embodiments disclosed herein, respectively. According to several embodiments disclosed herein, the result of the image thresholding step is a “black (pixel count = 0) and white (pixel count = 1)” image, where bright pixels represent pixels above the threshold. In the clustering and centroid calculations of several embodiments disclosed herein, neighboring pixels above the threshold are clustered together and a centroid is calculated.

[0058]

[0070] The input can be described as follows. For example, the thresholded_image can be an unsigned binary array having units of luminance level such as an image of "black (number of pixels = 0) and white (number of pixels = 1)". Here, bright pixels represent pixels that exceed the threshold value.

[0059]

[0071] The output can be described as follows. For example, the detection can be an integer array of [(x1,y1)…(x n ,y n )] in units of px. The integer array can be an array of zeros (e.g., default, initialization, etc.). Thus, the integer array can be, for example, a list of centroid (x, y) coordinate positions.

[0060]

[0072] As can be seen in the embodiment shown in FIG. 7A, according to the plurality of embodiments disclosed herein, the clustering technique is a kind of region growing algorithm used to build clusters in a continuous connection in an image, particularly in a binary (e.g., black and white) image where bright pixels exhibit values exceeding a specific threshold. FIG. 7A shows the region for growing the clustering of neighboring pixels exceeding the threshold, while FIG. 7B corresponds to the resulting image.

[0061]

[0073] According to the plurality of embodiments disclosed herein, the region growing algorithm can be described as follows. That is, 1. Black and White image: bright indicates pixels above threshold ->arrayOfBrightPixelsLocations 2. Build clusters 2.a. Move to the first element (Pivot Point) of the arrayOfBrightPixelsLocations buffer = P p11 2.a.1. findNeighbors for pivot point Pp1 ,e.g.:P p11 ,P p12 …P p1n assign found neighbors to cluster started from first pivot point,e.g.:P p1 remove assigned neighbors from arrayOfBrightPixelsLocations 2.a.2 findNeighbors for all neighbors in step 2.a.1 assign found neighbors to clusters started from the 1st pivot point,e.g.P p12 ,P p12 .. remove assigned neighbors from arrayOfBrightPixelsLocations ….. 2.a.n Continue findNeighbors to next level until no neighbors exists for all P p identified starting from step 2.a 2.b.Move to the next element of arrayOfBrightPixelsLocations and start from step 2.a 2.c Repeat 2.b until arrayOfBrightPixelsLocations is empty Where function findNeighbors() returns a list of neighbors of a given pixel that are still in arrayOfBrightPixelsLocations. findNeighbors(pixel, arrayOfBrightPixelsLocations): Initialize neighbors=[] (x,y)=pixel / / Check all possible adjacent neighbors for(dx,dy) <sqrt(2): neighbor=(x+dx, y+dy) if neighbor is in arrayOfBrightPixelsLocations: neighbors.append(neighbor) return neighbors The resulting neighboring nodes can be used to calculate the center of mass (x, y) position using a central mass center of mass algorithm.

[0062]

[0074] According to several embodiments disclosed herein, centroid calculation can be implemented as follows: For example, each point is in spatial coordinate x i and y i It has weight w i However, all points p belonging to the cluster are associated with a point (e.g., pixel intensity or importance). i =(x i ,y i By identifying the ), points within the cluster can be collected.

[0063]

[0075] Furthermore, the centroid (center of mass) is calculated. If each point has mass (pixel intensity), the centroid coordinates of the following coordinates can be calculated. That is, x C =( Σ i=1 N w i x i ) / (Σ i=1 N w i ) y C =( Σ i=1 N wi y i ) / (Σ i=1 N w i )

[0064]

[0076] Figure 8 is a block diagram of an exemplary implementation of an exemplary vision-based guidance analysis system 800. The vision-based guidance analysis system 800 in Figure 8 can be instantiated by programmable circuitry such as a central processing unit (CPU) that executes a first instruction (e.g., creating an instance, realizing, embodying, and implementing it over any length of time). Alternatively, the vision-based guidance analysis system 800 in Figure 8 can be instantiated by (i) application-specific integrated circuits (ASICs) and / or (ii) field-programmable gate arrays (FPGAs) that are built and / or configured in response to the execution of a second instruction to perform an operation corresponding to the first instruction (e.g., creating an instance, realizing, embodying, and implementing it over any length of time). Therefore, it should be understood that some or all of the circuitry in Figure 8 can be instantiated at the same or different times. Some or all of the circuitry in Figure 8 can be instantiated, for example, in hardware and / or in one or more threads running in series in hardware. Furthermore, in some embodiments, part or all of the circuit in Figure 8 may be implemented by a microprocessor circuit that executes instructions and / or an FPGA circuit that performs operations in order to implement one or more virtual machines and / or containers.

[0065]

[0077] An exemplary vision-based guidance system 800 includes an exemplary image interface circuit 802, an exemplary image processing circuit 804, an exemplary integrity monitoring circuit 806, an exemplary position / attitude guidance computer circuit 808, and data storage 810. According to several embodiments disclosed herein, the vision-based guidance system 800 includes (one or more) sensors 112 and / or a navigation controller 114, and / or is communicatively coupled thereto.

[0066]

[0078] An exemplary image interface circuit 802 is used to receive and / or access image data from (one or more) sensors 112. In this embodiment, the (one or more) sensors 112 are implemented as vehicles-carried cameras or image sensors. In the embodiment shown in Figure 8, the image data corresponds to an image acquired from the image sensor 112. In particular, the image corresponds to a captured image of light (e.g., RGB and NIR light) emitted from a ground-based light pattern array. In this embodiment, the light pattern array includes a plurality of light sources arranged along its periphery. According to some embodiments disclosed herein, the edges of the periphery include light sources having varying and / or irregular spacings between them. In some embodiments, the image interface circuit 802 is instantiated by a programmable circuit configured to execute image interface commands and / or operations such as those represented by the flowcharts in Figures 9 and 10.

[0067]

[0079] According to several embodiments disclosed herein, the image processing circuit 804 may be used to process and / or preprocess image data from (one or more) sensors 112. For example, the image processing circuit 804 may be used for spatial filtering. According to several embodiments disclosed herein, the image processing circuit 804 may be used for color-based deselection (e.g., four-color-based deselection). Furthermore or alternatively, the image processing circuit 804 performs clustering and / or centroid calculation of image data. In some embodiments, the image processing circuit 804 is instantiated by a programmable circuit configured to execute instructions and / or perform operations such as those represented by the flowcharts in Figures 9 and 10.

[0068]

[0080] The integrity monitoring circuit 806 in the illustrated embodiments is used to perform multilayer integrity verification and / or monitoring of the use of a vehicle's light pattern. According to several embodiments disclosed herein, the integrity monitoring circuit 806 is used to perform multilayer integrity verification by (i) performing clutter removal, (ii) shape-dimension-based candidate screening / integrity protection, (iii) forming and verifying candidate attitude / position solutions (or equivalent affine transformations) by internal consistency checks, and (iv) matching inertial data with image-based solution verification. Thus, by performing multilayer integrity verification, the integrity monitoring circuit 806 can enable, for example, authentication of automated guidance for vehicles with passengers. In some embodiments, the integrity monitoring circuit 806 is instantiated by a programmable circuit configured to execute integrity monitoring commands and / or perform actions such as those represented by the flowcharts in Figures 9 and 10.

[0069]

[0081] In this embodiment, a position / attitude guidance computer circuit 808 is implemented to guide the movement of the vehicle. According to several embodiments disclosed herein, a light pattern array is used to guide the vehicle in response to the verification of data / information associated with the light pattern array and / or the monitoring of the light pattern array for verification. In some embodiments, the position / attitude guidance computer circuit 808 calculates the relative position, attitude, and / or heading between the vehicle and the light pattern array. In some embodiments, the position / attitude guidance computer circuit 808 is instantiated by a programmable circuit configured to execute position / guidance computer instructions and / or perform actions such as those represented by the flowcharts in Figures 9 and 10.

[0070]

[0082] In some embodiments, the data storage 810 is implemented to store information related to pattern arrays, vertiports, spatial information, light source spacing information, and so on.

[0071]

[0083] An exemplary way of implementing the vision-based guidance analysis system 800 is shown in Figure 8, but one or more of the elements, processes, and / or devices shown in Figure 8 may be combined, divided, rearranged, omitted, excluded, and / or implemented in any other way. Furthermore, the exemplary image interface circuit 802, exemplary image processing circuit 804, exemplary integrity monitoring circuit 806, exemplary position / attitude guidance computer circuit 808, and / or more generally the exemplary vision-based guidance analysis system 800 in Figure 8 may be implemented by hardware alone or by software and / or firmware. Therefore, for example, any of the exemplary image interface circuit 802, exemplary processing circuit 804, exemplary integrity monitoring circuit 806, exemplary position / attitude guidance computer circuit 808, and / or more generally exemplary vision-based guidance analysis system 800 may be implemented by programmable circuits combined with machine-readable instructions (e.g., firmware or software), processor circuits, (one or more) analog circuits, (one or more) digital circuits, (one or more) logic circuits, (one or more) programmable circuits, (one or more) programmable microcontrollers, (one or more) graphics processing units ((one or more) GPUs), (one or more) digital signal processors ((one or more) DSPs), (one or more) ASICs, (one or more) programmable logic devices ((one or more) PLDs), and / or (one or more) field-programmable logic devices such as FPGAs ((one or more) FPLDs). Furthermore, the exemplary vision-based guidance analysis system 800 in Figure 8 may include, in addition to or instead of, those shown in Figure 8, one or more elements, processes, and / or devices, and / or two or more of any or all of the elements, processes, and devices shown.

[0072]

[0084] Figures 9 and 10 show flowcharts representing exemplary machine-readable instructions that may be executed by programmable circuits for implementing and / or instantiating the vision-based guidance analysis system 800 of Figure 8, and / or flowcharts representing exemplary operations that may be executed by programmable circuits for implementing and / or instantiating the vision-based guidance analysis system 800 of Figure 8. The machine-readable instructions may be one or more executable programs or (one or more) parts of one or more executable programs executed by programmable circuits, such as programmable circuit 1112 shown below in relation to Figure 11, and / or one or more functions or (one or more) parts of functions executed by exemplary programmable circuits (e.g., FPGAs) described below in relation to Figures 12 and / or 13. In some embodiments, the machine-readable instructions cause operations, tasks, etc., to be performed and / or carried out in an automated manner in the real world. As used herein, “automated” means without human intervention.

[0073]

[0085] A program may be implemented in instructions (e.g., software and / or firmware) stored in one or more non-transient computer-readable and / or machine-readable storage media, such as cache memory, magnetic storage devices or disks (e.g., floppy disks, hard disk drives (HDDs), etc.), optical storage devices or disks (e.g., Blu-ray discs, compact discs (CDs), digital multipurpose discs (DVDs), etc.), redundant arrays of independent disks (RAID), registers, ROMs, solid-state drives (SSDs), SSD memory, non-volatile memory (e.g., electrically erasable programmable read-only memory (EEPROM), flash memory, etc.), volatile memory (e.g., random access memory (RAM) of any kind), and / or any other storage device or storage disk. Instructions in non-transient computer-readable and / or machine-readable media may be programmed and / or executed by programmable circuits located within one or more hardware devices, but the entire program and / or parts thereof may be alternatively executed and / or instantiated by one or more hardware devices other than programmable circuits, and / or embodied in dedicated hardware. Machine-readable instructions may be distributed across multiple hardware devices and / or executed by two or more hardware devices (e.g., server and client hardware devices). For example, a client hardware device may be implemented by an endpoint client hardware device (e.g., a hardware device associated with a human and / or machine user) or an intermediate client hardware device gateway (e.g., a radio access network (RAN)). This intermediate client hardware device gateway may facilitate communication between the server and the endpoint client hardware device. Similarly, non-transient computer-readable storage media may include one or more media.Furthermore, while exemplary programs have been described with reference to the flowcharts shown in Figures 9 and 10, many other methods for implementing the exemplary vision-based guidance analysis system 800 may be used as alternatives. For example, the order in which the blocks of the flowchart are executed may be changed, and / or some of the described blocks may be modified, erased, or combined. Alternatively or additionally, any or all of the blocks of the flowchart may be implemented by one or more hardware circuits (e.g., processor circuits, discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) structured to perform the corresponding operations without executing software or firmware. The programmable circuits may be distributed across various network locations and / or may be local to one or more hardware devices (e.g., single-core processors (e.g., single-core CPUs), multi-core processors (e.g., multi-core CPUs, XPUs, etc.)). For example, a programmable circuit could be a CPU and / or FPGA located in the same package (e.g., the same integrated circuit (IC) package or two or more separate housings), one or more processors in a single machine, multiple processors distributed across multiple servers in a server rack, multiple processors distributed across one or more server racks, and / or any (one or more) combination thereof.

[0074]

[0086] The machine-readable instructions described herein may be stored in one or more of the following formats: compressed format, encrypted format, fragmented format, compiled format, executable format, packaged format, etc. The machine-readable instructions described herein may be stored as data (e.g., computer-readable data, machine-readable data, one or more bits (e.g., one or more computer-readable bits, one or more machine-readable bits, etc.), bitstreams (e.g., computer-readable bitstreams, machine-readable bitstreams, etc.)), or data structures (e.g., one or more parts of instructions, code, representations of code, etc.). This data or data structure may be used to create, manufacture, and / or generate machine-executable instructions. For example, machine-readable instructions may be fragmented and stored in one or more storage devices, disks, and / or computing devices (e.g., servers) located in a network or a collection of networks (e.g., in a cloud, in an edge device, etc.). Machine-readable instructions may require one or more of the following processes to become directly readable, interpretable, and / or executable by computing devices and / or other machines: installation, modification, adaptation, updating, synthesis, completion, configuration, decryption, expansion, decompression, distribution, reallocation, and compiling. For example, machine-readable instructions may be stored in multiple parts. Machine-readable instructions stored in such multiple parts may be individually compressed, encrypted, and stored on separate computing devices. When these are decrypted, expanded, and synthesized, they form a set of computer-executable and / or machine-executable instructions that perform one or more functions and / or operations that together may form a program as described herein.

[0075]

[0087] In another embodiment, machine-readable instructions may be stored in a readable state by a programmable circuit, but the execution of machine-readable instructions on a particular computing device or other device may require the addition of libraries (e.g., dynamic link libraries (DLLs)), software development kits (SDKs), application programming interfaces (APIs), etc. In yet another embodiment, it may be necessary to configure the machine-readable instructions (e.g., storing configurations, inputting data, recording network addresses, etc.) before the machine-readable instructions and / or corresponding programs (one or more) become fully or partially executable. Thus, machine-readable, computer-readable, and / or machine-readable media as used herein may include instructions and / or programs (one or more), regardless of the specific format or state of the machine-readable instructions and / or programs (one or more).

[0076]

[0088] The machine-readable instructions described herein may be expressed in any past, present, or future instruction language, scripting language, programming language, etc. For example, machine-readable instructions may be expressed using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, hypertext markup language (HTML), structured query language (SQL), Swift, etc.

[0077]

[0089] As described above, the exemplary operations in Figures 9 and 10 may be implemented using executable instructions (e.g., computer-readable and / or machine-readable instructions) stored in one or more non-transient computer-readable and / or machine-readable media. As used herein, non-transient computer-readable media, non-transient computer-readable storage media, non-transient machine-readable media, and / or non-transient machine-readable storage media are explicitly defined to include any type of computer-readable storage device and / or storage disk, but not to include propagated signals and not to include transmission media. Multiple examples of such non-transient computer-readable media, non-transient computer-readable storage media, non-transient machine-readable media, and / or non-transient machine-readable storage media include optical storage devices, magnetic storage devices, HDDs, flash memory, read-only memory (ROM), CDs, DVDs, caches, any type of RAM, registers, and / or any other storage devices or storage disks. Information is stored in these for any duration (e.g., for a long period, permanently, for a short period, for temporary buffering, and / or for caching information). As used herein, the terms “non-transient computer-readable storage medium” and “non-transient machine-readable storage medium” include any physical (mechanical, magnetic, and / or electrical) hardware that holds information over a period of time but does not include propagating signals and does not include a transmission medium. Multiple examples of non-transient computer-readable storage medium and / or non-transient machine-readable storage medium include any type of random-access memory, any type of read-only memory, solid-state memory, flash memory, optical disks, magnetic disks, disk drives, and / or redundant array (RAID) systems of independent disks. As used herein, the term “device” refers to a physical structure such as mechanical and / or electrical equipment, hardware, and / or circuits, which may or may not be composed of computer-readable instructions, machine-readable instructions, etc., and / or may be manufactured to execute computer-readable instructions, machine-readable instructions, etc.

[0078]

[0090] Figure 9 is a flowchart representing exemplary machine-readable instructions and / or exemplary actions 900 that may be executed, instantiated, and / or performed by a programmable circuit to detect light sources of a light pattern array in order to guide the movement of a vehicle carrying an image sensor (e.g., a camera). The exemplary machine-readable instructions and / or actions 900 in Figure 9 begin in block 902, in which the image interface circuit 802 retrieves and / or accesses image data / outputs from an image sensor (e.g., a camera) corresponding to light sources / lights of the vertiport and / or light pattern array. In the embodiment shown in Figure 9, the image interface circuit 802 captures or accesses images of the light pattern array corresponding to the field of view of the vehicle carrying the image sensor. In some embodiments, the vehicle's image sensor utilizes an RGB camera and an NIR camera to detect the aforementioned light sources.

[0079]

[0091] In block 904, the image processing circuit 804 performs image preprocessing on the image of the light pattern array captured by the aforementioned image sensor. According to several embodiments disclosed herein, the image processing circuit 804 performs spatial filtering of the image. In this embodiment, spatial filtering makes the features of the light source more easily visible.

[0080]

[0092] In block 906, the image processing circuit 804 performs clutter removal from the image (e.g., color-based removal). For example, four-color-based removal is performed. In some such embodiments, the image may include an image taken by a four-color camera carried by a vehicle.

[0081]

[0093] In block 908, the image processing circuit 804 performs clustering. The clustering performed by the image processing circuit 804 may correspond to clustering neighboring pixels together.

[0082]

[0094] In block 910, the image processing circuit 804 performs centroid calculation. According to some embodiments disclosed herein, the centroid coordinates of the aforementioned neighboring pixels are identified and / or calculated.

[0083]

[0095] In block 912, as will be described in more detail below in relation to Figure 10, an exemplary integrity monitoring circuit 806 performs multilayer integrity verification and monitoring of information (e.g., location information) corresponding to the output from the image sensor carried by the vehicle.

[0084]

[0096] In block 914, an exemplary position / attitude guidance computer circuit 808 determines the vehicle's relative position, heading, attitude, orientation, and / or altitude relative to the light source pattern array.

[0085]

[0097] In block 916, an exemplary position / attitude guidance computer circuit 808 guides the vehicle based on the vehicle's relative position to the light pattern array, its heading, and / or altitude. In some embodiments, the position / attitude guidance computer circuit 808 guides the vehicle to land on the light pattern array (for example, the light pattern array defines a vertiport).

[0086]

[0098] In block 918, an exemplary integrity monitoring circuit 806 and / or exemplary position / attitude guidance computer circuit 808 determine whether to repeat the process. If the process is repeated (block 918), control of the process returns to block 902. Otherwise, the process terminates. The determination may be based, for example, on whether the vehicle requires further guidance and / or whether the vehicle has landed on a vertiport.

[0087]

[0099] Figure 10 is a flowchart representing exemplary machine-readable instructions and / or exemplary operations 912 that may be executed, instantiated, and / or performed by a programmable circuit to verify data associated with a light pattern array in order to guide a vehicle based on the light pattern array. The exemplary machine-readable instructions and / or exemplary operations 912 in Figure 10 begin in block 1002, and exemplary integrity monitoring circuit 806 performs clutter removal and / or integrity protection. In some embodiments disclosed herein, four-color based clutter removal is performed.

[0088]

[0100] In block 1004, the integrity monitoring circuit 806 of the illustrated embodiment performs shape dimension-based candidate screening. The shape dimension / pattern / template matching may correspond, for example, to a line segment ratio. For example, the line segment ratio may correspond to, for example, the expected / inherent distance between light sources that uniquely identify a vertiport.

[0089]

[0101] In block 1006, an exemplary integrity monitoring circuit 806 identifies and verifies candidate solutions. For example, the integrity monitoring circuit 806 verifies image sensor / camera data and / or calculated data corresponding to the light pattern array. According to several embodiments disclosed herein, the redundancy of light sources in the light pattern array can be utilized (e.g., more than four light sources can be used to cross-validate the measurements). In this embodiment, the predicted positions of the light sources are matched to the measured positions of the light by utilizing database positions of the light sources and converting those database positions into camera frames using camera poses and positions identified using line-of-sight calculations for four (or more) light sources. When the measured and predicted positions of the light sources match, the light sources are consistent (e.g., with the expected configuration of the light sources). However, when the measured and predicted positions of the light sources do not match, it may indicate an error or that the candidate solution is incorrect.

[0090]

[0102] In block 1008, in this embodiment, the integrity monitoring circuit 806 matches image sensor / camera data with inertial data (e.g., INS data). In this embodiment, the integrity monitoring circuit 806 matches the position / or orientation calculated and / or identified using the camera data with an inertial solution based on IMU measurements (e.g., IMU measurements stored in the history) and previously verified camera updates in the history. Thus, the matching of camera position and orientation data with the inertial solution can be used to determine whether the position and orientation identified from the camera and / or image sensor data can be verified. In some embodiments, IMU voting (e.g., voting using three or more IMUs) is used to verify the inertial solution.

[0091]

[0103] In block 1010, the exemplary integrity monitoring circuit 806 determines whether to repeat the process. If the process is repeated (block 1010), control of the process returns to block 1002. The determination may be based on whether the solution has sufficient validity and / or completeness. Otherwise, the process terminates / returns.

[0092]

[0104] Figure 11 is a block diagram of an exemplary programmable circuit platform 1100 constructed to execute and / or instantiate exemplary machine-readable instructions and / or exemplary actions of Figures 9 and 10 for implementing the vision-based guidance analysis system 800 of Figure 8. The programmable circuit platform 1100 could be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a mobile phone, a smartphone, a tablet such as iPad®), a personal digital assistant (PDA), an internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, a headset or other wearable device, or any other type of computing and / or electronic device.

[0093]

[0105] The programmable circuit platform 1100 of the illustrated embodiment includes a programmable circuit 1112. The programmable circuit 1112 of the illustrated embodiment is hardware. For example, the programmable circuit 1112 may be implemented by one or more integrated circuits, logic circuits, FPGAs, microprocessors, CPUs, GPUs, DSPs, and / or microcontrollers from any desired affiliate or manufacturer. The programmable circuit 1112 may be implemented by one or more semiconductor-based (e.g., silicon-based) devices. In this embodiment, the programmable circuit 1112 implements an exemplary image interface circuit 802, an exemplary image processing circuit 804, an exemplary integrity monitoring circuit 806, and an exemplary position / attitude guidance computer circuit 808.

[0094]

[0106] The programmable circuit 1112 of the illustrated embodiment includes local memory 1113 (e.g., cache, registers, etc.). The programmable circuit 1112 of the illustrated embodiment communicates with main memory 1114, 1116 via bus 1118. Main memory 1114, 1116 includes volatile memory 1114 and non-volatile memory 1116. Volatile memory 1114 may be implemented by synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS® dynamic random access memory (RDRAM®), and / or any other type of RAM device. Non-volatile memory 1116 may be implemented by flash memory and / or any other desired type of memory device. Access to main memory 1114, 1116 of the illustrated embodiment is controlled by memory controller 1117. In some embodiments, the memory controller 1117 may be implemented by one or more integrated circuits, logic circuits, or any other type of circuitry for managing the flow of data to and from the main memories 1114, 1116 from any desired affiliate or manufacturer.

[0095]

[0107] The programmable circuit 1100 of the illustrated embodiment also includes an interface circuit 1120. The interface circuit 1120 may be implemented by hardware compliant with any type of interface standard, such as an Ethernet interface, Universal Serial Bus (USB), Bluetooth® interface, Near Field Communication (NFC) interface, Peripheral Interconnect (PCI) interface, and / or Peripheral Interconnect Express (PCIe).

[0096]

[0108] In the illustrated embodiment, one or more input devices 1122 are connected to the interface circuit 1120. The (one or more) input devices 1122 allow a user (e.g., a human user, a machine user, etc.) to input data and / or commands into the programmable circuit 1112. The (one or more) input devices 1122 may be implemented by, for example, a voice sensor, a microphone, a camera (still image or video), a keyboard, a button, a mouse, a touchscreen, a trackpad, a trackball, an isopoint device, and / or a voice recognition system.

[0097]

[0109] One or more output devices 1124 are also connected to the interface circuit 1120 of the illustrated embodiment. The (one or more) output devices 1124 may be implemented by, for example, display devices (e.g., light-emitting diodes (LEDs), organic light-emitting diodes (OLEDs), liquid crystal displays (LCDs), cathode ray tube displays (CRTs), positional switching (IPS) displays, touchscreens, etc.), haptic output devices, printers, and / or speakers. Thus, the interface circuit 1120 of the illustrated embodiment typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor (such as a GPU).

[0098]

[0110] The interface circuit 1120 in the illustrated embodiment also includes communication devices such as transmitters, receivers, transceivers, modems, resident gateways, wireless access points, and / or network interfaces, facilitating data exchange with external machines (e.g., any type of computing device) via the network 1126. Communication may be via, for example, Ethernet connections, digital subscriber line (DSL) connections, telephone line connections, coaxial cable systems, satellite systems, beyond-line-of-site wireless systems, line-of-sight wireless systems, cellular systems, optical connections, and the like.

[0099]

[0111] The programmable circuit 1100 of the illustrated embodiment also includes one or more mass storage disks or devices 1128 for storing firmware, software, and / or data. Multiple examples of such mass storage disks or devices 1128 include magnetic storage devices (e.g., floppy disks, drives, HDDs, etc.), optical storage devices (e.g., Blu-ray discs, CDs, DVDs, etc.), RAID systems, and / or solid-state storage disks or devices such as flash memory devices and / or SSDs.

[0100]

[0112] The machine-readable instructions 1132 may be implemented by the machine-readable instructions in Figures 9 and 10 and may be stored in the mass storage device 1128, in the volatile memory 1114, in the non-volatile memory 1116, and / or in at least one non-transient computer-readable storage medium such as a removable CD or DVD.

[0101]

[0113] Figure 12 is a block diagram of an exemplary implementation of the programmable circuit 1112 of Figure 11. In this embodiment, the programmable circuit 1112 of Figure 11 is implemented by a microprocessor 1200. For example, the microprocessor 1200 may be a general-purpose microprocessor (e.g., a general-purpose microprocessor circuit). The microprocessor 1200 executes some or all of the machine-readable instructions in the flowcharts of Figures 9 and 10 so as to effectively instantiate the circuit of Figure 8 as a logic circuit to perform the operations corresponding to these machine-readable instructions. In several such embodiments, the circuit of Figure 8 is instantiated by the hardware circuit of the microprocessor 1200 combined with the machine-readable instructions. For example, the microprocessor 1200 may be implemented by a multicore hardware circuit such as a CPU, DSP, GPU, or XPU. It may include any number of exemplary cores 1202 (e.g., one core), but the microprocessor 1200 in this embodiment is a multicore semiconductor device containing N cores. The cores 1202 of the microprocessor 1200 may operate independently or in cooperation to execute machine-readable instructions. For example, machine code, embedded software programs, or software programs corresponding to a firmware program may be executed by one of the cores 1202, or by multiple cores 1202, sometimes the same and sometimes different. In some embodiments, the machine code, embedded software programs, or software programs corresponding to a firmware program are divided into threads and executed in parallel by two or more cores 1202. The software programs correspond to some or all of the machine-readable instructions and / or operations represented by the flowcharts in Figures 9 and 10.

[0102]

[0114] The core 1202 may communicate by a first exemplary bus 1204. In some embodiments, the first bus 1204 may be implemented by a communication bus to perform communication associated with one or more of the cores 1202. For example, the first bus 1204 may be implemented by at least one of an inter-integrated circuit (I2C) bus, a serial peripheral interface (SPI) bus, a PCI bus, or a PCIe bus. Further or alternatively, the first bus 1204 may be implemented by any other type of computing or electrical bus. The core 1202 may receive data, instructions, and / or signals from one or more external devices by an exemplary interface circuit 1206. The core 1202 may output data, instructions, and / or signals to one or more external devices by the interface circuit 1206. The core 1202 in this embodiment includes exemplary local memory 1220 (e.g., a level 1 (L1) cache which can be divided into an L1 data cache and an L1 instruction cache), and the microprocessor 1200 also includes exemplary shared memory 1210 which can be shared by a core (e.g., a level 2 (L2 cache)) for high-speed access to data and / or instructions. Data and / or instructions can be transferred (e.g., shared) by reading from and writing to the shared memory 1210. Each of the local memory 1220 and shared memory 1210 of the core 1202 may be part of a hierarchy of storage devices which includes multiple levels of cache memory and main memory (e.g., main memories 1114, 1116 in Figure 11). Typically, higher levels of memory in the hierarchy exhibit less access time and have smaller storage capacity than lower levels of memory. The changes in the various levels of the cache hierarchy are managed (e.g., coordinated) by a cache coherency policy.

[0103]

[0115] Each core 1202 may be called a CPU, DSP, GPU, or any other type of hardware circuit. Each core 1202 includes a control unit circuit 1214, an arithmetic and logic (AL) circuit (often called an ALU) 1216, several registers 1218, local memory 1220, and a second exemplary bus 1222. Other structures may be present. For example, each core 1202 may include a vector unit circuit, a single-instruction multiplexed data (SIMD) unit circuit, a load / store unit (LSU) circuit, a branch / jump unit circuit, a floating-point unit (FPU) circuit, etc. The control unit circuit 1214 includes semiconductor-based circuitry built to control (e.g., coordinate) data movement within the corresponding core 1202. The AL circuit 1216 includes semiconductor-based circuitry built to perform one or more mathematical and / or logical operations on data within the corresponding core 1202. In some embodiments, the AL circuit 1216 performs integer-based operations. In other embodiments, the AL circuit 1216 also performs floating-point arithmetic. In several other embodiments, the AL circuit 1216 may include a first AL circuit that performs integer-based arithmetic and a second AL circuit that performs floating-point arithmetic. In some embodiments, the AL circuit 1216 may be called an arithmetic logic unit.

[0104]

[0116] Register 1218 is a semiconductor-based structure for storing data and / or instructions, such as one or more results of operations performed by the AL circuit 1216 of the corresponding core 1202. For example, register 1218 may include (one or more) vector registers, (one or more) SIMD registers, (one or more) general-purpose registers, (one or more) flag registers, (one or more) segment registers, (one or more) machine-specific registers, (one or more) control registers, (one or more) debug registers, (one or more) memory management registers, (one or more) machine check registers, etc. Register 1218 may be arranged in a bank as shown in Figure 12. Alternatively, register 1218 may be organized in any other arrangement, format, or structure, such as being distributed throughout the core 1202 to reduce access time. The second bus 1222 may be implemented by at least one of the I2C bus, SPI bus, PCI bus, or PCIe bus.

[0105]

[0117] Each core 1202 and / or more generally a microprocessor 1200 may include additional and / or alternative structures to those illustrated and described above. For example, one or more clock circuits, one or more power supplies, one or more power gates, one or more cache home agents (CHAs), one or more convergence / common mesh stops (CMS), one or more shifters (e.g., one or more barrel shifters), and / or other circuits may be present. The microprocessor 1200 is a semiconductor device manufactured to include many transistors interconnected to implement the structures described above within one or more integrated circuits (ICs) housed in one or more packages.

[0106]

[0118] The microprocessor 1200 may include and / or cooperate with one or more accelerators (e.g., accelerating circuits, hardware accelerators, etc.). In some embodiments, the accelerator may be implemented by logic circuits to perform a particular task faster and / or more efficiently than could be done by a general-purpose processor. Several examples of accelerators include ASICs and FPGAs, such as those described herein. GPUs, DSPs, and / or other programmable devices may also be accelerators. The accelerator may be mounted on the microprocessor 1200, may be in the same chip package as the microprocessor 1200, and / or may be in one or more separate packages from the microprocessor 1200.

[0107]

[0119] Figure 13 is a block diagram of another exemplary implementation of the programmable circuit 1112 of Figure 11. In this embodiment, the programmable circuit 1112 is implemented by an FPGA circuit 1300. For example, the FPGA circuit 1300 may be implemented by an FPGA. The FPGA circuit 1300 may be used to perform operations that would normally be performed by the exemplary microprocessor 1200 of Figure 12, which executes the corresponding machine-readable instructions. However, once configured, the FPGA circuit 1300 can instantiate operations and / or functions corresponding to machine-readable instructions in hardware and therefore can often perform operations / functions faster than they could be performed by a general-purpose microprocessor that executes the corresponding software.

[0108]

[0120] More specifically, in contrast to the microprocessor 1200 in Figure 12 described above (a general-purpose device that can be programmed to execute some or all of the machine-readable instructions represented by the flowcharts in Figures 9 and 10, but whose interconnects and logic circuits are fixed once manufactured), the FPGA circuit 1300 in the embodiment of Figure 13 includes interconnects and logic circuits that can be configured, built, programmed, and / or interconnected in various ways after manufacturing to instantiate some or all of the operations / functions corresponding to the machine-readable instructions represented by the flowcharts in Figures 9 and 10, for example. In particular, the FPGA circuit 1300 can be thought of as an array of logic gates, interconnects, and switches. The switches can be programmed to change the way in which the logic gates are interconnected by the interconnects, effectively forming one or more dedicated logic circuits (unless the FPGA circuit 1300 is reprogrammed). The configured logic circuits enable the logic gates to cooperate in various ways to perform various operations on data received by the input circuits. These operations may correspond to some or all of the instructions (e.g., software and / or firmware) represented by the flowcharts in Figures 9 and 10. Therefore, the FPGA circuit 1300 may be configured and / or constructed as a dedicated logic circuit for executing operations / functions corresponding to these software instructions in a dedicated manner similar to that of an ASIC, effectively instantiating some or all of the operations / functions corresponding to the machine-readable instructions in the flowcharts of Figures 9 and 10. Thus, the FPGA circuit 1300 may execute operations / functions corresponding to some or all of the machine-readable instructions in Figures 9 and 10 faster than a general-purpose microprocessor could.

[0109]

[0121] In the embodiment of Figure 13, the FPGA circuit 1300 is configured and / or built in accordance with being programmed (and / or reprogrammed one or more times) based on a binary file. In some embodiments, the binary file may be edited and / or generated based on instructions in a hardware description language (HDL), such as Lucid, VHSIC (Very High Speed ​​Integrated Circuits), VHDL (Hardware Description Language), or Verilog. For example, a user (e.g., a human user, a machine user, etc.) may write code or a program in the HDL that corresponds to one or more operations / functions, the code / program may be translated into a lower-level language as needed, and the code / program (e.g., code / program in a lower-level language) may be converted into a binary file (e.g., by a compiler, a software application, etc.). In some embodiments, a binary file may be accessed and / or loaded to configure and / or build the FPGA circuit 1300 in Figure 13 so that the FPGA circuit 1300 in Figure 13 performs one or more operations / functions. For example, a binary file may be implemented by a bitstream (e.g., one or more computer-readable bits, one or more machine-readable bits, etc.), data (e.g., computer-readable data, machine-readable data, etc.), and / or machine-readable instructions that are accessible to the FPGA circuit 1300 in Figure 13 in order to configure and / or construct the FPGA circuit 1300 or (one or more) parts thereof.

[0110]

[0122] In some embodiments, a binary file is edited, generated, converted, and / or output from a uniform software platform used to program the FPGA. For example, the uniform software platform can convert a first instruction (e.g., code or program) corresponding to one or more operations / functions in a high-level language (e.g., C, C++, Python, etc.) into a second instruction corresponding to one or more operations / functions in HDL. In some such embodiments, a binary file is edited, generated, and / or output from the uniform software platform based on the second instruction. In some embodiments, a binary file can be accessed and / or loaded to configure and / or build the FPGA circuit 1300 in Figure 13 so that the FPGA circuit 1300 in Figure 13 performs one or more operations / functions. For example, a binary file may be implemented by a bitstream (e.g., one or more computer-readable bits, one or more machine-readable bits, etc.), data (e.g., computer-readable data, machine-readable data, etc.), and / or machine-readable instructions that are accessible to the FPGA circuit 1300 in Figure 13 in order to configure and / or construct the FPGA circuit 1300 or (one or more) parts thereof.

[0111]

[0123] The FPGA circuit 1300 in Figure 13 includes an exemplary input / output (I / O) circuit 1302 for acquiring and / or outputting data to and from an exemplary configuration circuit 1304 and / or external hardware 1306. For example, the configuration circuit 1304 may be implemented by an interface circuit. This interface circuit may acquire a binary file. This binary file may be implemented by bitstreams, data, and / or machine-readable instructions to constitute the FPGA circuit 1300 or one or more parts thereof. In some such embodiments, the configuration circuit 1304 may acquire the binary file from a user, a machine (e.g., hardware circuitry (e.g., programmable or dedicated circuitry) that may implement an artificial intelligence / machine learning (AI / ML) model to generate the binary file), and / or any one or more combinations thereof. In some embodiments, the external hardware 1306 may be implemented by an external hardware circuit. For example, the external hardware 1306 may be implemented by the microprocessor 1200 in Figure 12.

[0112]

[0124] The FPGA circuit 1300 also includes an array of logic gate circuits 1308, a plurality of exemplary configurable interconnects 1310, and an exemplary storage circuit 1312. The logic gate circuits 1308 and the configurable interconnects 1310 are configurable to instantiate one or more operations / functions that can correspond to at least some of the machine-readable instructions and / or other desired operations shown in Figures 9 and 10. The logic gate circuits 1308 shown in Figure 13 are manufactured in blocks or groups. Each block includes a semiconductor-based electrical structure that can be configured into a logic circuit. In some embodiments, the electrical structure includes logic gates (e.g., AND gates, OR gates, Nor gates, etc.) that provide basic building blocks for the logic circuit. Electrically controllable switches (e.g., transistors) are present within each of the logic gate circuits 1308, enabling the configuration of the electrical structure and / or logic structure to form a circuit for performing a desired operation / function. The logic gate circuit 1308 may include other electrical structures such as lookup tables (LUTs), registers (e.g., flip-flops or latches), and multiplexers.

[0113]

[0125] The configurable interconnections 1310 in the illustrated embodiment include conductive paths, traces, vias, etc., which can have their state changed by programming a desired logic circuit by activating or deactivating one or more connections between one or more logic gate circuits 1308 (for example, using an HDL instruction language).

[0114]

[0126] The storage circuit 1312 in the illustrated embodiment is constructed to store one or more (one or more) results of operations performed by the corresponding logic gates. The storage circuit 1312 may be implemented by registers or the like. In the illustrated embodiment, the storage circuit 1312 is distributed among the logic gate circuits 1308 to facilitate access and increase execution speed.

[0115]

[0127] The exemplary FPGA circuit 1300 in Figure 13 also includes exemplary dedicated operation circuit 1314. In this embodiment, the dedicated operation circuit 1314 includes a special-purpose circuit 1316. This special-purpose circuit 1316 may be invoked to perform commonly used functions in order to avoid the need to program these functions in the field. Several examples of such special-purpose circuits 1316 include memory (e.g., DRAM) controller circuits, PCIe controller circuits, clock circuits, transceiver circuits, memory, and multiplier storage circuits. Other types of special-purpose circuits may exist. In some embodiments, the FPGA circuit 1300 may also include exemplary general-purpose programmable circuits 1318, such as an exemplary CPU 1320 and / or an exemplary DSP 1322. Other general-purpose programmable circuits 1318, such as a GPU, XPU, etc., which can be programmed to perform other operations, may be further or alternatively present.

[0116]

[0128] Figures 12 and 13 show two exemplary implementations of the programmable circuit 1112 of Figure 11, but many other approaches are considered. For example, the FPGA circuit may include an onboard CPU, such as one or more of the exemplary CPUs 1320 of Figure 12. Thus, the programmable circuit 1112 of Figure 11 may be further implemented by combining at least the exemplary microprocessor 1200 of Figure 12 and the exemplary FPGA circuit 1300 of Figure 13. In some such hybrid embodiments, one or more cores 1202 of Figure 12 may execute a first portion of machine-readable instructions represented by the flowcharts of Figures 9 and 10 to perform a first (one or more) operation / (one or more) function. The FPGA circuit 1300 of Figure 13 may be configured and / or constructed to perform a second (one or more) operation / (one or more) function corresponding to a second portion of machine-readable instructions represented by the flowcharts of Figures 9 and 10. Furthermore / or, the ASIC may be configured and / or constructed to perform a third (one or more) operation / (one or more) function corresponding to a third portion of the machine-readable instructions represented by the flowcharts in Figures 9 and 10.

[0117]

[0129] Therefore, it should be understood that some or all of the circuit in Figure 8 may be instantiated at the same or different times. For example, the same and / or different (one or more) parts of the microprocessor in Figure 12 may be programmed to execute (one or more) parts of machine-readable instructions at the same and / or different times. In some embodiments, the same and / or different (one or more) parts of the FPGA circuit 1300 in Figure 13 may be configured and / or constructed to perform operations / functions corresponding to (one or more) parts of machine-readable instructions at the same and / or different times.

[0118]

[0130] In some embodiments, some or all of the circuitry in Figure 8 may be instantiated, for example, in one or more threads running concurrently and / or in series. For example, the microprocessor 1200 in Figure 12 may execute machine-readable instructions in one or more threads running concurrently and / or in series. In some embodiments, the FPGA circuit 1300 in Figure 13 may be configured and / or constructed to perform operations / functions concurrently and / or in series. Furthermore, in some embodiments, some or all of the circuitry in Figure 8 may be implemented in one or more virtual machines and / or containers running on the microprocessor 1200 in Figure 12.

[0119]

[0131] In some embodiments, the programmable circuit 1112 in Figure 11 may be contained within one or more packages. For example, the microprocessor 1200 in Figure 12 and / or the FPGA circuit 1300 in Figure 13 may be contained within one or more packages. In some embodiments, the XPU may be implemented by the programmable circuit 1112 in Figure 11. This programmable circuit 1112 may be contained within one or more packages. For example, the XPU may contain a CPU (microprocessor 1200 in Figure 12, CPU 1320 in Figure 13, etc.) in one package, a DSP (DSP 1322 in Figure 13) in another package, a GPU in yet another package, and an FPGA (FPGA circuit 1300 in Figure 13) in yet another package.

[0120]

[0132] The terms “including” and “comprising” (and all their forms and tenses) are used herein as open-ended terms. Therefore, whenever a claim uses any form of “include” or “comprise” (e.g., “comprises,” “includes,” “comprising,” “including,” “having,” etc.) as a preamble or within any enumeration of claims, it should be understood that further elements, terms, etc., may exist without departing from the scope of the corresponding claim or enumeration. As used herein, the expression “at least” is open-ended, just as the words “comprise” and “include” are open-ended, for example, when used as a transition term in the preamble of claims. For example, when the term “and / or” is used in the form of A, B, and / or C, it means any combination or subset of A, B, and / or C, such as (1) A only, (2) B only, (3) C only, (4) A and B, (5) A and C, (6) B and C, and (7) A, B, and C. When used herein, in the context of describing a structure, component, article, object, and / or thing, the expression “at least one of A and B” is intended to refer to an embodiment that includes (1) at least one A, (2) at least one B, and (3) either at least one A or at least one B. Similarly, when used herein, in the context of describing a structure, component, article, object, and / or thing, the expression “at least one of A or B” is intended to refer to an embodiment that includes (1) at least one A, (2) at least one B, and (3) either at least one A or at least one B.When used herein, in a context describing the implementation or execution of a process, instruction, operation, and / or step, the expression “at least one of A and B” is intended to refer to embodiments comprising (1) at least one A, (2) at least one B, and (3) either at least one A or at least one B. Similarly, when used herein, in a context describing the implementation or execution of a process, instruction, operation, and / or step, the expression “at least one of A or B” is intended to refer to embodiments comprising (1) at least one A, (2) at least one B, and (3) either at least one A or at least one B.

[0121]

[0133] As used herein, singular references (e.g., “a,” “an,” “first,” “second,” etc.) do not exclude the plural. When used herein, the term “one (a)” or “one (an)” object refers to one or more of those objects. The terms “one (a) (or “one (an)”),” “one or more,” and “at least one” are used interchangeably herein. Furthermore, even if listed individually, multiple means, elements, or actions may be performed, for example, by the same entity or object. In addition, individual features may be included in various examples or claims, but such features may also be combined, and inclusion in various examples or claims does not imply that the combination of features is not feasible and / or beneficial.

[0122]

[0134] As used herein, unless otherwise specified, the term “above” describes the relationship between two parts with respect to the Earth. The first part is above the second part if the second part has at least one part between the Earth and the first part. Similarly, as used herein, the first part is “below” the second part if the first part is closer to the Earth than the second part. As described above, the first part can be above or below the second part in one or more of the following ways: there is another part between them, there is no other part between them, the first and second parts are in contact, or the first and second parts are not in direct contact with each other.

[0123]

[0135] When used in this patent application, the statement that any part (e.g., a layer, film, area, region, or plate) is present on another part in any way (e.g., placed, positioned, arranged, formed, etc.) indicates that the part referred to is in contact with the other part, or that the part referred to is present above the other part with one or more intermediate parts positioned between them.

[0124]

[0136] When used herein, references to connections (e.g., attached, linked, connected, and joined) may, unless otherwise indicated, include intermediate material between the elements referred to by the reference to connections and / or relative movement between those elements. Thus, references to connections do not necessarily imply that two elements are directly connected and / or have a fixed relationship with one another. When used herein, the statement that any part is "in contact" with another part is defined to mean that there is no intermediate part between the two parts.

[0125]

[0137] Unless otherwise specified, descriptions such as “first,” “second,” and “third” either indicate priority, physical order, placement within a list, and / or order in any manner, or they are used herein without any indication of any of these meanings, but merely as labels and / or arbitrary names to distinguish elements for the purpose of facilitating understanding of the disclosed embodiments. In some embodiments, the description “first” in a mode for carrying out the invention may be used to refer to an element, while the same element may be referred to by different descriptions such as “second” or “third” in the claims. In several such instances, it should be understood that such descriptors are used simply to clearly identify elements in the context of the discussion (e.g., within the claims) that might otherwise share the same name.

[0126]

[0138] When used herein, “approximately” and “about” modify their subject / values ​​to acknowledge the potential existence of variations that may occur in real-world applications. For example, “approximately” and “about” may modify dimensions that may not be accurate due to manufacturing tolerances and / or other real-world imperfections, as understood by those skilled in the art. For example, “approximately” and “about” may indicate that, unless otherwise specified herein, such dimensions may be within a tolerance of ±10%.

[0127]

[0139] As used herein, "substantially real time" refers to an event occurring in a nearly instantaneous manner, acknowledging that real-world delays may exist in computing time, transmission, etc. Therefore, unless otherwise specified, "substantially real time" means real time plus one second.

[0128]

[0140] As used herein, the expression “in communication,” including its variations, encompasses direct communication and / or indirect communication through one or more intermediate components, and does not necessarily have to be direct and physical (e.g., wired) communication and / or constant communication, but rather includes selective communication, additionally, at periodic intervals, scheduled intervals, aperiodic intervals and / or one-time events.

[0129]

[0141] As used herein, “programmable circuit” is defined as including (i) one or more special-purpose electrical circuits (e.g., application-specific integrated circuits (ASICs)) configured to perform a particular operation and comprising one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors), and / or (ii) one or more general-purpose semiconductor-based electrical circuits that are programmable by instructions to perform one or more specific functions and one or more operations, and which comprise one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors). Multiple examples of programmable circuits include programmable microprocessors, such as a central processing unit (CPU) capable of executing a first instruction to perform one or more operations and / or functions; a field-programmable gate array (FPGA) that can be programmed with a second instruction to result in the configuration and / or construction of an FPGA in order to instantiate one or more operations and / or functions corresponding to the first instruction; a graphics processor unit (GPU) capable of executing a first instruction to perform one or more operations and / or functions; a digital signal processor (DSP) capable of executing a first instruction to perform one or more operations and / or functions; and one or more microcontrollers capable of executing a first instruction to perform one or more operations and / or functions and / or integrated circuits, such as an XPU, a network processing unit (NPU), or an ASIC. For example, an XPU may be implemented by a heterogeneous computing system that includes multiple types of programmable circuits (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more NPUs, one or more DSPs, and / or any (one or more) combination thereof) and orchestration techniques (e.g., one or more application programming interfaces) that can assign (one or more) computing tasks to one or more of the multiple types of programmable circuits suitable for and available for (one or more) computing tasks.

[0130]

[0142] As used herein, an integrated circuit / network is defined as one or more semiconductor packages containing one or more circuit elements, such as transistors, capacitors, inductors, resistors, current paths, and diodes. For example, an integrated circuit may be implemented as one or more of the following: an ASIC, FPGA, chip, microchip, programmable circuit, semiconductor substrate combining multiple circuit elements, or a system-on-a-chip (SoC).

[0131]

[0143] Exemplary methods, apparatus, systems, and articles that enable robust vision-based guidance for vehicles are disclosed herein. Further embodiments and combinations thereof include:

[0132]

[0144] Embodiment 1 includes a system, the image sensor carried by a vehicle, the image sensor for identifying light sources of a ground-based pattern display, the spatial arrangement of the light sources at various distances between them defining a light pattern array, and the image sensor includes at least one processor circuit for performing multilayer integrity verification corresponding to the light pattern array based on the identified light sources for guidance of the vehicle via the light pattern array.

[0133]

[0145] Example 2 includes the system defined in Example 1. In this case, the image sensor is a four-color image sensor, and one or more of the at least one processor circuit is for performing four-color based clutter removal based on the output of the four-color image sensor.

[0134]

[0146] Example 3 includes the system defined in Example 1. In this case, the first light source of the pattern display is for emitting light of a first emission spectrum, and the second light source of the pattern display is for emitting light of a second emission spectrum different from the first emission spectrum.

[0135]

[0147] Example 4 includes the system defined in Example 3. In this case, the first light source emits near-field infrared (NIR) light, and the second light source emits red-green-blue (RGB) light.

[0136]

[0148] Example 5 includes the system defined in Example 1. In this case, the pattern display includes the arrangement of the light source on a rectangular periphery.

[0137]

[0149] Example 6 includes the system defined in Example 5. In this case, the light sources are arranged along the periphery, and at least one of the light sources is arranged with non-uniform spacing from another of the light sources.

[0138]

[0150] Example 7 includes the system defined in Example 5. In this case, the distance between the light sources varies between the edges of the rectangular periphery.

[0139]

[0151] Embodiment 8 includes an apparatus for verifying data corresponding to a ground-based light pattern array for vehicle guidance, the apparatus including an interface circuit communicatively coupled to an image sensor supported by the vehicle, machine-readable instructions, and a programmable circuit which instantiates or executes the machine-readable instructions for at least one of the following: identifying a light source of the ground-based light pattern array, the arrangement of which defines the light pattern array, based on the output from the image sensor; verifying the data corresponding to the light pattern array based on the identified light source via multilayer integrity verification; and enabling vehicle guidance via the light pattern array in accordance with the data corresponding to the verified light pattern array.

[0140]

[0152] Example 9 includes the apparatus defined in Example 8. In this case, the programmable circuit instantiates or executes machine-readable instructions for performing the multilayer integrity verification of the data corresponding to the optical pattern array by performing clutter removal, performing shape-dimension-based candidate screening, performing image-based solution verification, and matching inertial data with the image-based solution.

[0141]

[0153] Example 10 includes the apparatus defined in Example 8, in which the programmable circuit instantiates or executes the machine-readable instructions for performing color-based clutter removal.

[0142]

[0154] Example 11 includes the apparatus defined in Example 8. In this case, the programmable circuit instantiates or executes the machine-readable instructions for performing clustering and centroid calculation of images corresponding to the output from the image sensor.

[0143]

[0155] Example 12 includes the apparatus defined in Example 8. In this case, the programmable circuit instantiates or executes the machine-readable instructions for performing spatial filtering of the image corresponding to the output of the image sensor.

[0144]

[0156] Example 13 includes the apparatus defined in Example 9. In this case, the verification of the light pattern includes verifying the image sensor data using inertial Kalman filter residue.

[0145]

[0157] Example 14 includes the apparatus defined in Example 8. In this case, the programmable circuit instantiates or executes the machine-readable instructions for performing a ransack with respect to the verification of the pattern.

[0146]

[0158] Example 15 includes the apparatus defined in Example 8. In this case, the programmable circuit instantiates or executes the machine-readable instructions for estimating the affine transformation corresponding to the estimated attitude and position of the vehicle.

[0147]

[0159] Example 16 includes the apparatus defined in Example 8. In this case, the programmable circuit instantiates or executes the machine-readable instructions for performing clutter removal by image filtering of the output from the image sensor.

[0148]

[0160] Example 17 includes at least one non-transient machine-readable medium, the at least one non-transient machine-readable medium including a machine-readable instruction, the machine-readable instruction causing at least one processor circuit to at least identify a light source for a pattern display that defines a pattern arrangement, based on the output from an image sensor of the vehicle; verify data corresponding to the pattern arrangement in a multilayer integrity verification based on the identified light source; and determine the position of the vehicle relative to the pattern arrangement in accordance with the verified data.

[0149]

[0161] Example 18 includes at least one non-transient machine-readable medium as defined in Example 17. In this case, the machine-readable instruction causes one or more of the at least one processor circuits to perform color-based decolorization of the output of the image sensor.

[0150]

[0162] Example 19 includes at least one non-transient machine-readable medium as defined in Example 18. In this case, the machine-readable instruction causes one or more of the at least one processor circuits to perform at least one of clustering and centroid calculation of the output of the image sensor.

[0151]

[0163] Example 20 includes at least one non-transient machine-readable medium as defined in Example 17. In this case, the machine-readable instruction causes one or more of the at least one processor circuits to verify the data corresponding to the pattern arrangement by performing clutter removal, performing shape-dimension-based candidate screening, verifying the solution corresponding to the image sensor, and matching the solution corresponding to the image sensor with the vehicle's inertia data.

[0152]

[0164] Example 21 includes at least one non-transient machine-readable medium as defined in Example 17. In this case, the machine-readable instruction causes one or more of the at least one processor circuits to identify the light source based on the identification of different types of light sources in the pattern arrangement.

[0153]

[0165] Example 22 includes at least one non-transient machine-readable medium as defined in Example 17. In this case, the machine-readable instruction causes one or more of the at least one processor circuits to guide the vehicle to the pattern arrangement for landing.

[0154]

[0166] Example 23 includes at least one non-transient machine-readable medium as defined in Example 17. In this case, the machine-readable instruction causes one or more of the at least one processor circuits to identify a two-dimensional pattern of the landing pad corresponding to the pattern display, based on the pattern arrangement.

[0155]

[0167] Example 24 includes at least one non-transient, machine-readable medium as defined in Example 17. In this case, the verification of the data corresponding to the pattern arrangement includes verifying the output of the image sensor using inertial Kalman filter residue.

[0156]

[0168] Example 25 includes a method, which includes identifying light sources of a ground-based light pattern array based on output from an image sensor carried by a vehicle, wherein the spatial arrangement of the light sources defines the light pattern array; verifying data corresponding to the light pattern array through multilayer integrity verification; and enabling guidance for the vehicle via the light pattern array in accordance with the data corresponding to the verified light pattern array.

[0157]

[0169] Example 26 includes the method defined in Example 25. In this case, verifying the data corresponding to the optical pattern array includes performing clutter removal, performing shape-dimension-based candidate screening, verifying the solution corresponding to the image sensor, and matching the solution corresponding to the image sensor with the vehicle's inertial data.

[0158]

[0170] As described above, it will be understood that exemplary systems, apparatus, articles, and methods are disclosed that enable effective guidance of vehicles. Several embodiments disclosed herein may enable accurate verification of data corresponding to ground-based pattern displays such as vertiports. Several embodiments disclosed herein may enable meeting the requirements necessary for the autonomous operation of passenger vehicles such as passenger aircraft. Furthermore, several embodiments disclosed herein may also save weight and / or space in vehicles such as aircraft, ships, or spacecraft. The disclosed systems, apparatus, articles, and methods improve the efficiency of using computing devices by reducing the computational requirements typically associated with high-precision guidance systems. The disclosed systems, apparatus, articles, and methods therefore address one or more improvements in the operation of machines such as computers or other electronic and / or mechanical devices. Furthermore, several embodiments disclosed herein may enable the use of relatively inexpensive cameras, in contrast to high-resolution cameras commonly used in vision-based guidance.

[0159]

[0171] The following claims are incorporated into the embodiments for carrying out this invention by this reference. Certain exemplary systems, apparatus, articles, and methods have been disclosed herein, but the scope of this patent application is not limited to these. Rather, this patent application covers all systems, apparatus, articles, and methods that fairly fall within the scope of the claims of this patent application.

Claims

1. The system is (100, 200), An image sensor (202) carried by a vehicle (100), wherein the image sensor is for identifying light sources (134) of a ground-based pattern display (102), and the spatial arrangement of various distances between the light sources defines the light pattern array, and A system comprising at least one processor circuit (1112) for performing multilayer integrity verification corresponding to the light pattern array based on the identified light source for guiding the vehicle via the light pattern array.

2. The system according to claim 1, wherein the image sensor is a four-color image sensor, and one or more of the at least one processor circuit performs four-color based clutter removal based on the output of the four-color image sensor.

3. The system according to claim 1, wherein the first light source of the pattern display emits light of a first emission spectrum, and the second light source of the pattern display emits light of a second emission spectrum different from the first emission spectrum.

4. The system according to claim 3, wherein the first light source emits near-infrared (NIR) light, and the second light source emits red-green-blue (RGB) light.

5. The system according to claim 1, wherein the pattern display includes the arrangement of the light source on a rectangular periphery.

6. The system according to claim 5, wherein the light sources are arranged along the periphery, and at least one of the light sources is arranged with a non-uniform space between it and another of the light sources.

7. The system according to claim 5, wherein the distance between the light sources varies between the edges of the rectangular periphery.

8. A device for verifying data corresponding to a ground-based optical pattern array for vehicle guidance, An interface circuit (1206) that is communicatively coupled to an image sensor supported by the vehicle, Machine-readable instructions (1132), and Includes a programmable circuit, the programmable circuit is Based on the output from the image sensor, identify the light source of the ground-based light pattern array, wherein the arrangement of the light source defines the light pattern array. Through multilayer integrity verification, the data corresponding to the light pattern array is verified based on the identified light source, and A device that instantiates or executes machine-readable instructions to enable guidance of the vehicle via the light pattern array in accordance with the data corresponding to the verified light pattern array.

9. The programmable circuit is Perform clutter removal. Perform a shape and dimension-based candidate screening. Perform image-based solution validation, and The apparatus according to claim 8, wherein at least one of the following is instantiating or executing a machine-readable instruction for performing the multilayer integrity verification of the data corresponding to the optical pattern array by matching the inertial data with the image-based solution.

10. The apparatus according to claim 8, wherein the programmable circuit instantiates or executes the machine-readable instructions for performing color-based clutter removal.

11. The apparatus according to claim 8, wherein the programmable circuit instantiates or executes machine-readable instructions for performing clustering and centroid calculation of an image corresponding to the output from the image sensor.

12. The apparatus according to claim 8, wherein the programmable circuit instantiates or executes machine-readable instructions for performing spatial filtering of an image corresponding to the output of the image sensor.

13. The apparatus according to claim 8, wherein the verification of the light pattern includes verifying the data from the image sensor using inertial Kalman filter residue.

14. The apparatus according to claim 8, wherein the programmable circuit instantiates or executes the machine-readable instructions for performing a ransack with respect to the verification of the pattern.

15. The apparatus according to claim 8, wherein the programmable circuit instantiates or executes machine-readable instructions for estimating affine transformations corresponding to estimated candidate attitudes and positions of the vehicle.

16. The apparatus according to claim 8, wherein the programmable circuit instantiates or executes machine-readable instructions for performing clutter removal by image filtering of the output from the image sensor.

17. A non-transient machine-readable medium containing a machine-readable instruction, wherein the machine-readable instruction is provided to at least one processor circuit, Based on the output from the vehicle's image sensors, identify the light source for the pattern display that defines the pattern arrangement. Based on the identified light source, verify the data corresponding to the pattern arrangement in a multilayer integrity verification, and A non-transient, machine-readable medium that causes the system to determine the position of the vehicle relative to the pattern arrangement in accordance with the verified data.

18. The machine-readable instruction causes one or more of the at least one processor circuits to perform color-based removal of the output of the image sensor, according to claim 17, for at least one non-transient machine-readable medium.

19. The machine-readable instruction causes one or more of the at least one of the at least one processor circuits to perform clustering of the output of the image sensor and centroid calculation, as described in claim 18, for at least one non-transient machine-readable medium.

20. The machine-readable instruction is provided to one or more of the at least one processor circuits. Perform clutter removal. Perform a shape and dimension-based candidate screening. To verify the solution corresponding to the aforementioned image sensor, and The at least one non-transient machine-readable medium according to claim 17, wherein the data corresponding to the pattern arrangement is verified by matching the solution corresponding to the image sensor with the inertial data of the vehicle.

21. The machine-readable instruction causes one or more of the at least one processor circuits to identify the light source based on the identification of different types of light sources in the pattern arrangement, according to claim 17, for at least one non-transient machine-readable medium.

22. The machine-readable instruction causes one or more of the at least one processor circuits to guide the vehicle to the pattern arrangement for landing, according to claim 17, the at least one non-transient machine-readable medium.

23. The machine-readable instruction causes one or more of the at least one processor circuits to identify a two-dimensional pattern of a landing pad corresponding to the pattern display based on the pattern arrangement, the at least one non-transient machine-readable medium according to claim 17.

24. The verification of the data corresponding to the pattern arrangement includes verifying the output of the image sensor using inertial Kalman filter residue, according to at least one non-transient, machine-readable medium according to claim 17.

25. Identifying light sources of a ground-based light pattern array based on output from an image sensor carried by a vehicle, wherein the spatial arrangement of the light sources defines the light pattern array. Verify the data corresponding to the optical pattern array through multilayer integrity verification, and A method comprising enabling guidance for the vehicle via the light pattern array in accordance with the data corresponding to the verified light pattern array.

26. Verifying the data corresponding to the aforementioned optical pattern array is, Perform clutter removal. Perform a shape and dimension-based candidate screening. To verify the solution corresponding to the aforementioned image sensor, and The method according to claim 25, further comprising matching the solution corresponding to the image sensor with the inertial data of the vehicle.

Images