Image-based approach navigation system and aircraft

Abstract

This patent discloses an image-based approach navigation system and aircraft, relating to the fields of digital image processing and aviation. Approach is one of the most critical flight phases in the operation of fixed-wing aircraft. During approach using machine vision-based automatic takeoff and landing systems, it is necessary to identify the airport runway location based on images and to identify approach-related navigation information based on those images. This invention provides an image-based approach navigation system and aircraft to address the problems of poor real-time performance and high computational requirements commonly found in existing technologies when identifying runway locations using images and outputting approach navigation information. The image-based approach navigation system includes an image acquisition device and an image processing unit. Its key feature is that the system uses a machine learning model to receive forward-facing external view images from the aircraft, identifies the runway in the images, and simultaneously outputs landing guidance information for the aircraft.

Description

Technical Field

[0001] This invention relates to the fields of digital image processing and aviation, and particularly to an image-based approach navigation system and aircraft. Background Technology

[0002] In the field of civil aviation, machine vision-based automated takeoff and landing (ATO) systems have become a future development trend. Civil aircraft utilize onboard image recognition technology to achieve fully automated flight based on vision, enabling autonomous taxiing, takeoff, and landing. Machine vision-based ATO systems can help pilots reduce their focus on aircraft operation and concentrate more on strategic decision-making and mission management. Applying this technology to large civil aircraft can enhance future aircraft operation potential and further improve aircraft safety performance.

[0003] Approach is one of the most critical phases of flight for fixed-wing aircraft. When using machine vision-based automatic takeoff and landing systems, the approach process requires identifying the airport runway location from images and determining approach-related navigation information, such as glide slope deviation information for precise navigation and DEM information for non-precise navigation. Existing technologies include deep learning-based runway detection and identification, as well as image-based pose calculation for the landing phase. However, none of these technologies simultaneously identify the runway location and output approach navigation information. Furthermore, existing technologies place high demands on image real-time processing and equipment computing power. Summary of the Invention

[0004] This invention provides an image-based approach navigation system and aircraft, which solves the problems of poor real-time performance and high computing power requirements of existing technologies when using images to identify runway positions and output approach navigation information.

[0005] This invention provides an image-based approach navigation system, including an image acquisition device and an image processing unit. The system uses a machine learning model to receive forward-facing external view images of the aircraft, identifies the runway in the images, and simultaneously outputs landing guidance information for the aircraft.

[0006] Preferably, the aircraft landing guidance information includes precision approach guidance data and non-precision approach guidance data.

[0007] Preferably, the precision approach guidance data includes ILS horizontal deviation data and ILS vertical deviation data.

[0008] Preferred non-precision approach guidance data includes DME data.

[0009] Preferably, the machine learning model segments the input image into gratings.

[0010] Preferably, the machine learning model uses the grid with the highest confidence level to identify the runway in the image and simultaneously outputs aircraft landing guidance information.

[0011] Preferably, the machine learning model in the image-based approach navigation system is trained using the following method:

[0012] S1. Create a training dataset, including forward-facing external view images of the aircraft, approach guidance data of the current image frame, and manually labeled runway target bounding boxes.

[0013] S2, Construct the machine learning model;

[0014] The input to the machine learning model is the forward-facing external view image of the aircraft, and the output of the machine learning model is a tensor containing image information, runway detection box information, and landing information.

[0015] S3, training machine learning models;

[0016] The loss functions of the machine learning model include the confidence data loss function, the runway box loss function, and the landing information loss function.

[0017] Preferably, in step S2, the machine learning model segments the input image into gratings.

[0018] Preferably, in step S2, the runway detection frame information includes confidence data, runway detection frame center coordinates, and runway detection frame size.

[0019] Preferably, the confidence score data is the intersection-union ratio (IUGR) between the predicted runway detection boxes and the actual runway detection boxes.

[0020] Preferably, when training the machine learning model, the runway detection box information is trained first, and then the landing information is trained.

[0021] This invention also provides an aircraft that uses the above-described image-based approach navigation system. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of an image-based approach navigation system according to an embodiment of the present invention;

[0024] Figure 2 This is a flowchart of a machine learning model training method used in an image-based approach navigation system according to an embodiment of the present invention;

[0025] Figure 3 and Figure 4 This is a test verification result diagram of an image-based approach navigation system in one embodiment of the present invention; Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] In one embodiment of the present invention, an image-based approach navigation system is provided, including an image acquisition device and an image processing unit. The system is characterized in that it uses a machine learning model to receive forward-facing external view images of the aircraft, identifies the runway in the images, and simultaneously outputs landing guidance information for the aircraft.

[0028] The machine learning model takes as input a forward-facing external view image of the aircraft and outputs a tensor. The model first divides the input image into S×S grids, and the output tensor corresponds to this, with a size of S×S×N, where N represents the landing guidance information corresponding to one grid, specifically including:

[0029] 1. Runway detection frame information, including confidence level data, center coordinates (x, y) of the runway detection frame, and dimensions of the runway detection frame (width and height);

[0030] 2. Landing information, including glide slope deviation values ​​(horizontal deviation, vertical deviation) from precise approach information, and DME values ​​from non-precise approach information.

[0031] The image-based approach navigation system compares the confidence information in each row and column of the tensor output by the machine learning model, and extracts the landing guidance information corresponding to the grid with the highest confidence as the final landing guidance information.

[0032] The image-based approach navigation system reconstructs the runway detection box information from the final landing guidance information in the input image, that is, displays the runway detection box in the input image. The image containing the runway detection box is output directly, with or without processing, and displayed on the aircraft cockpit display panel or similar device to provide guidance for the pilot to operate and monitor the aircraft.

[0033] In one embodiment of the present invention, the landing information in the landing guidance information is input to the flight control system as input to the aircraft's autopilot system.

[0034] In one embodiment of the present invention, the landing information is displayed on the cockpit display panel of the aircraft, providing a reference for the pilot to operate and monitor the aircraft.

[0035] The image-based approach navigation system provided by this invention has been verified in an X-plane simulation environment, and the results are shown in Figures 3 and 4. Figure 3 This image, taken in a simulation environment, shows the aircraft positioned slightly to the left of the glide path. A white runway identification box accurately marks the runway's location in the image. The upper right corner displays the aircraft's landing guidance information. Perpendicular line segments represent the projections of the glide path's two-dimensional plane and the runway's two-dimensional plane onto the image plane. The horizontal and vertical deviations of the aircraft on the glide path can be indicated by the degree of deviation between the intersection of these perpendicular line segments and the gray intersecting line segments in the display window. The black solid line represents the actual value, the white solid line represents the guidance information output by the image-based approach navigation system, and the gray solid line represents the center of the image. The guidance information output by the navigation system indicates that the aircraft has deviated to the left of the glide path and is below its altitude. The error between the actual value and the navigation system's output is within an acceptable range.

[0036] Appendix Figure 4 The image shown is a frame of an aircraft approaching along the glide slope in a simulation environment. The white runway identification box accurately marks the location of the runway in the image. The upper right corner of the screen is the landing guidance information display window for the aircraft. The black true value line almost coincides with the white line output by the system, and this error is within an acceptable range.

[0037] The above-mentioned attached diagram is presented as a test example. Based on the results of multiple tests, it can be concluded that the average processing time per frame of the image-based approach navigation system is 0.017 seconds, the overall average running time of the algorithm is 0.083 seconds per frame, the average prediction error of ILS vertical deviation and horizontal deviation ranges from 0.05 to 0.3, and the average prediction error of DME ranges from 0.05 to 0.1 (unit: nautical miles). The prediction error fluctuates depending on different approach and landing scenarios and trajectories.

[0038] Simulation experiments show that the image-based approach navigation system has high real-time performance and low computing power requirements when using images to identify runway positions and output approach navigation information.

[0039] The modules in the image-based approach navigation system described above can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0040] Since the core of the image-based approach navigation system is a machine learning model, the specific description of the image-based approach navigation system can be referred to the description of the training method of the machine learning model in the system below. Unless otherwise stated, the descriptions of the same technical content are the same in both. Under the guidance of the prior art, inferences or deductions can be made on the discrepancies between the descriptions in the two documents by combining the prior art or common knowledge. The technical content obtained by inference and deduction also falls within the scope of protection claimed in this application.

[0041] The machine learning model used in the image-based approach navigation system is trained using the following method:

[0042] Step S1: Create a training dataset, including forward-facing external view images of the aircraft, approach guidance data of the current image frame, and manually marked runway target bounding boxes.

[0043] The approach guidance data for the current image frame includes precise approach guidance data and non-precise approach guidance data for the corresponding moment of the image. In one embodiment, precise approach guidance data includes ILS horizontal deviation data and ILS vertical deviation data, while non-precise approach guidance data includes DME data.

[0044] The runway target frame data is used to characterize the shape, position and other information of the runway target frame. In one embodiment, the runway target frame is a rectangle. The runway target frame data includes the runway target frame category, the x-coordinate and y-coordinate of the middle point of the runway target frame in the outward view image of the aircraft, and the width and height of the runway rectangle.

[0045] Step S2, construct the machine learning model;

[0046] In one embodiment, the machine learning model is built on Mobilenetv2. The input to the machine learning model is a forward-facing external view image of the aircraft, with an image size of 448×448. The output of the machine learning model is a 7×7×8 tensor, containing 7×7 raster image information, as well as runway detection box information and landing information in a total of 8 dimensions.

[0047] The runway detection box information in the eight dimensions includes:

[0048] 1. Airport runway confidence data;

[0049] 2. The x-coordinate of the point in the runway target frame in the outward-facing view of the aircraft;

[0050] 3. The coordinates of the point in the runway target frame in the y-direction of the outward-facing view image from the front of the aircraft;

[0051] 4. The width of the runway frame;

[0052] 5. Runway frame height;

[0053] The landing information includes:

[0054] 6. Level deviation value of ILS-like cells;

[0055] 7. Vertical deviation value of ILS-like structures;

[0056] 8. DME value.

[0057] In one embodiment, the airport runway confidence data is the intersection-union ratio (IUGR) of the predicted runway detection boxes and the actual runway detection boxes.

[0058] Step S3: Train the machine learning model;

[0059] The specific method includes: inputting the forward-facing external view image of the aircraft into the machine learning model to obtain an SxSxN target value estimation tensor; calculating the grid with the highest confidence in the model output, comparing the target value estimation tensor with the corresponding data in the training set, and calculating the confidence data loss function, runway box loss function, and landing information loss function for that grid; for the remaining grids, only the confidence data loss function is calculated. Backpropagation is then used to update the network parameters, ultimately obtaining the trained model.

[0060] The confidence level data loss function is as follows:

[0061]

[0062] In the formula, λ noobj This represents the confidence coefficient for the grid without the runway target box, which is set to 0.5 in one embodiment; Indicates whether the grid does not contain the target; 1 indicates no target, 0 indicates target. i Indicates the true confidence level of the target; This indicates the confidence level of the target prediction.

[0063] The runway detection box loss function is as follows:

[0064] loc = 1 - iou

[0065] In the formula, iou represents the intersection-union ratio (IU) between the predicted object detection box and the ground truth object detection box.

[0066] The landing information loss function is as follows:

[0067]

[0068] In the formula, S represents the number of grid cells; Indicates whether the grid contains a runway; 1 indicates a runway, 0 indicates no runway. appr The loss factor representing the aircraft approach and landing data is set to 0.8 in one embodiment.

[0069] The final loss function is the sum of the airport runway confidence loss function, the runway detection box loss function, and the landing information loss function.

[0070] During the model training process, the runway detection box information is first trained to obtain the runway detection model. In one embodiment of the present invention, the landing information loss function is initially set to 0, and only the confidence data and the runway detection box loss function are calculated. After convergence, the landing information loss function is restored to continue model training.

[0071] The landing information is then trained based on the runway detection model to obtain the machine learning model in the image-based approach navigation system. In one embodiment of the present invention, when retrieving the landing information and continuing training, the learning rate of all layers except the last layer of the network is multiplied by 0.1 to avoid losing the learned runway detection parameters.

[0072] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0073] The present invention also provides an embodiment of an aircraft using the aforementioned image-based approach navigation system. The aircraft relies on an onboard image-based approach navigation system to provide runway position information in the aircraft's cockpit display panel or a similar device, providing guidance for the pilot to operate and monitor the aircraft.

[0074] In one embodiment of the present invention, the aircraft relies on landing guidance information provided by an onboard image-based approach navigation system as input to the aircraft's autopilot system.

[0075] In one embodiment of the present invention, the aircraft relies on an onboard image-based approach navigation system to display landing navigation information on the aircraft cockpit display panel, providing a reference for the pilot to operate and monitor the aircraft.

[0076] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An image-based approach navigation system, comprising an image acquisition device and an image processing unit, characterized in that, The system uses a machine learning model to receive forward-facing external view images of the aircraft, identify the runway in the images, and simultaneously output landing guidance information for the aircraft. It compares the confidence information in each row and column of the tensor output by the machine learning model, and extracts the landing guidance information corresponding to the grid with the highest confidence as the final landing guidance information. The machine learning model is trained using the following method: S1. Create a training dataset, including forward-facing external view images of the aircraft, approach guidance data of the current image frame, and manually labeled runway target bounding boxes. S2, Construct the machine learning model; The input to the machine learning model is the forward-facing external view image of the aircraft, and the output of the machine learning model is a tensor containing runway detection box information and landing information. S3, training machine learning models; Calculate the confidence data loss function, runway box loss function, and landing information loss function for the grid with the highest confidence output by the machine learning model; calculate the confidence data loss function for the remaining grids, and backpropagate to update the network parameters; During the training of the machine learning model, the final loss function is the sum of the airport runway confidence loss function, the runway detection box loss function, and the landing information loss function. First, the runway detection box information is trained to obtain the runway detection model. The landing information loss function is set to 0, and only the confidence data and the runway detection box loss function are calculated. After convergence, the landing information loss function is restored to continue model training. Then, the landing information is trained based on the runway detection model to obtain the machine learning model in the image-based approach navigation system. When the landing information is restored for further training, the learning rate of all layers except the last layer of the network is multiplied by 0.1 to avoid losing the learned runway detection parameters.

2. The image-based approach navigation system according to claim 1, characterized in that, The aircraft landing guidance information includes precision approach guidance data and non-precision approach guidance data.

3. The image-based approach navigation system according to claim 2, characterized in that, The precision approach guidance data includes ILS horizontal deviation data and ILS vertical deviation data.

4. The image-based approach navigation system according to claim 2, characterized in that, Non-precision approach guidance data includes DME data.

5. The image-based approach navigation system according to claim 2, characterized in that, The machine learning model segments the input image into gratings.

6. The image-based approach navigation system according to claim 5, characterized in that, The machine learning model uses the grid with the highest confidence level to identify the runway in the image and simultaneously outputs aircraft landing guidance information.

7. The image-based approach navigation system according to claim 1, characterized in that, When training the machine learning model, the runway detection box information is trained first, and then the landing information is trained.

8. An aircraft using an image-based approach navigation system as claimed in any one of claims 1-6.

Images