Glide slope deviation determination system, method and aircraft

Abstract

This patent discloses a glide slope deviation determination system, method, and aircraft, belonging to the fields of digital image processing and aviation. It provides an image-based solution for approach and landing guidance, distinct from ILS and GBAS systems. The glide slope deviation determination system is applicable to civil fixed-wing aircraft and is part of a vision-based piloting system. It addresses the problem of using image recognition technology to obtain aircraft glide slope deviation data with similar accuracy to existing sensors and to apply this data to actual aircraft. The main technical solutions of the invention include: an image acquisition device, an image processing unit, and a posture calculation unit.

Description

Technical Field

[0001] The present invention relates to the fields of digital image processing and aviation, and in particular to a glide slope deviation determination system and an aircraft. Background Art

[0002] Landing is one of the most critical stages of flight for fixed-wing aircraft. During landing, the deviation between the aircraft and the glide path is an essential parameter. Existing domestic approach and landing guidance systems are mostly based on ILS. However, ILS suffers from several issues: Signals rely on ground reflection, making them susceptible to interference from the ground and surrounding objects. Snow, water, and overgrown grass can cause abnormal reflections; the operating channel is close to FM broadcast frequencies, making it susceptible to interference from FM stations; an ILS is required for each runway entrance, and an ILS system provides only one vertical path, with all aircraft landing at the same point; capital investment and maintenance costs are high. Therefore, research on GBAS-based approach and landing guidance has been conducted both domestically and internationally to replace and supplement ILS functions. However, the application of GBAS-based approach and landing guidance methods is not yet fully mature, and their integrity and reliability require improvement.

[0003] In the civil aircraft sector, machine vision-based automatic takeoff and landing systems have become a future development trend. Leveraging onboard image recognition technology, civil aircraft can achieve autonomous taxiing, takeoff, and landing through fully automated vision-based flight. This machine vision-based automatic takeoff and landing system allows pilots to focus less on aircraft operations and more on strategic decision-making and mission management. Applying this technology to large civil aircraft can enhance the potential for future aircraft operations and further improve aircraft safety. Summary of the Invention

[0004] The present invention provides a glide slope deviation determination system and method and an aircraft equipped with the system, which provides a machine vision-based solution for approach landing guidance that is different from the ILS system and the GBAS system.

[0005] The glide slope deviation determination system is applicable to civil fixed-wing aircraft and is part of a vision-assisted piloting system. It is used to solve the problem of how to obtain aircraft glide slope deviation data with an accuracy close to that of existing sensors through image recognition technology and to actually apply it to real aircraft.

[0006] In order to solve the problem of applying the glide slope deviation determination system in a real aircraft, the present invention provides a glide slope deviation determination system, which specifically includes:

[0007] Image acquisition equipment, including optical image acquisition equipment and infrared image acquisition equipment, the image acquisition equipment is installed on the aircraft and is used to collect image data containing key information of the runway;

[0008] An image processing unit, configured to pre-process the image data;

[0009] a posture solving unit, configured to determine a glide path deviation of the aircraft based on the image data, including a glide path deviation prediction model based on machine learning;

[0010] In the aircraft attitude determination system, the technical issue of how the attitude calculation unit can obtain glide path deviation data with an accuracy close to that of existing sensors from image data is particularly important.

[0011] Based on the above-mentioned general inventive concept, the present invention proposes an end-to-end model that, by training a machine learning model with appropriate methods, can estimate an aircraft's glide path deviation data directly from images. Because the calculation of the glide path deviation data does not rely on additional landmark information, nor is it solved based on the features of multiple frames before and after, the end-to-end model ensures higher accuracy and faster computing speed, making it more suitable for practical applications and possessing greater robustness. In practical applications, the present invention also does not require additional ground information identification or the use of multiple frames of images before and after, thereby reducing the amount of computation required.

[0012] Based on the above general inventive concept, the collection and processing of model training data is even more important to ensure the accuracy and robustness of the end-to-end model. From a practical engineering perspective, this invention considers the design and actual application scenarios of vision-assisted driving systems and provides two typical approaches to collecting model training data: data collected from a simulation environment and data collected from real-world scenarios.

[0013] The two types of training data mentioned above each have their advantages and disadvantages. Using only data collected from simulation environments to train the model provides a large amount of low-cost training data, improving model accuracy through the use of large amounts of simulation data. Furthermore, simulation data collected from existing, mature industrial flight simulation software can capture data under various weather conditions, playing a crucial role in the early design and initial verification of vision-assisted pilot systems. Furthermore, models trained using simulation data can be directly applied to real-world aircraft with relatively simple adjustments.

[0014] The advantage of using data collected from real flight scenarios for model training is that the training set data comes from the real environment, so the model trained in this way is more adaptable to the input data in the real environment. In addition, the aircraft glide path and related data in the real flight scenario can be directly obtained from the aircraft bus. Since there is no problem of aligning the image time with the timestamp of the aircraft attitude and its related data, the data collection difficulty is lower. However, it is obvious that since multiple real flights are required to collect enough data, the collection cost is significantly higher than collecting in a simulation environment.

[0015] In order to further balance the efficiency of training data collection and the final aircraft attitude prediction effect of the model, data from two sources can be used simultaneously during training.

[0016] Based on the above aircraft attitude determination system solution, the following features can also be included:

[0017] Optionally, the glide slope deviation prediction model based on machine learning is an end-to-end convolutional neural network model.

[0018] Optionally, the glide slope deviation model based on machine learning is trained by the following method:

[0019] Step S1, collecting data from a simulation environment and / or a real environment; the data includes image data from the driver's perspective and onboard data;

[0020] Step S2, pre-processing the data;

[0021] Step S3: construct a neural network, which takes the image obtained by the image processing unit as input and takes the glide path deviation data and the heading deviation data as output; the neural network includes 33 convolutional layers and 1 fully connected layer.

[0022] Optionally, when the data collected in step S1 is derived from a real environment, the airborne data includes attitude deviation data (ILS data and / or GBAS data), attitude data, barometric altitude, wheel speed, and IRS data information collected by other airborne sensors; the airborne data is obtained from the aircraft bus;

[0023] Optionally, in step S2, the pre-processing includes data augmentation, which does not include rotation and translation. Because the specific location of the runway in the image corresponds to the ILS data, random rotation or translation of the input image will prevent the model from being trained correctly.

[0024] Optionally, a method for determining a glide slope deviation includes:

[0025] Obtain an image containing a runway;

[0026] Preprocessing the image;

[0027] Input the processed image into the machine learning model;

[0028] Get the glide slope deviation value.

[0029] Optionally, the machine learning model is an end-to-end model, the input of the model is an image, and the output is a glide slope deviation value.

[0030] Optionally, the glide slope deviation value includes glide slope deviation data and heading deviation data.

[0031] This technology was validated in an X-Plane simulation environment and offline data from real-world scenarios. Images were fed into a trained machine learning model, which then used forward inference to output predicted heading and glide path deviation guidance values. Simultaneously, actual ILS heading and glide path deviation guidance values ​​were collected. By comparing the predicted values ​​obtained from the images with the actual ILS values, the average error per frame for heading deviation was calculated to be 0.04, and for glide path deviation to be 0.08.

[0032] The verification results in real-world offline data are as follows: Figure 4 As shown, it is displayed on an offline computer screen. The value marked with "real" on the right is the actual value collected by the ILS, and the value marked with "pre" is the estimated value of the machine learning model. It can be found that the deviation between the two is small, and the solution claimed in the present invention has a good positive effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0034] Figure 1 This is a schematic diagram of the glide slope deviation determination system in one embodiment of the present invention.

[0035] Figure 2 is a flow chart of a method for determining a glide slope deviation in one embodiment of the present invention;

[0036] Figure 3 is a flow chart of a glide slope deviation model training method according to one embodiment of the present invention;

[0037] Figure 4 This is a diagram showing the verification results of the method for determining the glide slope deviation in one embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0039] Example 1

[0040] In one embodiment of the present invention, a system for determining an aircraft attitude is provided. The system includes an image acquisition device, an image processing unit, and a posture solving unit.

[0041] The image acquisition device, mounted on the aircraft, is used to capture image data containing key runway information. The image acquisition device utilizes a CCD camera and must comply with the requirements of GJB 7083. Furthermore, the image pixels captured by the image acquisition device must be at least 680 x 680. In this embodiment, the image data captured by the image acquisition device is captured from the pilot's perspective, ensuring that each frame of image covers the runway throughout the advanced landing process. The focal length ensures a clear view of the runway.

[0042] The image processing unit pre-processes the image data and can process the image of the image acquisition device and display it in a suitable form in the cockpit instrument; the image processing unit pre-processes the image data collected by the image acquisition device to make it better suitable for the machine learning model in the posture solution unit.

[0043] The posture solving unit is used to determine the glide path deviation of the aircraft based on the image data, including a glide path deviation prediction model based on machine learning.

[0044] The machine learning-based glide slope deviation prediction model in the attitude solver is trained using the following method:

[0045] Step S1 collects data from the simulation environment; this data includes pilot-perspective image data and airborne data. To achieve data diversity, simulated approaches and landings are conducted from a distance of 3 nautical miles from the runway threshold under various simulated weather conditions, including clear skies, overcast skies, low visibility, and stormy weather, as well as at three different times of day: morning, midday, and evening. The corresponding airborne data is then acquired. This airborne data is output by professional flight simulation software, including ILS data.

[0046] Since the time points of the onboard data output by professional flight simulation software cannot be aligned one-to-one with the time points of the image data, it is necessary to align the timestamps of the image data and the onboard data to form a data set containing the image data and the onboard data at the corresponding time. In general, the operation of aligning timestamps can be summarized as follows:

[0047] Extract a frame of imagery captured by the image acquisition device and obtain the timestamp T of that frame. Based on timestamp T, find the glide path deviation and heading deviation data in the airborne ILS data whose timestamp is closest to T but no earlier than T. Name the frame of imagery using an index, and also name the extracted ILS data using the same index to ensure a one-to-one correspondence between the airborne imagery data and the airborne ILS data.

[0048] Since the aircraft's approach attitude is relatively stable during the approach process, the changes in the glide deviation and heading deviation data within the above time difference are not enough to affect the overall training of the model. The above method effectively reduces the processing workload of the model training data and improves the efficiency of model training.

[0049] Step S2: pre-processing the data, including downsampling the image, expanding the data set, normalizing the data, and reorganizing the data.

[0050] in,

[0051] Downsampling includes: in order to reduce the computational cost, the image resolution is downsampled to 224x 224 and stored in BGR format, that is, each channel represents blue, green and red respectively, and its pixel value range is [0,255].

[0052] Augmenting the dataset involves randomizing image brightness, contrast, saturation, and hue to simulate varying lighting conditions and camera levels. However, random rotation or translation is not permitted in this invention because the specific locations of the runway in the image correspond to the ILS data, and random rotation or translation of the input image would prevent the model from training correctly.

[0053] Data normalization: Divide all image data by 255 to normalize the data to [0, 1].

[0054] Data reorganization: In order to eliminate the time series correlation of the dataset, the labeled data is reorganized to improve adaptability and generalization performance.

[0055] Step S3: constructing a neural network, wherein the neural network takes the image obtained by the image processing unit as input and takes the glide slope deviation data as output; the neural network includes 33 convolutional layers and 1 fully connected layer;

[0056] The specific architecture of the neural network includes:

[0057] The convolutional neural network used in the present invention is the ResNet network, and a 34-layer model is used. The network contains 33 convolutional layers and 1 fully connected layer, wherein a maximum pooling layer is added after the first convolution layer, and an average pooling layer is added before the last fully connected layer. Among them, each convolution layer is followed by the BatchNorm method and the Relu activation function, and residual processing is performed between each two convolution layers to form a residual block. Based on ResNet, the output channel of the last fully connected layer is changed to 2, with a one-dimensional channel corresponding to the heading deviation and a one-dimensional channel corresponding to the glide deviation.

[0058] The model training process also includes comparing the guided estimate output by the convolutional neural network with the true value based on ILS or GBAS to obtain the mean square error, which is used as the loss function; backpropagation is performed based on the loss function, and the network parameters are updated. The goal of the update is to minimize the loss function; after multiple rounds of training, an end-to-end network model can be obtained in which the loss function reaches the expected value.

[0059] After training, the convolutional neural network model can be put into use. When used, by inputting extravehicular images, the approach and landing guidance values ​​can be directly obtained.

[0060] The effectiveness of the neural network model in predicting glide slope deviation values ​​was tested by inputting newly acquired videos from both simulation and real-world environments into the network. The newly acquired videos were completely independent of the previously trained data and were also acquired under various lighting and weather conditions.

[0061] Newly captured airborne images from the simulation environment were fed into the trained convolutional neural network model. The model then used forward inference to output heading and glide path deviation data predicted based on the image. Simultaneously, actual ILS heading and glide path deviation data were collected. By comparing the predicted values ​​obtained from the images with the actual ILS values, the average error per frame for heading deviation was calculated to be 0.04, and for glide path deviation to be 0.08.

[0062] The verification results in real-world offline data are as follows: Figure 4 As shown in the figure, images collected during a C919 passenger aircraft test flight are input into the model provided in this embodiment, and the model output is displayed in the image. The values ​​marked with "real" on the right are the actual values ​​collected by the ILS, and the values ​​marked with "pre" are the estimated values ​​by the machine learning model. It can be seen that the deviation between the two is small, indicating that the solution claimed in this invention has a good and positive effect.

[0063] Example 2

[0064] Example 2 shares the same glide slope deviation system architecture and components as Example 1, differing in the training method for the machine learning-based glide slope deviation prediction model in the posture calculation unit. Example 2 uses data collected from a real environment during training. The pre-training data collection process for the model is as follows:

[0065] Data is collected from real-world environments, including pilot-perspective imagery and onboard data. To achieve data diversity, video footage of the aircraft approaching and landing is captured from a distance of 3 nautical miles from the runway threshold in various weather conditions, including clear skies, overcast skies, low visibility, and stormy weather, as well as at three different times of the day: morning, midday, and evening. The onboard data corresponding to these images is then retrieved from the aircraft bus. This onboard data includes ILS data (attitude deviation data) detected and output by the aircraft's existing sensors. To ensure training accuracy, flight attitude, pressure altitude, wheel speed, and IRS data are also collected.

[0066] The present invention also provides a method for determining a glide slope deviation, comprising: obtaining an image containing a runway; preprocessing the image; inputting the processed image into a machine learning model; and obtaining a glide slope deviation value. The machine learning model is an end-to-end model, wherein the input of the model is an image and the output is a glide slope deviation value. The glide slope deviation value includes glide slope deviation data and heading deviation data.

[0067] The glide slope deviation determination method and the glide slope deviation determination system have the same working principle, and are not described again here.

[0068] In addition, an embodiment provided by the present invention also includes an aircraft equipped with the above-mentioned glide slope deviation determination system, particularly a fixed-wing aircraft.

[0069] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0070] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A glide slope deviation determination system, characterized in that: include: Image acquisition equipment, including optical image acquisition equipment and infrared image acquisition equipment, the image acquisition equipment is installed on the aircraft and is used to collect image data containing key information of the runway; An image processing unit, configured to pre-process the image data; a posture solving unit, configured to determine a glide path deviation of the aircraft based on the image data, including a glide path deviation prediction model based on machine learning; The glide slope deviation model based on machine learning is trained by the following method: Step S1, collecting data from a simulation environment and a real environment; the data includes image data from the driver's perspective and airborne data; aligning the timestamps of the image data and the airborne data to form a data set including the image data and the airborne data at the corresponding time; Step S2, pre-processing the data; Step S3: constructing a neural network, wherein the neural network takes the image obtained by the image processing unit as input and takes the glide path deviation data and the heading deviation data as output.

2. The glide slope deviation determination system according to claim 1, characterized in that: The glide slope deviation prediction model based on machine learning is an end-to-end convolutional neural network model.

3. The glide slope deviation determination system according to claim 1 or 2, characterized in that: The neural network includes 33 convolutional layers and 1 fully connected layer.

4. The glide slope deviation determination system according to claim 3, characterized in that: When the data collected in step S1 is derived from a real environment, the airborne data includes ILS deviation data and / or GBAS data, attitude data, barometric altitude, wheel speed and / or IRS data information collected by other airborne sensors; the airborne data is obtained from the aircraft bus.

5. The glide slope deviation determination system according to claim 3, characterized in that: In step S2, the pre-processing includes data enhancement processing; the data enhancement processing does not include rotation and translation.

6. A method for determining a glide slope deviation, comprising: Obtain an image containing a runway; Preprocessing the image; The processed image is input into the machine learning model to output the glide slope deviation value; The machine learning model is trained by the following method: Step S1, collecting data from a simulation environment and a real environment; the data includes image data from the driver's perspective and airborne data; aligning the timestamps of the image data and the airborne data to form a data set including the image data and the airborne data at the corresponding time; Step S2, pre-processing the data; Step S3: constructing a neural network, wherein the neural network takes the image obtained by the image processing unit as input and takes the glide path deviation data and the heading deviation data as output.

7. The method for determining the glide slope deviation according to claim 6, wherein: The machine learning model is an end-to-end model, the input of the model is an image, and the output is a glide slope deviation value.

8. The method for determining the glide slope deviation according to claim 7, wherein: The glide slope deviation value includes glide slope deviation data and heading deviation data.

9. An aircraft, characterized in that: A glide slope deviation determination system as described in claims 1-5 is installed.

Images