Calculation method for identifying airport runway image, electronic equipment and medium

Abstract

The invention discloses a calculation method for identifying an airport runway image, electronic equipment and a medium. The method can comprise the following steps: acquiring a to-be-processed image frame of a video stream in an airplane foresight infrared camera, and carrying out preprocessing operation; constructing a deep learning model of the image frame and the mask of the runway marker; according to the preprocessed image frame, predicting a mask of an airplane runway marker through a deep learning model; and performing a plurality of judgment rules according to the predicted mask of the runway marker, and obtaining an airport runway image after passing the judgment rules. The method is based on a structure combining a deep learning algorithm and discrimination according to actual conditions, has remarkable technical advantages and application values, improves the recognition accuracy, enhances the response capability, has high anti-interference capability, and optimizes data fusion and time sequence processing.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and more specifically, to a computing method, electronic equipment and medium for recognizing airport runway images. Background Art

[0002] At present, drones achieve positioning and navigation based on a combination of satellites, radio and inertia. However, in terms of visual perception, there is no system framework to control the perception of drones. The camera of the drone's forward perspective can provide rich image information, guiding the drone to contribute to important target recognition, intelligent control and other aspects. Therefore, in terms of drone landing, the advantages of visual information can be used to assist drones in landing accurately by identifying runway images. At present, the on-board lane line recognition technology is based on the car's fixed height and angle camera for recognition. The lane line has a large area in the image and is easy to identify. However, from the top-down perspective of the aircraft, the pixel area occupied by the runway is small, the runway line is not clear, and it is difficult to directly apply the relevant algorithms for lane line recognition. At present, there is a lack of algorithms for accurately and quickly identifying the runway from a top-down perspective.

[0003] There is still a need to develop a computational method to recognize airport runway images.

[0004] The information disclosed in the background technology section of the present invention is only intended to deepen the understanding of the general background technology of the present invention, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art already known to those skilled in the art. Summary of the invention

[0005] The present invention proposes a computing method, electronic device and medium for identifying airport runway images. The method is based on a deep learning algorithm and a combination of judgments based on actual conditions. It has significant technical advantages and application value, improves recognition accuracy, enhances responsiveness, has high anti-interference capabilities, and optimizes data fusion and timing processing.

[0006] In a first aspect, an embodiment of the present disclosure provides a computing method for identifying an airport runway image, comprising:

[0007] Obtain the image frames to be processed from the video stream of the aircraft's forward-looking infrared camera and perform preprocessing operations;

[0008] A deep learning model that constructs masks of image frames and runway landmarks;

[0009] Predicting the mask of the runway marker through the deep learning model according to the preprocessed image frame;

[0010] Multiple judgment rules are applied to the mask of the predicted runway markers, and the airport runway image is obtained after passing the rules.

[0011] Preferably, the preprocessing operation includes:

[0012] Denosing and enhancing the image frame by Gaussian filtering and histogram equalization.

[0013] Preferably, the multiple determination rules include a position determination rule, a slope determination rule, a radar altitude determination rule, and a ground speed determination rule.

[0014] Preferably, the position determination rule is:

[0015] X_left < X_middle < X_right

[0016] where X_left, X_middle, and X_right respectively represent the coordinates on the X-axis of a certain fixed Y value corresponding to the left lane line, the middle lane line, and the right lane line in the image.

[0017] Preferably, the slope determination rule includes:

[0018] Obtaining the slopes of the left and right lane lines and the slopes of the left and right edges of the runway according to the mask, where the slope of the left lane line is equal to the slope of the left edge of the runway, and the slope of the right lane line is equal to the slope of the right edge of the runway.

[0019] Preferably, the radar altitude determination rule includes:

[0020] Reading the radar altitude data of the drone, calculating the size of the pixels occupied by the aircraft runway according to the camera internal parameters and the altitude of the drone, and the error from the actual runway is less than the corresponding threshold.

[0021] Preferably, the ground speed determination rule includes:

[0022] Reading the ground speed data of the drone, calculating the running trajectory of the marker in the image frame, and the error from the runway is less than the corresponding threshold.

[0023] Preferably, it further includes:

[0024] According to the differential Gaussian model algorithm, comparing the current image frame with the image frame of the previous calculation result, judging the similarity difference of the images. If the difference is less than the set threshold, then fusing the data of the current image frame and the image frame of the previous calculation result.

[0025] In a second aspect, an embodiment of the present disclosure further provides an electronic device, which includes:

[0026] A memory storing executable instructions;

[0027] A processor that runs the executable instructions in the memory;

[0028] A graphics processor, a hardware device for performing a large number of parallel computations, which speeds up the execution speed of the instructions in the image operation part.

[0029] In a third aspect, embodiments of the present disclosure further provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described calculation method for identifying airport runway images.

[0030] The method and apparatus of the present invention have other characteristics and advantages, which will be apparent from the accompanying drawings incorporated herein and the subsequent detailed description, or will be described in detail in the accompanying drawings incorporated herein and the subsequent detailed description. These drawings and detailed description together are used to explain the specific principles of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] By describing the exemplary embodiments of the present invention in more detail in conjunction with the accompanying drawings, the above and other objects, features, and advantages of the present invention will become more apparent. In the exemplary embodiments of the present invention, the same reference numerals generally represent the same components.

[0032] Figure 1 FIG. shows a flowchart of steps of a calculation method for identifying airport runway images according to an embodiment of the present invention.

[0033] Figure 2 FIG. shows a compliance determination flowchart for airport images according to an embodiment of the present invention.

[0034] Figure 3 FIG. shows a schematic diagram of an airport runway mask according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The preferred embodiments of the present invention will be described in more detail below. Although the preferred embodiments of the present invention are described below, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein.

[0036] Figure 1 FIG. shows a flowchart of steps of a calculation method for identifying airport runway images according to an embodiment of the present invention.

[0037] As Figure 1 shown, the calculation method for identifying airport runway images includes:

[0038] Step 101, obtaining an image frame to be processed from a video stream in a forward-looking infrared camera of an aircraft and performing a preprocessing operation;

[0039] Step 102, constructing a deep learning model of a mask of the image frame and an airport runway marker;

[0040] Step 103: Predict the mask of the aircraft runway marker based on the preprocessed image frame;

[0041] Step 104: Apply multiple determination rules to the predicted mask of the aircraft runway marker, and obtain the airport runway image after passing.

[0042] In one example, the preprocessing operation includes:

[0043] Perform denoising and enhancement operations on the image frame through Gaussian filtering and histogram equalization.

[0044] In one example, the multiple determination rules include a position determination rule, a slope determination rule, a radar altitude determination rule, and a ground speed determination rule.

[0045] In one example, the position determination rule is:

[0046] X left < X middle < X right

[0047] Wherein, X left, X middle, and X right respectively represent the coordinates on the X-axis corresponding to a certain fixed Y value in the middle of the left lane line, the middle lane line, and the right lane line in the image.

[0048] In one example, the slope determination rule includes:

[0049] Obtain the slopes of the left and right lane lines and the slopes of the left and right edges of the runway based on the mask. The slope of the left lane line is equal to the slope of the left edge of the runway, and the slope of the right lane line is equal to the slope of the right edge of the runway.

[0050] In one example, the radar altitude determination rule includes:

[0051] Read the radar altitude data of the drone, calculate the size of the pixels occupied by the aircraft runway based on the camera internal parameters and the altitude of the drone, and the error from the actual runway is less than the corresponding threshold.

[0052] In one example, the ground speed determination rule includes:

[0053] Read the ground speed data of the drone, calculate the running trajectory of the marker in the image frame, and the error from the runway is less than the corresponding threshold.

[0054] In one example, it further includes:

[0055] According to the differential Gaussian model algorithm, compare the current image frame with the image frame of the previously calculated result, judge the similarity difference of the images. If the difference is less than the set threshold, fuse the data of the current image frame and the image frame of the previously calculated result.

[0056] Specifically, obtain each frame of the image to be processed in the forward-looking infrared camera of the aircraft; perform preprocessing operations on the image frame, perform image denoising and image enhancement operations through Gaussian filtering and histogram equalization methods, and at the same time compare with the previous frame of image through the frame difference method.

[0057] Obtain the positions of the key signs on the aircraft runway through deep learning model calculation. The input of the deep learning model is the image frame to be processed, and the output is the mask of the aircraft runway markers. The deep learning model is composed of an encoder-decoder architecture combination. The specific network structure is comprehensively considered according to the image size and the performance of the image processor. The common structures are convolutional neural network, YOLO network and ViT network structure.

[0058] Perform multiple compliance judgments to determine whether the positions of the identified key signs conform to the actual situation and eliminate unqualified positions:

[0059] 1. The position determination rule is X left < X middle < X right, where X left, X middle, and X right are within (X min , X max ), where X left, X middle, and X right respectively represent the coordinates on the X-axis corresponding to a certain fixed Y value in the middle of the left lane line, the middle lane line, and the right lane line. X min and X max are the minimum and maximum values of the coordinates on the X-axis corresponding to the fixed Y value of the runway. Judge three times at reasonable positions according to the value range of the sign on the Y-axis.

[0060] 2. Slope determination rule: Obtain the slopes of the left and right lane lines and the slopes of the left and right edges of the runway according to the mask. The slope of the left lane line is equal to the slope of the left edge of the runway, and the slope of the right lane line is equal to the slope of the right edge of the runway.

[0061] 3. Radar altitude determination rule: Read the radar altitude data of the UAV. According to the camera internal parameters and the altitude of the UAV, calculate the size of the pixels occupied by the aircraft runway according to the imaging principle, and compare it with the actual runway. The error should be less than 10%.

[0062] 4. Ground speed determination rule: Read the ground speed data of the UAV. According to the optical flow trajectory, calculate the running trajectory of the marker in the video frame and match it with the result obtained by the model. The error on the runway should be less than 20 pixels.

[0063] According to the differential Gaussian model algorithm, compare the current image frame with the image frame of the previously calculated result to judge the similarity difference of the images. The result with a small difference can be regarded as a small change in the position of the aircraft runway. Using the Kalman filtering method, compare the position calculated by the current model with the result of the previous image frame to complete data fusion and draw the runway position on the current image.

[0064] The present invention also provides an electronic device, which includes: a memory storing executable instructions; and a processor that runs the executable instructions in the memory to implement the above-described calculation method for recognizing airport runway images.

[0065] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described calculation method for recognizing airport runway images.

[0066] To facilitate understanding of the solutions and effects of the embodiments of the present invention, the following gives three specific application examples. Those skilled in the art should understand that these examples are only for facilitating the understanding of the present invention, and any specific details are not intended to limit the present invention in any way.

[0067] Example 1

[0068] Obtain each frame of the image to be processed in the video stream of the forward-looking infrared camera of the aircraft when the unmanned aerial vehicle (UAV) lands on the runway. Regard the task of recognizing airport runway images as a common image segmentation task in deep learning. According to the different heights of the UAV landing, collect image data at a step size of 30 meters; after the UAV height is less than 100 meters, collect image data at a step size of 2 seconds, and then divide all the image data into a training set and a test set according to a ratio of 8:2.

[0069] After labeling the data, select a suitable deep learning network model for training. Practice has proved that most of the commonly used segmentation network models at present, such as convolutional neural networks, YOLO networks, and ViT networks, can initially achieve the algorithm effect. Considering the comprehensive factors of time efficiency and accuracy, the YOLO network is currently adopted.

[0070] Figure 2 Shows a compliance determination flowchart for airport images according to an embodiment of the present invention.

[0071] After training the deep learning model, obtain the image to be processed in the video stream of the forward-looking infrared camera of the aircraft; based on the image to be processed, obtain the mask of the key signs on the aircraft runway through the deep learning model, and perform multiple determination rules, such as Figure 2 as shown:

[0072] 1. The position determination rule is X left < X middle < X right, where X left, X middle, and X right are between (X min , X max ), and X left, X middle, and X right respectively represent the coordinates on the X-axis of a certain fixed Y value in the middle of the left lane line, the middle lane line, and the right lane line, X min and X maxThe minimum and maximum values of the X-axis coordinates corresponding to the fixed Y value of the runway are judged three times at reasonable positions according to the value range of the marker on the Y-axis.

[0073] 2. Slope determination rule: Obtain the slopes of the left and right lane lines and the slopes of the left and right edges of the runway according to the mask. The slope of the left lane line is equal to the slope of the left edge of the runway, and the slope of the right lane line is equal to the slope of the right edge of the runway.

[0074] 3. Radar altitude determination rule: Read the radar altitude data of the UAV. According to the internal parameters of the camera and the altitude of the UAV, and based on the imaging principle, calculate the size of the pixels occupied by the aircraft runway, and compare it with the actual runway. The error should be less than 10%.

[0075] 4. Ground speed determination rule: Read the ground speed data of the UAV. According to the optical flow trajectory, calculate the running trajectory of the marker in the video frame, and match it with the result obtained by the model. The error on the runway should be less than 20 pixels.

[0076] Figure 3 The schematic diagram of the airport runway mask according to an embodiment of the present invention is shown.

[0077] According to the difference Gaussian model algorithm, compare the current image frame with the image frame of the result calculated last time to judge the similarity difference of the images. The result with a small difference can be regarded as a slight change in the position of the aircraft runway. Using the Kalman filtering method, compare the position calculated by the current model with the result of the previous image frame to complete data fusion, and draw the runway position on the current image, as Figure 3 shown.

[0078] Example 2

[0079] The present disclosure provides an electronic device, which includes: a memory storing executable instructions; a processor that runs the executable instructions in the memory, and a graphics processor that accelerates the image data processing speed to implement the above-mentioned calculation method for identifying airport runway images.

[0080] The electronic device according to an embodiment of the present disclosure includes a memory, a processor, and a graphics processor.

[0081] The memory is used to store non-temporary computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0082] The processor can be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions. In one embodiment of the present disclosure, the processor is used to run the computer-readable instructions stored in the memory.

[0083] The graphics processing unit (GPU) is a hardware device for performing a large number of parallel computations, having numerous computing units, and supporting multiple threads to execute the same instruction in the same cycle. In the examples of the present disclosure, the graphics processing unit is used to process image computations of the deep learning module.

[0084] Those skilled in the art should understand that, in order to solve the technical problem of how to obtain good user experience effects, this embodiment may also include well-known structures such as communication buses, interfaces, etc., and these well-known structures should also be included in the protection scope of the present disclosure.

[0085] For a detailed description of this embodiment, reference may be made to the corresponding descriptions in the foregoing embodiments, and details will not be repeated here.

[0086] Example 3

[0087] The embodiments of the present disclosure provide a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor and a graphics processing unit, the computing method for recognizing airport runway images as described above is implemented.

[0088] According to the computer-readable storage medium of the embodiments of the present disclosure, non-temporary computer-readable instructions are stored thereon. When the non-temporary computer-readable instructions are run by a processor, all or part of the steps of the methods of the foregoing embodiments of the present disclosure are executed, and the graphics processing unit can accelerate the execution speed of some instructions.

[0089] The above-mentioned computer-readable storage medium includes but is not limited to: optical storage media (such as CD-ROMs and DVDs), magneto-optical storage media (such as MOs), magnetic storage media (such as magnetic tapes or external hard drives), media with built-in rewritable non-volatile memories (such as memory cards), and media with built-in ROMs (such as ROM cartridges).

[0090] Those skilled in the art should understand that the purpose of the above description of the embodiments of the present invention is only to exemplarily illustrate the beneficial effects of the embodiments of the present invention, and is not intended to limit the embodiments of the present invention to any of the examples given.

[0091] The embodiments of the present invention have been described above. The above description is exemplary and not exhaustive, and is also not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A computational method for identifying airport runway images, characterized in that: include: Obtain the image frames to be processed from the video stream of the aircraft's forward-looking infrared camera and perform preprocessing operations; A deep learning model that constructs masks of image frames and runway landmarks; Predicting the mask of the runway marker through the deep learning model according to the preprocessed image frame; Multiple judgment rules are applied to the mask of the predicted runway markers, and the airport runway image is obtained after passing the rules.

2. The calculation method for identifying an airport runway image according to claim 1, wherein: The pre-processing operation includes: The image frame is subjected to denoising and enhancement operations by Gaussian filtering and histogram averaging.

3. The calculation method for identifying an airport runway image according to claim 1, wherein: The multiple determination rules include a position determination rule, a slope determination rule, a thunder height determination rule, and a ground speed determination rule.

4. The calculation method for identifying an airport runway image according to claim 3, wherein: The position determination rule is: X 左 <X 中 <X 右 Among them, X 左 , X 中 , X 右 Respectively represent the coordinates on the X-axis of the image corresponding to a fixed Y value in the middle of the left lane line, the middle lane line, and the right lane line.

5. The calculation method for identifying an airport runway image according to claim 3, wherein: The slope determination rules include: The slopes of the left and right lane lines and the slopes of the left and right edges of the runway are obtained according to the mask. The slopes of the left lane line and the left edge of the runway are equal, and the slopes of the right lane line and the right edge of the runway are equal.

6. The calculation method for identifying airport runway images according to claim 3, wherein: The thunder height determination rules include: Read the drone radar height data, calculate the size of the pixels occupied by the runway based on the camera internal parameters and the drone's altitude, and the error with the actual runway is less than the corresponding threshold.

7. The calculation method for identifying airport runway images according to claim 3, wherein: The ground speed determination rules include: The ground speed data of the UAV is read and the running trajectory of the marker in the image frame is calculated, and the error with the runway is less than the corresponding threshold.

8. The calculation method for identifying airport runway images according to claim 1, wherein: Also includes: According to the differential Gaussian model algorithm, the current image frame is compared with the image frame whose result was calculated last time to determine the difference in image similarity. If the difference is less than the set threshold, the data of the current image frame and the image frame whose result was calculated last time are fused.

9. An electronic device, characterized in that: The electronic device comprises: A memory storing executable instructions; A processor, wherein the processor runs the executable instructions in the memory to implement the computing method for identifying an airport runway image according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computing method for identifying an airport runway image according to any one of claims 1 to 8 is implemented.

Images