Intelligent aircraft inspection method, device and system, unmanned aerial vehicle and electronic equipment

Abstract

The invention provides an intelligent aircraft inspection method, device and system, an unmanned aerial vehicle and electronic equipment, and the method comprises the steps: responding to an inspection task instruction of an aircraft on a parking apron, and carrying out the self flight waypoint line calibration based on the current position of the aircraft on the parking apron; an inspection task of each waypoint on the calibrated flight waypoint line is started in sequence, data acquisition is performed on a plurality of detection areas on the aircraft based on recording point locations of the inspection tasks, and target position pictures and radar point cloud data of the corresponding detection areas are acquired; identifying the target position pictures and the radar point cloud data of the plurality of detection areas of each recording point location, and generating an inspection image report of the detection area corresponding to each recording point location based on an identification result; and sending the inspection image report of each recording point to a data medium station in real time for visual display. The unmanned aerial vehicle is utilized to inspect the aircraft, and compared with traditional manual inspection, the manpower cost and the time cost are reduced, and the working efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft maintenance, and in particular, to an intelligent aircraft inspection method, device, system, unmanned aerial vehicle and electronic device. Background Art

[0002] With the development of the aviation industry, the safe flight of aircraft has become a key consideration in the industry. As an important link to ensure flight safety, the accuracy and efficiency of aircraft inspection are crucial. The traditional inspection method relies on manual labor. Inspectors need to visually check the surface of the aircraft, which is often time-consuming and laborious, and is easily affected by human factors, and there may be missed inspections or misinspections. Therefore, manual inspection not only increases the inspection cost, but also has potential safety hazards and low work efficiency.

[0003] Therefore, how to reduce the inspection cost and improve the accuracy and efficiency of aircraft inspection is a problem to be solved at present. Summary of the Invention

[0004] The present invention provides an intelligent aircraft inspection method, device, system, unmanned aerial vehicle and electronic device to at least solve the problem in the related art that manual aircraft inspection leads to increased inspection cost, potential safety hazards and low work efficiency. The technical solution of the present invention is as follows:

[0005] According to the first aspect of the embodiments of the present invention, an intelligent aircraft inspection method is provided. The method is applied to an unmanned aerial vehicle and includes:

[0006] In response to an inspection task instruction for an aircraft on the apron, calibrate its own flight waypoint route based on the current position of the aircraft on the apron;

[0007] Sequentially start the inspection tasks for each waypoint on the calibrated flight waypoint route, and collect data for multiple detection areas on the aircraft based on the recorded points of the inspection task, to obtain target position pictures and radar point cloud data for the corresponding detection areas;

[0008] Identify the target position pictures and radar point cloud data for the multiple detection areas at each recorded point respectively, and generate an inspection image report for the detection area corresponding to each recorded point based on the identification results;

[0009] Send the inspection image report for each recorded point to the data center in real time for visual display.

[0010] Optionally, the step of calibrating its own flight waypoint route based on the current position of the aircraft on the apron in response to an inspection task instruction for the aircraft on the apron includes:

[0011] Receive the inspection task instruction of the aircraft on the inspection apron sent by the UAV control center through the data middle platform;

[0012] Based on the inspection task instruction, take images of the head and side wings of the aircraft within the apron;

[0013] Determine the offset coordinates between the current position and the preset position of the aircraft according to the images of the head and side wings;

[0014] Calibrate its own flight waypoint line based on the offset coordinates.

[0015] Optionally, collect data from multiple detection areas on the aircraft based on the recording points of the inspection task, and obtain target position pictures and radar point cloud data of the corresponding detection areas, including:

[0016] Based on the recording points of the inspection task, for multiple detection areas on the aircraft, take visible light or infrared multi-sensor photos according to the requirements of each detection area, and send the photos to the trained neural convolutional network CNN target detection model to obtain target position pictures of visible light or infrared of the corresponding detection areas, where the target detection model is pre-annotated and trained on a picture set taken by visible light and infrared; and

[0017] Based on the recording points of the inspection task, for multiple detection areas on the aircraft, perform point cloud scanning with a lidar sensor according to the requirements of each detection area to obtain radar point cloud data of the corresponding detection areas.

[0018] Optionally, identify the target position pictures and radar point cloud data of multiple detection areas at each recording point respectively, and generate an inspection image report for the corresponding detection areas of each recording point, including:

[0019] Identify the target position pictures and radar point cloud data of multiple detection areas at each recording point by itself respectively, and generate an inspection image report for the corresponding detection areas of each recording point based on the corresponding identification results; and / or

[0020] Send the target position pictures and radar point cloud data of multiple detection areas at each recording point to the algorithm calculation center for identification; so that the algorithm calculation center identifies the received target position pictures and radar point cloud data, generates an inspection image report for the corresponding detection areas of each recording point based on the identification results, and sends the inspection image report to the data middle platform for visual display.

[0021] Optionally, the step of separately identifying the target position pictures and the radar point cloud data of multiple detection areas at each recording point by itself includes: identifying the target position pictures of multiple detection areas at each recording point by its visible light algorithm or infrared thermal imaging algorithm to obtain a first identification result; and identifying the radar point cloud data of multiple detection areas at each recording point by its lidar algorithm to obtain a second identification result;

[0022] The step of generating an inspection image report for the detection area corresponding to each recording point based on the identification result includes: comparing the first identification result with the position picture of the corresponding preset detection area, and generating an inspection image report on whether the detection area corresponding to each recording point is abnormal according to the comparison result of the position picture; and comparing the second identification result with the cloud data of the corresponding preset detection area, and generating an inspection image report on whether the detection area corresponding to each recording point is abnormal according to the comparison result of the cloud data.

[0023] Optionally, the method further includes:

[0024] When the inspection image report for the corresponding detection area is an abnormal report, an alarm event for the corresponding detection area is issued.

[0025] According to the second aspect of the embodiments of the present invention, there is provided an intelligent inspection aircraft device, characterized in that the device is applied to an unmanned aircraft and includes:

[0026] A calibration module, configured to respond to an inspection task instruction of the aircraft on the apron, and calibrate its own flight waypoint line based on the current position of the aircraft on the apron;

[0027] An acquisition module, configured to sequentially start the inspection tasks of each waypoint on the calibrated flight waypoint line, collect data for multiple detection areas on the aircraft based on the recording points of the inspection tasks, and obtain the target position pictures and radar point cloud data of the corresponding detection areas;

[0028] An identification module, configured to separately identify the target position pictures and radar point cloud data of multiple detection areas at each recording point, and generate an inspection image report for the detection area corresponding to each recording point based on the identification result;

[0029] A sending module, configured to real-time send the inspection image report of each recording point to a data center for visual display.

[0030] Optionally, the calibration module includes:

[0031] An instruction receiving module, configured to receive an inspection task instruction of the aircraft on the inspection apron through the data center;

[0032] The shooting module is used to take images of the head and side wings of the aircraft within the apron based on the inspection task instruction;

[0033] The determination module is used to determine the offset coordinates between the current position and the preset position of the aircraft according to the images of the head and side wings;

[0034] The avionics calibration module is used to calibrate its own flight waypoint line based on the offset coordinates.

[0035] Optionally, the acquisition module includes:

[0036] The first sensor acquisition module is used to take visible light or infrared multi-sensor photos of multiple detection areas on the aircraft according to the requirements of each detection area based on the recording points of the inspection task, and send the photos to the trained CNN target detection model to obtain the target position pictures of visible light or infrared for the corresponding detection areas, where the target detection model is pre-annotated and trained on the picture set taken by visible light and infrared; and / or

[0037] The second sensor acquisition module is used to perform point cloud scanning with a lidar sensor on multiple detection areas on the aircraft according to the requirements of each detection area based on the recording points of the inspection task to obtain the radar point cloud data of the corresponding detection areas.

[0038] Optionally, the recognition module includes:

[0039] The algorithm recognition module is used to respectively recognize the target position pictures and radar point cloud data of multiple detection areas at each recording point by itself, and generate an inspection image report for the corresponding detection areas of each recording point based on the recognition results; and / or

[0040] The algorithm call module is used to send the target position pictures and radar point cloud data of multiple detection areas at each recording point to the algorithm calculation center for recognition, and generate an inspection image report for the corresponding detection areas of each recording point based on the recognition results received from the algorithm calculation center.

[0041] Optionally, the algorithm recognition module includes:

[0042] The first algorithm recognition module is used to recognize the target position pictures of multiple detection areas at each recording point through its own visible light algorithm or infrared thermal imaging algorithm to obtain the first recognition result;

[0043] The first report generation module is used to compare the first recognition result with the position pictures of the corresponding preset detection areas, and generate an inspection image report on whether there is an abnormality in the corresponding detection areas of each recording point according to the comparison results of the position pictures.

[0044] The second algorithm recognition module is used to recognize the radar point cloud data of multiple detection areas at each recording point through its own lidar algorithm to obtain a second recognition result;

[0045] The second report generation module is used to compare the second recognition result with the cloud data of the corresponding preset detection area, and generate an inspection image report on whether the detection area corresponding to each recording point is abnormal according to the comparison result of the cloud data.

[0046] Optionally, the algorithm call module includes:

[0047] The data sending module is used to send the target position pictures and radar point cloud data of multiple detection areas at each recording point to the algorithm calculation center for recognition;

[0048] The data receiving module is used to receive the recognition result sent by the algorithm calculation center;

[0049] The third report generation module is used to generate an inspection image report on whether the detection area corresponding to each recording point is abnormal based on the recognition result received by the data receiving module.

[0050] Optionally, the device further includes:

[0051] The alarm module is used to send an alarm event for the corresponding detection area when the inspection image report of the corresponding detection area of the first report generation module, the first report generation module and / or the third report generation module is an abnormal report.

[0052] According to the third aspect of the embodiments of the present invention, an intelligent inspection aircraft system is provided, including: a data middle platform module, a ramp map data center and a drone control center, and a drone controlled by the drone control center, wherein,

[0053] The data middle platform module is used to issue an inspection task instruction for the inspection aircraft to the ramp map data center and the drone control center according to an operation instruction;

[0054] The ramp map data center is used to store and manage the positions of the aircraft parked on the airport ramp, and when receiving the inspection task instruction, send the file of the inspection aircraft and the position information on the ramp map to the drone control center;

[0055] The drone control center is used to control the flight and landing of the drone when receiving the inspection task instruction and the file and position information of the inspection aircraft;

[0056] The drone is used to respond to the inspection task instruction of the aircraft on the apron sent by the drone control center, and calibrate its own flight waypoint route based on the current position of the aircraft on the apron; sequentially start the inspection task of each waypoint on the flight waypoint route after calibration, collect data for multiple detection areas on the aircraft based on the recording points of the inspection task, obtain the target position image and radar point cloud data of the corresponding detection area, respectively identify the target position image and radar point cloud data of multiple detection areas of each recording point, and generate an inspection image report of the detection area corresponding to each recording point based on the identification result, and send the inspection image report of each recording point to the data middle station module in real time;

[0057] The data middle platform module is also used to visually display the inspection image report received from the drone.

[0058] According to a fourth aspect of an embodiment of the present invention, there is provided an intelligent inspection aircraft system, comprising: a data middle platform module, an apron map data center, an unmanned aerial vehicle control center, a multi-sensor fusion recognition module and an algorithm calculation center, and an unmanned aerial vehicle controlled by the unmanned aerial vehicle control center, wherein:

[0059] The data middle platform module is used to issue inspection task execution instructions to the apron map data center and the drone control center according to the operation instructions;

[0060] The apron map data center is used to store and manage the locations of aircraft parked on the airport apron, and upon receiving the inspection task instruction, send the files of the inspection aircraft and the location information on the apron map to the UAV control center;

[0061] The UAV control center is used to control the UAV to fly and land when receiving the inspection task instruction and the file and location information of the inspection aircraft;

[0062] The drone is used to respond to the inspection task instruction of the aircraft on the apron sent by the drone control center, calibrate its own flight waypoint route based on the current position of the aircraft on the apron; sequentially start the inspection task of each waypoint on the flight waypoint route after calibration, collect data for multiple detection areas on the aircraft based on the recording points of the inspection task, obtain the target position picture and radar point cloud data of the corresponding detection area, and send the target position picture and radar point cloud data to the multi-sensor fusion recognition module;

[0063] The multi-sensor fusion recognition module is used to respectively recognize the target position pictures and radar point cloud data of the multiple detection areas received for each recording point;

[0064] The algorithm calculation center is used to compare the picture recognition result and the cloud data recognition result of the multi-sensor fusion recognition module with the position pictures and cloud data of the corresponding preset detection areas respectively, and generate an inspection image report for the detection area corresponding to each recording point according to the comparison results; and send the inspection image report of each recording point to the data middle platform module in real time.

[0065] The data middle platform module is further used to perform visual display on the inspection image report received from the algorithm calculation center.

[0066] According to the fifth aspect of the embodiments of the present invention, a drone is provided, which executes the intelligent inspection aircraft method as described above; and / or is configured with the intelligent inspection aircraft device as described above.

[0067] According to the sixth aspect of the embodiments of the present invention, an electronic device is provided, including:

[0068] A processor;

[0069] A memory for storing executable instructions of the processor;

[0070] Wherein, the processor is configured to execute the instructions to implement the intelligent inspection aircraft method as described above.

[0071] According to the seventh aspect of the embodiments of the present invention, a computer-readable storage medium is provided, when the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can execute the intelligent inspection aircraft method as described above.

[0072] According to the eighth aspect of the embodiments of the present invention, a computer program product is provided, including a computer program or instructions, and when the computer program or instructions are executed by the processor of the electronic device, the intelligent inspection aircraft method as described above is implemented.

[0073] The technical solutions provided by the embodiments of the present invention at least bring the following beneficial effects:

[0074] In the embodiments of the present invention, in response to the inspection task instruction of the aircraft on the apron, the flight waypoint line of the aircraft is calibrated based on the current position of the aircraft on the apron; the inspection tasks of each waypoint on the flight waypoint line after calibration are sequentially started, and data collection is performed on multiple detection areas on the aircraft based on the recording points of the inspection tasks, and the target position pictures and radar point cloud data of the corresponding detection areas are obtained; the target position pictures and radar point cloud data of the multiple detection areas of each recording point are respectively identified, and an inspection image report of the detection area corresponding to each recording point is generated based on the identification results; the inspection image reports of each recording point are sent to the data center in real time for visual display. That is to say, in the embodiments of the present invention, the UAV can quickly and accurately locate through its own flight waypoint line, and identify the collected images and data by itself, and can complete the inspection task in a short time. Compared with the traditional manual inspection, it not only reduces the labor cost and time cost, but also greatly improves the work efficiency.

[0075] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present invention. Brief Description of the Drawings

[0076] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention and do not constitute an improper limitation to the present invention. In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0077] Figure 1 It is a flowchart of a method for an intelligent inspection aircraft provided by an embodiment of the present invention.

[0078] Figure 2 It is a flowchart of an application example of a method for an intelligent inspection aircraft provided by an embodiment of the present invention.

[0079] Figure 3 It is a flowchart of a recording point provided by an embodiment of the present invention.

[0080] Figure 4 It is an interaction flowchart of an algorithm provided by an embodiment of the present invention.

[0081] Figure 5 It is a flowchart of visible light algorithm identification provided by an embodiment of the present invention.

[0082] Figure 6It is a flowchart for infrared imaging algorithm recognition provided by an embodiment of the present invention.

[0083] Figure 7A It is a schematic diagram of normal lines provided by an embodiment of the present invention.

[0084] Figure 7B It is a schematic diagram of abnormal low temperature of lines provided by an embodiment of the present invention.

[0085] Figure 8A It is a schematic diagram of normal lines extracted by using a skeleton thinning algorithm provided by an embodiment of the present invention.

[0086] Figure 8B It is a schematic diagram of abnormal low temperature of lines extracted by using a skeleton thinning algorithm provided by an embodiment of the present invention.

[0087] Figure 9 It is a flowchart for lidar algorithm recognition provided by an embodiment of the present invention.

[0088] Figure 10 It is a block diagram of an intelligent inspection aircraft device provided by an embodiment of the present invention.

[0089] Figure 11 It is a block diagram of an intelligent inspection aircraft system provided by an embodiment of the present invention.

[0090] Figure 12 It is another block diagram of an intelligent inspection aircraft system provided by an embodiment of the present invention.

[0091] Figure 13 It is an application block diagram of an intelligent inspection aircraft system provided by an embodiment of the present invention.

[0092] Figure 14 It is a block diagram of a drone provided by an embodiment of the present invention.

[0093] Figure 15 It is a block diagram of an electronic device provided by an embodiment of the present invention.

[0094] Figure 16 It is a block diagram of a device for an intelligent inspection aircraft provided by an embodiment of the present invention. Specific embodiments

[0095] In order to enable those of ordinary skill in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0096] It should be noted that the terms "first", "second", etc. in the description, claims and above-mentioned drawings of the present invention are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that such data used can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0097] Technical terms:

[0098] The region growing algorithm is an image segmentation method based on pixel similarity. It realizes image segmentation by combining similar pixel points into a region.

[0099] The neural convolutional network (CNN, Convolutional Neural Network) is used to process image grid structure data. It extracts features through convolutional layers and reduces the data dimension through pooling layers, and is widely used in image recognition and object detection.

[0100] DilatedFCN, the semantic segmentation algorithm is mainly based on the dilated fully convolutional network (dilatedFCN). This network uses dilated convolutions in the backbone network to extract high-resolution feature maps to achieve high-performance segmentation performance.

[0101] The HSV color space, the parameters of the color in HSV are: hue (H), saturation (S), value (V). It is an intuitive color model for users. One can start with a pure color, that is, specify the color angle H and let V = S = 1, and then the required color can be obtained by adding black and white to it. Adding black can reduce V while S remains unchanged, and similarly adding white can reduce S while V remains unchanged.

[0102] Open3D is an open-source library that supports the rapid development of software for processing 3D data. It provides rich functions for processing point cloud data, including operations such as point cloud reading and writing, visualization, filtering, feature extraction, segmentation, surface reconstruction, and spatial transformation.

[0103] A point cloud is a point data set collected by a laser scanning device. That is to say, a point cloud is data automatically scanned by various 3D laser scanners (airborne, vehicle-mounted, tripod-mounted, handheld, etc.) for the measurement object, including the three-dimensional coordinates, backscatter intensity, color and other information of each point. Due to the huge and dense amount of data, it is called a point cloud. It can be used in many fields such as topographic surveying, 3D modeling, building and cultural relics surveying, power line inspection, and deformation monitoring.

[0104] On the basis of understanding the above technical terms, please also refer to the following embodiments.

[0105] Under normal circumstances, before and after an aircraft flight, it is usually necessary for inspection personnel to conduct an external inspection of the aircraft in a timely manner to detect damage or faults on the aircraft surface, such as cracks, detachment, deformation, etc., and perform timely repairs and replacements to avoid the impact of these faults on flight safety. However, traditional aircraft external inspections usually require a large amount of manpower and time, resulting in high labor costs. Based on this problem, the embodiments of the present invention provide an intelligent inspection method for aircraft. Through the daily automatic intelligent inspection of the aircraft by an unmanned aerial vehicle (UAV), the UAV can quickly and accurately locate and complete the inspection task within a short period of time. Compared with traditional manual inspections, UAV inspections can greatly improve work efficiency. Compared with traditional inspection methods, UAV inspections can reduce labor costs and time costs. An efficient UAV can replace multiple inspectors, thereby reducing the operating costs of inspections.

[0106] Please refer to Figure 1 , which is a flowchart of an intelligent inspection method for aircraft provided by the embodiments of the present invention. The method is applied to a UAV, as Figure 1 shown, and the method includes the following steps:

[0107] Step 101: In response to the inspection task instruction of the aircraft on the apron, calibrate the flight waypoint line based on the current position of the aircraft on the apron.

[0108] Step 102: Sequentially start the inspection tasks for each waypoint on the calibrated flight waypoint line, collect data for multiple detection areas on the aircraft based on the recorded points of the inspection tasks, and obtain the target position pictures and radar point cloud data of the corresponding detection areas.

[0109] Step 103: Identify the target position pictures and radar point cloud data of multiple detection areas for each recorded point respectively, and generate an inspection image report for the corresponding detection area of each recorded point based on the identification results.

[0110] Step 104: Send the inspection image report of each recorded point to the data center in real time for visual display.

[0111] The intelligent inspection method for aircraft described in the present invention can be applied to terminals, etc., without limitation here. The terminal implementation devices can be electronic devices such as UAVs, without limitation here.

[0112] In the embodiments of the present invention, the drone can quickly and accurately locate through its own flight waypoint route, and perform self-identification on the collected images and data, and can complete the inspection task in a short time. Compared with the traditional manual inspection, it not only reduces the labor cost and time cost, but also greatly improves the work efficiency. A high-efficiency drone can replace multiple people for inspection, thereby reducing the operation cost of inspection. A high-efficiency drone can replace multiple people for inspection, thereby reducing the operation cost of inspection.

[0113] The following combines Figure 1 , and details the specific implementation steps of a method for an intelligent inspection aircraft provided by the embodiments of the present invention.

[0114] In step 101, in response to the inspection task instruction of the aircraft on the apron, the self-flight waypoint route is calibrated based on the current position of the aircraft on the apron.

[0115] In this step, the user can send an inspection task instruction to the apron map data center and the drone control center through the data middle platform. When the drone control center receives the inspection task instruction, it controls the drone to take off for inspection. That is, the drone receives the inspection task instruction of the aircraft on the apron sent by the drone control center through the data middle platform. Based on the inspection task instruction, images of the head and flank of the aircraft are taken within the apron, and the offset coordinates between the current position and the preset position of the aircraft are determined according to the images of the head and flank; based on the offset coordinates, the self-flight waypoint route is calibrated, and the purpose of the calibration is to make the current flight waypoint route of the drone match the originally preset drone flight waypoint route.

[0116] In step 102, the inspection tasks of each waypoint on the calibrated flight waypoint route are sequentially started, and data is collected from multiple detection areas on the aircraft based on the recording points of the inspection tasks, and target position pictures and radar point cloud data of the corresponding detection areas are obtained.

[0117] In this step, after the UAV calibrates its own flight waypoint route, the inspection tasks for each waypoint on the calibrated flight waypoint route are sequentially started. For multiple detection areas on the aircraft based on the recording points of the inspection tasks, visible light or infrared multi-sensor photos are taken according to the requirements of each detection area, and the photos are sent to a trained convolutional neural network (CNN) object detection model to obtain the target position pictures of visible light or infrared for the corresponding detection areas. Among them, the object detection model is pre-annotated and trained on a set of pictures taken by visible light and infrared. And for multiple detection areas on the aircraft based on the recording points of the inspection tasks, lidar sensors are used to perform point cloud scanning according to the requirements of each detection area to obtain the radar point cloud data for the corresponding detection areas.

[0118] Among them, in this embodiment, a waypoint is a point of interest collected by the inspection aircraft, the position of a point. That is to say, a waypoint is an important location and landmark recorded in the Global Positioning System (GPS). A flight route is a route connected by waypoints and consists of two or more points.

[0119] In step 103, the target position pictures and radar point cloud data for multiple detection areas at each recording point are respectively identified, and an inspection image report for the corresponding detection area at each recording point is generated based on the identification results.

[0120] In this step, in the first identification method, the UAV can respectively identify the target position pictures and radar point cloud data for multiple detection areas at each recording point by itself, and generate an inspection image report for the corresponding detection area at each recording point based on the identification results.

[0121] In the second identification method, the UAV can send the target position pictures and radar point cloud data for multiple detection areas at each recording point to an algorithm calculation center for identification; so that the algorithm calculation center can identify the received target position pictures and radar point cloud data, generate an inspection image report for the corresponding detection area at each recording point based on the identification results, and send the inspection image report to the data middle platform for visual display.

[0122] The third recognition method, that is, combining the above two methods, means that the UAV can identify the partial simple target position pictures and radar point cloud data of multiple detection areas at each recording point by itself, and generate an inspection image report for the detection area corresponding to each recording point based on the recognition result; and send the partial complex target position pictures and radar point cloud data of multiple detection areas at each recording point to the algorithm calculation center for recognition; so that the algorithm calculation center can recognize the received partial complex target position pictures and radar point cloud data, generate an inspection image report for the detection area corresponding to each recording point based on the recognition result, and send the inspection image report to the data middle platform for visual display.

[0123] It should be noted that in this embodiment, the method of separately recognizing the target position pictures and radar point cloud data can be at least one of the above three methods.

[0124] Among them, the process of the UAV itself recognizing the target position pictures of multiple detection areas at each recording point and generating an inspection image report for the detection area corresponding to each recording point based on the recognition result includes: recognizing the target position pictures of multiple detection areas at each recording point through its own visible light algorithm or infrared thermal imaging algorithm to obtain a first recognition result; comparing the first recognition result with the position pictures of the corresponding preset detection areas, and generating an inspection image report on whether the detection area corresponding to each recording point is abnormal according to the comparison result of the position pictures.

[0125] The process of the UAV itself recognizing the radar point cloud data of multiple detection areas at each recording point and generating an inspection image report for the detection area corresponding to each recording point based on the recognition result includes: recognizing the radar point cloud data of multiple detection areas at each recording point through its own lidar algorithm to obtain a second recognition result; comparing the second recognition result with the cloud data of the corresponding preset detection areas, and generating an inspection image report on whether the detection area corresponding to each recording point is abnormal according to the comparison result of the cloud data.

[0126] It should be noted that in this embodiment, since the position and direction angle of the aircraft parked on the apron are fixed, it is necessary to pre-enter each type of aircraft parameter into the system in advance, and also need to establish independent files in the system for the sizes and other information of different types of aircraft parked on the apron in advance, as well as preset the position pictures and preset cloud data of each detection area to be inspected by the aircraft in advance, so that the UAV can collect the images or data of each detection area on the aircraft for matching to determine whether there is an abnormality.

[0127] In step 104, the inspection image report of each recording point is sent to the data middle platform in real time for visual display.

[0128] In this step, the UAV sends the inspection image report of each recording point to the data center in real time, so that the data center can visually display the inspection image report. The abnormal information is included in the visualized inspection image report, which is convenient for obtaining the corresponding processing decision according to the abnormal information.

[0129] In the embodiment of the present invention, in response to the inspection task instruction of the aircraft on the apron, the flight waypoint line of the UAV is calibrated based on the current position of the aircraft on the apron; the inspection tasks of each waypoint on the flight waypoint line after calibration are started in sequence, and data collection is performed on multiple detection areas on the aircraft based on the recording points of the inspection tasks, and the target position pictures and radar point cloud data of the corresponding detection areas are obtained; the target position pictures and radar point cloud data of multiple detection areas at each recording point are respectively identified, and an inspection image report of the detection area corresponding to each recording point is generated based on the identification result; the inspection image report of each recording point is sent to the data center in real time for visual display. That is to say, in the embodiment of the present invention, the UAV can quickly and accurately locate through its own flight waypoint line, and perform self-identification on the collected images and data, and can complete the inspection task in a short time. Compared with the traditional manual inspection, it not only reduces the labor cost and time cost, but also greatly improves the inspection work efficiency.

[0130] Optionally, in another embodiment, on the basis of the above embodiment, the method may further include: when the inspection image report of the corresponding detection area is an abnormal report, an alarm event of the corresponding detection area is issued.

[0131] That is to say, in this embodiment, for this abnormal report, an alarm event is issued, so that the maintenance personnel can give corresponding processing decisions for this abnormal report, further improving the accuracy of aircraft inspection and abnormal handling.

[0132] Please also refer to Figure 2 , which is a flowchart of an application example of an intelligent inspection aircraft method provided by the embodiment of the present invention. The execution entities involved in the method include: a UAV, a data center (which can also be called a data center module, a front-end and back-end module or a background system module, etc.) and an algorithm calculation center module (which can also be simply called an algorithm calculation center). Specifically, it includes:

[0133] Step 201: The data center module sends the received inspection task instruction of the inspection aircraft issued by the user to the UAV control center.

[0134] Step 202: The UAV control center sends the inspection task instruction to the UAV.

[0135] Step 203: The drone takes off and flies into the apron to capture images of the nose and wing of the aircraft.

[0136] Step 204: The drone sends the captured images of the nose and wing to the algorithm calculation center.

[0137] Step 205: Based on the triangular coordinate positioning principle formed by the nose and wing, the algorithm calculation center obtains the offset coordinates of the current position of the aircraft from the originally preset position.

[0138] Among them, the triangular coordinate positioning principle is to use multiple detectors to detect the target orientation at different positions, and then use the triangular geometry principle to determine the position and distance of the target; that is, according to the characteristics of a triangle, as long as the length of one side and the angles between this side and the other two sides are known, the third side can be calculated.

[0139] Step 206: The algorithm calculation center sends the offset coordinates to the drone.

[0140] Step 207: Based on the offset coordinates, the drone calibrates its own flight waypoint route by compensating the coordinate position, so as to match the originally preset drone flight waypoint route.

[0141] Step 208: After the waypoint route calibration of the drone is completed, the inspection tasks of each waypoint on the flight waypoint route are started.

[0142] Step 209: The drone collects data on multiple detection areas on the aircraft based on the recording points of the inspection task, and obtains the target position images and radar point cloud data of the corresponding detection areas.

[0143] In this step, the drone performs actions such as taking pictures with the camera and scanning the lidar point cloud for the waypoint task, and asynchronously transmits the collected target position images and radar point cloud data to the algorithm calculation center;

[0144] Step 210: The drone asynchronously transmits the collected target position images and radar point cloud data to the algorithm calculation center.

[0145] Step 211: After completing the inspection task of the waypoint route, the drone returns to the nest of the drone control center for charging.

[0146] Step 212: The algorithm calculation center respectively identifies the target position images and radar point cloud data of multiple detection areas at each recording point received, and generates an inspection image report for the corresponding detection area at each recording point based on the identification results.

[0147] In this step, different algorithms can be used to identify the target location images and lidar point cloud data synchronously or asynchronously. The algorithms can include: visible light algorithms, infrared thermal imaging algorithms, lidar algorithms, etc. In specific applications, it is not limited to this, and other similar algorithms can also be included. This embodiment does not make any restrictions.

[0148] Among them, in the specific implementation of step 211 and step 212, there is no order of precedence, and they can also be executed simultaneously. This embodiment does not make any restrictions.

[0149] Step 213: The algorithm calculation center sends the inspection image report of each recording point to the data middle platform module in real time.

[0150] Step 214: The data middle platform module visually displays the received inspection image report.

[0151] In the embodiment of the present invention, the drone can quickly and accurately locate by calibrating its own flight waypoint route, and take the example of sending the collected images and data to the algorithm calculation center for identification (the drone can also perform its own identification, etc.). It can complete the inspection task in a short time. Compared with the traditional manual inspection, it not only reduces the labor cost and time cost, but also greatly improves the inspection work efficiency.

[0152] Please also refer to Figure 3 , which is a flowchart of a recording point provided by the embodiment of the present invention. In this embodiment, the drone records the waypoints and measurement points (the measurement points are the picture-taking actions required after reaching the waypoints) of the aircraft on the apron under manual intervention, so that during the execution of the next inspection task, it will take pictures and identify them in sequence according to the recorded points. Then, the pictures and lidar point cloud data obtained through the measurement points. In this embodiment, the visible light and infrared pictures are labeled and trained to obtain a target detection model, that is, the target location can be accurately determined only during the task execution. And after scanning the lidar point cloud data through the measurement points and performing point cloud segmentation and feature extraction actions, during the execution of the next task, it can be compared and detected with this feature data. Specifically, it includes:

[0153] Step 301: The drone records the waypoints and measurement points (i.e., the recorded points) according to the civil aviation detection points;

[0154] Step 302: The drone collects visible light and infrared pictures and lidar point cloud scans according to the recorded points to obtain corresponding visible light and infrared data sets, and lidar point cloud data sets;

[0155] Step 303: The visible light and infrared data sets may include: screw loss holes, bumps and pits on the fuselage, APU extinguishing bottle access door, locking status of the access door to the THS compartment, angle of attack transducer, crew oxygen release status, radome, green status of the indicating window when the hatch is closed and locked, status of the drinking water discharge panel switch, status of the air-conditioning connection handle, fuel tank dipstick area, static dischargers, etc. However, in specific applications, it is not limited to this.

[0156] Step 304: Label the collected visible light and infrared data sets using the labeling tool;

[0157] Step 305: Input the labeled visible light and infrared data sets into the neural convolutional network for training to obtain a trained target detection model, and then execute Step 308.

[0158] In this embodiment, a training set is obtained. The training set includes: visible light data sets and infrared data sets, as well as visible light data sets and infrared data sets with labels. The training set is input into the neural convolutional network detection model for training. During the training process, the visible light data and infrared data output by the neural convolutional network detection model are respectively compared with the input visible light data sets and infrared data sets with labels, the gap between the two is calculated, and this gap is used as the loss value. Based on the loss value, iterative training is performed through the backpropagation mechanism, and the parameters of the neural convolutional network detection model are adjusted until convergence to obtain a trained target detection model. This target detection model can also be called a trained neural convolutional network target detection model.

[0159] Step 306: The radar point cloud data sets may include: detection of unblocked full temperature probe access holes, detection of unblocked airspeed tube inlets, detection of unblocked exhaust grilles, detection of unblocked NACA vents, etc.

[0160] It should be noted that Steps 303 and 306, in specific applications, do not have a sequential order and can also be executed simultaneously. This embodiment does not make any restrictions.

[0161] Step 307: Perform point cloud segmentation and point cloud feature extraction on the radar point cloud data sets as the initial data of the radar point cloud data. After collecting the radar point cloud data of the next inspection task, compare it with this initial data to determine whether the collected data is abnormal.

[0162] In the embodiment of the present invention, the UAV collects visible light and infrared photos and radar point cloud scans according to the recorded points to obtain corresponding visible light and infrared data sets, as well as radar point cloud data sets, labels the visible light and infrared data sets, and the radar point cloud data sets, and inputs them into the neural convolutional network for training to obtain a trained target detection model.

[0163] Please also refer to Figure 4 which is an interaction flowchart of multiple algorithms provided by an embodiment of the present invention. In this embodiment, after the drone receives a mission instruction and reaches a waypoint, it parses the instruction to perform visible light, infrared photography, and lidar point cloud scanning, and then sends them to the algorithm calculation center. The algorithm calculation center uses the corresponding visible light algorithm, infrared thermal imaging algorithm, and lidar algorithm for identification and analysis to generate an inspection image report. Finally, the calculated inspection image report is sent to the data middle platform (i.e., the front-end and back-end systems) for visual display and storage. The method includes:

[0164] Step 401: The drone reaches the specified waypoint;

[0165] Step 402: The drone respectively uses visible light photography, infrared thermal imaging photography, and lidar scanning according to the inspection task to obtain the corresponding visible light target position picture, infrared thermal imaging target image, and radar point cloud data, and sends the visible light target position picture, infrared thermal imaging target image, and radar point cloud data to the algorithm calculation center.

[0166] In this step, the drone respectively uses visible light photography to obtain the corresponding visible light target position picture for each detection area of each recording point in the inspection task; and uses infrared thermal imaging photography to obtain the corresponding infrared thermal imaging target image, and uses lidar scanning to obtain the corresponding radar point cloud data, and can send the visible light target position picture, infrared thermal imaging target image, and radar point cloud data to the algorithm calculation center; of course, in this embodiment, the drone can also use the visible light algorithm, infrared thermal imaging algorithm, and lidar algorithm stored in itself for identification, and the identification process is the same as that of each algorithm in the algorithm calculation center. For the specific implementation process of the corresponding algorithm, please refer to the following. This embodiment takes sending to the algorithm calculation center for identification as an example.

[0167] In this step, for different inspection tasks, the drone can use different shooting methods to obtain the target position pictures and radar point cloud data of each detection area of each recording point.

[0168] Step 403: The algorithm calculation center respectively uses the visible light algorithm, infrared thermal imaging algorithm, and lidar algorithm for identification to obtain the visible light identification result, infrared imaging identification result, and radar point cloud identification result.

[0169] That is to say, in this step, the algorithm calculation center respectively uses the visible light algorithm to identify the received visible light target position pictures to obtain the visible light recognition results; and uses the infrared thermal imaging algorithm to identify the received infrared thermal imaging target position pictures to obtain the infrared recognition results; and uses the lidar algorithm to identify the received radar point cloud data target position pictures to obtain the radar point cloud recognition results.

[0170] Step 404: The algorithm calculation center compares the visible light recognition results with the position pictures of the corresponding preset detection areas, and generates an inspection image report on whether the corresponding detection areas are abnormal according to the comparison results of the position pictures; and compares the infrared recognition results with the position pictures of the corresponding preset detection areas, and generates an inspection image report on whether the corresponding detection areas are abnormal according to the comparison results of the position pictures; and compares the radar point cloud recognition results with the cloud data of the corresponding preset detection areas, and generates an inspection image report on whether the corresponding detection areas are abnormal according to the comparison results of the cloud data.

[0171] Step 405: The algorithm calculation center synthesizes all the inspection image reports on whether the corresponding detection areas are abnormal to obtain all the inspection image reports on whether the corresponding detection areas of each recording point are abnormal.

[0172] Step 406: The algorithm calculation center sends all the inspection image reports on whether the corresponding detection areas of each recording point are abnormal to the data middle platform.

[0173] Step 407: The data middle platform visually displays and stores the received inspection image reports.

[0174] In the embodiment of the present invention, the drone sends the collected images and data to the algorithm calculation center for identification, which can complete the inspection task in a short time. Compared with the traditional manual inspection, it not only reduces the labor cost and time cost, but also greatly improves the inspection work efficiency.

[0175] Please also refer to Figure 5 , which is a flowchart of a visible light algorithm recognition provided by the embodiment of the present invention. The method includes:

[0176] Step 501: The drone reaches the designated waypoints of the inspection aircraft and takes pictures of multiple detection areas of each recording point.

[0177] Step 502: The CNN target detection module classifies the pictures of multiple detection areas of each recording point taken, and uses different algorithms to identify the pictures of different categories.

[0178] Step 503: The first type of pictures includes: screw loss holes, concavities and convexities on the fuselage, APU fire extinguisher access door, locking status of the THS compartment access door, status of the fuselage oxygen release flap, green status of the indicator window when the cabin door is closed and locked, and status of the air conditioning interface handle;

[0179] Step 504: Determine whether the status results in the first type of pictures are the same as the pre-stored correct data. If they are the same, it indicates normal; otherwise, it indicates abnormal. Further, when there is an abnormality, an alarm can be issued.

[0180] Step 505: The second type of pictures includes: angle of attack sensor, radome, fuel tank dipstick area, static discharger brush, and status of the drinking water drain panel switch;

[0181] It should be noted that for Step 503 and Step 505, in specific implementation, the order can be selected arbitrarily or they can be executed simultaneously. This embodiment does not make any restrictions.

[0182] Step 506: For the angle of attack sensor and the static discharger brush in the second type of pictures, determine whether the number of sensors, the number of static discharger brushes, and the relative distance recorded at the recording point are matched with the pre-stored ones. If they are matched, it indicates normal; otherwise, it indicates a shortage. Further, when there is a shortage, an alarm can be issued.

[0183] Step 507: For the radome in the second type of pictures, determine whether there is any foreign object covering or blocking when taking pictures and identifying the HSV reflection brightness in all directions of the UAV. If there is, it indicates abnormal; otherwise, it indicates normal. Further, when there is an abnormality, an alarm can be issued.

[0184] Step 508: For the fuel tank dipstick area in the second type of pictures, in the cropped area, extract the height distance ratio between the black vernier caliper and the square panel through RGB to determine whether the fuel tank dipstick meets the standard. If it does, it indicates normal; otherwise, it indicates abnormal. Further, when there is an abnormality, an alarm can be issued.

[0185] Step 509: For the status of the drinking water drain panel switch in the second type of pictures, use the DilatedFCN segmentation algorithm to segment the pointer, and use the least squares method to calculate the angle of the pointer to obtain the switch status of the pointer, that is, the pointer angle. Determine whether the pointer angle is the same as the pre-stored pointer angle. If it is the same, it indicates normal; otherwise, it indicates abnormal. Further, when there is an abnormality, an alarm can be issued.

[0186] It should be noted that for Step 506 to Step 509, in specific implementation, there is no order requirement and they can also be executed simultaneously. This embodiment does not make any restrictions.

[0187] That is to say, in this embodiment, as Figure 5 shown:

[0188] In steps 501 to 504, after the UAV reaches the designated waypoint, it takes pictures of the measurement points and sends them to the algorithm calculation center for target detection and recognition. After identifying the results of the screw loss holes, body bumps and pits, APU fire extinguisher access door, locking status of the THS compartment access door, status of the crew oxygen release tablets, green status of the indicator window when the cabin door is closed and locked, and air-conditioning interface handle status, the identified results are then compared with the previously recorded data to determine whether there is an abnormality.

[0189] In steps 501, 502, and 506, for the identification of the missing angle-of-attack sensors and discharge brushes, after the algorithm receives the pictures, it performs target detection to obtain the position coordinates, and then matches the relative coordinate templates of the angle-of-attack sensors and discharge brushes during recording. If the match is inconsistent, an alarm is issued.

[0190] In steps 501, 502, and 507, for the radome area which is relatively large and triangular, it is impossible to capture all the details in one shot. That is, it takes multiple shots from multiple angles and then compares the HSV color space. If there is a large difference in the HSV color space, it is determined that there is an abnormality in this area, and an alarm is issued.

[0191] In steps 501, 502, and 508, to identify the relative scale ratio value of the fuel tank dipstick. After the target detects the fuel tank dipstick area, it extracts the black dipstick through RGB color extraction, calculates the angle, obtains the perspective transformation correction parameters to correct the image. Finally, it calculates the height distance ratio of the dipstick relative to the square panel to obtain the position of the dipstick.

[0192] In steps 501, 502, and 509, for the identification of the switch state of the drinking water drain panel. After the target detects the panel switch pointer, it uses the DilatedFCN segmentation algorithm to segment the pointer, and finally uses the least squares method to calculate the angle of the pointer to obtain the pointer angle, that is, the switch state.

[0193] Please also refer to Figure 6 , which is a flowchart of an infrared thermal imaging algorithm identification provided by an embodiment of the present invention, specifically including:

[0194] Step 601: After the UAV reaches the designated waypoint, it takes infrared thermal imaging pictures (such as no overheating of the brake actuating pipe, front landing gear - front wheel well pipeline, wire detection);

[0195] Step 602: Use the infrared thermal imaging algorithm to convert the original infrared thermal imaging image to obtain the temperature value and the pseudo-color image.

[0196] Step 603: Determine whether the temperature value exceeds the preset threshold. If so, execute step 604; otherwise, execute step 609, that is, end;

[0197] Step 604: Send an alarm event for abnormal temperature points, and then execute Step 609;

[0198] Step 605: Based on the pseudo-color image, use the HSV color space to identify the circuit diagram above the threshold, and binaryize it into a black and white image;

[0199] Step 606: Use the region growing algorithm to extract the circuit diagram;

[0200] Step 607: Determine whether the extracted circuit diagram is consistent with the template matching the excitation through the skeleton thinning algorithm. If not, execute Step 608; otherwise, execute Step 609:

[0201] Step 608: Send a circuit alarm event;

[0202] Step 609: End.

[0203] That is to say, in this embodiment, as Figure 6 shown in the infrared thermal imaging algorithm recognition flowchart shown:

[0204] In Steps 601 and 602, after the UAV reaches the specified waypoint, it takes an infrared image, and then converts the temperature value and the pseudo-color image through the distance from the measured point.

[0205] In Steps 603 and 604, if the converted temperature value is higher than the set threshold, immediately send an alarm event for abnormal temperature points.

[0206] In Steps 605 and 606, based on the pseudo-color image, use the HSV color space to identify the circuit diagram above the threshold, binaryize it into a black and white image, and then use the region growing algorithm to extract the circuit, specifically as Figure 7A and Figure 7B shown, Figure 7A is a schematic diagram of a normal circuit provided by an embodiment of the present invention, Figure 7B is a schematic diagram of an abnormal low temperature of a circuit provided by an embodiment of the present invention. Under normal circumstances, the higher the temperature, the whiter the thick white line in the figure.

[0207] For another example Figure 8A and Figure 8B , Figure 8A is a schematic diagram of a normal circuit extracted by using the skeleton thinning algorithm provided by an embodiment of the present invention, Figure 8B is a schematic diagram of an abnormal low temperature of a circuit extracted by using the skeleton thinning algorithm provided by an embodiment of the present invention.

[0208] That is to say, since the circuit is basically fixed, the recorded circuit is used as a template. Whether the circuit extracted by using the skeleton thinning algorithm is consistent with the recorded matching template is used to determine whether to give an alarm.

[0209] For another example, when the line is normal and there are abnormal low-temperature points on the line, etc., the judgment method is similar to the above-mentioned low temperature, and will not be elaborated here specifically.

[0210] Please also refer to Figure 9 , which is a flowchart for laser radar algorithm recognition provided by an embodiment of the present invention, specifically including:

[0211] Step 901: After the drone reaches the designated waypoint, 3D radar data is scanned (such as detecting whether the sampling holes of the full-temperature probe are unblocked, whether the air intake of the pitot tube is unblocked, whether the exhaust grille is unblocked, whether the NACA ventilation port is unblocked, etc.)

[0212] Step 902: After Open3d point cloud filtering, point cloud matching is performed with the recorded point (i.e., video recording point) data record to obtain an accurate region of interest.

[0213] Step 903: Take out the point with the farthest distance in the region of interest, and then use the region growing algorithm to judge whether the elevation difference between two points is less than the threshold. If so, the region continues to grow; otherwise, the region stops growing.

[0214] Step 904: Judge whether the difference between the point statistics after region growth and the recorded point statistics is less than the threshold. If so, execute Step 905; otherwise, execute Step 906.

[0215] Step 905: Send a normal event record.

[0216] Step 906: Send an alarm event and request manual review.

[0217] Step 907: End.

[0218] That is to say, in this embodiment, as Figure 9 shown,

[0219] In Steps 901 and 902, after the drone reaches the waypoint, the data obtained by scanning the recorded position is sent to the algorithm for point cloud filtering and registration to accurately obtain the region of interest.

[0220] In Steps 903 to 906, take out the point with the farthest distance in the region of interest, and then use the region growing algorithm to judge whether the elevation difference between two points is less than the threshold. If so, the region continues to grow, otherwise, the region stops growing. Then, continue to use the region growing algorithm to judge whether the difference between the point statistics after growth and the recorded fiber statistics is less than the threshold. If it is less, it is determined to be normal, otherwise an alarm event is issued and manual review is requested.

[0221] It should be noted that in this embodiment, examples of other inspection operations on the aircraft that need to be identified will not be elaborated one by one.

[0222] It should be noted that for the method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present disclosure is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to the present invention.

[0223] Please also refer to Figure 10 , which is a block diagram of an intelligent inspection aircraft device provided by an embodiment of the present invention. The device includes: a calibration module 1001, a collection module 1002, an identification module 1003, and a transmission module 1004, where

[0224] The calibration module 1001 is configured to respond to an inspection task instruction of an aircraft on the apron and calibrate its own flight waypoint line based on the current position of the aircraft on the apron;

[0225] The collection module 1002 is configured to sequentially start the inspection tasks of each waypoint on the flight waypoint line after calibration, collect data from multiple detection areas on the aircraft based on the recording points of the inspection tasks, and obtain target position pictures and radar point cloud data of the corresponding detection areas;

[0226] The identification module 1003 is configured to respectively identify the target position pictures and radar point cloud data of multiple detection areas at each recording point, and generate an inspection image report for the corresponding detection area at each recording point based on the identification results;

[0227] The transmission module 1004 is configured to real-time transmit the inspection image report of each recording point to the data middle platform for visual display.

[0228] Optionally, in another embodiment, based on the above embodiment, the calibration module includes:

[0229] An instruction receiving module, configured to receive an inspection task instruction of an aircraft on the apron through the data middle platform;

[0230] A photographing module, configured to photograph images of the head and wings of the aircraft within the apron based on the inspection task instruction;

[0231] A determination module, configured to determine the offset coordinates between the current position of the aircraft and a preset position according to the images of the head and wings;

[0232] An avionics calibration module for calibrating its own flight waypoint line based on the offset coordinates.

[0233] Optionally, in another embodiment, based on the above embodiment, the acquisition module includes:

[0234] A first sensor acquisition module for taking visible light or infrared multi-sensor photos of multiple detection areas on the aircraft according to the requirements of each detection area based on the recording points of the inspection task, and sending the photos to a trained CNN target detection model to obtain target position pictures of visible light or infrared for the corresponding detection areas, where the target detection model is pre-annotated and trained on a picture set taken by visible light and infrared; and / or

[0235] A second sensor acquisition module for performing point cloud scanning with a lidar sensor on multiple detection areas on the aircraft according to the requirements of each detection area based on the recording points of the inspection task to obtain radar point cloud data for the corresponding detection areas.

[0236] Optionally, in another embodiment, based on the above embodiment, the recognition module includes:

[0237] An algorithm recognition module for respectively recognizing the target position pictures and radar point cloud data of multiple detection areas at each recording point by itself, and generating an inspection image report for the corresponding detection area of each recording point based on the recognition results; and / or

[0238] An algorithm call module for sending the target position pictures and radar point cloud data of multiple detection areas at each recording point to an algorithm calculation center for recognition, and generating an inspection image report for the corresponding detection area of each recording point based on the recognition results received from the algorithm calculation center.

[0239] Optionally, in another embodiment, based on the above embodiment, the algorithm recognition module includes:

[0240] A first algorithm recognition module for recognizing the target position pictures of multiple detection areas at each recording point through its own visible light algorithm or infrared thermal imaging algorithm to obtain a first recognition result;

[0241] A first report generation module for comparing the first recognition result with the position pictures of the corresponding preset detection areas, and generating an inspection image report on whether the corresponding detection area of each recording point is abnormal according to the comparison results of the position pictures;

[0242] A second algorithm recognition module, configured to recognize the lidar point cloud data of multiple detection areas at each recording point through its own lidar algorithm, and obtain a second recognition result;

[0243] A second report generation module, configured to compare the second recognition result with the cloud data of the corresponding preset detection area, and generate an inspection image report on whether the detection area corresponding to each recording point is abnormal according to the comparison result of the cloud data.

[0244] Optionally, in another embodiment, based on the above embodiment, the algorithm call module includes:

[0245] A data sending module, configured to send the target position pictures and lidar point cloud data of multiple detection areas at each recording point to an algorithm calculation center for recognition;

[0246] A data receiving module, configured to receive the recognition result sent by the algorithm calculation center;

[0247] A third report generation module, configured to generate an inspection image report on whether the detection area corresponding to each recording point is abnormal based on the recognition result received by the data receiving module.

[0248] Optionally, in another embodiment, based on the above embodiment, the device further includes:

[0249] An alarm module, configured to issue an alarm event for the corresponding detection area when the inspection image report of the corresponding detection area of the first report generation module, the first report generation module and / or the third report generation module is an abnormal report.

[0250] Please also refer to Figure 11 , which is a block diagram of an intelligent inspection aircraft system provided by an embodiment of the present invention. The system includes: a data middle platform module 1101, and a parking apron map data center 1102, a drone control center 1103, and a drone 1104 that are respectively connected to the data middle platform module 1101 for communication, and the drone 1104 controlled by the drone control center 1103. Among them,

[0251] The data middle platform module 1101 is configured to issue an inspection task instruction for the inspection aircraft to the parking apron map data center and the drone control center according to an operation instruction;

[0252] The parking apron map data center 1102 is configured to store and manage the positions of the aircraft parked on the airport parking apron, and when receiving the inspection task instruction, send the file of the inspection aircraft and the position information on the parking apron map to the drone control center;

[0253] The UAV control center 1103 is used to control the UAV to fly and land when receiving the inspection task instruction and the file and position information of the inspection aircraft;

[0254] The UAV 1104 is used to respond to the inspection task instruction of the aircraft on the apron sent by the UAV control center, calibrate its own flight waypoint line based on the current position of the aircraft on the apron; sequentially start the inspection tasks of each waypoint on the flight waypoint line after calibration, collect data for multiple detection areas on the aircraft based on the recording points of the inspection task, obtain the target position pictures and radar point cloud data of the corresponding detection areas, respectively identify the target position pictures and radar point cloud data of the multiple detection areas at each recording point, generate an inspection image report for the detection area corresponding to each recording point based on the identification results, and send the inspection image report of each recording point to the data middle platform module 1101 in real time;

[0255] The data middle platform module 1101 is further used to visually display the inspection image report received from the UAV 1104.

[0256] Please also refer to Figure 12 For another block diagram of an intelligent inspection aircraft system provided by an embodiment of the present invention, the system includes: a data middle platform module 1201, and a parking apron map data center 1202 respectively connected to the data middle platform module 1201, a UAV control center 1203, a multi-sensor acquisition module 1204 and an algorithm calculation center 1205 communicate with each other respectively, and a UAV 1206 controlled by the UAV control center 1203, and the algorithm calculation center 1205 communicates with the multi-sensor acquisition module 1204 and the UAV 1206 respectively, wherein,

[0257] The data middle platform module 1201 is used to issue an inspection task instruction to the parking apron map data center and the UAV control center according to an operation instruction;

[0258] The parking apron map data center 1202 is used to store and manage the positions of the aircraft parked on the airport apron, and when receiving the inspection task instruction, send the file of the inspection aircraft and the position information on the apron map to the UAV control center;

[0259] The UAV control center 1203 is used to control the UAV to fly and land when receiving the inspection task instruction and the file and position information of the inspection aircraft;

[0260] The drone 1206 is used to respond to the inspection task instruction of the aircraft on the apron sent by the drone control center, and calibrate its own flight waypoint line based on the current position of the aircraft on the apron; sequentially start the inspection tasks of each waypoint on the calibrated flight waypoint line, collect data for multiple detection areas on the aircraft based on the recording points of the inspection tasks, obtain the target position pictures and radar point cloud data of the corresponding detection areas, and send the target position pictures and radar point cloud data to the multi-sensor fusion recognition module;

[0261] The multi-sensor acquisition module 1204 is used to respectively identify the target position pictures and radar point cloud data of multiple detection areas for each recording point received.

[0262] The algorithm calculation center 1205 is used to compare the picture recognition result and cloud data recognition result of the multi-sensor fusion recognition module with the position pictures and cloud data of the corresponding preset detection areas respectively, and generate an inspection image report for the detection area corresponding to each recording point according to the comparison result; send the inspection image report of each recording point to the data middle platform module in real time;

[0263] The data middle platform module 1201 is further used to visually display the inspection image report received from the algorithm calculation center.

[0264] Please also refer to Figure 13 , which is an application block diagram of an intelligent inspection aircraft system provided by an embodiment of the present invention, specifically including: a user 1301, a data middle platform 1302, an apron map data center 1303, a multi-sensor acquisition data center 1304, an algorithm calculation center 1305, a drone control center 1306, and a drone 1307. Among them, the data middle platform 1302 is respectively communicatively connected with the user 1301, the apron map data center 1303, the multi-sensor acquisition data center 1304, the algorithm calculation center 1305, and the drone control center 1306, and the drone control center 1306 controls the drone 1307 to take off, inspect, and land, and the algorithm calculation center 1305 is respectively communicatively connected with the multi-sensor acquisition data center 1304 and the drone 1307.

[0265] The system schematic diagram of the intelligent inspection aircraft based on the drone is as Figure 13As shown in the figure. The (dashed box) represents the data middle platform (i.e., the data middle platform module), through which users can interact with the apron map data center, the UAV control center, the multi-sensor disability data center, and the algorithm calculation center. As shown by the labels 1, 2, 3, and 4 in the figure, users can issue instructions to execute inspection tasks to the apron map data center and the UAV control center through the data middle platform, calibrate their own flight waypoint routes based on the current position of the aircraft on the apron, collect data from multiple detection areas on the aircraft based on the recorded points of the inspection task, obtain target position pictures and radar point cloud data of the corresponding detection areas, send the target position pictures and radar point cloud data of the multiple detection areas at each recorded point to the calculation center for recognition, and receive the generated inspection image report in feedback, and can view the status through real-time video backhaul. As shown by the labels 1 and 4, users can perform manual intervention in case of emergency and control the UAV to fly, land, etc. through the UAV control center. As shown by the labels 1, 5, 6, and 7, users can identify with the multi-sensor data acquisition center and the algorithm calculation center through the data middle platform, generate an inspection image report based on the recognition result, and send it to the data middle platform in real time for visual display to the users.

[0266] Please also refer to Figure 14 , which is a block diagram of a UAV provided by an embodiment of the present invention. The UAV is configured with an intelligent inspection aircraft device 1401. The modules included in the intelligent inspection aircraft device 1401 are as described above in detail and will not be elaborated here.

[0267] In the embodiment of the present invention, the UAV can quickly and accurately locate and complete the inspection task within a short time. Compared with traditional manual inspection, UAV inspection can greatly improve work efficiency. UAV inspection can liberate operators from dangerous environments and reduce the risk of injury to inspection personnel. In addition, the development of UAV technology enables inspection tasks to be performed in harsh environments, ensuring the quality and reliability of inspections. Compared with traditional inspection methods, UAV inspection in the embodiment of the present invention can reduce labor costs and time costs. An efficient UAV can replace multiple people for inspection, thereby reducing the operating cost of inspection.

[0268] Optionally, the embodiment of the present invention further provides an electronic device, including:

[0269] A processor;

[0270] A memory for storing executable instructions of the processor;

[0271] Wherein, the processor is configured to execute the instructions to implement the intelligent inspection aircraft method as described above.

[0272] Optionally, an embodiment of the present invention further provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the intelligent inspection aircraft method as described above.

[0273] Optionally, an embodiment of the present invention further provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a processor of an electronic device, the intelligent inspection aircraft method as described above is implemented.

[0274] Regarding the device in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0275] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0276] Figure 15 is a block diagram of an electronic device 1500 provided by an embodiment of the present invention Figure 15 . As shown in the figure, it includes a processor 1501, a communication interface 702, a memory 1503, and a communication bus 1504. Among them, the processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through the communication bus 1504;

[0277] The memory 1503 is used to store executable instructions of the processor;

[0278] The processor 1501 is used to implement the method as described above when executing the executable instructions on the memory 1503.

[0279] Among them, the communication bus in this embodiment may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0280] The communication interface is used for communication between the above electronic device and other devices.

[0281] The memory may include a Random Access Memory (RAM), or may also include a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0282] The aforementioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0283] In another embodiment provided by the present invention, a computer-readable storage medium is further provided. When the instructions in the computer-readable storage medium are executed by the processor of an electronic device, the electronic device can execute the intelligent inspection aircraft method as described above. For example, the computer-readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0284] In another embodiment provided by the present invention, a computer program product is further provided, including a computer program or instructions. When the computer program or instructions are executed by the processor of an electronic device, the intelligent inspection aircraft method as described above is implemented.

[0285] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0286] Figure 16 FIG. 4 is a block diagram of a device 1600 for an intelligent inspection aircraft provided by an embodiment of the present invention. For example, the device 1600 can be provided as a server. Referring to Figure 16 FIG. 4, the device 1600 includes a processing component 1622, which further includes one or more processors, and memory resources represented by a memory 1632 for storing instructions executable by the processing component 1622, such as application programs. The application programs stored in the memory 1632 can include one or more modules each corresponding to a set of instructions. In addition, the processing component 1622 is configured to execute instructions to perform the above method.

[0287] The device 1600 may further include a power supply component 1626 configured to perform power management of the device 1600, a wired or wireless network interface 1650 configured to connect the device 1600 to a network, and an input / output (I / O) interface 1658. The device 1600 can operate based on an operating system stored in the memory 1632, such as Windows ServerTM, MacOS XTM, UnixTM, LinuxTM, FreeBSDTM, or the like.

[0288] Other embodiments of the present invention will be readily apparent to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following the general principles of the invention and including known common general knowledge or conventional technical means in the technical field not disclosed herein. The specification and examples are only to be considered as exemplary, and the true scope and spirit of the invention are pointed out by the following claims.

[0289] It should be understood that the present invention is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. An intelligent aircraft inspection method, characterized in that: The method is applied to a drone, comprising: In response to an inspection task instruction of an aircraft on the apron, calibrate its own flight waypoint route based on the current position of the aircraft on the apron; Sequentially start the inspection task of each waypoint on the flight waypoint route after calibration, collect data for multiple detection areas on the aircraft based on the recording points of the inspection task, and obtain target position images and radar point cloud data of the corresponding detection areas; Recognize the target position images and radar point cloud data of multiple detection areas of each recording point respectively, and generate an inspection image report of the detection area corresponding to each recording point based on the recognition results; The inspection image report of each recording point is sent to the data center in real time for visual display.

2. The intelligent aircraft inspection method according to claim 1, characterized in that: The step of responding to the inspection task instruction of the aircraft on the apron and calibrating the flight waypoint route of the aircraft based on the current position of the aircraft on the apron includes: Receive inspection mission instructions for aircraft on the apron from the drone control center through the data center; Based on the inspection task instruction, taking images of the head and wing of the aircraft in the apron; Determining offset coordinates between a current position of the aircraft and a preset position according to the images of the head and wing; Based on the offset coordinates, the flight waypoint route is calibrated.

3. The intelligent aircraft inspection method according to claim 1, characterized in that: The data collection for multiple detection areas on the aircraft based on the recording points of the inspection task to obtain target position images and radar point cloud data of the corresponding detection areas includes: Based on the recorded points of the inspection task, multiple inspection areas on the aircraft are photographed by visible light or infrared multi-sensors according to the needs of each inspection area, and the photos are sent to a trained neural convolution network (CNN) target detection model to obtain visible light or infrared target position pictures of the corresponding inspection area, wherein the target detection model is pre-annotated and trained on a set of pictures taken by visible light and infrared; and Based on the recorded points of the inspection task, a laser radar sensor is used to perform point cloud scanning on multiple inspection areas on the aircraft according to the needs of each inspection area to obtain radar point cloud data of the corresponding inspection area.

4. The intelligent aircraft inspection method according to claim 1, characterized in that: The target position images and radar point cloud data of the multiple detection areas of each recording point are respectively identified, and an inspection image report of the detection area corresponding to each recording point is generated based on the identification result, including: Identify the target location images and radar point cloud data of multiple detection areas of each recording point by itself, and generate an inspection image report of the detection area corresponding to each recording point based on the corresponding identification results; and / or The target position pictures and radar point cloud data of the multiple detection areas of each recording point are sent to the algorithm computing center for identification; so that the algorithm computing center can identify the received target position pictures and radar point cloud data, and generate an inspection image report of the detection area corresponding to each recording point based on the identification result, and send the inspection image report to the data center for visual display.

5. The intelligent aircraft inspection method according to claim 4, characterized in that: The identifying of the target position pictures and radar point cloud data of the multiple detection areas of each recording point by itself respectively includes: identifying the target position pictures of the multiple detection areas of each recording point by its own visible light algorithm or infrared thermal imaging algorithm to obtain a first identification result; and identifying the radar point cloud data of the multiple detection areas of each recording point by its own laser radar algorithm to obtain a second identification result; The method of generating an inspection image report of the detection area corresponding to each recording point based on the recognition result includes: comparing the first recognition result with the position picture of the corresponding preset detection area, and generating an inspection image report of whether the detection area corresponding to each recording point is abnormal based on the comparison result of the position picture; and comparing the second recognition result with the cloud data of the corresponding preset detection area, and generating an inspection image report of whether the detection area corresponding to each recording point is abnormal based on the comparison result of the cloud data.

6. An intelligent inspection aircraft device, characterized in that: The device is applied to a drone, and includes: A calibration module, for responding to an inspection task instruction of an aircraft on the apron, and performing self-flight waypoint route calibration based on the current position of the aircraft on the apron; The acquisition module is used to sequentially start the inspection task of each waypoint on the flight waypoint route after calibration, collect data for multiple detection areas on the aircraft based on the recording points of the inspection task, and obtain the target position image and radar point cloud data of the corresponding detection area; An identification module, used to identify the target position images and radar point cloud data of multiple detection areas of each recording point respectively, and generate an inspection image report of the detection area corresponding to each recording point based on the identification results; The sending module is used to send the inspection image report of each recording point to the data center in real time for visual display.

7. An intelligent aircraft inspection system, characterized in that: include: The data center module, the apron map data center and the drone control center, and the drone control center controls the drone, wherein: The data middle platform module is used to issue inspection task instructions of the inspection aircraft to the apron map data center and the drone control center according to the operation instructions; The apron map data center is used to store and manage the locations of the aircraft parked on the airport apron, and upon receiving the inspection task instruction, send the files of the inspection aircraft and the location information on the apron map to the UAV control center; The UAV control center is used to control the UAV to fly and land when receiving the inspection task instruction and the file and location information of the inspection aircraft; The drone is used to respond to the inspection task instruction of the aircraft on the apron sent by the drone control center, and calibrate its own flight waypoint route based on the current position of the aircraft on the apron; sequentially start the inspection task of each waypoint on the flight waypoint route after calibration, collect data for multiple detection areas on the aircraft based on the recording points of the inspection task, obtain the target position image and radar point cloud data of the corresponding detection area, respectively identify the target position image and radar point cloud data of multiple detection areas of each recording point, and generate an inspection image report of the detection area corresponding to each recording point based on the identification result, and send the inspection image report of each recording point to the data middle station module in real time; The data middle platform module is also used to visually display the inspection image report received from the drone.

8. An intelligent aircraft inspection system, characterized in that: include: Data center module, apron map data center, drone control center, multi-sensor fusion recognition module and algorithm calculation center, and drones controlled by the drone control center, among which, The data middle platform module is used to issue inspection task execution instructions to the apron map data center and the drone control center according to the operation instructions; The apron map data center is used to store and manage the locations of aircraft parked on the airport apron, and upon receiving the inspection task instruction, send the files of the inspection aircraft and the location information on the apron map to the UAV control center; The UAV control center is used to control the UAV to fly and land when receiving the inspection task instruction and the file and location information of the inspection aircraft; The drone is used to respond to the inspection task instruction of the aircraft on the apron sent by the drone control center, calibrate its own flight waypoint route based on the current position of the aircraft on the apron; sequentially start the inspection task of each waypoint on the flight waypoint route after calibration, collect data for multiple detection areas on the aircraft based on the recording points of the inspection task, obtain the target position picture and radar point cloud data of the corresponding detection area, and send the target position picture and radar point cloud data to the multi-sensor fusion recognition module; The multi-sensor fusion recognition module is used to respectively recognize the target position pictures and radar point cloud data of the multiple detection areas received for each recording point; The algorithm computing center is used to compare the image recognition results and cloud data recognition results of the multi-sensor fusion recognition module with the location images and cloud data of the corresponding preset detection areas, respectively, and generate an inspection image report of the detection area corresponding to each recording point according to the comparison results; and send the inspection image report of each recording point to the data middle station module in real time; The data middle platform module is also used to visualize the inspection image report received from the algorithm computing center.

9. A drone, characterized in that: Execute the intelligent inspection aircraft method according to any one of claims 1 to 5; and / or be equipped with the intelligent inspection aircraft device according to claim 6.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the intelligent aircraft inspection method according to any one of claims 1 to 5.

Images