Image recognition-based intelligent inspection operation system for aircraft routes

Abstract

The application provides an airplane route intelligent inspection operation system based on image recognition. The system comprises a data acquisition module, a video processing module, an image recognition module, an interactive terminal and a monitoring background. The data acquisition module acquires video images of the airplane route inspection operation process and transmits the video images to the video processing module in the processor in real time. Each frame of video image after coding and decoding processing is transmitted to the image recognition module. The image recognition module automatically identifies defects and faults of airplane equipment components in the inspection operation area, and missed inspection and illegal behaviors through a neural network image recognition analysis algorithm, and transmits the inspection video images and the image recognition analysis result data to the interactive terminal and the monitoring background in real time. The interactive terminal realizes voice broadcast reminding and detail viewing of the defects, missed inspection and illegal behaviors. The monitoring background realizes remote real-time monitoring of the inspection operation behavior and guidance of fault elimination operation, and builds an airplane route inspection operation information index database for backtracking analysis and query of the inspection process historical data.

Description

TECHNICAL FIELD

[0001] The present application relates to the aircraft route operation industry, in particular to an aircraft route intelligent inspection operation system based on image recognition. BACKGROUND

[0002] The current aircraft route inspection operation includes short stop, pre-flight, post-flight and departure inspection operation, which is a traditional visual inspection operation mode, full manual detection and paper and pen record work sheet, and the equipment components in different areas of the aircraft are checked one by one according to the around aircraft inspection operation process, such as whether there are defects and faults in the aircraft nose, engine, landing gear, wheelhouse, wing and tail.

[0003] Because there are too many equipment components to be checked in the around aircraft inspection operation process, and the operation time window is short, the quality of the traditional visual inspection operation completely depends on the experience and responsibility of the inspection personnel, so there are problems that the inspection operation completion quality cannot be traced, whether there is missing inspection cannot be confirmed, and the work sheet cannot be signed, which often causes economic losses such as aircraft returning to the field and flight delay, and even leads to flight safety accidents.

[0004] The aircraft route intelligent inspection operation system based on image recognition of the present application provides real-time operation guidance for external inspection personnel to avoid missing inspection, improves the efficiency and quality of the inspection operation, provides an effective supervision tool for quality and safety management personnel to realize safety accident traceability and risk avoidance, can unify and integrate the route aircraft basic database, behavior label record database and defect fault feature database, realizes statistical analysis of big data, and effectively improves the comprehensive management ability of route operation data. SUMMARY

[0005] In order to overcome the above problems or at least partially solve the above problems, the embodiments of the present application provide an aircraft route intelligent inspection operation system based on image recognition.

[0006] The embodiments of the present application are implemented as follows:

[0007] An image recognition-based aircraft route intelligent inspection operation system, comprising: a data acquisition module, configured to acquire video images of aircraft equipment components and inspection operation behaviors in a route inspection operation process and transmit the video images to a processor in real time; the processor, configured to process, store and analyze the video images transmitted by the data acquisition module, and automatically identify defects and failures of the aircraft equipment components, omissions and irregular operation behaviors in the video images in real time, and transmit the image recognition results to an interactive terminal and a supervision background in real time; the interactive terminal, configured to control start and stop of the processor, view the image recognition results of the inspection operation video images through the processor, and convert text prompt information in the image recognition results into corresponding voice content for real-time voice broadcast and reminding; and the supervision background, configured to remotely view real-time video images of the inspection operation through the processor, build an aircraft route inspection operation information index database, and realize backtracking and viewing of historical data records of the inspection operation.

[0008] In some embodiments of the present application, an image recognition-based aircraft route intelligent inspection operation system, the processor comprises a video processing module, configured to receive video image information of aircraft equipment components and inspection operation behaviors, perform H.265 compression encoding storage, video stream MJEPG continuous frame screenshot, video stream real-time transcoding and transmission protocol format conversion and distribution processing, and transmit the video image information processed by data to an image recognition module in real time; the image recognition module is configured to use a pre-trained feature model library of aircraft equipment components, defects and failures and operation behaviors to perform feature matching and recognition judgment on the input video stream MJEPG continuous frame screenshot based on a neural network image recognition analysis algorithm according to the video image information transmitted by the image processing module, and output image recognition results.

[0009] In some embodiments of the present application, an image recognition-based aircraft route intelligent inspection operation system, the video processing module comprises at least a feature model library of aircraft equipment components, defects and failures and operation behaviors, the aircraft equipment components comprise at least feature models of a nose, a radar cover, an engine, a landing gear, a tire, a wheel cabin, a wing, a tail, a fuselage and a hatch door; the defects and failures comprise at least feature models of pits, lightning strikes, stains, damages and bird strikes; and the operation behaviors comprise at least feature models of missing inspection targets, insufficient or overtime inspection duration and incomplete inspection coverage area.

[0010] In some embodiments of the present application, an image recognition-based aircraft route intelligent inspection operation system, the image recognition results in the image recognition module comprise at least names and areas of aircraft equipment components, classification names of defects and failures, classification names of irregular behaviors, video screenshot images with positioning labels, time stamps and text annotation prompt information.

[0011] In some embodiments of the present application, an image recognition-based aircraft route intelligent inspection operation system, the aircraft route inspection operation information index database constructed by the supervision background at least includes unified integrated inspection data, establishes a public basic database, a route aircraft basic database, an inspection process behavior database, a defect fault database, etc., and the information index data at least includes aircraft number, work order number, work card number, operation start and end time, inspection video file access path and image recognition result.

[0012] Some embodiments of the present application have at least the following advantages or beneficial effects:

[0013] 1. The system of the present application uses neural network image recognition technology to realize automatic real-time recognition of defects and faults of aircraft equipment components in route inspection operation, and full-process voice instant reminding, which improves the recognition efficiency of aircraft component defects and greatly reduces the error rate and missed detection rate of manual recognition.

[0014] 2. The system of the present application realizes remote real-time full-process monitoring of inspection operation behavior, automatically identifies irregular operation behavior, and can effectively improve the work quality of inspection operation and greatly reduce the flight safety risks and hidden dangers caused by irregular operation.

[0015] 3. The system of the present application realizes the retrieval and query of video image data of route inspection operation behavior and aircraft equipment faults and defects, can quickly view the history record of faults and defects, thereby greatly improving the work efficiency of inspection operation and solving the problem of safety accident traceability.

[0016] 4. The system of the present application improves the data management capability, unifies the aircraft route operation and maintenance data, establishes a route aircraft basic database, an inspection operation behavior database, a defect fault database, etc., and realizes the statistical analysis of route operation and maintenance big data.

[0017] 5. The system of the present application has flexible scalability, rich interfaces and expansion capability, and can flexibly expand the defect fault recognition type and provide general recognition service to the three-party system. DETAILED DESCRIPTION

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only illustrate some embodiments of the present application, and therefore should not be considered as a limitation to the scope, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0019] Fig. 1 is a structure diagram of an image recognition-based aircraft route intelligent inspection operation system of the present application; DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0023] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0024] Example

[0025] Referring to Figure 1, this embodiment provides an intelligent aircraft flight path inspection system based on image recognition, including:

[0026] The data acquisition module is used to collect video images of aircraft equipment components such as the nose, radome, wings, engines, tail, landing gear, engine room, hatches, and antennas in real time during the line inspection operation, as well as video image information of inspection operation behavior such as equipment inspection area, duration, and operation steps. The invention uses a mechanically image-stabilized ear-hook camera, which is worn by inspection personnel during the operation and can transmit 4K high-definition resolution, MJPEG format video stream data in real time.

[0027] The processor, connected to the data acquisition module, processes, stores, and analyzes video images transmitted by the data acquisition module. It automatically and in real-time identifies aircraft equipment component defects, missed inspections, and violations in the video images, and transmits the image recognition results to the interactive terminal and monitoring backend in real time. The processor includes a video processing module and an image recognition module. The video processing module receives 4K high-definition MJPEG format video stream data transmitted by the data acquisition module, performs H.265 compression encoding to 1920*1080 resolution, 30 frames per second high-definition video, stores it on the local disk, and transmits it to the interactive terminal and monitoring backend in real-time according to the RTSP transmission protocol. It also captures each frame of the video stream and transmits the 4K high-definition MJPEG format continuous frame screenshots to the image recognition module in real-time. The image recognition module, based on the MJPEG continuous frame screenshots transmitted by the image processing module, uses a neural network-based image recognition analysis algorithm and a pre-trained feature model library of aircraft equipment components, defects, and operational behaviors to perform feature matching and recognition on the input MJPEG continuous frame screenshots, outputting the image recognition results. The image recognition results include aircraft equipment component names and regions, fault / defect classification names, video screenshots with location tags, timestamps, and text annotation prompts. This invention uses a portable edge computing host to achieve 4K high-definition video stream input and real-time encoding / decoding conversion output, realizes neural network-based image recognition analysis algorithms and GPU real-time processing calculations, and outputs image recognition results.

[0028] An interactive terminal connects to a processor and controls the processor's start and stop by sending signaling messages. It communicates with an image recognition module via HTTP to obtain image and text information from the image recognition results, decodes and views video screenshots in the image and text information in real time, and converts text prompts into corresponding voice content for instant voice broadcast. This invention uses a waterproof, dustproof, and drop-proof rugged smartphone terminal, which connects to the processor via the smartphone terminal's WiFi network to control the processor's start and stop, view image and text information from the image recognition results, and output voice messages.

[0029] The monitoring backend connects to the processor and communicates with the video processing module via the RTSP transmission protocol to decode and display the inspection operation video stream image in real time at a resolution of 1920*1080, 30 frames per second, and H.265 format, with a video image latency of less than 600ms. It also communicates with the image recognition module via the HTTP transmission protocol to acquire and store image recognition results. Based on these results, an aircraft route inspection operation information index database is constructed, enabling retrospective viewing of historical data records for inspection operations. The information index database includes at least the aircraft number, work order number, work card number, operation start and end time, inspection video file access path, and image recognition results. This invention utilizes a cloud server, remotely connected to the processor via network, to achieve real-time remote monitoring of aircraft inspection operations, assisting in determining defect and fault levels and guiding troubleshooting operations, as well as retrospectively analyzing and querying historical information data on defects, faults, and operational behaviors during the inspection process.

[0030] Furthermore, image recognition technology based on neural networks can enable key capabilities such as intelligent acquisition and analysis of inspection operation video images, as well as automatic output of image data analysis results.

[0031] Furthermore, the system enables intelligent real-time identification of defects and malfunctions in aircraft equipment components during line inspection operations, providing real-time voice prompts throughout the process. This improves the efficiency of aircraft component defect identification and greatly reduces the error rate and missed detection rate of manual identification.

[0032] Furthermore, the system enables remote real-time monitoring of violations during inspection operations and facilitates multi-party communication, effectively improving the quality of inspection work and greatly reducing flight safety risks and hazards caused by violations.

[0033] Furthermore, the system enables the source analysis and retrieval of video image data related to line inspection operations and aircraft equipment malfunctions and defects. It can quickly view historical records of malfunctions and defects and guide troubleshooting operations, thereby greatly improving the efficiency of inspection operations and solving the problem of being unable to trace the source of safety accidents.

[0034] Furthermore, the system enhances data management capabilities, unifies and integrates aircraft route maintenance data, and establishes a basic database of routes and aircraft, a database of inspection operation behavior, and a database of defects and troubleshooting guidelines, thereby enabling statistical analysis of big data in route maintenance.

[0035] Furthermore, the system is flexible and scalable, with rich interfaces and expansion capabilities, which can flexibly expand the types of defects identified and connect with third-party systems to provide general identification services.

[0036] Furthermore, the system enables real-time tracking and monitoring of aircraft component status and inspection activities; and provides unified and standardized management of inspection plans and operational processes. It automatically learns and accumulates a database of defect and hazard feature images, enabling accurate identification and verification of defects and hazards, thus promoting the application and development of intelligent defect and hazard identification technology in the field of line maintenance.

[0037] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0038] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. An intelligent aircraft flight path inspection system based on image recognition, characterized in that, include The data acquisition module is used to acquire video images of aircraft equipment components and inspection activities during the route inspection process and transmit them to the processor in real time. The acquisition module includes a camera, which is worn and used by inspection personnel during the operation to transmit video stream data in real time. The processor is used to process, store and analyze video images transmitted by the data acquisition module, and automatically identify in real time defects, malfunctions, missed inspections and violations of regulations in aircraft equipment components in the video images, and provide real-time voice reminders throughout the process, and transmit the image recognition results to the interactive terminal and the monitoring backend in real time. The video processing module includes a feature model library of work behaviors, and the work behaviors include at least the feature models of missing targets, insufficient or excessive inspection time, and incomplete inspection coverage area. The interactive terminal is used to control the start and stop of the processor, connect to the processor to view the video image recognition results of the inspection operation, and convert the text prompts in the image recognition results into corresponding voice content for real-time voice broadcast reminders; The monitoring backend is used to connect to the processor to remotely view real-time video images of the inspection operation, build an index database of aircraft route inspection operation information, and enable retrospective viewing of historical data records of the inspection operation.

2. The intelligent aircraft flight path inspection system based on image recognition according to claim 1, characterized in that, The processor includes a video processing module and an image recognition module, wherein: The video processing module is used to receive video image information of aircraft equipment components and inspection operations, perform H.265 compression encoding and storage, MJEPG continuous frame screenshotting of video stream, real-time transcoding of video stream, and transmission protocol format conversion and distribution processing, and transmit the processed video image information to the image recognition module in real time. The image recognition module is used to perform feature matching and recognition on the continuous frame screenshots of the input video stream MJEPG based on the video image information transmitted by the image processing module, using a neural network-based image recognition analysis algorithm and a pre-trained feature model library of aircraft equipment components, defects, and operational behaviors, and output the image recognition result.

3. The intelligent aircraft flight path inspection system based on image recognition according to claim 2, characterized in that, The video processing module also includes at least a feature model library of aircraft equipment components and defects; the aircraft equipment components include at least feature models of the nose, radome, engine, landing gear, tires, engine room, wings, tail, fuselage, and hatch doors; the defects include at least feature models of dents, lightning strikes, stains, damage, and bird strikes.

4. The intelligent aircraft flight path inspection system based on image recognition according to claim 2, characterized in that, The image recognition results in the image recognition module include at least the names and regions of aircraft equipment components, the names of fault and defect categories, the names of violation categories, video screenshots with location tags, timestamps, and text annotation prompts.

5. The intelligent aircraft flight path inspection system based on image recognition according to claim 1, characterized in that, The aircraft route inspection operation information index database constructed by the regulatory backend includes at least the unified integration of inspection data, the establishment of a public basic database, a route and aircraft basic database, an inspection process behavior database, and a defect and fault database. The information index data includes at least the aircraft number, work order number, work card number, operation start and end time, inspection video file access path, and image recognition results.

Images