Aircraft overhaul field detection method, system and equipment and storage medium

Abstract

The invention discloses an aircraft maintenance field detection method, system and device and a storage medium, and relates to the field of aviation maintenance, the method comprises the steps that a target video stream is acquired in real time according to a target maintenance task, and the target maintenance task comprises a task execution stage and a task execution scene; identifying the target video stream according to the task execution stage and the task execution scene to obtain a target detection result; and feeding back the target detection result to the cloud platform. And analyzing a task execution stage and a task execution scene in the video stream to capture a key target and a key state in a maintenance process so as to identify any abnormal condition, so that early warning is given out in time, and potential safety risks are avoided. Through integration of the cloud platform, remote monitoring and decision making are supported, so that managers can know field conditions in real time and respond to any abnormal conditions in time.

Description

Technical Field

[0001] The present invention relates to the field of aviation detection, and in particular to an aircraft maintenance on-site detection method, system, equipment and storage medium. Background Art

[0002] Aircraft maintenance sites, as the cornerstone of airlines' operational support systems, play a crucial role in ensuring safe ground operations. They constitute the first and strongest line of defense for flight safety. However, current maintenance sites generally face management challenges due to their multifaceted nature. Traditional methods of monitoring abnormalities at maintenance sites appear inadequate and lack effective advanced technological support. The "person-to-person" supervision model is not only inefficient but also ill-suited to the rapid development and high standards of the modern aviation industry.

[0003] To improve flight safety and reduce errors during pre- and post-flight inspections, various monitoring and improvement measures are currently being implemented, including the use of checklists, cross-checks, flight data monitoring systems, enhanced flight crew training, regular and irregular audits and inspections, and the establishment of feedback and improvement mechanisms. However, these traditional methods still rely on manual operations and lack the support of advanced technology, resulting in inefficient oversight and difficulties in effectively preventing and controlling maintenance errors. This makes oversight more difficult for regulatory authorities, inadequate corporate management tools, and often places maintenance personnel under significant pressure. Summary of the Invention

[0004] The primary objective of this invention is to provide an on-site aircraft maintenance inspection method, system, device, and storage medium. This method analyzes the mission execution phases and scenarios captured in video streams to capture key objectives and key states during the maintenance process, identifying any anomalies and providing timely warnings to mitigate potential safety risks. Through cloud platform integration, remote monitoring and decision-making are supported, enabling managers to understand on-site conditions in real time and respond promptly to any anomalies.

[0005] In order to achieve the above objectives, the embodiments of the present application provide the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a method for on-site inspection of aircraft maintenance is provided, the method comprising:

[0007] Acquire a target video stream in real time according to a target maintenance task, wherein the target maintenance task includes a task execution phase and a task execution scenario;

[0008] Identify the target video stream according to the task execution stage and the task execution scenario to obtain a target detection result;

[0009] The target detection result is fed back to the cloud platform.

[0010] Optionally, identifying the target video stream according to the task execution stage and the task execution scenario to obtain a target detection result includes:

[0011] Identifying objects in the target video stream to obtain a number of target objects and corresponding position information;

[0012] Based on the detection conditions corresponding to the mission execution phase and the mission execution scenario, as well as a number of target objects and corresponding location information, a target detection result is obtained; the mission execution phase includes a mission start phase and a mission end phase, and the mission execution scenario includes a pre-flight inspection and a post-flight inspection.

[0013] Optionally, the target objects are safety pins and landing gear; based on the detection conditions corresponding to the mission execution phase and the mission execution scenario, as well as a plurality of target objects and corresponding position information, obtaining a target detection result includes:

[0014] Calculating an intersection-over-union ratio of detection frames of the safety pin and the landing gear according to position information of the safety pin and the landing gear;

[0015] If the detection frame intersection-over-union ratio exceeds a set intersection-over-union ratio threshold, the target detection result is determined as a detection of an installed safety pin;

[0016] If the mission is in post-flight inspection, and the detection frame intersection-in-union ratio does not exceed the set intersection-in-union ratio threshold at the end of the mission execution, the target detection result is determined to be an abnormality in which the safety pin is not installed;

[0017] If it is during pre-flight inspection, after the mission execution starts, if the detection frame intersection-in-union ratio exceeds the set intersection-in-union ratio threshold, the target detection result is determined to be an abnormality in which the safety pin is not removed.

[0018] Optionally, the target object is a certificate; based on the detection conditions corresponding to the task execution stage and the task execution scenario, as well as a plurality of target objects and corresponding location information, a target detection result is obtained, including:

[0019] Obtaining a certificate identification result based on whether the certificate is identified, wherein the certificate identification result includes whether the certificate is successfully identified or not identified;

[0020] If all target certificates are successfully identified during the task execution phase, the target detection result is determined to be certificate ready;

[0021] If not all target certificates are identified during the task execution phase, the target detection result is determined to be insufficient certificate preparation.

[0022] Optionally, after feeding back the target detection result to the cloud platform, the method further includes:

[0023] The cloud platform sends the target detection results to the corresponding inspection instruments and user devices, and performs an abnormal reminder process.

[0024] Optionally, acquiring a target video stream in real time according to a target maintenance task includes:

[0025] Obtain the target maintenance task sent by the cloud platform, wherein the target maintenance task also includes relevant information and a video stream address of a patrol instrument, wherein the relevant information includes work order information, department information, maintenance personnel information, and patrol instrument identification information;

[0026] The target video stream is acquired in real time based on the inspection instrument video stream address.

[0027] Optionally, before acquiring the target video stream in real time according to the target maintenance task, the method further includes:

[0028] The cloud platform verifies the interface access rights based on the login information of the user's device, and executes the subsequent steps if the verification is successful;

[0029] Receive signals from inspection instruments, record the start and end status of inspection tasks, and obtain and maintain a list of online inspection instruments at a fixed frequency;

[0030] The video data uploaded by the inspection instrument is archived and associated with the inspection task information.

[0031] According to a second aspect of an embodiment of the present application, there is provided an aircraft maintenance on-site detection system, the system comprising:

[0032] A target video stream acquisition module is used to acquire a target video stream in real time according to a target maintenance task, wherein the target maintenance task includes a task execution phase and a task execution scenario;

[0033] A target detection module is used to identify the target video stream according to the task execution stage and task execution scenario to obtain a target detection result;

[0034] The feedback module is used to feed back the target detection result to the cloud platform.

[0035] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0036] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. The computer-readable instructions can be executed by a processor to implement the method described in the first aspect above.

[0037] In summary, the embodiments of the present application provide an aircraft maintenance on-site detection method, system, device and storage medium, which obtains a target video stream in real time according to a target maintenance task, wherein the target maintenance task includes a task execution stage and a task execution scene; identifies the target video stream according to the task execution stage and the task execution scene to obtain a target detection result; and feeds back the target detection result to the cloud platform. The task execution stage and the task execution scene in the video stream are analyzed to capture the key targets and key states in the maintenance process to identify any abnormal situation, thereby issuing an early warning in time to avoid potential safety risks. Through the integration of the cloud platform, remote monitoring and decision-making are supported, allowing managers to understand the on-site situation in real time and respond to any abnormal situation in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0039] The structures, proportions, sizes, etc. illustrated in this specification are intended solely to complement the contents disclosed herein and to facilitate understanding and reading by persons skilled in the art. They are not intended to limit the conditions under which the present invention may be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, without affecting the efficacy and objectives of the present invention, shall remain within the scope of the technical contents disclosed herein.

[0040] Figure 1 A flowchart of an on-site inspection method for aircraft maintenance provided in an embodiment of the present application;

[0041] Figure 2 Detailed application process of the maintenance site anomaly detection system based on cloud-edge-end collaboration provided in the embodiment of this application;

[0042] Figure 3 A logical diagram of an anomaly detection algorithm provided in an embodiment of the present application;

[0043] Figure 4 A functional block diagram of the software architecture provided in the embodiment of the present application;

[0044] Figure 5 This is an architecture diagram of an aircraft maintenance on-site detection system provided in an embodiment of the present application;

[0045] Figure 6A structural diagram of an electronic device provided in an embodiment of the present application is shown;

[0046] Figure 7 A diagram showing a computer-readable storage medium provided in an embodiment of the present application.

[0047] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0049] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0050] In addition, the terms "first," "second," and so on, used in this disclosure are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referenced. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this disclosure, "plurality" means at least two, such as two or three, unless otherwise specifically defined.

[0051] In the present invention, unless otherwise specified or limited, the terms "connection" and "fixation" should be understood in a broad sense. For example, "fixation" can mean fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; internal communication between two elements or interaction between two elements, unless otherwise specified. Those skilled in the art will be able to understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0052] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in this field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0053] There are many abnormal situations at the maintenance site, and the present invention mainly targets the following two: 1. The aircraft landing gear safety pin is a key device to ensure the safety of the aircraft on the ground. When the aircraft is parked or undergoing system testing, the insertion of the safety pin is crucial to prevent the landing gear from accidentally retracting. Before takeoff, it is necessary to ensure that all safety pins have been correctly removed to avoid affecting the aircraft's aerodynamic layout and flight performance. However, the storage and retrieval process of the safety pins is often prone to errors, and the traditional inspection process relies on manual operation, which is inefficient and prone to errors. 2. The aircraft's nationality registration certificate, airworthiness certificate and radio station license are necessary documents for the legal flight of the aircraft, known as the "three certificates". In the pre-flight inspection, confirming whether the three certificates are complete is a key step, and any omission may result in the aircraft being unable to fly legally. Traditional inspection methods rely on manual verification, which is not only time-consuming, but also prone to errors due to human negligence.

[0054] In view of the above abnormal conditions at the maintenance site, the present invention needs to further propose a fast, efficient and accurate automatic detection method for abnormal conditions at the aircraft maintenance site.

[0055] Figure 1 An aircraft maintenance on-site detection method provided by an embodiment of the present application is shown, the method comprising:

[0056] Step 101: acquiring a target video stream in real time according to a target maintenance task, wherein the target maintenance task includes a task execution phase and a task execution scenario;

[0057] Step 102: Identify the target video stream according to the task execution stage and task execution scenario to obtain a target detection result;

[0058] Step 103: Feedback the target detection result to the cloud platform.

[0059] In a possible implementation, before acquiring the target video stream in real time according to the target maintenance task in step 101, the method further includes:

[0060] The cloud platform verifies interface access rights based on the login information of the user device, and executes subsequent steps if the verification is passed; receives signals from the inspection meter, records the start and end status of the inspection task, and obtains and maintains the online inspection meter list at a fixed frequency; archives the video data uploaded by the inspection meter and associates it with the inspection task information.

[0061] The cloud platform verifies interface access permissions, ensuring only authorized users and devices can access and perform maintenance tasks, enhancing system security and data protection. Task progress is tracked to ensure accurate record of task start and end points, while online inspection meters are updated in real time to facilitate task allocation and management. Video data is saved and associated with task information, providing a basis for future review, analysis, and anomaly investigation.

[0062] Imagine an aircraft requires a pre-flight inspection. A patrol meter (an integrated maintenance personnel wearable device) worn by the maintenance personnel begins recording a real-time video stream. The cloud platform first verifies the maintenance personnel's identity and permissions, then records the start status of the inspection task and begins updating the online patrol meter list at a fixed frequency. During the inspection, the patrol meter transmits the real-time video stream, which the edge intelligent algorithm platform analyzes to identify whether the safety pin is correctly installed or removed, and to verify that the three certificates are complete. If anomalies are detected, such as an incomplete safety pin or missing certificates, the system immediately feeds these test results back to the cloud platform. Upon receiving these test results, the cloud platform can immediately notify the maintenance personnel to make corrections and record these anomalies for subsequent analysis. This ensures that all safety measures are in place before the aircraft takes off, thus ensuring flight safety. This approach significantly improves safety and efficiency at aircraft maintenance sites.

[0063] In a possible implementation, in step 101, acquiring a target video stream in real time according to a target maintenance task includes:

[0064] Obtain the target maintenance task sent by the cloud platform, the target maintenance task also includes relevant information and the patrol instrument video stream address, the relevant information includes work order information, department information, maintenance personnel information and patrol instrument identification information; obtain the target video stream in real time based on the patrol instrument video stream address.

[0065] Detailed information about the target maintenance task is obtained from the cloud platform, including work order information, department information, maintenance personnel information, and patrol meter identification information. This helps ensure that maintenance work is executed according to the planned schedule and requirements, and provides context for subsequent video stream analysis. The target video stream is obtained in real time based on the patrol meter's video stream address. This allows the system to monitor the maintenance process in real time, ensuring that every moment of the maintenance process is captured, providing real-time data for subsequent anomaly detection and analysis.

[0066] For example, let's say an aircraft is about to undergo a pre-flight inspection. The cloud platform sends the target inspection task to the edge intelligent algorithm platform, including detailed task information and the video stream address of the inspection meter. This information may include: Work order information: The specific content and requirements of the inspection task; Department information: The department responsible for the inspection; Maintenance personnel information: The list and qualifications of the personnel involved in the inspection; Inspection meter identification information: The identifier used to identify the specific inspection meter.

[0067] Based on this information, the edge intelligent algorithm platform connects to the designated inspection instrument video stream address and begins acquiring the video stream in real time. During the inspection process, if the deep learning model detects any anomalies in the video stream (for example, a safety pin not installed correctly), the system immediately feeds the detection results back to the cloud platform, which then notifies on-site maintenance personnel to take necessary corrective measures. This implementation ensures real-time monitoring and rapid response to maintenance work, improving the safety and efficiency of aircraft maintenance.

[0068] In a possible implementation, in step 102, identifying the target video stream according to the task execution stage and the task execution scenario to obtain a target detection result includes:

[0069] Objects in the target video stream are identified to obtain a number of target objects and corresponding location information; target detection results are obtained based on the detection conditions corresponding to the task execution stage and the task execution scenario, as well as the number of target objects and corresponding location information; the task execution stage includes a task start stage and a task end stage, and the task execution scenario includes a pre-flight inspection and a post-flight inspection.

[0070] Identify objects in the target video stream to obtain key target objects and their location information within the maintenance site. Identified targets are further analyzed and evaluated based on the specific detection conditions of the mission execution phase (mission start and mission end) and mission execution scenario (pre-flight maintenance and post-flight maintenance). Target detection results are generated based on the target object's location information and the mission execution phase and scenario. These results are used to determine whether the maintenance task was executed correctly according to established procedures and whether any anomalies exist.

[0071] In one possible implementation, in step 102, the target objects are safety pins and landing gear. Based on the detection conditions corresponding to the mission execution phase and the mission execution scenario, as well as a plurality of target objects and corresponding position information, a target detection result is obtained, including:

[0072] Based on the position information of the safety pin and the landing gear, the detection frame intersection-and-union ratio of the safety pin and the landing gear is calculated; if the detection frame intersection-and-union ratio exceeds the set intersection-and-union ratio threshold, the target detection result is determined to be that the installed safety pin is detected; if it is in post-flight inspection, the detection frame intersection-and-union ratio does not exceed the set intersection-and-union ratio threshold at the end of mission execution, the target detection result is determined to be that the safety pin is not installed abnormally; if it is in pre-flight inspection, the detection frame intersection-and-union ratio exceeds the set intersection-and-union ratio threshold after the start of mission execution, the target detection result is determined to be that the safety pin is not removed abnormally.

[0073] Identifying and acquiring the location information of the shear pin and landing gear at the inspection site provides the basis for subsequent IoU calculations and anomaly detection. By calculating the IoU of the detection frames of the shear pin and landing gear, the spatial relationship between them is determined. This is a key step in determining whether the shear pin is correctly installed or removed. Based on the IoU results and the task execution phase and scenario, it is determined whether there is an anomaly where the shear pin is not installed or removed. This helps to promptly identify and correct potential safety risks.

[0074] Suppose an aircraft is undergoing post-flight maintenance. The system identifies the positions of the safety pin and landing gear through the video stream and generates corresponding detection frames. The system calculates the intersection-and-union ratio of the safety pin and landing gear detection frames. If the intersection-and-union ratio exceeds the set threshold (for example, 20%), the safety pin is considered to be correctly installed. If the intersection-and-union ratio does not exceed the threshold at the end of the task execution, the system will determine that the safety pin is not installed abnormally and immediately notify the maintenance personnel to check and correct it. If the intersection-and-union ratio exceeds the threshold after the pre-flight maintenance task begins, the system will determine that the safety pin is not removed abnormally and notify the maintenance personnel to handle it. In this way, the safety and efficiency of the aircraft maintenance site are significantly improved, while also ensuring the airworthiness of the aircraft.

[0075] In one possible implementation, in step 102, the target object is a certificate; based on the detection conditions corresponding to the task execution phase and the task execution scenario, as well as a plurality of target objects and corresponding location information, a target detection result is obtained, including:

[0076] Based on whether the certificate is recognized, a certificate recognition result is obtained, and the certificate recognition result includes successful certificate recognition and unrecognized certificate; if all target certificates are successfully recognized during the task execution phase, the target detection result is judged as certificate readiness; if all target certificates are not recognized during the task execution phase, the target detection result is judged as insufficient certificate preparation.

[0077] By identifying certificates in the video stream, the system ensures that all necessary certificates are detected during the maintenance mission. This ensures compliance with aviation safety regulations and ensures the aircraft is legal for flight. Based on whether all target certificates are successfully identified, the system generates a certificate identification result, which helps determine whether the aircraft meets pre-flight documentation requirements. The system assesses the certificate readiness to determine whether the aircraft can continue with the subsequent flight mission or whether it needs to be suspended to complete the missing certificates.

[0078] Suppose an aircraft is about to undergo a pre-flight inspection. The system uses the video stream to identify all target certificates at the inspection site, such as nationality registration certificates, airworthiness certificates, and radio station licenses. The system divides the identification results into two categories: certificates successfully identified and certificates not identified. If the system successfully identifies all target certificates during the mission execution phase, the target detection result is judged as "certificate ready", indicating that the aircraft can continue with subsequent flight missions. If the system does not identify all target certificates during the mission execution phase, the target detection result is judged as "inadequate certificate preparation", and the system will notify the maintenance personnel to check and supplement the missing certificates. In this way, the safety and compliance of the aircraft maintenance site are significantly improved, while also ensuring the airworthiness and flight safety of the aircraft.

[0079] In summary, this application can monitor the access status of the safety pin in real time, automatically detect whether the safety pin is correctly inserted or removed through sensors and image recognition technology, and immediately issue an alarm when an abnormality is found, thereby reducing human error and improving safety. Through electronic scanning and database comparison technology, the integrity and validity of the three certificates can be quickly verified. The system should be able to automatically identify certificate information and match it with the records in the database to ensure that all certificates meet the requirements. At the same time, it will automatically notify relevant personnel when an abnormality is found to ensure the legality and safety of the flight. However, the above are only examples of target objects, and the rest of the target detection at the aircraft site is within the scope of protection.

[0080] In a possible implementation, after feeding back the target detection result to the cloud platform in step 103, the method further includes:

[0081] The cloud platform sends the target detection results to the corresponding inspection instruments and user devices, and performs an abnormal reminder process.

[0082] Target detection results are sent to corresponding inspection instruments and user devices, ensuring that all relevant personnel and systems have real-time access to detection information for subsequent action. The cloud platform's central role enhances collaboration between different devices and users, ensuring the flow of information and the consistency of task execution.

[0083] Assume that during the aircraft's pre-flight inspection, step 103 has already fed back the target detection results to the cloud platform. The cloud platform then sends the target detection results, such as the status of the safety pin and landing gear, as well as the certificate identification results, to the inspection instrument and user device responsible for the inspection. If the detection results indicate an abnormality, such as the safety pin not being removed or the certificate being missing, the cloud platform triggers an abnormality reminder process, immediately notifying the maintenance personnel and management personnel through sound, light, text message, or app notification. After receiving the abnormality reminder, the maintenance personnel can immediately check and correct the problem, such as rechecking the status of the safety pin or searching for missing certificates. Management personnel can adjust their work plans based on the detection results and abnormality reminders to ensure that all issues are resolved before the aircraft takes off. In this way, the safety and efficiency of the aircraft inspection site are significantly improved, while also ensuring the aircraft's airworthiness and flight safety.

[0084] The term "cloud-edge-end" generally refers to the collaborative working model of cloud computing, edge computing, and terminal devices in data processing and storage. The following is a brief description of the three:

[0085] (1) Cloud computing: Cloud computing refers to a variety of services provided through the Internet ("the cloud"), including software, storage, databases, networks, analytics, etc. Users can access these services through cloud service providers without having to maintain hardware and software themselves.

[0086] (2) Edge computing: Edge computing moves data processing and analysis tasks from centralized data centers to the edge of the network, closer to the data source. This reduces data transmission delays and improves response speed, making it suitable for applications requiring real-time processing.

[0087] (3) Terminal devices: Terminal devices refer to devices that users directly use, such as smartphones, personal computers, sensors, etc. These devices usually have data processing capabilities, can perform some basic data processing tasks, and send data to the edge or cloud for more complex processing.

[0088] In the "cloud-edge-end" model, terminal devices are responsible for collecting data and performing preliminary processing; edge computing nodes are responsible for handling tasks that require quick response, reducing the burden on the cloud; and cloud computing centers are responsible for large-scale data processing and storage, as well as complex analysis and decision support.

[0089] Figure 2 The detailed application process of the maintenance site anomaly detection system based on cloud-edge-end collaboration is shown. Through this process, fast, efficient, and accurate automatic detection of maintenance site anomalies can be achieved, improving flight safety and maintenance efficiency. This system includes the following devices:

[0090] Terminal recorder: that is, inspection video recorder, worn by relevant management personnel and maintenance personnel, connected to a camera, and equipped with network communication capabilities.

[0091] The edge platform consists of an edge collection station and an edge intelligent algorithm platform. The edge collection station, located near the maintenance site, provides functions such as data entry and binding for terminal recorders, video upload, and collection. It transmits information to the terminal recorders via a physical interface and communicates with the cloud server via the network. The edge intelligent algorithm platform is an edge server with sufficient AI computing power. It is used to deploy anomaly detection algorithms, receive real-time images from terminal recorders within a certain range, and perform algorithmic processing. It communicates with the terminal recorders via the 4G network and with the cloud server via the network.

[0092] Cloud Management Platform: This platform provides services using powerful cloud servers for multiple management systems, including the work dispatch system and audio and video management system. This platform is responsible for the operation of multiple management systems, including the work dispatch system and audio and video management system, using cloud servers to provide services.

[0093] Based on the above devices and platforms, the application process specifically includes the following stages:

[0094] Phase 1: Inspection tasks are issued.

[0095] Step 1: The maintenance personnel arrive at the collection station, enter information such as work number, and complete the binding with the inspection instrument.

[0096] Step 2: The dispatching system in the cloud management platform sends the work order task data to the inspection meter, displaying the personal maintenance task list.

[0097] The dispatch system within the cloud management platform records relevant work order data, allowing cloud platform administrators to perform relevant management operations. When maintenance personnel arrive at the collection station to claim a recorder, they enter their work number and other information, binding it to the work order. This completes the binding between the maintenance personnel and the patrol meter, and the meter displays a list of their individual maintenance tasks.

[0098] Phase 2: Execution of inspection tasks.

[0099] Step 1: The maintenance personnel selects the task to be executed in the inspection instrument.

[0100] Step 2: After selecting the task, the inspection instrument starts recording real-time images through the head-mounted camera.

[0101] Step 3: Maintenance personnel perform maintenance inspection tasks.

[0102] Step 4: After the task is completed, the maintenance personnel terminate the task in the inspection meter, the recording ends, and the video data is temporarily archived in the inspection meter.

[0103] After selecting the task you want to perform in the recorder, you can start the maintenance inspection task. The inspection instrument will then start recording the real-time image through the camera. After the task is completed, perform the relevant operation in the recorder to terminate the task. At this time, the image recording ends and the video data is temporarily archived in the inspection instrument.

[0104] The third stage: inspection data collection.

[0105] Step 1: Upload the video data stored in the inspection instrument to the cloud management platform.

[0106] Step 2: After the upload is completed, the maintenance personnel are no longer bound to the inspection meter, and the inspection meter is returned to the collection station.

[0107] Step 3: The uploaded video data is archived in the cloud management platform and associated with the corresponding inspection task information.

[0108] Data is collected by the recorder and related equipment at an edge collection station. The video data stored in the inspection meter is uploaded to the cloud management platform through the collection station. Once the collection is complete, the maintenance personnel are unlinked from the equipment, and the recorder is returned to the collection station. The video data uploaded to the cloud management platform is archived and associated with the task information.

[0109] Phase 4: Anomaly detection and risk warning.

[0110] Step 1: The edge intelligent algorithm platform interacts with the cloud management platform to obtain information such as recorder status and work order content.

[0111] Step 2: The recorder returns the recorded image to the edge intelligent algorithm platform in real time.

[0112] Step 3: The anomaly detection algorithm module processes the image frame, inputs it into the deep learning network, and obtains the target detection result.

[0113] Step 4: Combine auxiliary judgment information to detect abnormal situations.

[0114] Step 5: When an abnormality is detected, an early warning is given to maintenance personnel through voice broadcast or other means.

[0115] The edge intelligent algorithm platform interacts with the cloud management platform to obtain relevant information about online recorders, such as recorder status and work order content, as scene information for the anomaly detection algorithm module to assist in judgment. While performing maintenance tasks, the recorder returns the recorded images to the edge intelligent algorithm platform's anomaly detection module in real time. The anomaly detection algorithm module processes the image frames in real time, inputs them into the relevant deep learning network for processing, obtains preliminary target detection results, and uses the auxiliary judgment information to detect anomalies. If the algorithm detects an anomaly in the maintenance work using the recorder, it will issue an early warning message to the recorder and provide a voice prompt.

[0116] Maintenance sites can encounter numerous abnormalities. This invention primarily addresses abnormalities in the storage and retrieval of landing gear safety pins before and after flight, as well as abnormalities in the pre-flight three-certificate inspection. It further proposes a fast, efficient, and accurate, non-manual method for automatically detecting abnormalities in maintenance sites. By integrating wearable devices (end) worn by maintenance personnel, an artificial intelligence algorithm platform (edge), and an equipment management cloud platform (cloud), multi-dimensional digital intelligent monitoring technology integrating video perception and IoT technologies is applied to the maintenance site management and monitoring system. This allows for the effective detection of abnormalities in the maintenance site without human intervention, achieving fast, efficient, and accurate optimization goals.

[0117] Figure 3 The following is a logical diagram of the anomaly detection algorithm, which includes the following steps:

[0118] Step 1: Get device information and video stream address to access real-time video frames.

[0119] The edge intelligent algorithm platform obtains the device information of the terminal recorder that is online and has started the task through the cloud management platform, obtains the RTSP stream address of the real-time video frame of the terminal recorder, and reads the real-time video frame of the recorder.

[0120] Step 2: Get task information.

[0121] Mission information includes mission execution phases and mission execution scenarios. Mission execution phases include mission start phases and mission end phases. Mission execution scenarios include pre-flight maintenance and post-flight maintenance.

[0122] By analyzing task-related information, such as work order data and timestamps, we can determine the start and end signals of the task, as well as the scenario to which the task belongs.

[0123] Step 3: Combine the video frame and task information to perform target detection and obtain the target detection result.

[0124] Step 3-1: Analyze the video frames through the deep learning network to identify and locate key targets in the video (safety pin, landing gear, three certificates).

[0125] Step 3-2-1: Determine safety pin abnormality: including the safety pin not being installed abnormally, and the safety pin not being removed abnormally.

[0126] When the mission scenario is post-flight inspection, if the installed safety pin is not detected until the end of the mission, it is considered that a safety pin is not installed anomaly has occurred.

[0127] When the mission scenario is pre-flight inspection, if the safety pin is found to be installed after the mission begins, it is considered that the safety pin has not been removed.

[0128] Specifically, the system analyzes the positions of the safety pin and landing gear detected in the video frame and determines the positional relationship between the safety pin and the landing gear target detection frame. If the intersection-over-union (IOU) of the safety pin and landing gear detection frames exceeds 20%, indicating that there is some spatial overlap between the safety pin and the landing gear, the system will detect the installed safety pin target.

[0129] If the system does not detect the installed safety pin at the end of the post-flight inspection (i.e., the IOU does not exceed 20%), the system will determine that the safety pin is not installed abnormally. If the system detects the installed safety pin after the pre-flight inspection begins (i.e., the IOU exceeds 20%), the system will determine that the safety pin has not been removed abnormally.

[0130] Using video analysis technology, the system automatically detects and determines the installation and removal of safety pins during aircraft maintenance to ensure aircraft safety. A set IOU threshold (20%) is used to determine whether the safety pin is installed. The system also applies different abnormality detection logic based on the mission scenario (pre-flight or post-flight). This automated detection reduces human error and improves the safety and efficiency of maintenance work.

[0131] Step 3-2-2: Determine if there are any abnormalities in the three certificates (Nationality Registration Certificate, Airworthiness Certificate and Radio Station License):

[0132] If the system still fails to detect the existence of the three certificates at the end of the task, it is considered a three-certificate unchecked exception.

[0133] To ensure that the aircraft's legal flight documents are properly inspected and confirmed, and to avoid flight safety issues caused by the lack of legal documents, automated video analysis and object detection technology can improve the efficiency and accuracy of inspections and reduce the possibility of human negligence.

[0134] Specifically, a deep learning framework (such as TensorFlow or PyTorch) can be used to build an object detection model that can identify safety pins, landing gear, and three certificates. Computer vision techniques such as object detection and image recognition are used to analyze video frames.

[0135] Step 4: Abnormal warning: The above three types of abnormalities will be warned on the terminal recorder through different voice alarm prompts.

[0136] The RTSP (Real-Time Streaming Protocol) protocol enables real-time transmission of video streams, ensuring that the edge intelligent algorithm platform can promptly obtain real-time video data from the scene. Once an abnormality is detected, the system can quickly send warning information to the inspection instrument through the warning information distribution interface, providing real-time warning prompts. While maintenance personnel are performing maintenance work in real time, the system can detect anomalies and provide timely warning prompts to ensure the safety of maintenance work. Integrating video perception and IoT technologies enables multi-dimensional digital intelligent monitoring, improving the comprehensiveness and accuracy of detection. Through automated detection methods, human participation is reduced, detection efficiency is improved, and the possibility of human error is reduced.

[0137] Figure 4 The following is a functional block diagram of the software architecture applicable to the embodiment of this application. The edge intelligent algorithm platform mainly obtains and returns data through the interfaces provided by the inspection instrument management service and audio and video management service of the cloud platform. The following is a specific implementation method of the example, which is described according to the flow direction of the data flow:

[0138] Step 1: Log in and authenticate on the cloud platform.

[0139] The edge intelligent algorithm platform first accesses the cloud platform's login interface and authenticates using a username and password. After a successful login, the platform returns the PHPSESSID cookie value, which the edge intelligent algorithm platform uses to access subsequent interfaces to obtain access rights.

[0140] Step 2: Get a list of online inspection instruments.

[0141] The edge intelligent algorithm platform queries the online inspection meter list through the inspection meter list acquisition interface at a fixed frequency, and updates and maintains the online inspection meter list connected to the edge intelligent algorithm platform based on the returned data.

[0142] Step 3: Upload and obtain the inspection instrument status.

[0143] After the maintenance personnel select and start the task, the inspection meter uploads the start and end status of the work to the cloud platform. For the devices in the online inspection meter list, the list of inspection meters in working state is obtained and maintained, and the actual video stream is obtained.

[0144] Step 4: Upload and obtain the inspection instrument working information.

[0145] Once the inspection meter enters operation, it begins uploading relevant information to the cloud platform. This information includes work order information, department information, maintenance personnel information, the inspection meter ID, and the RTSP address of the real-time video stream. The edge intelligent algorithm platform obtains this information to prepare for subsequent video stream processing and anomaly detection.

[0146] Step 5: Video stream processing and anomaly detection.

[0147] For inspection devices with established video stream acquisition, the edge intelligent algorithm platform creates a new thread. The acquired video frames are fed into a deep learning model for inference. Combining target detection results, operational information, and relevant algorithm configurations, it determines abnormal conditions at the maintenance site.

[0148] Step 6: Send alarm event and voice broadcast.

[0149] Based on different abnormal situations, the edge intelligent algorithm platform inputs warning information through the cloud platform's warning information distribution interface. The cloud platform sends the warning information to the inspection meter, which broadcasts the warning information in real time, alerting maintenance personnel to the abnormal situation.

[0150] Through this process, the edge intelligent algorithm platform can achieve fast, efficient and accurate automatic detection of abnormal conditions at the maintenance site, and provide early warning information to maintenance personnel in a timely manner, thereby improving the safety and efficiency of maintenance work.

[0151] The present invention applies multi-dimensional digital intelligent monitoring technology that integrates video perception and Internet of Things technology to the maintenance site management and inspection system through integrated maintenance personnel wearable devices (end), artificial intelligence algorithm platform (edge) and equipment management cloud platform (cloud). It effectively detects abnormal situations in the maintenance site in a non-human way, achieving fast, efficient and accurate optimization goals.

[0152] The edge-end-cloud joint system not only reduces retrofit costs but also meets real-time requirements, enabling fast, efficient, and accurate maintenance site anomaly detection. The application of this system can significantly improve the safety and efficiency of maintenance work and reduce human error, representing a significant innovation in maintenance site management and monitoring systems. Existing inspection instruments do not require large-scale retrofits, as real-time recording is already an integral part of the maintenance workflow. This means existing hardware resources can be utilized, reducing investment in additional equipment. By establishing an edge intelligent algorithm platform, combined with existing device-side recorders and a cloud-side management platform, a joint edge-end-cloud system is formed. This integration approach effectively utilizes existing resources and reduces system retrofit costs. Detection results can effectively supplement manual inspections, improving accuracy and efficiency while reducing the burden of manual inspections.

[0153] In summary, the embodiment of the present application provides an aircraft maintenance on-site detection method, which obtains a target video stream in real time according to a target maintenance task, wherein the target maintenance task includes a task execution stage and a task execution scene; identifies the target video stream according to the task execution stage and the task execution scene to obtain a target detection result; and feeds back the target detection result to the cloud platform. The task execution stage and the task execution scene in the video stream are analyzed to capture the key targets and key states in the maintenance process to identify any abnormal situation, thereby issuing an early warning in time to avoid potential safety risks. Through the integration of the cloud platform, remote monitoring and decision-making are supported, allowing managers to understand the on-site situation in real time and respond to any abnormal situation in a timely manner.

[0154] Based on the same technical concept, the embodiment of the present application also provides an aircraft maintenance on-site detection system, such as Figure 5 As shown, the system includes:

[0155] The target video stream acquisition module 501 is used to acquire the target video stream in real time according to the target maintenance task, wherein the target maintenance task includes a task execution stage and a task execution scenario;

[0156] The target detection module 502 is used to identify the target video stream according to the task execution stage and the task execution scenario to obtain a target detection result;

[0157] The feedback module 503 is used to feed back the target detection result to the cloud platform.

[0158] The present application also provides an electronic device corresponding to the method provided in the above embodiment. Figure 6 , which shows an electronic device provided by some embodiments of the present application. The electronic device 20 may include: a processor 200, a memory 201, a bus 202, and a communication interface 203. The processor 200, the communication interface 203, and the memory 201 are connected via the bus 202. The memory 201 stores a computer program executable on the processor 200. When the processor 200 executes the computer program, it executes the method provided by any of the aforementioned embodiments of the present application.

[0159] The memory 201 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element and at least one other network element are connected via at least one physical port (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0160] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs. The processor 200 executes the programs upon receiving execution instructions. The methods disclosed in any of the aforementioned embodiments of the present application may be applied to or implemented by the processor 200.

[0161] The processor 200 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 200 or by software instructions. The above processor 200 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 201 , and the processor 200 reads the information in the memory 201 and completes the steps of the above method in combination with its hardware.

[0162] The electronic device provided in the embodiments of the present application and the method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented by them.

[0163] The present application also provides a computer-readable storage medium corresponding to the method provided in the above embodiment. Figure 7 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method provided by any of the aforementioned embodiments is executed.

[0164] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0165] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0166] It should be noted that the above embodiments illustrate rather than limit the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbols placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0167] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0168] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made by using the contents of the present invention description and drawings under the concept of the present invention, or direct / indirect application in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. An aircraft maintenance on-site detection method, characterized in that: The method comprises: Acquire a target video stream in real time according to a target maintenance task, wherein the target maintenance task includes a task execution phase and a task execution scenario; Identify the target video stream according to the task execution stage and the task execution scenario to obtain a target detection result; The target detection result is fed back to the cloud platform.

2. The method according to claim 1, wherein The identifying the target video stream according to the task execution stage and the task execution scenario to obtain a target detection result includes: Identifying objects in the target video stream to obtain a number of target objects and corresponding position information; Based on the detection conditions corresponding to the mission execution phase and the mission execution scenario, as well as a number of target objects and corresponding location information, a target detection result is obtained; the mission execution phase includes a mission start phase and a mission end phase, and the mission execution scenario includes a pre-flight inspection and a post-flight inspection.

3. The method according to claim 2, wherein The target objects are safety pins and landing gear. Based on the detection conditions corresponding to the mission execution phase and the mission execution scenario, as well as a number of target objects and corresponding position information, a target detection result is obtained, including: Calculating an intersection-over-union ratio of detection frames of the safety pin and the landing gear according to position information of the safety pin and the landing gear; If the detection frame intersection-over-union ratio exceeds a set intersection-over-union ratio threshold, the target detection result is determined as a detection of an installed safety pin; If the mission is in post-flight inspection, and the detection frame intersection-in-union ratio does not exceed the set intersection-in-union ratio threshold at the end of the mission execution, the target detection result is determined to be an abnormality in which the safety pin is not installed; If it is during pre-flight inspection, after the mission execution starts, if the detection frame intersection-in-union ratio exceeds the set intersection-in-union ratio threshold, the target detection result is determined to be an abnormality in which the safety pin is not removed.

4. The method according to claim 2, wherein The target object is a certificate; Based on the detection conditions corresponding to the task execution stage and the task execution scenario, as well as a number of target objects and corresponding location information, a target detection result is obtained, including: Obtaining a certificate identification result based on whether the certificate is identified, wherein the certificate identification result includes whether the certificate is successfully identified or not identified; If all target certificates are successfully identified during the task execution phase, the target detection result is determined to be certificate ready; If not all target certificates are identified during the task execution phase, the target detection result is determined to be insufficient certificate preparation.

5. The method according to any one of claims 1 to 4, characterized in that After feeding back the target detection result to the cloud platform, the method further includes: The cloud platform sends the target detection results to the corresponding inspection instruments and user devices, and performs an abnormal reminder process.

6. The method according to claim 1, wherein The real-time acquisition of a target video stream according to a target maintenance task includes: Obtain the target maintenance task sent by the cloud platform, wherein the target maintenance task also includes relevant information and a video stream address of a patrol instrument, wherein the relevant information includes work order information, department information, maintenance personnel information, and patrol instrument identification information; The target video stream is acquired in real time based on the inspection instrument video stream address.

7. The method according to claim 1, wherein Before acquiring the target video stream in real time according to the target maintenance task, the method further includes: The cloud platform verifies the interface access rights based on the login information of the user's device, and executes the subsequent steps if the verification is successful; Receive signals from inspection instruments, record the start and end status of inspection tasks, and obtain and maintain a list of online inspection instruments at a fixed frequency; The video data uploaded by the inspection instrument is archived and associated with the inspection task information.

8. An aircraft maintenance on-site detection system, characterized in that: The system comprises: A target video stream acquisition module is used to acquire a target video stream in real time according to a target maintenance task, wherein the target maintenance task includes a task execution phase and a task execution scenario; A target detection module is used to identify the target video stream according to the task execution stage and task execution scenario to obtain a target detection result; The feedback module is used to feed back the target detection result to the cloud platform.

9. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement the method according to any one of claims 1 to 7.

Images