A visual aircraft maintenance method, system, electronic equipment and medium

Abstract

The present invention belongs to the field of aircraft maintenance technology, and its purpose is to provide a visual aircraft maintenance method, system, electronic equipment and medium. Based on the present invention, maintenance personnel can directly view the precise correspondence between the three-dimensional model of aviation equipment and the actual equipment structure in the augmented reality device, and complete each step of the maintenance operation in sequence through the maintenance guidance information in the final three-dimensional model, thereby reducing the maintenance personnel's dependence on the maintenance manual, while avoiding the problem of maintenance operation errors, which is conducive to greatly improving the intuitiveness and efficiency of maintenance; secondly, the present invention can automatically update the final three-dimensional model based on the dynamic update of the real-time scene, thereby generating detailed dynamic maintenance step instructions for maintenance personnel, providing real-time guidance to maintenance personnel during the maintenance process, and reducing the risk of operational errors during the maintenance process.

Description

Technical Field

[0001] The present invention belongs to the technical field of aircraft maintenance, and in particular relates to a visualized aircraft maintenance method, system, electronic equipment and medium. Background Art

[0002] With the rapid development of civil aviation technology, the complexity and precision of aircraft are increasing year by year, and the repair and maintenance of aviation equipment is becoming more and more complicated and delicate. Traditional aviation equipment maintenance work mainly relies on maintenance manuals, two-dimensional drawings and the experience of maintenance personnel. Maintenance personnel usually need to carry a large number of paper or digital maintenance manuals with them and operate manually according to the instructions in the maintenance manuals.

[0003] However, in the process of using the prior art, the inventors found that the prior art has at least the following problems:

[0004] In the prior art, the maintenance manual of aviation equipment usually exists in the form of static documents, and maintenance personnel need to manually flip through the maintenance manual to find the corresponding step information. This method makes maintenance time-consuming and is prone to maintenance errors due to misunderstandings or improper operations by maintenance personnel, resulting in low maintenance efficiency. In addition, since aviation equipment has many key components and its structure is usually complex, fault analysis and prediction methods are still mainly based on the experience of maintenance personnel, making it difficult for maintenance personnel to accurately locate the source of equipment failure in a short period of time, increasing the complexity and uncertainty of maintenance operations, resulting in extended equipment downtime and further reduced maintenance efficiency.

[0005] At present, in the field of civil aviation equipment maintenance, the application of computer-aided design (CAD) technology is becoming increasingly popular. Maintenance personnel can view the structure and layout of parts through two-dimensional or three-dimensional equipment models to assist maintenance operations. However, the display of equipment three-dimensional models still relies on traditional computer screens and cannot be directly compared with actual equipment. Maintenance personnel still need to constantly switch perspectives between the screen and the actual equipment during the maintenance process. Especially when facing large-scale aviation equipment with complex structures and many parts, traditional display methods are difficult to effectively meet the visualization needs of equipment details and overall layout during maintenance operations, resulting in limited improvement in maintenance efficiency.

[0006] Therefore, it is necessary to study a visual aircraft maintenance method, system, electronic equipment and medium that can effectively improve maintenance efficiency. Summary of the invention

[0007] The present invention aims to solve the above technical problems at least to a certain extent, and provides a visual aircraft maintenance method, system, electronic equipment and medium.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a visual aircraft maintenance method, comprising:

[0010] Acquire aircraft image information of a designated aircraft, and obtain an aircraft three-dimensional model of the designated aircraft according to the aircraft image information;

[0011] Obtaining a maintenance manual for the designated aircraft, and constructing a maintenance instruction database for the designated aircraft based on the maintenance manual;

[0012] Based on a preset augmented reality device, real-time scene information of the designated aircraft is collected, and an initial three-dimensional model matching the real-time scene information is extracted from the three-dimensional model of the aircraft;

[0013] Acquire maintenance task information corresponding to the real-time scene information, and extract maintenance instruction information matching the real-time scene information from the maintenance instruction database according to the maintenance task information;

[0014] Integrating the maintenance instruction information into the initial three-dimensional model to obtain a final three-dimensional model;

[0015] The final three-dimensional model is visualized based on the augmented reality device so that maintenance personnel can perform maintenance work on the aviation equipment corresponding to the real-time scene information in the designated aircraft based on the final three-dimensional model; wherein the final three-dimensional model is displayed superimposed on the real-time scene information.

[0016] In a possible design, the aircraft image information includes device image information of each aviation device in the designated aircraft and device position information of each aviation device in the designated aircraft; correspondingly, obtaining the aircraft three-dimensional model of the designated aircraft according to the aircraft image information includes:

[0017] Generating a three-dimensional equipment model of each aviation equipment in the designated aircraft according to the equipment image information of each aviation equipment;

[0018] An aircraft three-dimensional model of the designated aircraft is generated according to the equipment three-dimensional model of each aviation equipment and the equipment position information of each aviation equipment in the designated aircraft.

[0019] In a possible design, when generating the 3D equipment model of each aviation equipment in the specified aircraft based on the equipment image information of each aviation equipment, a NURBS curve is used to generate a 3D surface conversion algorithm or a point cloud-based geometry reconstruction algorithm.

[0020] In a possible design, constructing a maintenance instruction database for the designated aircraft based on the maintenance manual includes:

[0021] Digitally processing the maintenance manual and extracting the step text information in the maintenance manual;

[0022] Performing instruction analysis on the step text information to obtain the instruction text in the step text information;

[0023] Decomposing and structuring the step text information according to the instruction text to obtain maintenance guidance information;

[0024] A maintenance instruction database for the designated aircraft is constructed based on all maintenance instruction information in the maintenance manual.

[0025] In a possible design, extracting an initial three-dimensional model matching the real-time scene information from the aircraft three-dimensional model includes:

[0026] Performing feature recognition processing on the real-time scene information to obtain feature point information in the real-time scene information;

[0027] Performing feature alignment processing on the feature point information and the three-dimensional model of the aircraft to obtain relative position information and relative angle information between the three-dimensional model of the aircraft and the real-time scene information;

[0028] According to the relative position information and the relative angle information, an initial three-dimensional model that coincides with the real-time scene information is generated based on the aircraft three-dimensional model.

[0029] In a possible design, obtaining maintenance task information corresponding to the real-time scene information includes:

[0030] Collecting real-time equipment operation data of the specified aviation equipment in the real-time scene information, and obtaining historical equipment operation data matching the specified aviation equipment;

[0031] Performing fault diagnosis processing on the designated aviation equipment according to the real-time equipment operation data and the historical equipment operation data to obtain fault source information of the designated aviation equipment;

[0032] Maintenance task information corresponding to the real-time scene information is generated according to the fault source information.

[0033] In a possible design, performing fault diagnosis processing on the designated aviation equipment according to the real-time equipment operation data and the historical equipment operation data to obtain fault source information of the designated aviation equipment includes:

[0034] Performing data mining on the historical equipment operation data to obtain historical fault information of the designated aviation equipment;

[0035] Constructing an initial fault prediction model, and training the initial fault prediction model according to the historical equipment operation data and the historical fault information to obtain a trained fault prediction model;

[0036] The real-time equipment operation data is input into the trained fault prediction model so as to obtain the fault source information of the designated aviation equipment based on the trained fault prediction model.

[0037] In a second aspect, the present invention provides a visual aircraft maintenance system, which is used to implement a visual aircraft maintenance method as described in any one of the above; the visual aircraft maintenance system includes:

[0038] A base data acquisition module, used to acquire aircraft image information of a designated aircraft, and obtain an aircraft three-dimensional model of the designated aircraft based on the aircraft image information; and also used to acquire a maintenance manual of the designated aircraft, and construct a maintenance instruction database of the designated aircraft based on the maintenance manual;

[0039] A scene information acquisition module, which is in communication with the base data acquisition module and is used to acquire real-time scene information of the designated aircraft based on a preset augmented reality device, and to extract an initial three-dimensional model matching the real-time scene information from the three-dimensional model of the aircraft;

[0040] a task information acquisition module, which is in communication with the scene information acquisition module and is used to acquire maintenance task information corresponding to the real-time scene information, and to extract maintenance instruction information matching the real-time scene information from the maintenance instruction database according to the maintenance task information;

[0041] A model visualization module is communicatively connected to the task information acquisition module, and is used to integrate the maintenance instruction information into the initial three-dimensional model to obtain a final three-dimensional model; it is also used to visualize the final three-dimensional model based on the augmented reality device, so that maintenance personnel can perform maintenance operations on the aviation equipment corresponding to the real-time scene information in the designated aircraft based on the final three-dimensional model; wherein the final three-dimensional model is displayed superimposed on the real-time scene information.

[0042] In a third aspect, the present invention provides an electronic device, comprising:

[0043] a memory for storing computer program instructions; and,

[0044] A processor is used to execute the computer program instructions to complete the operation of a visual aircraft maintenance method as described in any one of the above.

[0045] In a fourth aspect, the present invention provides a computer program product, comprising a computer program or instructions, wherein when the computer program or the instructions are executed by a computer, the computer program or the instructions implement a visual aircraft maintenance method as described in any one of the above.

[0046] The beneficial effects of the present invention are:

[0047] The present invention discloses a visualized aircraft maintenance method, system, electronic device and medium, which can effectively improve maintenance efficiency. Specifically, during the implementation of the present invention, first, aircraft image information of a designated aircraft is obtained, so as to obtain an aircraft three-dimensional model of the designated aircraft according to the aircraft image information, and the maintenance manual of the designated aircraft is obtained, so as to construct a maintenance instruction database of the designated aircraft according to the maintenance manual; then, the real-time scene information of the designated aircraft is collected based on a preset augmented reality device, and an initial three-dimensional model matching the real-time scene information is extracted from the aircraft three-dimensional model; then, maintenance task information corresponding to the real-time scene information is obtained, and maintenance instruction information matching the real-time scene information is extracted from the maintenance instruction database according to the maintenance task information, and then the maintenance instruction information is integrated into the initial three-dimensional model to obtain a final three-dimensional model; finally, the final three-dimensional model is visualized based on the augmented reality device, so that maintenance personnel can perform maintenance operations on the aviation equipment corresponding to the real-time scene information in the designated aircraft based on the final three-dimensional model. Based on this, maintenance personnel can directly view the precise correspondence between the three-dimensional model of aviation equipment and the actual equipment structure in the augmented reality device, and complete each step of the maintenance operation in sequence through the maintenance instruction information in the final three-dimensional model, thereby reducing the maintenance personnel's dependence on the maintenance manual, and at the same time avoiding the problem of maintenance operation errors, which is conducive to greatly improving the intuitiveness and efficiency of maintenance; secondly, the present invention can automatically update the final three-dimensional model based on the dynamic update of the real-time scene, thereby generating detailed dynamic maintenance step instructions for maintenance personnel, providing real-time guidance for maintenance personnel during the maintenance process, and reducing the risk of operational errors during the maintenance process.

[0048] Other beneficial effects of the present invention will be further described in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of a visual aircraft maintenance method in an embodiment;

[0050] Figure 2is a module block diagram of a visual aircraft maintenance system in an embodiment;

[0051] Figure 3 It is a module block diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0053] Embodiment 1:

[0054] This embodiment discloses a visualized aircraft maintenance method, which can be executed by, but is not limited to, a computer device or a virtual machine with certain computing resources, such as a personal computer, a smart phone, a personal digital assistant, or a wearable device, or by a virtual machine.

[0055] like Figure 1 As shown, a visual aircraft maintenance method may include but is not limited to the following steps:

[0056] S1. Acquire aircraft image information of a designated aircraft, and obtain an aircraft three-dimensional model of the designated aircraft based on the aircraft image information.

[0057] Specifically, in this embodiment, the aircraft image information includes device image information of each aviation device in the designated aircraft and device position information of each aviation device in the designated aircraft; correspondingly, in step S1, obtaining the aircraft three-dimensional model of the designated aircraft according to the aircraft image information includes:

[0058] S101. Generate a three-dimensional equipment model of each aviation equipment in the specified aircraft based on the equipment image information of each aviation equipment; specifically, in this embodiment, the equipment image information of any aviation equipment adopts a CAD (Computer-Aided Design) drawing extracted from the design document of any aviation equipment, which generally includes each component in any aviation equipment and the three-dimensional geometric information of the components.

[0059] Specifically, in step S101, when generating the equipment 3D model of each aviation equipment in the specified aircraft according to the equipment image information of each aviation equipment, a NURBS (Non-Uniform Rational B-Splines) curve is used to generate a 3D surface conversion algorithm or a point cloud-based geometric body reconstruction algorithm or other computer graphics conversion algorithms are used. It should be noted that in this embodiment, the equipment 3D model is in a 3D format such as STL (a 3D file format) or OBJ (a standard file format for representing 3D geometric shapes).

[0060] In addition, it should be noted that in this embodiment, after generating the three-dimensional equipment model of each aviation equipment in the designated aircraft, a three-dimensional model database of the designated aircraft is constructed based on the three-dimensional equipment model of each aviation equipment for subsequent call.

[0061] S102. Generate a three-dimensional aircraft model of the designated aircraft based on the three-dimensional equipment model of each aviation equipment and the equipment position information of each aviation equipment in the designated aircraft.

[0062] S2. Obtain the maintenance manual of the designated aircraft, and construct a maintenance instruction database for the designated aircraft based on the maintenance manual; specifically, in this embodiment, the maintenance manual of the designated aircraft includes maintenance information in paper or electronic maintenance manuals of multiple aviation equipment in the designated aircraft, which is not limited here.

[0063] In step S2 of this embodiment, constructing a maintenance instruction database for the designated aircraft according to the maintenance manual includes:

[0064] S201. The maintenance manual is digitized, and the step text information in the maintenance manual is extracted by using, for example, an OCR (Optical Character Recognition) technique, and the image information in the maintenance manual is extracted by using, for example, an image segmentation and recognition algorithm, so as to generate maintenance instruction information with image information later;

[0065] S202. Use Natural Language Processing (NLP) technology to perform instruction analysis on the step text information to obtain the instruction text in the step text information; specifically, in this embodiment, the instruction text, such as specified operation verbs and part nouns, is determined according to the fault conditions of each aviation equipment in the maintenance manual and is not limited here.

[0066] S203. Decompose and structure the step text information according to the instruction text to obtain maintenance guidance information; specifically, in this embodiment, the maintenance guidance information includes specific operation step guidance information such as disassembly, inspection and installation, and the maintenance guidance information is presented in the form of a logical flow chart to provide maintenance personnel with intuitive maintenance guidance.

[0067] S204. Construct a maintenance instruction database for the designated aircraft based on all maintenance instruction information in the maintenance manual.

[0068] S3. Based on a preset augmented reality device, real-time scene information of the designated aircraft is collected, and an initial three-dimensional model matching the real-time scene information is extracted from the aircraft three-dimensional model; it should be noted that, in step S3 of this embodiment, the real-time scene information is extracted from the real-time video stream collected by the augmented reality device; after the initial three-dimensional model matching the real-time scene information is extracted from the aircraft three-dimensional model, the initial three-dimensional model can also be directly visualized, which is not limited here.

[0069] It should be understood that the initial three-dimensional model is a global aircraft three-dimensional model or a local three-dimensional model in the aircraft three-dimensional model that matches the local aviation equipment, depending on the scene range corresponding to the real-time scene information shown. If the overall structure of the designated aircraft is in the real-time scene information, then the initial three-dimensional model is the aircraft three-dimensional model of the designated aircraft; if the local aviation equipment structure of the designated aircraft is in the real-time scene information, then the initial three-dimensional model is the local three-dimensional model in the designated aircraft that corresponds to the local aviation equipment structure.

[0070] Specifically, in step S3, extracting an initial three-dimensional model matching the real-time scene information from the aircraft three-dimensional model includes:

[0071] S301. Perform feature recognition processing on the real-time scene information to obtain feature point information in the real-time scene information; specifically, in this embodiment, feature recognition processing is performed on the real-time scene information using a feature point matching algorithm such as a SIFT (Scale Invariant Feature Transform) algorithm or a SURF (Speeded Up Robust Features) feature extraction algorithm, which is not limited here.

[0072] S302. Perform feature alignment processing on the feature point information and the three-dimensional model of the aircraft to obtain relative position information and relative angle information between the three-dimensional model of the aircraft and the real-time scene information;

[0073] S303. Generate an initial three-dimensional model that overlaps with the real-time scene information based on the aircraft three-dimensional model according to the relative position information and the relative angle information.

[0074] S4. Acquire maintenance task information corresponding to the real-time scene information, and extract maintenance instruction information matching the real-time scene information from the maintenance instruction database according to the maintenance task information.

[0075] It should be noted that, in one implementation of step S4 of this embodiment, the maintenance task information corresponding to the real-time scene information can be set by the maintenance personnel themselves. Specifically, the maintenance personnel can set the maintenance task information themselves according to the aviation equipment failure conditions they have known in advance.

[0076] In another implementation of step S4 of this embodiment, in order to further improve the automation level of aviation equipment maintenance and reduce the maintenance workload of maintenance personnel, obtaining maintenance task information corresponding to the real-time scene information includes:

[0077] S401. Collect real-time equipment operation data of the aviation equipment specified in the real-time scene information, and obtain historical equipment operation data matching the specified aviation equipment; specifically, in this embodiment, the real-time equipment operation data includes but is not limited to key indicator data related to the operation of the specified aviation equipment, such as temperature, pressure and current, which are collected through various sensors installed in the specified aviation equipment and are basic data for fault diagnosis; the number of specified aviation equipment in the real-time scene information is one or more, which is not limited here.

[0078] S402. Perform fault diagnosis processing on the designated aviation equipment according to the real-time equipment operation data and the historical equipment operation data to obtain fault source information of the designated aviation equipment.

[0079] Specifically, in step S402, fault diagnosis processing is performed on the designated aviation equipment according to the real-time equipment operation data and the historical equipment operation data to obtain fault source information of the designated aviation equipment, including:

[0080] S4021. Perform data mining on the historical equipment operation data to obtain historical fault information of the designated aviation equipment; specifically, in this embodiment, the historical fault information includes information such as the fault mode and fault cause corresponding to the historical fault problems of the designated aviation equipment. During the implementation process, data mining techniques such as cluster analysis and association rule mining can be used to identify historical cases similar to the real-time equipment operation data, and further confirm whether the designated aviation equipment has a fault problem.

[0081] S4022. Construct an initial fault prediction model, and train the initial fault prediction model according to the historical equipment operation data and the historical fault information to obtain a trained fault prediction model; specifically, in this embodiment, the initial fault prediction model adopts a machine learning model (such as a support vector machine or a neural network model), and performs fault prediction and diagnosis based on the trained fault prediction model, so as to prompt maintenance personnel of possible faulty components and their causes.

[0082] S4023. Input the real-time equipment operation data into the trained fault prediction model, so as to obtain the fault source information of the designated aviation equipment based on the trained fault prediction model. Specifically, in this embodiment, the fault source information includes the identification information of the faulty component in the designated aviation equipment, the fault mode and the fault cause, etc. Accordingly, after the maintenance instruction information is integrated into the initial three-dimensional model, the component three-dimensional model of the faulty component in the final three-dimensional model obtained can be set to a specified format, and when the final three-dimensional model is visualized, the component three-dimensional model of the faulty component is highlighted, so that maintenance personnel can quickly locate the fault point, thereby helping to further improve maintenance efficiency.

[0083] S403. Generate maintenance task information corresponding to the real-time scene information according to the fault source information.

[0084] With the development of the Internet of Things and sensor technology, more and more aviation equipment can collect data on its operating status through sensors. However, most existing technologies are limited to passive monitoring of data, and fail to combine real-time operating data with historical fault information of the equipment for intelligent analysis, making it difficult to achieve intelligent positioning of potential equipment faults. To this end, this embodiment integrates historical equipment operating data and real-time equipment operating data of aviation equipment based on the above steps S401-S403, and can intelligently analyze the operating status of specified aviation equipment and provide fault location prompts, thereby enabling maintenance personnel to quickly and accurately identify potential problems of equipment and perform targeted maintenance operations when faced with complex equipment failures, thereby significantly improving the efficiency of aviation equipment maintenance.

[0085] S5. Integrate the maintenance instruction information into the initial three-dimensional model to obtain a final three-dimensional model.

[0086] S6. Visually display the final three-dimensional model based on the augmented reality device so that maintenance personnel can perform maintenance operations on the aviation equipment corresponding to the real-time scene information in the designated aircraft based on the final three-dimensional model; wherein the final three-dimensional model is displayed superimposed on the real-time scene information. It should be noted that in this embodiment, the maintenance guidance information is used to provide maintenance personnel with real-time maintenance step guidance.

[0087] Specifically, in step S3 of the present embodiment, real-time scene information of the designated aircraft is collected in real time through a preset augmented reality device such as AR (Augmented Reality) glasses or a smart tablet; correspondingly, in step S6 of the present embodiment, the final three-dimensional model is visualized through the augmented reality device so that maintenance personnel can view the final three-dimensional model of the aviation equipment within their field of view in real time during the maintenance process; in the present embodiment, the augmented reality device preferably adopts AR glasses that can be worn on the head of the maintenance personnel, thereby ensuring that the final three-dimensional model is displayed in superposition with the real-time scene information within the user's field of view, thereby facilitating the maintenance personnel to quickly and accurately perform maintenance operations when a failure occurs in the aviation equipment within their field of view.

[0088] In addition, in this embodiment, based on the scene conversion of the maintenance personnel, the real-time scene information will be updated accordingly, and then the final three-dimensional model matching the updated real-time scene information will be dynamically generated, thereby realizing the dynamic update of the maintenance guidance information, thereby guiding the maintenance personnel to gradually troubleshoot and repair the faulty components.

[0089] This embodiment discloses a visual aircraft maintenance method based on augmented reality technology, and proposes a series of improvements to the technical limitations of traditional maintenance methods. The augmented reality technology is used to assist in the maintenance of civil aircraft, which can effectively improve the maintenance efficiency. Specifically, during the implementation of this embodiment, first, the aircraft image information of the designated aircraft is obtained, so as to obtain the aircraft three-dimensional model of the designated aircraft according to the aircraft image information, and the maintenance manual of the designated aircraft is obtained, so as to construct a maintenance instruction database of the designated aircraft according to the maintenance manual; then, the real-time scene information of the designated aircraft is collected based on the preset augmented reality device, and the initial three-dimensional model matching the real-time scene information is extracted from the aircraft three-dimensional model; then, the maintenance task information corresponding to the real-time scene information is obtained, and the maintenance instruction information matching the real-time scene information is extracted from the maintenance instruction database according to the maintenance task information, and then the maintenance instruction information is integrated into the initial three-dimensional model to obtain the final three-dimensional model; finally, the final three-dimensional model is visualized based on the augmented reality device, so that the maintenance personnel can perform maintenance operations on the aviation equipment corresponding to the real-time scene information in the designated aircraft based on the final three-dimensional model. Based on this, maintenance personnel can directly view the precise correspondence between the three-dimensional model of aviation equipment and the actual equipment structure in the augmented reality device, and complete each step of the maintenance operation in sequence through the maintenance instruction information in the final three-dimensional model, thereby reducing the maintenance personnel's dependence on the maintenance manual, and at the same time avoiding the problem of maintenance operation errors, which is conducive to greatly improving the intuitiveness and efficiency of maintenance; secondly, this embodiment can automatically update the final three-dimensional model based on the dynamic update of the real-time scene, thereby generating detailed dynamic maintenance step instructions for maintenance personnel, providing real-time guidance to maintenance personnel during the maintenance process, and reducing the risk of operational errors during the maintenance process.

[0090] Embodiment 2:

[0091] This embodiment discloses a visualized aircraft maintenance system for implementing the visualized aircraft maintenance method in Embodiment 1; Figure 2 As shown, the visual aircraft maintenance system includes:

[0092] A base data acquisition module, used to acquire aircraft image information of a designated aircraft, and obtain an aircraft three-dimensional model of the designated aircraft based on the aircraft image information; and also used to acquire a maintenance manual of the designated aircraft, and construct a maintenance instruction database of the designated aircraft based on the maintenance manual;

[0093] A scene information acquisition module, which is in communication with the base data acquisition module and is used to acquire real-time scene information of the designated aircraft based on a preset augmented reality device, and to extract an initial three-dimensional model matching the real-time scene information from the three-dimensional model of the aircraft;

[0094] a task information acquisition module, which is in communication with the scene information acquisition module and is used to acquire maintenance task information corresponding to the real-time scene information, and to extract maintenance instruction information matching the real-time scene information from the maintenance instruction database according to the maintenance task information;

[0095] A model visualization module is communicatively connected to the task information acquisition module, and is used to integrate the maintenance instruction information into the initial three-dimensional model to obtain a final three-dimensional model; it is also used to visualize the final three-dimensional model based on the augmented reality device, so that maintenance personnel can perform maintenance operations on the aviation equipment corresponding to the real-time scene information in the designated aircraft based on the final three-dimensional model; wherein the final three-dimensional model is displayed superimposed on the real-time scene information.

[0096] It should be noted that the working process, working details and technical effects of the visual aircraft maintenance system provided in this embodiment 2 can be found in embodiment 1 and will not be described in detail here.

[0097] Embodiment 3:

[0098] Based on Embodiment 1 or 2, this embodiment discloses an electronic device, which may be a smart phone, a tablet computer, a laptop computer, or a desktop computer. The electronic device may be referred to as a user terminal, a portable terminal, a desktop terminal, etc. Figure 3 As shown, the electronic equipment includes:

[0099] a memory for storing computer program instructions; and,

[0100] A processor is used to execute the computer program instructions to complete the operation of a visual aircraft maintenance method as described in any one of Embodiment 1.

[0101] Specifically, the processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen.

[0102] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 301 to implement the visual aircraft maintenance method provided in Example 1 of the present application.

[0103] In some embodiments, the terminal may further optionally include: a communication interface 303 and at least one peripheral device. The processor 301, the memory 302 and the communication interface 303 may be connected via a bus or a signal line. Each peripheral device may be connected to the communication interface 303 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 304, a display screen 305 and a power supply 306.

[0104] The communication interface 303 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 301 and the memory 302. In some embodiments, the processor 301, the memory 302, and the communication interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302, and the communication interface 303 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0105] The radio frequency circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 304 communicates with a communication network and other communication devices through electromagnetic signals.

[0106] The display screen 305 is used to display a UI (User Interface). The UI may include any combination of graphics, text, icons, and videos.

[0107] The power supply 306 is used to supply power to various components in the electronic device.

[0108] Embodiment 4:

[0109] Based on any one of Embodiments 1 to 3, this embodiment discloses a computer program product, including a computer program or instructions, which, when executed by a computer, implements a visualized aircraft maintenance method as described in any one of Embodiments 1. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0110] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual aircraft maintenance method, characterized in that: include: Acquire aircraft image information of a designated aircraft, and obtain an aircraft three-dimensional model of the designated aircraft according to the aircraft image information; Obtaining a maintenance manual for the designated aircraft, and constructing a maintenance instruction database for the designated aircraft based on the maintenance manual; Based on a preset augmented reality device, real-time scene information of the designated aircraft is collected, and an initial three-dimensional model matching the real-time scene information is extracted from the three-dimensional model of the aircraft; Acquire maintenance task information corresponding to the real-time scene information, and extract maintenance instruction information matching the real-time scene information from the maintenance instruction database according to the maintenance task information; Integrating the maintenance instruction information into the initial three-dimensional model to obtain a final three-dimensional model; Visually displaying the final three-dimensional model based on the augmented reality device so that maintenance personnel can perform maintenance operations on the aviation equipment corresponding to the real-time scene information in the designated aircraft based on the final three-dimensional model; wherein the final three-dimensional model is displayed in a superimposed manner with the real-time scene information; Extracting an initial three-dimensional model matching the real-time scene information from the aircraft three-dimensional model includes: Performing feature recognition processing on the real-time scene information to obtain feature point information in the real-time scene information; Performing feature alignment processing on the feature point information and the three-dimensional model of the aircraft to obtain relative position information and relative angle information between the three-dimensional model of the aircraft and the real-time scene information; According to the relative position information and the relative angle information, an initial three-dimensional model that coincides with the real-time scene information is generated based on the aircraft three-dimensional model.

2. A visual aircraft maintenance method according to claim 1, characterized in that: The aircraft image information includes device image information of each aviation device in the designated aircraft and device position information of each aviation device in the designated aircraft; correspondingly, obtaining the aircraft three-dimensional model of the designated aircraft according to the aircraft image information includes: Generating a three-dimensional equipment model of each aviation equipment in the designated aircraft according to the equipment image information of each aviation equipment; An aircraft three-dimensional model of the designated aircraft is generated according to the equipment three-dimensional model of each aviation equipment and the equipment position information of each aviation equipment in the designated aircraft.

3. A visual aircraft maintenance method according to claim 2, characterized in that: When generating the three-dimensional equipment model of each aviation equipment in the specified aircraft according to the equipment image information of each aviation equipment, a NURBS curve is used to generate a three-dimensional surface conversion algorithm or a point cloud-based geometric reconstruction algorithm is used.

4. A visual aircraft maintenance method according to claim 1, characterized in that: According to the maintenance manual, a maintenance instruction database for the designated aircraft is constructed, including: Digitally processing the maintenance manual and extracting the step text information in the maintenance manual; Performing instruction analysis on the step text information to obtain the instruction text in the step text information; Decomposing and structuring the step text information according to the instruction text to obtain maintenance guidance information; A maintenance instruction database for the designated aircraft is constructed based on all maintenance instruction information in the maintenance manual.

5. A visual aircraft maintenance method according to claim 1, characterized in that: Acquiring maintenance task information corresponding to the real-time scene information, including: Collecting real-time equipment operation data of the specified aviation equipment in the real-time scene information, and obtaining historical equipment operation data matching the specified aviation equipment; Performing fault diagnosis processing on the designated aviation equipment according to the real-time equipment operation data and the historical equipment operation data to obtain fault source information of the designated aviation equipment; Maintenance task information corresponding to the real-time scene information is generated according to the fault source information.

6. A visual aircraft maintenance method according to claim 5, characterized in that: According to the real-time equipment operation data and the historical equipment operation data, a fault diagnosis process is performed on the designated aviation equipment to obtain fault source information of the designated aviation equipment, including: Performing data mining on the historical equipment operation data to obtain historical fault information of the designated aviation equipment; Constructing an initial fault prediction model, and training the initial fault prediction model according to the historical equipment operation data and the historical fault information to obtain a trained fault prediction model; The real-time equipment operation data is input into the trained fault prediction model so as to obtain the fault source information of the designated aviation equipment based on the trained fault prediction model.

7. A visual aircraft maintenance system, characterized in that: Used to implement a visualized aircraft maintenance method according to any one of claims 1 to 6; the visualized aircraft maintenance system comprises: A base data acquisition module, used to acquire aircraft image information of a designated aircraft, and obtain an aircraft three-dimensional model of the designated aircraft based on the aircraft image information; and also used to acquire a maintenance manual of the designated aircraft, and construct a maintenance instruction database of the designated aircraft based on the maintenance manual; A scene information acquisition module, which is in communication with the base data acquisition module and is used to acquire real-time scene information of the designated aircraft based on a preset augmented reality device, and to extract an initial three-dimensional model matching the real-time scene information from the three-dimensional model of the aircraft; a task information acquisition module, which is in communication with the scene information acquisition module and is used to acquire maintenance task information corresponding to the real-time scene information, and to extract maintenance instruction information matching the real-time scene information from the maintenance instruction database according to the maintenance task information; a model visualization module, which is in communication connection with the task information acquisition module, and is used to integrate the maintenance instruction information into the initial three-dimensional model to obtain a final three-dimensional model; and is also used to visualize the final three-dimensional model based on the augmented reality device, so that maintenance personnel can perform maintenance operations on the aviation equipment corresponding to the real-time scene information in the designated aircraft based on the final three-dimensional model; wherein the final three-dimensional model is displayed in a superimposed manner with the real-time scene information; Extracting an initial three-dimensional model matching the real-time scene information from the aircraft three-dimensional model includes: Performing feature recognition processing on the real-time scene information to obtain feature point information in the real-time scene information; Performing feature alignment processing on the feature point information and the three-dimensional model of the aircraft to obtain relative position information and relative angle information between the three-dimensional model of the aircraft and the real-time scene information; According to the relative position information and the relative angle information, an initial three-dimensional model that coincides with the real-time scene information is generated based on the aircraft three-dimensional model.

8. An electronic device, characterized in that: include: a memory for storing computer program instructions; as well as, A processor is used to execute the computer program instructions to complete the operation of the visual aircraft maintenance method according to any one of claims 1 to 6.

9. A computer program product comprising a computer program or instructions, characterized in that When the computer program or the instructions are executed by a computer, the visual aircraft maintenance method according to any one of claims 1 to 6 is implemented.

Images