Aircraft error-prevention maintenance far-end support system based on AR intelligent glasses

Abstract

The invention discloses an aircraft error prevention maintenance far-end support system based on AR intelligent glasses, and relates to the technical field of aircraft operation and maintenance management. According to the aircraft error-prevention maintenance far-end support system based on the AR intelligent glasses, a virtual model is constructed according to real aircraft part information, aircraft part images are acquired by using an AR glasses terminal in a field environment, and related feature extraction is performed on different aircraft part images in cooperation with an AI image recognition technology, so that the error-prevention maintenance far-end support system based on the AR intelligent glasses is obtained. According to the method and the system, the intelligent identification effect of the aircraft parts is achieved, meanwhile, intelligent judgment of possible abnormal parts of the aircraft parts can be achieved through comparison of the position relation between related characteristics, the error prevention advantage is provided for aircraft safety investigation, communication interconnection between a plurality of AR intelligent glasses terminals is achieved through 5G communication, and the communication efficiency is improved. Distance interference is effectively avoided, and convenient conditions are provided for technical experts to remotely guide field staff to operate, maintain and overhaul airplanes in a real-time image transmission mode.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft operation and maintenance management, and in particular to an aircraft error-proofing maintenance remote support system based on AR smart glasses. Background Art

[0002] Usually, a safety hazard check is carried out before the plane takes off. However, the conventional checking method is usually carried out manually, which may result in missed detections and false detections. When problems occur, there is also the problem of poor information channels, which makes it impossible to solve the problems quickly and effectively.

[0003] Based on the retrieval of the above information, an aircraft error-proofing maintenance remote support system based on AR smart glasses is proposed. A digital model of aircraft parts is constructed, and AR glasses are used as a visual interactive terminal to ensure smooth communication while realizing intelligent safety inspection of aircraft parts. It has the advantage of error-proofing and effectively avoids the interference of distance on problem-solving efficiency. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides an aircraft error-proof maintenance remote support system based on AR smart glasses, which solves the problems raised in the above-mentioned background technology.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: an aircraft error-proofing maintenance remote support system based on AR smart glasses, including an AR data center and several AR glasses terminals, wherein the AR data center and the several AR glasses terminals are connected via 5G communication, and the AR glasses terminals are used to provide a human-computer interaction interface and collect image information of aircraft parts, and the AR data center includes:

[0006] An AI intelligent image library, which is used to establish corresponding virtual models based on real aircraft parts;

[0007] An image recognition module, wherein the image recognition module is used to perform image recognition on the collected aircraft parts images using AI image recognition technology to obtain relevant feature information in the aircraft parts images;

[0008] A 5G communication module, wherein the 5G communication module is used to establish a 5G communication channel between a plurality of AR glasses terminals and an AR data center;

[0009] A fault identification module, which is used to locate the virtual model of the corresponding aircraft component according to the image recognition result, and to determine whether the aircraft component has a fault;

[0010] A work order triggering module is used to generate maintenance according to the aircraft component maintenance work order.

[0011] The present invention is further configured as follows: the AI ​​intelligent image library includes a model building unit and a feature mapping unit;

[0012] The model building unit is used to build a virtual model according to the aircraft parts information. During the process, the same parts are segmented according to different materials, and after the segmented virtual models are established, they are assembled to obtain the virtual model corresponding to the final parts;

[0013] The feature mapping unit is used to establish a mapping relationship between the corresponding virtual model and related features according to the component feature information.

[0014] The present invention is further configured as follows: the fault identification module includes a similarity comparison module, an abnormality location unit and a fault judgment unit, the similarity comparison module is connected to the abnormality location unit, and the abnormality location unit is connected to the fault judgment unit.

[0015] The present invention is further configured as follows: the similarity comparison module is used to locate the corresponding virtual model according to the mapping relationship based on the relevant feature information in the image recognition result;

[0016] According to the specific position information of the relevant feature information in the aircraft component image, the angle of the virtual model is adjusted by using the matrix transformation technology to obtain a virtual model that fits the angle of the aircraft component image, and the position information of the relevant features in the virtual model at this time is compared with the position information of the relevant features in the aircraft component image for coincidence;

[0017] The abnormality positioning unit is used to mark the relevant features corresponding to the incompletely overlapped parts in the aircraft component image when the overlap comparison result shows that the parts are not completely overlapped, and record them as abnormal features;

[0018] The fault judgment unit is used to display the relevant features corresponding to the abnormal features in the normal state on the same screen when abnormal features appear, so that the wearer of the AR glasses terminal can perform fault analysis.

[0019] The present invention is further configured as follows: the work order trigger module includes a work order recording unit, a scene trigger unit and a maintenance identification unit, the work order recording unit is connected to the scene trigger unit, and the scene trigger unit is connected to the maintenance identification unit.

[0020] The present invention is further configured as follows: the work order recording unit is used to record the maintenance log of aircraft parts;

[0021] The scene trigger unit is used to trigger the maintenance inspection instruction and start retrieving the corresponding aircraft component maintenance log when the aircraft component image information collected by the AR glasses terminal is identified as an aircraft component recorded in the maintenance log;

[0022] The maintenance identification unit is used to obtain maintenance location information from the aircraft component maintenance log and to identify and display it in the aircraft component image information.

[0023] The present invention is further configured such that: when an abnormal feature appears at a sign displaying part during the sign displaying process, the maintenance sign unit issues an alarm for an unresolved fault.

[0024] The present invention is further configured as follows: the AR glasses terminal includes a terminal worn by maintenance personnel and a terminal worn by experts. When an unresolved fault alarm occurs, the AR data center establishes a 5G communication channel between the terminal worn by the expert and the terminal worn by the maintenance personnel.

[0025] The present invention provides an aircraft error-proof maintenance remote support system based on AR smart glasses. It has the following beneficial effects:

[0026] The present invention realizes the digital transformation of aircraft parts by constructing virtual models based on real aircraft parts information, uses AR glasses terminals to collect images of aircraft parts in a field environment, and cooperates with AI image recognition technology to extract relevant features of different aircraft parts images, thereby achieving the intelligent recognition effect of aircraft parts. At the same time, by comparing the positional relationship between relevant features, intelligent judgment of possible abnormal parts of aircraft parts can be realized, which provides an error-proof advantage for aircraft safety inspections. 5G communication is used to realize communication and interconnection between several AR smart glasses terminals, effectively avoiding distance interference, and real-time image transmission is used to provide convenient conditions for technical experts to remotely guide on-site staff in aircraft operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the system architecture of the present invention;

[0028] Figure 2 A schematic diagram of the system architecture of the AR data center of the present invention;

[0029] Figure 3 A schematic diagram of the system architecture of the AI ​​intelligent image library of the present invention;

[0030] Figure 4 A schematic diagram of the system architecture of a fault identification module of the present invention;

[0031] Figure 5 Schematic diagram of the system architecture of the work order trigger module of the present invention.

[0032] In the figure:

[0033] 1. AR data center;

[0034] 2. AR glasses terminal;

[0035] 3. AI intelligent image library; 301. Model building unit; 302. Feature mapping unit;

[0036] 4. Image recognition module;

[0037] 5. 5G communication module;

[0038] 6. Fault identification module; 601. Similarity comparison module; 602. Abnormal location unit; 603. Fault judgment unit;

[0039] 7. Work order trigger module; 701. Work order recording unit; 702. Scenario trigger unit; 703. Maintenance identification unit. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0041] See also Figure 1-5 , the embodiment of the present invention provides the following technical solutions:

[0042] Embodiment 1: An aircraft error-proof maintenance remote support system based on AR smart glasses includes an AR data center 1 and several AR glasses terminals 2. The AR data center 1 and the several AR glasses terminals 2 are connected via 5G communication. The AR glasses terminals 2 include terminals worn by maintenance personnel and terminals worn by experts. The AR glasses terminals 2 are used to provide a human-computer interaction interface and collect image information of aircraft parts. The terminals worn by maintenance personnel and terminals worn by experts can communicate via 5G through the AR data center 1.

[0043] As a preferred solution, the AR data center 1 is composed of an AI intelligent image library 3, an image recognition module 4, a 5G communication module 5 and a fault identification module 6. In order to realize the digital transformation of aircraft operation and maintenance detection, the AI ​​intelligent image library 3 includes a model building unit 301 and a feature mapping unit 302. The model building unit 301 is used to build a virtual model according to the aircraft parts information. During the process, the same part is segmented according to different materials. After the segmented virtual model is established, it is assembled to obtain the virtual model corresponding to the final part. The feature mapping unit 302 is used to establish a mapping relationship between the corresponding virtual model and the relevant features according to the part feature information. In this way, the judgment of the virtual model corresponding to the real aircraft part can be realized according to the judgment of the relevant features.

[0044] As a preferred solution, when the AR glasses terminal 2 is at the operation and maintenance site to collect image information of aircraft parts, the image information it obtains is recognized by the image recognition module 4 using AI image recognition technology to obtain relevant feature information in the image of the aircraft parts, wherein the relevant feature information includes but is not limited to size, color, shape and typical attachment position.

[0045] As a preferred solution, in order to ensure real-time and efficient transmission of image information between the AR data center 1 and several AR glasses terminals 2, a 5G communication channel between several AR glasses terminals 2 and the AR data center 1 is established using a 5G communication module 5.

[0046] As a preferred solution, in order to realize intelligent judgment on whether the aircraft parts are normal after the on-site maintenance personnel collect the aircraft parts information through the AR glasses terminal 2, the fault identification module 6 includes a similarity comparison module 601, an abnormality positioning unit 602 and a fault judgment unit 603. The similarity comparison module 601 is used to locate the corresponding virtual model according to the mapping relationship based on the relevant feature information in the image recognition result, and also adjust the angle of the virtual model according to the matrix transformation technology based on the specific position information of the relevant feature information in the aircraft parts image, so as to obtain a virtual model that fits the angle of the aircraft parts image, and compare the position information of the relevant features in the virtual model with the position information of the relevant features in the aircraft parts image for overlap.

[0047] As a detailed description, the method of obtaining a virtual model that fits the image angle of the aircraft parts includes:

[0048] In the process of adjusting the angle of the virtual model using matrix transformation technology, relevant features are used as reference. When the overlap between the relevant features in the virtual model after the angle adjustment and the relevant features in the aircraft component image exceeds 50%, it can be determined that the angle adjustment of the virtual model is completed. This is used to avoid the situation where there are several relevant features on an aircraft component and manual inspection and judgment are solely relied upon, which is prone to missed detection and misjudgment. It can be imagined that the more relevant features there are, the greater the workload of manual judgment and the easier it is to make mistakes. Conversely, the fewer relevant features there are, the more accurate the manual judgment and the less likely it is to make mistakes.

[0049] As a preferred solution, in order to achieve guided comparison of abnormal features, the similarity comparison module 601 is connected to the abnormal positioning unit 602. The abnormal positioning unit 602 is used to mark the relevant features corresponding to the incompletely overlapping parts in the aircraft parts image when the overlap comparison results show that there is incomplete overlap, and record them as abnormal features. The abnormal positioning unit 602 is connected to the fault judgment unit 603. The fault judgment unit 603 is used to display the relevant features of the corresponding abnormal features in the normal state on the same screen when abnormal features appear, so that the wearer of the AR glasses terminal 2 can perform fault analysis.

[0050] It is further explained that when performing fault analysis, relevant image information can be simultaneously displayed in the expert wearable terminal and the maintenance personnel wearable terminal, which facilitates the expert wearable terminal to remotely provide guidance to the on-site maintenance personnel wearable terminal work.

[0051] In this embodiment, not only can intelligent and efficient error prevention of aircraft parts be achieved, but also convenient conditions are provided for the training and instruction of on-site maintenance personnel, and it has good scalability.

[0052] Embodiment 2. This embodiment is an improvement of the previous embodiment. The aircraft error-proof maintenance remote support system based on AR smart glasses also includes a work order trigger module 7 composed of a work order recording unit 701, a scene trigger unit 702 and a maintenance identification unit 703, wherein the work order recording unit 701 is used to record the maintenance log of aircraft parts, and the maintenance log includes but is not limited to the specific aircraft parts to be repaired, the specific maintenance method and whether the maintenance is completed.

[0053] It should be noted that after the inspection and maintenance of aircraft parts is completed, the virtual model corresponding to the inspected parts needs to be updated.

[0054] As a preferred solution, in order to provide detailed practical teaching to on-site maintenance personnel, the work order recording unit 701 is connected to the scene triggering unit 702. When the aircraft parts image information collected by the AR glasses terminal 2 is identified as the aircraft parts recorded in the maintenance log, the scene triggering unit 702 triggers the maintenance detection instruction and starts to retrieve the corresponding aircraft parts maintenance log. The scene triggering unit 702 is connected to the maintenance identification unit 703. The maintenance identification unit 703 obtains the maintenance part information from the aircraft parts maintenance log and displays it in the aircraft parts image information. In this way, the memory can be deepened and the learning efficiency can be improved through the observation of the actual objects on site.

[0055] As a preferred solution, in order to ensure the safe operation and use of aircraft parts after maintenance, when the maintenance identification unit 703 displays the identification part in the AR glasses terminal 2, the abnormal feature judgment is performed according to the abnormal feature judgment method proposed in Example 1. When abnormal features appear in the identification display part, an unresolved fault alarm is issued. At this time, the 5G communication module 5 establishes a 5G communication channel between the terminal worn by the expert and the terminal worn by the maintenance personnel, so that on-site maintenance personnel can communicate with the experts online, provide timely guidance to the on-site maintenance personnel, and ensure the efficiency of resolving abnormal features.

[0056] The advantage of the second embodiment over the first embodiment is that it can not only realize the secondary inspection of whether the aircraft parts can operate normally after maintenance, but also provide timely and effective guidance for the work of on-site maintenance learners, effectively reducing the errors caused by human factors.

[0057] The method of using the above-mentioned aircraft error-proofing maintenance remote support system based on AR smart glasses includes:

[0058] S1. According to the aircraft parts information, the model building unit 301 is used to build a virtual model. During the process, the same parts are segmented according to different materials. After the segmented virtual models are established, they are assembled to obtain the virtual models corresponding to the final parts. The post-feature mapping unit 302 establishes a mapping relationship between the corresponding virtual model and the relevant features according to the part feature information;

[0059] S2, the AR glasses terminal 2 corresponding to the on-site end collects the image information of the aircraft parts and transmits it to the image recognition module 4 in real time through the 5G communication module 5. The image recognition module 4 uses AI image recognition technology to perform image recognition on the collected aircraft parts images and obtains relevant feature information in the aircraft parts images. The similarity comparison module 601 locates the corresponding virtual model according to the mapping relationship based on the relevant feature information;

[0060] S3, the similarity comparison module 601 adjusts the angle of the virtual model using matrix transformation technology according to the specific position information of the relevant feature information in the aircraft component image. After the overlap between the relevant features in the virtual model after the angle adjustment and the relevant features in the aircraft component image exceeds 50%, a virtual model that fits the angle of the aircraft component image is obtained, and the position information of the relevant features in the virtual model at this time is compared with the position information of the relevant features in the aircraft component image for overlap. If the overlap comparison result shows that the virtual model is not completely overlapped, the process goes to S4. Otherwise, if the virtual model is completely overlapped, the process goes to S6.

[0061] S4. The abnormality positioning unit 602 marks the relevant features corresponding to the incompletely overlapped parts in the aircraft component image and records them as abnormal features. The fault judgment unit 603 transmits the relevant feature information of the corresponding abnormal feature in the normal state and the current abnormal feature information to the AR glasses terminal 2 through the 5G communication module 5 for display on the same screen. At this time, the 5G communication module 5 also establishes a 5G communication channel between the expert wearing terminal and the maintenance personnel wearing terminal for the wearer of the AR glasses terminal 2 to perform fault analysis;

[0062] S5, the work order recording unit 701 records the maintenance log of the aircraft parts. When the aircraft parts image information collected by the AR glasses terminal 2 is identified as the aircraft parts recorded in the maintenance log, the scene triggering unit 702 triggers the maintenance detection instruction and starts to retrieve the corresponding aircraft parts maintenance log. The maintenance identification unit 703 obtains the maintenance part information from the aircraft parts maintenance log, and identifies it in the aircraft parts image information, and displays it in the AR glasses terminal 2. The operation of S3 is repeated. The difference is that when an abnormal feature appears in the marked display part, an unresolved fault alarm is issued and the process goes to S4. On the contrary, when no abnormal feature appears in the marked display part, the process goes to S6.

[0063] S6. Complete the inspection of current aircraft parts.

[0064] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0065] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An aircraft error-proofing maintenance remote support system based on AR smart glasses, comprising an AR data center (1) and a plurality of AR glasses terminals (2), wherein the AR data center (1) and the plurality of AR glasses terminals (2) are connected via 5G communication, and the AR glasses terminals (2) are used to provide a human-computer interaction interface and collect image information of aircraft parts, and are characterized in that: The AR data center (1) comprises: An AI intelligent image library (3), wherein the AI ​​intelligent image library (3) is used to establish a corresponding virtual model based on real aircraft parts; An image recognition module (4), the image recognition module (4) being used to perform image recognition on the collected aircraft component images using AI image recognition technology to obtain relevant feature information in the aircraft component images; A 5G communication module (5), wherein the 5G communication module (5) is used to establish a 5G communication channel between a plurality of AR glasses terminals (2) and an AR data center (1); A fault identification module (6), the fault identification module (6) being used to locate the virtual model of the corresponding aircraft component according to the image recognition result, and to determine whether the aircraft component has a fault; A work order triggering module (7), wherein the work order triggering module (7) is used to generate a maintenance work order based on an aircraft component maintenance work order.

2. The aircraft error-proofing maintenance remote support system based on AR smart glasses according to claim 1 is characterized in that: The AI ​​intelligent image library (3) comprises a model building unit (301) and a feature mapping unit (302); The model building unit (301) is used to build a virtual model according to aircraft component information. During the process, the same component is segmented according to different materials, and after the segmented virtual models are established, they are assembled to obtain a virtual model corresponding to the final component. The feature mapping unit (302) is used to establish a mapping relationship between the corresponding virtual model and related features according to the component feature information.

3. The aircraft error-proofing maintenance remote support system based on AR smart glasses according to claim 2 is characterized in that: The fault identification module (6) comprises a similarity comparison module (601), an abnormality location unit (602) and a fault judgment unit (603); the similarity comparison module (601) is connected to the abnormality location unit (602), and the abnormality location unit (602) is connected to the fault judgment unit (603).

4. The aircraft error-proofing maintenance remote support system based on AR smart glasses according to claim 3 is characterized by: The similarity comparison module (601) is used to locate the corresponding virtual model according to the mapping relationship based on the relevant feature information in the image recognition result; According to the specific position information of the relevant feature information in the aircraft component image, the angle of the virtual model is adjusted by using the matrix transformation technology to obtain a virtual model that fits the angle of the aircraft component image, and the position information of the relevant features in the virtual model at this time is compared with the position information of the relevant features in the aircraft component image for coincidence; The abnormality positioning unit (602) is used to mark the relevant features corresponding to the incompletely overlapped parts in the aircraft component image and record them as abnormal features when the overlap comparison result shows that the parts are not completely overlapped. The fault judgment unit (603) is used to display the related features corresponding to the abnormal features in the normal state on the same screen when an abnormal feature appears, so that the wearer of the AR glasses terminal (2) can perform fault analysis.

5. The aircraft error-proofing maintenance remote support system based on AR smart glasses according to claim 4 is characterized in that: The work order trigger module (7) comprises a work order recording unit (701), a scene triggering unit (702) and a maintenance identification unit (703); the work order recording unit (701) is connected to the scene triggering unit (702), and the scene triggering unit (702) is connected to the maintenance identification unit (703).

6. The aircraft error-proofing maintenance remote support system based on AR smart glasses according to claim 5 is characterized in that: The work order recording unit (701) is used to record the maintenance log of aircraft parts; The scene trigger unit (702) is used to trigger a maintenance inspection instruction and start retrieving the corresponding aircraft component maintenance log when the aircraft component image information collected by the AR glasses terminal (2) is identified as an aircraft component recorded in the maintenance log; The maintenance identification unit (703) is used to obtain maintenance location information from the aircraft component maintenance log, and to identify and display the information in the aircraft component image information.

7. The aircraft error-proofing maintenance remote support system based on AR smart glasses according to claim 6 is characterized by: During the sign display process, when an abnormal feature appears at the sign display part, the maintenance sign unit (703) issues an alarm indicating that the fault has not been resolved.

8. The aircraft error-proofing maintenance remote support system based on AR smart glasses according to claim 7 is characterized in that: The AR glasses terminal (2) includes a terminal worn by maintenance personnel and a terminal worn by experts. When an unresolved fault alarm occurs, the AR data center (1) establishes a 5G communication channel between the terminal worn by the expert and the terminal worn by the maintenance personnel.

Images