An aircraft maintenance inspection assistance method and system based on digital twin

Abstract

The present invention relates to an aircraft maintenance assistance method and system based on digital twin. The method includes: receiving a maintenance trigger operation from a user for an aircraft to be inspected; in response to the maintenance trigger operation being a maintenance trigger operation for the power system of the aircraft to be inspected, constructing a first voxel model corresponding to the power system in a digital modeling manner to assist in maintenance; wherein, the first voxel model has different colors and transparencies at different positions; the power system includes at least one of an engine and an auxiliary power unit; in response to the maintenance trigger operation being a maintenance trigger operation for the control system of the aircraft to be inspected, constructing a second voxel model corresponding to the control system in a digital modeling manner to assist in maintenance; wherein, the second voxel model has different colors and transparencies at different positions; the control system includes at least one of a mechanical control system, a fly-by-wire control system, and a lift augmentation device.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft maintenance, and in particular to an aircraft maintenance assistance method and system based on digital twin. Background Art

[0002] Aircraft maintenance is a crucial part of aircraft maintenance. Aircraft maintenance mainly refers to the regular or irregular inspection, repair, and maintenance of aircraft and its related equipment, aiming to ensure that the safety, reliability, and performance of the aircraft meet the relevant standard requirements. The quality of aircraft maintenance is directly related to the safety performance and operation efficiency of the aircraft.

[0003] However, traditional maintenance methods rely on manual experience and have problems such as heavy workload, low efficiency, and strong subjectivity, making it difficult to meet the requirements of modern aircraft complexity and maintenance refinement. Therefore, it is particularly important to introduce intelligent and information-based assistance methods and systems.

[0004] Research has found that existing maintenance assistance methods are mostly based on AR (Augmented Reality) technology. For example, a multi-sensor combined with a historical maintenance database has been proposed to provide data visualization-based maintenance guidance to users through AR technology. However, for fields with high maintenance accuracy, such as aerospace, guidance based solely on historical maintenance data is far from sufficient. In addition, the fineness of the maintenance target visualization model constructed by AR is also poor, making it difficult to support the requirements of high-precision maintenance. Summary of the Invention

[0005] To solve the above-mentioned problems of the prior art, the present invention provides an aircraft maintenance assistance method and system based on digital twin.

[0006] In a first aspect, an embodiment of the present application provides an aircraft maintenance assistance method based on digital twin, including: receiving a maintenance trigger operation of a user for an aircraft to be inspected; in response to the maintenance trigger operation being a maintenance trigger operation for the power system of the aircraft to be inspected, constructing a first voxel model corresponding to the power system in a digital modeling manner to assist in maintenance; wherein, the first voxel model has different colors and transparencies at different positions; the power system includes at least one of an engine and an auxiliary power unit; in response to the maintenance trigger operation being a maintenance trigger operation for the control system of the aircraft to be inspected, constructing a second voxel model corresponding to the control system in a digital modeling manner to assist in maintenance; wherein, the second voxel model has different colors and transparencies at different positions; the control system includes at least one of a mechanical control system, a fly-by-wire control system, and a lift augmentation device.

[0007] Optionally, constructing the first voxel model corresponding to the power system through digital modeling includes: obtaining image data of multiple angles of the power system; converting the image data of multiple angles into a sparse point cloud model; converting the sparse point cloud model into an initial voxel model; inputting the features of each voxel in the initial voxel model into a first multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into a second multi-layer perceptron to obtain the transparency of the voxel; obtaining the first voxel model based on the initial voxel model, the output result of the first multi-layer perceptron, and the output result of the second multi-layer perceptron.

[0008] Optionally, inputting the features of each voxel in the initial voxel model into a first multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into a second multi-layer perceptron to obtain the transparency of the voxel includes: creating high-dimensional features, medium-dimensional features, and low-dimensional features for each voxel; based on a third multi-layer perceptron, combining the relative distance and direction between each voxel and the image acquisition device to obtain the weights of the three features of each voxel; fusing the high-dimensional features, medium-dimensional features, low-dimensional features of each voxel and their respective corresponding weights to obtain target features; inputting the target features corresponding to each voxel into the first multi-layer perceptron to obtain the colors of each voxel, and inputting the target features corresponding to each voxel into the second multi-layer perceptron to obtain the transparencies of each voxel.

[0009] Optionally, constructing the second voxel model corresponding to the control system through digital modeling includes: obtaining image data of multiple angles of the control system; converting the image data of multiple angles into a sparse point cloud model; converting the sparse point cloud model into an initial voxel model; inputting the features of each voxel in the initial voxel model into a fourth multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into a fifth multi-layer perceptron to obtain the transparency of the voxel; obtaining the second voxel model based on the initial voxel model, the output result of the fourth multi-layer perceptron, and the output result of the fifth multi-layer perceptron.

[0010] Optionally, inputting the features of each voxel in the initial voxel model into a fourth multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into a fifth multi-layer perceptron to obtain the transparency of the voxel includes: creating high-dimensional features, medium-dimensional features, and low-dimensional features for each voxel; based on a sixth multi-layer perceptron, combining the relative distance and direction between each voxel and the image acquisition device to obtain the weights of the three features of each voxel; fusing the high-dimensional features, medium-dimensional features, low-dimensional features of each voxel and their respective corresponding weights to obtain target features; inputting the target features corresponding to each voxel into the fourth multi-layer perceptron to obtain the colors of each voxel, and inputting the target features corresponding to each voxel into the fifth multi-layer perceptron to obtain the transparencies of each voxel.

[0011] Optionally, the method further includes: constructing a voxel model of other components of the aircraft to be detected; constructing a component association map based on the mutual control logic relationship of other components of the aircraft to be detected; when the first control instruction cannot be transmitted to the target component to complete the control task, based on the component association map, determining other voxel models of components on the transmission line of the first control instruction and displaying them.

[0012] Optionally, the method further includes: in response to the maintenance trigger operation being a maintenance trigger operation for the airframe structure of the aircraft to be detected, acquiring image information of the airframe structure; the airframe structure includes at least one of a fuselage, a wing, a tail, and a landing gear; inputting the image information into a pre-trained image segmentation and recognition model to obtain a defect mask and an airframe structure mask; calculating the damage degree of the aircraft to be detected based on the defect mask, the airframe structure mask, the body size and shape of the aircraft to be detected, and the proportion of the photographed airframe structure in the complete airframe in the image information.

[0013] Optionally, when the maintenance trigger operation is a maintenance trigger operation for the fuselage and wings in the airframe structure of the aircraft to be detected, the method further includes: combining the damage degree of the aircraft to be detected, the wing projected area, and the airframe structure mask to output the lift of the aircraft to be detected after identifying the defect.

[0014] Optionally, before receiving the maintenance trigger operation of the user for the aircraft to be detected, the method further includes: acquiring the number of the aircraft to be detected input by the user; based on the number of the aircraft to be detected, acquiring the basic information of the aircraft corresponding to the number of the aircraft to be detected from the maintenance database; determining the maintenance process of the aircraft to be detected based on the basic information of the aircraft corresponding to the number of the aircraft to be detected; where different maintenance processes correspond to different maintenance trigger operations.

[0015] Second aspect, an aircraft maintenance assistance system based on digital twin provided by an embodiment of the present application includes: a receiving module, configured to receive a maintenance trigger operation of a user for an aircraft to be detected; a first construction module, configured to, in response to the maintenance trigger operation being a maintenance trigger operation for a power system of the aircraft to be detected, construct a first voxel model corresponding to the power system by means of digital modeling to assist in maintenance; wherein, the first voxel model has different colors and transparencies at different positions; the power system includes at least one of an engine and an auxiliary power unit; a second construction module, configured to, in response to the maintenance trigger operation being a maintenance trigger operation for a control system of the aircraft to be detected, construct a second voxel model corresponding to the control system by means of digital modeling to assist in maintenance; wherein, the second voxel model has different colors and transparencies at different positions; the control system includes at least one of a mechanical control system, a fly-by-wire control system, and a lift augmentation device.

[0016] The beneficial effects of the present invention include: First, a maintenance method for aircraft maintenance based on digital twin is proposed, and the highlight of this method is to construct a voxel model with color and transparency through digital modeling. The voxel model can accurately represent the geometric shape and internal structure of the complex system of the aircraft. At the same time, by adding color and transparency settings to the voxel model, the modeling can be made more realistic and the visualization effect can be better. In a complex system, transparency can be used to display different components in layers, highlighting the key parts. Furthermore, it can help users observe, analyze, and maintain the control system more efficiently. Especially in a complex system, it can intuitively display the internal structure and highlight the key areas. That is, compared with the maintenance assistance using AR technology in the prior art, its accuracy and the fineness of the visualization model are higher, supporting higher-precision maintenance requirements.

[0017] Second, this method can be effectively applied to the two complex aircraft system structures of the power system and the control system. The maintenance of the power system and the control system is one of the most critical tasks in the aviation field, which involves the core performance and flight safety of the aircraft. The embodiment of the present application can provide more accurate and real-time maintenance assistance through the technical solution of digital twin and voxel model, help maintenance personnel effectively identify and repair potential faults, thereby improving safety, maintenance efficiency, and accuracy. Description of the Drawings

[0018] Figure 1 It is a flowchart of the steps of a method for aircraft maintenance assistance based on digital twin provided by an embodiment of the present invention;

[0019] Figure 2 It is a flowchart of the steps of another method for aircraft maintenance assistance based on digital twin provided by an embodiment of the present invention;

[0020] Figure 3 A flowchart of steps of another aircraft maintenance inspection assistance method based on digital twin provided by an embodiment of the present invention;

[0021] Figure 4 A block diagram of modules of an aircraft maintenance inspection assistance system based on digital twin provided by an embodiment of the present invention;

[0022] Figure 5 A block diagram of modules of an electronic device applying the aircraft maintenance inspection assistance method based on digital twin provided by an embodiment of the present invention. Detailed implementation manners

[0023] In the following description, specific details such as specific system architectures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0024] In addition, in the description of the specification of the present application and the appended claims, the terms "first", "second", "third", etc. are only used for differential description and cannot be construed as indicating or implying relative importance.

[0025] Research has found that existing maintenance inspection assistance methods are mostly based on AR (Augmented Reality) technology. For example, a method of combining multiple sensors with a historical maintenance inspection database to provide data visualization-based maintenance inspection guidance to users through AR technology has been proposed. However, for fields with high maintenance inspection accuracy, such as aerospace, guidance based only on historical maintenance inspection data is far from sufficient. In addition, the fineness of the visual model of the maintenance inspection target constructed by AR is also poor, making it difficult to support the needs of high-precision maintenance inspection.

[0026] In view of the above problems, the present application proposes the following embodiments to solve the above technical problems.

[0027] Please refer to Figure 1 , an embodiment of the present application provides an aircraft maintenance inspection assistance method based on digital twin, including: Step 101 to Step 103.

[0028] Step 101: Receive a maintenance inspection trigger operation from a user for an aircraft to be inspected.

[0029] Among them, the user can perform the above-mentioned maintenance inspection trigger operation through electronic products such as visual wearable devices or tablet computers. In the embodiment of the present application, the user may refer to maintenance inspection personnel.

[0030] After the user performs a maintenance trigger operation on the aircraft to be detected, identify the detection trigger operation. If the maintenance trigger operation is for the power system of the aircraft to be detected, perform step 102. If the maintenance trigger operation is for the control system of the aircraft to be detected, perform step 103.

[0031] Step 102: Construct a first voxel model corresponding to the power system by digital modeling to assist in maintenance.

[0032] Among them, the first voxel model has different colors and transparencies at different positions; the power system includes at least one of an engine and an auxiliary power unit.

[0033] Considering that the power system is precise and complex and requires a relatively delicate digital modeling instrument to assist the user in detection, therefore, in the embodiments of the present application, a first voxel model corresponding to the power system is constructed by digital modeling.

[0034] It should be noted that the voxel model is a three-dimensional space modeling method based on three-dimensional volume pixels. In the voxel model, the three-dimensional space is divided into multiple small cube units, and each cube unit is called a voxel. The geometric shape and internal structure of the power system can be accurately represented through the voxel model. At the same time, by adding color and transparency settings to the voxel model, the modeling can be made more realistic and the visualization effect can be better. In a complex system, the transparency can be used to display different components in layers, highlighting the key parts. Furthermore, it can help the user observe, analyze, and maintain the power system more efficiently, especially in a complex system, the internal structure can be intuitively displayed and the key areas can be highlighted.

[0035] In addition, during the actual assistance process, the first voxel model can be projected onto a two-dimensional image at a preset perspective and can be aligned with the actual structure diagram of the power system. When the user has doubts during maintenance, they can click on the corresponding position of the first voxel model. Then, in response to the user's click, the position on the two-dimensional map of the first voxel model is extracted, and through the aligned structure diagram, the actual structure of this position is shown to the user. This method can, on the one hand, help the user understand the complex structure of the power system and locate problems, and on the other hand, quickly locate the actual structure, improving the maintenance efficiency.

[0036] Of course, during the actual maintenance process, it is also necessary to obtain the real data of the power system in real time through various sensors and update the first voxel model.

[0037] Step 103: Construct a second voxel model corresponding to the control system by digital modeling to assist in maintenance.

[0038] Among them, the second voxel model has different colors and transparencies at different positions; the control system includes at least one of a mechanical control system, a fly-by-wire control system, and a lift device.

[0039] Considering that the control system is precise and complex and requires a relatively delicate digital modeling instrument to assist users in detection, in the embodiments of the present application, a second voxel model corresponding to the control system is constructed through digital modeling.

[0040] It should be noted that the voxel model is a three-dimensional space modeling method based on three-dimensional volume pixels. In the voxel model, the three-dimensional space is divided into multiple small cube units, and each cube unit is called a voxel. The geometric shape and internal structure of the control system can be accurately represented through the voxel model. At the same time, by adding color and transparency settings in the voxel model, the modeling can be made more realistic and the visualization effect can be better. In a complex system, the transparency can be used to display different components in layers, highlighting the key parts. Furthermore, it can help users observe, analyze, and repair the control system more efficiently, especially in a complex system, it can intuitively display the internal structure and highlight the key areas.

[0041] In addition, during the actual assistance process, the second voxel model can be projected onto a two-dimensional image at a preset perspective and can be aligned with the actual structure diagram of the control system. When the user has doubts during maintenance, the user can click on the corresponding position of the second voxel model, and in response to the user's click, the position on the two-dimensional image of the second voxel model is extracted, and through the aligned structure diagram, the actual structure of this position is shown to the user. In this way, on the one hand, it can help users understand the complex structure of the control system and locate problems, and on the other hand, it can quickly locate the actual structure, improving the maintenance efficiency.

[0042] Of course, during the actual maintenance process, real-time data of the power system also needs to be obtained through various sensors and the second voxel model needs to be updated.

[0043] In summary, the aircraft maintenance assistance method based on digital twin provided by the embodiments of the present application has the following beneficial effects, including:

[0044] First, a maintenance method for aircraft maintenance based on digital twin is proposed. The highlight of this method is to construct a voxel model with color and transparency through digital modeling. The voxel model can accurately represent the geometric shape and internal structure of the complex system of the aircraft. At the same time, by adding color and transparency settings to the voxel model, the modeling can be made more realistic and the visualization effect can be better. In a complex system, transparency can be used to display different components in layers, highlighting the key parts. Furthermore, it can help users observe, analyze, and maintain the control system more efficiently, especially in a complex system, it can intuitively display the internal structure and highlight the key areas. That is, compared with the maintenance assistance using AR technology in the prior art, its accuracy and the fineness of the visualization model are higher, supporting higher-precision maintenance requirements.

[0045] Second, this method can be effectively applied to two complex aircraft system structures, namely the power system and the control system. The maintenance of the power system and the control system is one of the most critical tasks in the aviation field, which is related to the core performance and flight safety of the aircraft. Through the technical solution of digital twin and voxel model in this application embodiment, more accurate and real-time maintenance assistance can be provided to help maintenance personnel effectively identify and repair potential faults, thereby improving safety, maintenance efficiency, and accuracy.

[0046] Please refer to Figure 2 , optionally, the above steps construct a first voxel model corresponding to the power system through digital modeling, including: step 201 to step 205.

[0047] Step 201: Obtain image data of multiple angles of the power system.

[0048] Since the power system of the aircraft is precise and complex, more delicate digital modeling is required to assist users in maintenance. Therefore, first, image acquisition equipment is used to obtain image data of multiple angles to ensure that all directions and angles of the entire power system can be covered.

[0049] Step 202: Convert the image data of multiple angles into a sparse point cloud model.

[0050] In the embodiment of this application, the image data of multiple angles can be input into COLMAP to generate a sparse point cloud model.

[0051] Step 203: Convert the sparse point cloud model into an initial voxel model.

[0052] Then, the sparse point cloud model is further processed and converted into a voxel form, that is, an initial voxel model is obtained.

[0053] Optionally, in one embodiment, the expression of voxel conversion is:

[0054] ;

[0055] In this formula, represents the voxel set, represents the sparse point cloud, represents the voxel size, represents floor function, represents reducing the irregularity of the point cloud. In this embodiment, the RANSAC algorithm can be specifically used to remove irregularities (outliers, noise points), and then the accuracy and precision of the data can be determined.

[0056] Step 204: Input the features of each voxel in the initial voxel model into the first multi-layer perceptron to obtain the color of the voxel, and input the features of each voxel in the initial voxel model into the second multi-layer perceptron to obtain the transparency of the voxel.

[0057] It should be noted that two multi-layer perceptrons with the same structure but different internal parameters (obtained through learning) are pre-constructed in this application. The first multi-layer perceptron is mainly used to output the color of the voxel, and the second multi-layer perceptron is mainly used to output the transparency of the voxel.

[0058] Step 205: Obtain the first voxel model based on the initial voxel model, the output result of the first multi-layer perceptron, and the output result of the second multi-layer perceptron.

[0059] Finally, combine the initial voxel model with the output result of the first multi-layer perceptron and the output result of the second multi-layer perceptron to obtain the first voxel model.

[0060] In summary, the embodiment of this application provides a complete construction method for the voxel model corresponding to the power system. It should be noted that in the embodiment of this application, image data from multiple angles of the power system are obtained to ensure that all directions and angles of the entire power system can be covered, and details of all parts of the power system can be fully captured. Then, the image data is first converted into a sparse point cloud model, and then the sparse point cloud model is converted into a voxel model. This method can ensure that the generated voxel model has richer details and a more coherent structure. Finally, two multi-layer perceptrons are used to accurately predict the color and transparency of the voxel to form a first voxel model with high precision.

[0061] Please refer to Figure 3 , optionally, inputting the features of each voxel in the initial voxel model into the first multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into the second multi-layer perceptron to obtain the transparency of the voxel includes: Steps 301 to 304.

[0062] Step 301: Create high-dimensional features, medium-dimensional features, and low-dimensional features for each voxel.

[0063] Specifically, for each (belonging to the above voxel set ) a learnable vector set can be created , , .

[0064] Among them, represents high-dimensional features, represents a real vector with a dimension of 32; represents medium-dimensional features, represents a real vector with a dimension of 16, represents low-dimensional features, represents a real vector with a dimension of 8.

[0065] Step 302: Based on the third multi-layer perceptron, combined with the relative distance and direction between each voxel and the image acquisition device, the weights of the three features of each voxel are obtained.

[0066] Among them, the third multi-layer perceptron is configured to generate weights of learnable vectors (three features).

[0067] First of all, the formula for the relative distance and direction between the image device and each voxel can be referred to as follows: ; ;

[0068] In the above formula, represents the voxel and the image acquisition device 's relative distance, represents the position of the image acquisition device, represents the position where the voxel center is located, represents the voxel and the direction of the image acquisition device; the symbol represents the Euclidean norm (L2 norm).

[0069] Using the third multi-layer perceptron to generate the weights of the three features can be expressed as:

[0070] ;

[0071] Among them, in this formula, represents the third multi-layer perceptron, , , are the weights corresponding to high-dimensional features, medium-dimensional features, and low-dimensional features respectively; Softmax here represents weight allocation, making the sum of weights equal to 1.

[0072] Step 303: Fuse the high-dimensional features, medium-dimensional features, low-dimensional features of each voxel and their respective corresponding weights to obtain the target features.

[0073] That is, fuse according to each feature and the weight of each feature to obtain the target features.

[0074] Step 304: Input the target features corresponding to each voxel into the first multi-layer perceptron to obtain the color of each voxel, and input the target features corresponding to each voxel into the second multi-layer perceptron to obtain the transparency of each voxel.

[0075] In summary, in the embodiment of the present application, the high-dimensional features, medium-dimensional features, and low-dimensional features of each voxel are extracted, and after determining the weights of each feature according to the relative distance and direction between each voxel and the image acquisition device, they are fused to be used as the input of the first multi-layer perceptron and the second multi-layer perceptron. In this way, comprehensive voxel features can be captured, the accuracy of the output color and transparency can be improved, that is, the visualization fineness can be enhanced.

[0076] Optionally, through digital modeling, a second voxel model corresponding to the control system is constructed, including: obtaining image data of multiple angles of the control system; converting the image data of multiple angles into a sparse point cloud model; converting the sparse point cloud model into an initial voxel model; inputting the features of each voxel in the initial voxel model into the fourth multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into the fifth multi-layer perceptron to obtain the transparency of the voxel; based on the initial voxel model, the output result of the fourth multi-layer perceptron, and the output result of the fifth multi-layer perceptron, the second voxel model is obtained.

[0077] It should be noted that the process of constructing the second voxel model can refer to the implementation process of constructing the first voxel model in the foregoing embodiment. The processing processes of the two are basically the same and can refer to each other, so it will not be elaborated here.

[0078] In summary, the embodiment of the present application provides a complete construction method for the voxel model corresponding to the control system. It should be noted that in the embodiment of the present application, image data of multiple angles of the control system is obtained to ensure that all direction angles of the entire control system can be covered, and the details of each part of the control system can be fully captured. Then the image data is first converted into a sparse point cloud model, and then the sparse point cloud model is converted into a voxel model. This method can ensure that the generated voxel model has richer details and more coherent structure. Finally, two multi-layer perceptrons are used to accurately predict the color and transparency of the voxels to form a second voxel model with high precision.

[0079] Optionally, input the features of each voxel in the initial voxel model into a fourth multi-layer perceptron to obtain the color of the voxel, and input the features of each voxel in the initial voxel model into a fifth multi-layer perceptron to obtain the transparency of the voxel, including: creating high-dimensional features, medium-dimensional features, and low-dimensional features for each voxel; based on a sixth multi-layer perceptron, combining the relative distance and direction between each voxel and the image acquisition device to obtain the weights of the three features of each voxel; fusing the high-dimensional features, medium-dimensional features, low-dimensional features of each voxel and their respective corresponding weights to obtain target features; inputting the target features corresponding to each voxel into the fourth multi-layer perceptron to obtain the colors of each voxel, and inputting the target features corresponding to each voxel into the fifth multi-layer perceptron to obtain the transparency of each voxel.

[0080] It should be noted that the above fusion process can also refer to the description of the dynamic system in the foregoing embodiments. The same parts can be referred to each other, and will not be elaborated here.

[0081] In summary, in the embodiments of the present application, high-dimensional features, medium-dimensional features, and low-dimensional features of each voxel are extracted, and after determining the weights of each feature according to the relative distance and direction between each voxel and the image acquisition device, they are fused to be used as the inputs of the fourth multi-layer perceptron and the fifth multi-layer perceptron. In this way, comprehensive voxel features can be captured, the accuracy of the output color and transparency can be improved, that is, the visualization fineness is enhanced.

[0082] Optionally, the method further includes: constructing a voxel model of other components of the aircraft to be detected; constructing a component association map based on the mutual control logic relationship of other components of the aircraft to be detected; when the first control instruction cannot be transmitted to the target component to complete the control task, based on the component association map, determining other voxel models of components on the transmission line of the first control instruction and displaying them.

[0083] It should be noted that since the maintenance standard of the control system is whether the target component can be controlled by a control instruction, in addition to constructing the voxel model, all components are also mapped according to preset rules (generally speaking, the aircraft manufacturer will provide the mutual control logic relationship between components, and this relationship can be used as the construction rule of the component association map). When the user performs maintenance on the control system, the way to do it is to send a control instruction from one component to another component and check whether the target component can complete the task expressed by the instruction. Most of the instruction transmissions between components are not direct transmissions, and usually pass through transfer components, which are mostly sensor components. When the target component cannot complete the expected task, the voxel model of the associated component will be given according to the component association map, so that the user can quickly troubleshoot.

[0084] Exemplarily, for instance, the air compressor and the turbine are strongly coupled. If there is a problem with the turbine, the air compressor will be affected, and then the information of the air compressor will also be pushed.

[0085] Optionally, the method further includes: in response to the maintenance trigger operation being a maintenance trigger operation for the airframe structure of the aircraft to be detected, acquiring image information of the airframe structure; the airframe structure includes at least one of a fuselage, a wing, a tail, and a landing gear; inputting the image information into a pre-trained image segmentation and recognition model to obtain a defect mask and an airframe structure mask; calculating the damage degree of the aircraft to be detected based on the defect mask, the airframe structure mask, the size and shape of the airframe of the aircraft to be detected, and the proportion of the airframe structure captured in the image information to the complete airframe.

[0086] In the embodiment of the present application, Efficientvit can be used as the image segmentation and recognition model. The image information is sent into the image segmentation and recognition model to obtain a defect mask and an airframe structure mask . Since information such as the size and shape of the airframe of the aircraft to be detected can be directly obtained, the damage degree of the airframe structure of the aircraft to be detected can be calculated. The calculation formula is as follows:

[0087] ;

[0088] In this formula, represents the loss length, represents the proportion of the airframe structure captured in the image to the complete airframe. The calculation formula of

[0089] ;

[0090] In this formula, represents the true length of the complete airframe, represents the projection calculation function, the input of is the coordinates of the point with the largest horizontal axis coordinate and the coordinates of the point with the smallest horizontal axis coordinate in . The calculation details of are as follows: after inputting the above coordinates, project the above pixel points into the real space according to the internal parameters of the image acquisition device, and then calculate the real space coordinate distance as the length of the part captured in the image. When using for projection, a depth information is required. In this embodiment, it is realized by a laser rangefinder installed at the lower end of the image acquisition device.

[0091] Since the maintenance of the airframe structure is to judge the structural integrity, only the physical simulation of the fuselage structure is required in the embodiment of the present application. That is, the embodiment of the present application provides a way to assist in the maintenance of the airframe structure through physical simulation.

[0092] In addition, considering that an aircraft may be struck by a bird, and the impact of a bird strike depends on the location and severity of the bird strike. Bird strikes at different locations will have different impacts on the operation of the aircraft, but AR technology can only display some historical data and cannot simulate or evaluate the actual impact of a bird strike at a specific location on the aircraft, thus making the AR-based method unable to complete the simulation of this impact and lacking sufficient accurate practical guidance. The present application also provides the following embodiments to solve the above problems:

[0093] Optionally, when the maintenance trigger operation is a maintenance trigger operation for the fuselage and wings in the airframe structure of the aircraft to be detected, the method further includes: combining the damage degree, wing projection area, and airframe structure mask of the aircraft to be detected, and outputting the lift of the aircraft to be detected after identifying the defect.

[0094] Among them, the calculation formula for lift is:

[0095] ;

[0096] In this formula, represents lift, represents air density, represents the relative fluid velocity, represents the wing projection area, represents the lift coefficient, represents the weight obtained according to Different weights can be set for different positions of the airframe.

[0097] That is, the embodiments of the present application also consider the impact of the defect location on lift to further assist the user in maintenance guidance.

[0098] Optionally, before receiving the maintenance trigger operation of the user for the aircraft to be detected, the method further includes: obtaining the number of the aircraft to be detected input by the user; based on the number of the aircraft to be detected, obtaining the basic information of the aircraft corresponding to the number of the aircraft to be detected from the maintenance database; and determining the maintenance process of the aircraft to be detected based on the basic information of the aircraft corresponding to the number of the aircraft to be detected; wherein, different maintenance processes correspond to different maintenance trigger operations.

[0099] Specifically, the user can first enter the ID of the aircraft to be inspected. Usually, the basic information of each aircraft, such as aircraft ID, model, aircraft component signals, aircraft age, and aircraft mileage, is stored in the maintenance database. Therefore, after obtaining the ID of the aircraft to be inspected entered by the user, the basic information of the aircraft corresponding to the ID of the aircraft to be inspected is extracted from the maintenance database. It should be noted that in the embodiments of the present application, for aircraft of different models, different ages, and different flight mileages, their maintenance processes are different. Therefore, in the embodiments of the present invention, the model, age, and flight mileage are used as constraint conditions to match the maintenance processes in the maintenance database (there are generally standard maintenance guides for aircraft maintenance), and a recommended guidance process is obtained.

[0100] It can be seen that through this method, different aircraft states can be adapted, the rationality and efficiency of maintenance can be improved, and resource utilization can be optimized.

[0101] In addition, after the user completes the maintenance, the target to be maintained needs to be tested to ensure normal function. In this process, the parameters of some functional targets are recorded through special equipment. The recorded parameters will also be synchronized. Since the relationships between the structural components of the maintenance target itself and the relationships between the maintenance target and other components have been constructed, the flow of data between components or structural components can be simulated and calculated, and it can be judged whether the component data feedback is consistent with the actual situation. Suppose there is an association between component A, component B, and component C after modeling. After the user tests component A after the maintenance, the data generated when component A works is A1. When simulating and calculating, A1 is input into component B and component C in sequence, and the data B1 and C1 generated by component B and component C are calculated. When the user tests component B and component C, the collected data will be compared with B1 and C1 to judge the consistency.

[0102] Please refer to Figure 4, based on the same inventive concept, an embodiment of the present application provides an aircraft maintenance assistance system 400 based on digital twin, including: a receiving module 401, configured to receive a maintenance trigger operation of a user for an aircraft to be detected; a first construction module 402, configured to, in response to the maintenance trigger operation being a maintenance trigger operation for the power system of the aircraft to be detected, construct a first voxel model corresponding to the power system by means of digital modeling to assist in maintenance; wherein, the first voxel model has different colors and transparencies at different positions; the power system includes at least one of an engine and an auxiliary power unit; a second construction module 403, configured to, in response to the maintenance trigger operation being a maintenance trigger operation for the control system of the aircraft to be detected, construct a second voxel model corresponding to the control system by means of digital modeling to assist in maintenance; wherein, the second voxel model has different colors and transparencies at different positions; the control system includes at least one of a mechanical control system, a fly-by-wire control system, and a lift augmentation device.

[0103] Please refer to Figure 5 , based on the same inventive concept, an embodiment of the present application provides a module housing of an electronic device 500 applying the above method. The electronic device 500 includes: at least one processor 501 ( Figure 5 only one is shown), a memory 502, and a computer program 503 stored in the memory 502 and executable on at least one processor 501. When the processor 501 executes the computer program 503, the steps of the method in any of the foregoing embodiments are implemented.

[0104] The electronic device 500 may be a server, a personal computer, a laptop computer, and so on.

[0105] Those skilled in the art can understand that Figure 5 merely examples of the electronic device 500 are provided, and do not constitute a limitation to the electronic device 500. It may include more or fewer components than shown in the figure, or combine certain components, or different components.

[0106] The so-called processor 501 may be a Central Processing Unit (CPU), and this processor 501 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0107] In some embodiments, the memory 502 may be an internal storage unit of the electronic device 500, such as the hard disk or memory of the electronic device 500. In some other embodiments, the memory 502 may also be an external storage device of the electronic device 500, such as a plug-in hard disk equipped on the electronic device 500, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 502 may also include both the internal storage unit of the electronic device 500 and the external storage device.

[0108] It should be noted that for the above-mentioned systems, devices, etc., since they are based on the same concept as the method embodiments of this application, the modules designed by the system and the steps and technical effects executed by the device can be referred to the method embodiment part, and will not be elaborated here.

[0109] Those skilled in the art can clearly understand that for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In practical applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated in one processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0110] An embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0111] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal is enabled to implement the steps in the above-mentioned various method embodiments when executed.

[0112] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-mentioned embodiment methods of the present application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc.

[0113] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0114] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0115] In the embodiments provided in the present application, it should be understood that the disclosed device / network device and method can be implemented in other ways. For example, the device / network device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0116] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0117] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An aircraft maintenance inspection assistance method based on digital twin, characterized in that Including: Receiving a maintenance trigger operation from a user for an aircraft to be detected; In response to the maintenance trigger operation being a maintenance trigger operation for the power system of the aircraft to be detected, acquiring image data of the power system from multiple angles; converting the image data from multiple angles into a sparse point cloud model; converting the sparse point cloud model into an initial voxel model; inputting the features of each voxel in the initial voxel model into a first multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into a second multi-layer perceptron to obtain the transparency of the voxel; Based on the initial voxel model, the output result of the first multi-layer perceptron, and the output result of the second multi-layer perceptron, obtaining a first voxel model; the first voxel model has different colors and transparencies at different positions; the power system includes at least one of an engine and an auxiliary power unit; In response to the maintenance trigger operation being a maintenance trigger operation for the control system of the aircraft to be detected, acquiring image data of the control system from multiple angles; converting the image data from multiple angles into a sparse point cloud model; converting the sparse point cloud model into an initial voxel model; inputting the features of each voxel in the initial voxel model into a fourth multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into a fifth multi-layer perceptron to obtain the transparency of the voxel; Based on the initial voxel model, the output result of the fourth multi-layer perceptron, and the output result of the fifth multi-layer perceptron, obtaining a second voxel model; the second voxel model has different colors and transparencies at different positions; the control system includes at least one of a mechanical control system, a fly-by-wire control system, and a lift augmentation device.

2. The maintenance assistance method for aircraft maintenance based on digital twin according to claim 1, wherein The step of inputting the features of each voxel in the initial voxel model into a first multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into a second multi-layer perceptron to obtain the transparency of the voxel includes: Creating high-dimensional features, medium-dimensional features, and low-dimensional features for each voxel; Based on a third multi-layer perceptron, combining the relative distance and direction between each voxel and the image acquisition device to obtain the weights of the three features of each voxel; Fusing the high-dimensional features, medium-dimensional features, low-dimensional features of each voxel and their respective corresponding weights to obtain target features; Inputting the target features corresponding to each voxel into the first multi-layer perceptron to obtain the color of each voxel, and inputting the target features corresponding to each voxel into the second multi-layer perceptron to obtain the transparency of each voxel.

3. The aircraft maintenance inspection assistance method based on digital twin according to claim 1, wherein The step of inputting the features of each voxel in the initial voxel model into a fourth multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into a fifth multi-layer perceptron to obtain the transparency of the voxel includes: Creating high-dimensional features, medium-dimensional features, and low-dimensional features for each voxel; Based on a sixth multi-layer perceptron, combining the relative distance and direction between each voxel and the image acquisition device to obtain the weights of the three features of each voxel; Fuse the high-dimensional features, medium-dimensional features, low-dimensional features of each voxel and their respective corresponding weights to obtain the target features; Input the target features corresponding to each voxel into the fourth multi-layer perceptron to obtain the color of each voxel, and input the target features corresponding to each voxel into the fifth multi-layer perceptron to obtain the transparency of each voxel.

4. The method for assisting aircraft maintenance based on digital twin according to claim 3, wherein The method further includes: Construct a voxel model of other components of the aircraft to be detected; Based on the mutual control logic relationship of other components of the aircraft to be detected, construct a component association graph; When the first control instruction cannot be transmitted to the target component to complete the control task, based on the component association graph, determine other component voxel models on the transmission line of the first control instruction and display them.

5. The aircraft maintenance inspection assistance method based on digital twin according to claim 1, wherein The method further includes: In response to the maintenance trigger operation being a maintenance trigger operation for the airframe structure of the aircraft to be detected, obtain image information of the airframe structure; the airframe structure includes at least one of a fuselage, a wing, a tail, and a landing gear; Input the image information into a pre-trained image segmentation and recognition model to obtain a defect mask and an airframe structure mask; Based on the defect mask, the airframe structure mask, the body size and shape of the aircraft to be detected, and the proportion of the airframe structure captured in the image information in the complete airframe, calculate the damage degree of the aircraft to be detected.

6. The digital-twin-based maintenance assistance method according to claim 5, wherein When the maintenance trigger operation is a maintenance trigger operation for the fuselage and wings in the airframe structure of the aircraft to be detected, the method further includes: Combine the damage degree of the aircraft to be detected, the wing projected area, and the airframe structure mask to output the lift of the aircraft to be detected after identifying the defect.

7. The aircraft maintenance inspection assistance method based on digital twin according to claim 1, wherein Before receiving the maintenance trigger operation of the user for the aircraft to be detected, the method further includes: Obtain the number of the aircraft to be detected input by the user; Based on the number of the aircraft to be detected, obtain the basic information of the aircraft corresponding to the number of the aircraft to be detected from the maintenance database; Based on the basic information of the aircraft corresponding to the number of the aircraft to be detected, determine the maintenance process of the aircraft to be detected; among them, different maintenance processes correspond to different maintenance trigger operations.

8. An aircraft maintenance assistance system based on digital twin, characterized in that, Include: A receiving module for receiving the maintenance trigger operation of the user for the aircraft to be detected; A first construction module for, in response to the maintenance trigger operation being a maintenance trigger operation for the power system of the aircraft to be detected, obtaining image data of multiple angles of the power system; converting the image data of multiple angles into a sparse point cloud model; converting the sparse point cloud model into an initial voxel model; inputting the features of each voxel in the initial voxel model into a first multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into a second multi-layer perceptron to obtain the transparency of the voxel; Based on the initial voxel model, the output result of the first multi-layer perceptron, and the output result of the second multi-layer perceptron, obtain a first voxel model; the first voxel model has different colors and transparencies at different positions; the power system includes at least one of an engine and an auxiliary power unit; A second construction module, configured to, in response to the maintenance trigger operation for the maintenance trigger operation of the control system of the aircraft to be detected, acquire image data of multiple angles of the control system; convert the image data of the multiple angles into a sparse point cloud model; convert the sparse point cloud model into an initial voxel model; input the features of each voxel in the initial voxel model into a fourth multi-layer perceptron to obtain the color of the voxel, and input the features of each voxel in the initial voxel model into a fifth multi-layer perceptron to obtain the transparency of the voxel; Based on the initial voxel model, the output result of the fourth multi-layer perceptron, and the output result of the fifth multi-layer perceptron, a second voxel model is obtained; the second voxel model has different colors and transparencies at different positions; the control system includes at least one of a mechanical control system, a fly-by-wire control system, and a lift augmentation device.

Images