Maintenance maintenance auxiliary method and system based on digital twinning

Abstract

The invention relates to a digital twinning-based maintenance auxiliary method and a digital twinning-based maintenance auxiliary system. The method comprises the following steps: receiving a maintenance trigger operation of a user for a to-be-detected aircraft; in response to the overhaul triggering operation as the overhaul triggering operation for the power system of the to-be-detected aircraft, constructing a first voxel model corresponding to the power system in a digital modeling mode to assist overhaul; wherein the first voxel model has different colors and transparency at different positions; the power system comprises at least one of an engine and an auxiliary power device; in response to the overhaul triggering operation as the overhaul triggering operation for the control system of the to-be-detected aircraft, constructing a second voxel model corresponding to the control system in a digital modeling mode to assist overhaul; wherein the second voxel model has different colors and transparency at different positions; the control system comprises at least one of a mechanical control system, a fly-by-wire control system and a high lift device.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft maintenance, and in particular to an aircraft maintenance assistance method and system based on digital twin. Background Art

[0002] Aircraft maintenance is a crucial part of aircraft maintenance. Aircraft maintenance mainly refers to the regular or irregular inspection, repair, maintenance, etc. of the aircraft and its related equipment, aiming to ensure that the safety, reliability, and performance of the aircraft meet the relevant standard requirements. The quality of aircraft maintenance is directly related to the safety performance and operation efficiency of the aircraft.

[0003] However, traditional maintenance methods rely on manual experience, which have problems such as large workload, low efficiency, and strong subjectivity, and are difficult to meet the needs of modern aircraft complexity and maintenance refinement. Therefore, it is particularly important to introduce intelligent and information-based assistance methods and systems.

[0004] Research has found that existing maintenance assistance methods are mostly based on AR (Augmented Reality) technology. For example, a method that combines multiple sensors with a historical maintenance database to provide data visualization-based maintenance guidance to users through AR technology has been proposed. However, for fields with high maintenance accuracy, such as aerospace, guidance based solely on historical maintenance data is far from sufficient. In addition, the fineness of the maintenance target visualization model constructed by AR is also poor, making it difficult to support the needs of high-precision maintenance. Summary of the Invention

[0005] To solve the above-mentioned problems of the prior art, the present invention provides an aircraft maintenance assistance method and system based on digital twin.

[0006] In a first aspect, an embodiment of the present application provides an aircraft maintenance assistance method based on digital twin, including: receiving a maintenance trigger operation of a user for an aircraft to be detected; in response to the maintenance trigger operation being a maintenance trigger operation for the power system of the aircraft to be detected, constructing a first voxel model corresponding to the power system by digital modeling to assist in maintenance; wherein, the first voxel model has different colors and transparencies at different positions; the power system includes at least one of an engine and an auxiliary power unit; in response to the maintenance trigger operation being a maintenance trigger operation for the control system of the aircraft to be detected, constructing a second voxel model corresponding to the control system by digital modeling to assist in maintenance; wherein, the second voxel model has different colors and transparencies at different positions; the control system includes at least one of a mechanical control system, a fly-by-wire control system, and a lift augmentation device.

[0007] Optionally, constructing the first voxel model corresponding to the power system by digital modeling includes: obtaining image data of multiple angles of the power system; converting the image data of multiple angles into a sparse point cloud model; converting the sparse point cloud model into an initial voxel model; inputting the features of each voxel in the initial voxel model into a first multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into a second multi-layer perceptron to obtain the transparency of the voxel; obtaining the first voxel model based on the initial voxel model, the output result of the first multi-layer perceptron, and the output result of the second multi-layer perceptron.

[0008] Optionally, inputting the features of each voxel in the initial voxel model into a first multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into a second multi-layer perceptron to obtain the transparency of the voxel includes: creating high-dimensional features, medium-dimensional features, and low-dimensional features for each voxel; based on a third multi-layer perceptron, combining the relative distance and direction between each voxel and the image acquisition device to obtain the weights of the three features of each voxel; fusing the high-dimensional features, medium-dimensional features, low-dimensional features of each voxel and their respective corresponding weights to obtain target features; inputting the target features corresponding to each voxel into the first multi-layer perceptron to obtain the colors of each voxel, and inputting the target features corresponding to each voxel into the second multi-layer perceptron to obtain the transparencies of each voxel.

[0009] Optionally, constructing the second voxel model corresponding to the control system by digital modeling includes: obtaining image data of multiple angles of the control system; converting the image data of multiple angles into a sparse point cloud model; converting the sparse point cloud model into an initial voxel model; inputting the features of each voxel in the initial voxel model into a fourth multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into a fifth multi-layer perceptron to obtain the transparency of the voxel; obtaining the second voxel model based on the initial voxel model, the output result of the fourth multi-layer perceptron, and the output result of the fifth multi-layer perceptron.

[0010] Optionally, inputting the features of each voxel in the initial voxel model into a fourth multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into a fifth multi-layer perceptron to obtain the transparency of the voxel includes: creating high-dimensional features, medium-dimensional features, and low-dimensional features for each voxel; based on a sixth multi-layer perceptron, combining the relative distance and direction between each voxel and the image acquisition device to obtain the weights of the three features of each voxel; fusing the high-dimensional features, medium-dimensional features, low-dimensional features of each voxel and their respective corresponding weights to obtain target features; inputting the target features corresponding to each voxel into the fourth multi-layer perceptron to obtain the colors of each voxel, and inputting the target features corresponding to each voxel into the fifth multi-layer perceptron to obtain the transparency of each voxel.

[0011] Optionally, the method further includes: constructing a voxel model of other components of the aircraft to be detected; constructing a component association graph based on the mutual control logic relationship of other components of the aircraft to be detected; when the first control instruction cannot be transmitted to the target component to complete the control task, based on the component association graph, determining other voxel models of components on the transmission line of the first control instruction and displaying them.

[0012] Optionally, the method further includes: in response to the maintenance trigger operation being a maintenance trigger operation for the airframe structure of the aircraft to be detected, acquiring image information of the airframe structure; the airframe structure includes at least one of a fuselage, a wing, a tail, and a landing gear; inputting the image information into a pre-trained image segmentation and recognition model to obtain a defect mask and an airframe structure mask; calculating the damage degree of the aircraft to be detected based on the defect mask, the airframe structure mask, the airframe size and shape of the aircraft to be detected, and the proportion of the photographed airframe structure in the complete airframe in the image information.

[0013] Optionally, when the maintenance trigger operation is a maintenance trigger operation for the fuselage and wings in the airframe structure of the aircraft to be detected, the method further includes: combining the damage degree of the aircraft to be detected, the wing projected area, and the airframe structure mask to output the lift of the aircraft to be detected after identifying the defect.

[0014] Optionally, before receiving the maintenance trigger operation of the user for the aircraft to be detected, the method further includes: acquiring the number of the aircraft to be detected input by the user; based on the number of the aircraft to be detected, acquiring the basic information of the aircraft corresponding to the number of the aircraft to be detected from the maintenance database; determining the maintenance process of the aircraft to be detected based on the basic information of the aircraft corresponding to the number of the aircraft to be detected; wherein, different maintenance processes correspond to different maintenance trigger operations.

[0015] Second aspect, an aircraft maintenance assistance system based on digital twin provided by an embodiment of the present application includes: a receiving module, configured to receive a maintenance trigger operation of a user for an aircraft to be detected; a first construction module, configured to, in response to the maintenance trigger operation being a maintenance trigger operation for a power system of the aircraft to be detected, construct a first voxel model corresponding to the power system by means of digital modeling to assist in maintenance; wherein, the first voxel model has different colors and transparencies at different positions; the power system includes at least one of an engine and an auxiliary power unit; a second construction module, configured to, in response to the maintenance trigger operation being a maintenance trigger operation for a control system of the aircraft to be detected, construct a second voxel model corresponding to the control system by means of digital modeling to assist in maintenance; wherein, the second voxel model has different colors and transparencies at different positions; the control system includes at least one of a mechanical control system, a fly-by-wire control system, and a lift augmentation device.

[0016] The beneficial effects of the present invention include: First, a maintenance method for aircraft maintenance based on digital twin is proposed, and the highlight of this method is to construct a voxel model with color and transparency through digital modeling. The voxel model can accurately represent the geometric shape and internal structure of complex systems of the aircraft. At the same time, by adding color and transparency settings to the voxel model, the modeling can be made more realistic and the visualization effect can be better. In complex systems, transparency can be used to display different components in layers, highlighting the key parts. Furthermore, it can help users observe, analyze, and maintain the control system more efficiently, especially in complex systems, it can intuitively display the internal structure and highlight the key areas. That is, compared with the maintenance assistance using AR technology in the prior art, its accuracy and the fineness of the visualization model are higher, supporting higher-precision maintenance requirements.

[0017] Second, this method can effectively be applied to two complex aircraft system structures, namely the power system and the control system. The maintenance of the power system and the control system is one of the most critical tasks in the aviation field, which involves the core performance and flight safety of the aircraft. The embodiment of the present application can provide more accurate and real-time maintenance assistance through the technical solution of digital twin and voxel model, helping maintenance personnel effectively identify and repair potential faults, thereby improving safety, maintenance efficiency, and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of steps of a method for aircraft maintenance assistance based on digital twin provided by an embodiment of the present invention; Figure 2 It is a flowchart of steps of another method for aircraft maintenance assistance based on digital twin provided by an embodiment of the present invention; Figure 3A step flowchart of another aircraft maintenance and repair assistance method based on digital twin provided by an embodiment of the present invention; Figure 4 A module block diagram of an aircraft maintenance and repair assistance system based on digital twin provided by an embodiment of the present invention; Figure 5 A module block diagram of an electronic device applying the aircraft maintenance and repair assistance method based on digital twin provided by an embodiment of the present invention. Detailed implementation manners

[0019] In the following description, specific details such as specific system architectures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0020] In addition, in the description of the specification and the appended claims of the present application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0021] Research has found that existing maintenance and repair assistance methods are mostly based on AR (Augmented Reality) technology. For example, a method that combines multiple sensors with a historical maintenance database to provide data visualization-based maintenance guidance to users through AR technology has been proposed. However, for fields with high maintenance accuracy requirements, such as aerospace, guidance based solely on historical maintenance data is far from sufficient. In addition, the fineness of the maintenance target visualization model constructed by AR is also poor, making it difficult to support the needs of high-precision maintenance.

[0022] In view of the above problems, the present application proposes the following embodiments to solve the above technical problems.

[0023] Please refer to Figure 1 , an embodiment of the present application provides an aircraft maintenance and repair assistance method based on digital twin, including: Step 101 to Step 103.

[0024] Step 101: Receive a maintenance trigger operation from the user for the aircraft to be inspected.

[0025] Among them, the user can perform the above-mentioned maintenance trigger operation through electronic products such as visualization wearable devices or tablets. In the embodiment of the present application, the user may refer to maintenance personnel.

[0026] After the user performs a maintenance trigger operation on the aircraft to be detected, identify the detection trigger operation. If the maintenance trigger operation is for the power system of the aircraft to be detected, execute step 102. If the maintenance trigger operation is for the control system of the aircraft to be detected, execute step 103.

[0027] Step 102: Construct a first voxel model corresponding to the power system by digital modeling to assist in maintenance.

[0028] Among them, the first voxel model has different colors and transparencies at different positions; the power system includes at least one of an engine and an auxiliary power unit.

[0029] Considering that the power system is precise and complex and requires a relatively delicate digital modeling instrument to assist the user in detection, therefore, in the embodiments of the present application, a first voxel model corresponding to the power system is constructed by digital modeling.

[0030] It should be noted that the voxel model is a three-dimensional space modeling method based on three-dimensional volume pixels. In the voxel model, the three-dimensional space is divided into multiple small cube units, and each cube unit is called a voxel. The geometric shape and internal structure of the power system can be accurately represented through the voxel model. At the same time, by adding color and transparency settings to the voxel model, the modeling can be made more realistic and the visualization effect can be better. In a complex system, the transparency can be used to display different components in layers, highlighting the key parts. Furthermore, it can help the user observe, analyze and maintain the power system more efficiently, especially in a complex system, it can intuitively display the internal structure and highlight the key areas.

[0031] In addition, during the actual assistance process, the first voxel model can be projected onto a two-dimensional image at a preset perspective and can be aligned with the actual structure diagram of the power system. When the user has doubts during maintenance, the user can click on the corresponding position of the first voxel model. In response to the user's click, the position on the two-dimensional map of the first voxel model is extracted, and through the aligned structure diagram, the actual structure of this position is displayed to the user. This method can, on the one hand, help the user understand the complex structure of the power system and locate problems, and on the other hand, can quickly locate the actual structure, improving the maintenance efficiency.

[0032] Of course, during the actual maintenance process, it is also necessary to use a variety of sensors to continuously obtain the real data of the power system and update the first voxel model.

[0033] Step 103: Construct a second voxel model corresponding to the control system by digital modeling to assist in maintenance.

[0034] Among them, the second voxel model has different colors and transparencies at different positions; the control system includes at least one of a mechanical control system, a fly-by-wire control system, and a lift device.

[0035] Considering that the control system is precise and complex and requires a relatively delicate digital modeling instrument to assist users in detection, in the embodiments of the present application, a second voxel model corresponding to the control system is constructed through digital modeling.

[0036] It should be noted that the voxel model is a three-dimensional spatial modeling method based on three-dimensional volume pixels. In the voxel model, the three-dimensional space is divided into multiple small cube units, and each cube unit is called a voxel. The geometric shape and internal structure of the control system can be accurately represented through the voxel model. At the same time, by adding color and transparency settings in the voxel model, the modeling can be made more realistic and the visualization effect can be better. In a complex system, the transparency can be used to display different components in layers, highlighting the key parts. Furthermore, it can help users observe, analyze, and repair the control system more efficiently, especially in a complex system, the internal structure can be intuitively displayed and the key areas can be highlighted.

[0037] In addition, during the actual assistance process, the second voxel model can be projected onto a two-dimensional image at a preset viewing angle and can be aligned with the actual structure diagram of the control system. When the user has doubts during maintenance, the corresponding position of the second voxel model can be clicked, and in response to the user's click, the position on the two-dimensional map of the second voxel model is extracted, and the actual structure of this position is displayed to the user through the aligned structure diagram. This method can, on the one hand, help users understand the complex structure of the control system and locate problems, and on the other hand, quickly locate the actual structure, improving the maintenance efficiency.

[0038] Of course, during the actual maintenance process, real-time data of the power system also needs to be obtained through various sensors and the second voxel model needs to be updated.

[0039] In summary, the aircraft maintenance assistance method based on digital twin provided by the embodiments of the present application has the following beneficial effects, including: First, a maintenance method for aircraft maintenance based on digital twin is proposed. The highlight of this method is to construct a voxel model with color and transparency through digital modeling. The voxel model can accurately represent the geometric shape and internal structure of the complex system of the aircraft. At the same time, by adding color and transparency settings to the voxel model, the modeling can be made more realistic and the visualization effect can be better. In a complex system, transparency can be used to display different components in layers, highlighting the key parts. Furthermore, it can help users observe, analyze, and maintain the control system more efficiently, especially in a complex system, it can intuitively display the internal structure and highlight the key areas. That is, compared with the maintenance assistance using AR technology in the prior art, its accuracy and the fineness of the visualization model are higher, supporting higher-precision maintenance requirements.

[0040] Second, this method can be effectively applied to the two complex aircraft system structures of the power system and the control system. The maintenance of the power system and the control system is one of the most critical tasks in the aviation field, which is related to the core performance and flight safety of the aircraft. The embodiment of the present application can provide more accurate and real-time maintenance assistance through the technical solution of digital twin and voxel model, helping maintenance personnel effectively identify and repair potential faults, thereby improving safety, maintenance efficiency, and accuracy.

[0041] Please refer to Figure 2 , optionally, the above steps construct a first voxel model corresponding to the power system through digital modeling, including: Step 201 to Step 205.

[0042] Step 201: Obtain image data of multiple angles of the power system.

[0043] Since the power system of the aircraft is precise and complex, more delicate digital modeling is required to assist users in maintenance. Therefore, first, image data of multiple angles are obtained through an image acquisition device to ensure that all directions and angles of the entire power system can be covered.

[0044] Step 202: Convert the image data of multiple angles into a sparse point cloud model.

[0045] In the embodiment of the present application, the image data of multiple angles can be input into COLMAP to generate a sparse point cloud model.

[0046] Step 203: Convert the sparse point cloud model into an initial voxel model.

[0047] Then, the sparse point cloud model is further processed and converted into a voxel form, that is, an initial voxel model is obtained.

[0048] Optionally, in one embodiment, the expression of voxel conversion is: ; In this formula, represents a set of voxels, represents a sparse point cloud, represents the voxel size, represents floor function, represents reducing the irregularity of the point cloud. In this embodiment, the RANSAC algorithm can be specifically used to remove irregularities (outliers, noise points), and then the accuracy and precision of the data can be determined.

[0049] Step 204: Input the features of each voxel in the initial voxel model into the first multi-layer perceptron to obtain the color of the voxel, and input the features of each voxel in the initial voxel model into the second multi-layer perceptron to obtain the transparency of the voxel.

[0050] It should be noted that two multi-layer perceptrons with the same structure but different internal parameters (obtained through learning) are pre-constructed in this application. The first multi-layer perceptron is mainly used to output the color of the voxel, and the second multi-layer perceptron is mainly used to output the transparency of the voxel.

[0051] Step 205: Obtain the first voxel model based on the initial voxel model, the output result of the first multi-layer perceptron, and the output result of the second multi-layer perceptron.

[0052] Finally, combine the initial voxel model with the output results of the first multi-layer perceptron and the second multi-layer perceptron to obtain the first voxel model.

[0053] In summary, the embodiment of this application provides a complete construction method for the voxel model corresponding to the dynamic system. It should be noted that in the embodiment of this application, image data from multiple angles of the dynamic system are obtained to ensure that all directions and angles of the entire dynamic system can be covered, and details of all parts of the dynamic system can be fully captured. Then, the image data is first converted into a sparse point cloud model, and then the sparse point cloud model is converted into a voxel model. This method can ensure that the generated voxel model has richer details and more coherent structure. Finally, two multi-layer perceptrons are used to accurately predict the color and transparency of the voxels to form a first voxel model with high precision.

[0054] Please refer to Figure 3 , optionally, inputting the features of each voxel in the initial voxel model into the first multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into the second multi-layer perceptron to obtain the transparency of the voxel includes: Steps 301 to 304.

[0055] Step 301: Create high-dimensional features, medium-dimensional features, and low-dimensional features for each voxel.

[0056] Specifically, for each (belonging to the above voxel set ), a set of learnable vectors can be created , , .

[0057] Among them, represents high-dimensional features, represents a real-valued vector with a dimension of 32; represents medium-dimensional features, represents a real-valued vector with a dimension of 16, represents low-dimensional features, represents a real-valued vector with a dimension of 8.

[0058] Step 302: Based on the third multi-layer perceptron, combined with the relative distance and direction between each voxel and the image acquisition device, obtain the weights of the three features of each voxel.

[0059] Among them, the third multi-layer perceptron is configured to generate weights of learnable vectors (three features).

[0060] First of all, the formulas for the relative distance and direction between the image device and each voxel can be referred to as follows: ; ; In the above formulas, represents the voxel and the image acquisition device 's relative distance, represents the position of the image acquisition device, represents the position where the voxel center is located, represents the voxel and the direction of the image acquisition device; the symbol represents the Euclidean norm (L2 norm).

[0061] Using the third multi-layer perceptron to generate the weights of the three features can be expressed as: ; Among them, in this formula, represents the third multi-layer perceptron, , , are the weights corresponding to the high-dimensional features, medium-dimensional features, and low-dimensional features respectively; Softmax here represents weight allocation so that the weights sum to 1.

[0062] Step 303: Fuse the high-dimensional features, medium-dimensional features, low-dimensional features of each voxel and their respective corresponding weights to obtain the target features.

[0063] That is, fusion is performed according to each feature and the weight of each feature to obtain the target feature.

[0064] Step 304: Input the target feature corresponding to each voxel into the first multi-layer perceptron to obtain the color of each voxel, and input the target feature corresponding to each voxel into the second multi-layer perceptron to obtain the transparency of each voxel.

[0065] In summary, in the embodiment of the present application, high-dimensional features, medium-dimensional features, and low-dimensional features of each voxel are extracted, and after determining the weights of each feature according to the relative distance and direction between each voxel and the image acquisition device, fusion is performed to be used as the input of the first multi-layer perceptron and the second multi-layer perceptron. In this way, comprehensive voxel features can be captured, the accuracy of the output color and transparency can be improved, that is, the visualization fineness is enhanced.

[0066] Optionally, through digital modeling, a second voxel model corresponding to the control system is constructed, including: obtaining image data of multiple angles of the control system; converting the image data of multiple angles into a sparse point cloud model; converting the sparse point cloud model into an initial voxel model; inputting the features of each voxel in the initial voxel model into the fourth multi-layer perceptron to obtain the color of the voxel, and inputting the features of each voxel in the initial voxel model into the fifth multi-layer perceptron to obtain the transparency of the voxel; based on the initial voxel model, the output result of the fourth multi-layer perceptron, and the output result of the fifth multi-layer perceptron, obtaining the second voxel model.

[0067] It should be noted that the process of constructing the second voxel model can refer to the implementation process of constructing the first voxel model in the foregoing embodiment. The processing processes of the two are basically the same and can be mutually referred to, so it will not be elaborated here.

[0068] In summary, the embodiment of the present application provides a complete construction method for the voxel model corresponding to the control system. It should be noted that in the embodiment of the present application, image data of multiple angles of the control system is obtained to ensure that all direction angles of the entire control system can be covered, and the details of each part of the control system can be fully captured. Then the image data is first converted into a sparse point cloud model, and then the sparse point cloud model is converted into a voxel model. This method can ensure that the generated voxel model has richer details and more coherent structure. Finally, two multi-layer perceptrons are used to accurately predict the color and transparency of the voxel to form a second voxel model with high precision.

[0069] Optionally, input the features of each voxel in the initial voxel model into a fourth multi-layer perceptron to obtain the color of the voxel, and input the features of each voxel in the initial voxel model into a fifth multi-layer perceptron to obtain the transparency of the voxel, including: creating high-dimensional features, medium-dimensional features, and low-dimensional features for each voxel; based on a sixth multi-layer perceptron, combining the relative distance and direction between each voxel and the image acquisition device to obtain the weights of the three features of each voxel; fusing the high-dimensional features, medium-dimensional features, low-dimensional features of each voxel and their respective corresponding weights to obtain target features; inputting the target features corresponding to each voxel into the fourth multi-layer perceptron to obtain the colors of each voxel, and inputting the target features corresponding to each voxel into the fifth multi-layer perceptron to obtain the transparency of each voxel.

[0070] It should be noted that the above fusion process can also refer to the description of the dynamic system in the foregoing embodiments. For the same parts, reference can be made to each other, and details will not be elaborated here.

[0071] In summary, in the embodiments of the present application, high-dimensional features, medium-dimensional features, and low-dimensional features of each voxel are extracted, and after determining the weights of each feature according to the relative distance and direction between each voxel and the image acquisition device, they are fused to be used as the inputs of the fourth multi-layer perceptron and the fifth multi-layer perceptron. In this way, comprehensive voxel features can be captured, the accuracy of the output color and transparency can be improved, that is, the visualization fineness can be enhanced.

[0072] Optionally, the method further includes: constructing a voxel model of other components of the aircraft to be detected; constructing a component association graph based on the mutual control logic relationship of other components of the aircraft to be detected; when the first control instruction cannot be transmitted to the target component to complete the control task, based on the component association graph, determining other voxel models of components on the transmission line of the first control instruction and displaying them.

[0073] It should be noted that since the maintenance standard of the control system is whether the target component can be controlled by a control instruction, in addition to constructing the voxel model, all components are also constructed into a graph according to preset rules (generally speaking, the aircraft manufacturer will provide the mutual control logic relationship between components, and this relationship can be used as the construction rule of the component association graph). When the user performs maintenance on the control system, the method that can be adopted is to send a control instruction from one component to another component and check whether the target component can complete the task expressed by the instruction. Most of the instruction transmissions between components are not direct transmissions and usually pass through intermediate components, which are mostly sensor components. When the target component cannot complete the expected task, the voxel model of the associated component will be given according to the component association graph to facilitate the user to quickly troubleshoot.

[0074] Exemplarily, for instance, the air compressor and the turbine are strongly coupled. If there is a problem with the turbine, the air compressor will be affected, and then the information of the air compressor is also pushed.

[0075] Optionally, the method further includes: in response to the maintenance trigger operation being a maintenance trigger operation for the airframe structure of the aircraft to be detected, acquiring image information of the airframe structure; the airframe structure includes at least one of the fuselage, wings, tail, and landing gear; inputting the image information into a pre-trained image segmentation and recognition model to obtain a defect mask and an airframe structure mask; calculating the damage degree of the aircraft to be detected based on the defect mask, the airframe structure mask, the size and shape of the airframe of the aircraft to be detected, and the proportion of the photographed airframe structure in the complete airframe in the image information.

[0076] In the embodiment of the present application, Efficientvit can be used as the image segmentation and recognition model. The image information is sent into the image segmentation and recognition model to obtain a defect mask and an airframe structure mask . Since information such as the size and shape of the airframe of the aircraft to be detected can be directly obtained, the damage degree of the airframe structure of the aircraft to be detected can be calculated. The calculation formula is as follows: ; In this formula, represents the loss length, represents the proportion of the photographed airframe structure in the complete airframe in the image. The calculation formula of is as follows: In this formula, represents the true length of the complete airframe, represents the projection calculation function, the input of is the coordinates of the point with the largest horizontal axis coordinate and the coordinates of the point with the smallest horizontal axis coordinate in . The calculation details of are as follows: after inputting the above coordinates, the above pixel points are projected into the real space according to the internal parameters of the image acquisition device, and then the distance of the real space coordinates is calculated as the length of the photographed part of the image. When using

[0077] for projection, a depth information is required. In this embodiment, it is realized by using a laser rangefinder installed at the lower end of the image acquisition device.

[0078] In addition, considering that an aircraft may be struck by a bird, and the impact of a bird strike depends on the location and severity of the bird strike. Bird strikes at different locations will have different impacts on the operation of the aircraft, but AR technology can only display some historical data and cannot simulate or evaluate the actual impact of a bird strike at a specific location on the aircraft. As a result, the AR-based method cannot complete the simulation of this impact and lacks sufficient accurate practical guidance. The present application also provides the following embodiments to solve the above problems: Optionally, when the maintenance trigger operation is a maintenance trigger operation for the fuselage and wings in the airframe structure of the aircraft to be detected, the method further includes: combining the damage degree, wing projection area, and airframe structure mask of the aircraft to be detected, and outputting the lift of the aircraft to be detected after identifying defects.

[0079] Among them, the calculation formula for lift is: ; In this formula, represents lift, represents air density, represents the relative fluid velocity, represents the wing projection area, represents the lift coefficient, represents the weight obtained according to , and different weights can be set for different positions of the airframe.

[0080] That is, the embodiments of the present application also consider the impact of the defect location on lift to further assist the user in maintenance guidance.

[0081] Optionally, before receiving the maintenance trigger operation of the user for the aircraft to be detected, the method further includes: obtaining the number of the aircraft to be detected input by the user; based on the number of the aircraft to be detected, obtaining the basic information of the aircraft corresponding to the number of the aircraft to be detected from the maintenance database; based on the basic information of the aircraft corresponding to the number of the aircraft to be detected, determining the maintenance process of the aircraft to be detected; among them, different maintenance processes correspond to different maintenance trigger operations.

[0082] Specifically, the user can first enter the number (ID) of the aircraft to be detected. Usually, the basic information of each aircraft, such as aircraft number, model, aircraft component signal, aircraft age, aircraft mileage, etc., will be stored in the maintenance database. Therefore, after obtaining the number of the aircraft to be detected input by the user, the basic information of the aircraft corresponding to the number of the aircraft to be detected is extracted from the maintenance database. It should be noted that in the embodiments of the present application, for aircraft of different models, different ages, and different flight mileages, their maintenance processes are different. Therefore, in the embodiments of the present invention, the maintenance processes in the maintenance database are matched using the model, aircraft age, and flight mileage as constraint conditions (there are generally standard maintenance guides for aircraft maintenance), and the recommended guidance process is obtained.

[0083] It can be seen that through this method, different aircraft states can be adapted, the rationality and efficiency of maintenance and repair can be improved, and resource utilization can be optimized.

[0084] In addition, after the user completes the maintenance and repair, it is necessary to test the target to be maintained to ensure normal function. In this process, the parameters of some functional targets will be recorded through special equipment. The recorded parameters will also be synchronized. Since the relationships between the structural components of the maintenance target itself and the relationships between the maintenance target and the remaining components have been constructed, the flow of data between components or between structural components can be simulated and calculated, and it can be judged whether the component data feedback is consistent with the actual situation. Suppose there is an association between component A, component B, and component C after modeling. After the user completes the maintenance and repair of component A, the data generated when component A works is A1. When performing simulation calculations, A1 is input into component B and component C in sequence, and the data B1 and C1 generated by component B and component C are calculated. When the user tests component B and component C, the collected data will be compared with B1 and C1 to judge the consistency.

[0085] Please refer to Figure 4 , based on the same inventive concept, an embodiment of the present application provides a digital twin-based maintenance assistance system 400 for aircraft maintenance, including: a receiving module 401, configured to receive a maintenance trigger operation of a user for an aircraft to be detected; a first construction module 402, configured to, in response to the maintenance trigger operation being a maintenance trigger operation for the power system of the aircraft to be detected, construct a first voxel model corresponding to the power system in a digital modeling manner to assist in maintenance; wherein, the first voxel model has different colors and transparencies at different positions; the power system includes at least one of an engine and an auxiliary power unit; a second construction module 403, configured to, in response to the maintenance trigger operation being a maintenance trigger operation for the control system of the aircraft to be detected, construct a second voxel model corresponding to the control system in a digital modeling manner to assist in maintenance; wherein, the second voxel model has different colors and transparencies at different positions; the control system includes at least one of a mechanical control system, a fly-by-wire control system, and a lift augmentation device.

[0086] Please refer to Figure 5 , based on the same inventive concept, an embodiment of the present application provides a module housing of an electronic device 500 that applies the above method. The electronic device 500 includes: at least one processor 501 ( Figure 5 only one is shown), a memory 502, and a computer program 503 stored in the memory 502 and executable on at least one processor 501. When the processor 501 executes the computer program 503, the steps of the method in any of the foregoing embodiments are implemented.

[0087] The electronic device 500 can be a server, a personal computer, a laptop computer, and so on.

[0088] Those skilled in the art can understand that Figure 5 this is merely an example of the electronic device 500, which does not constitute a limitation on the electronic device 500. It may include more or fewer components than shown in the figure, or combine certain components, or different components.

[0089] The so-called processor 501 can be a central processing unit (CPU). The processor 501 can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0090] In some embodiments, the memory 502 can be an internal storage unit of the electronic device 500, such as the hard disk or memory of the electronic device 500. In some other embodiments, the memory 502 can also be an external storage device of the electronic device 500, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 500. Further, the memory 502 can also include both the internal storage unit and the external storage device of the electronic device 500.

[0091] It should be noted that for the above content of the system, device, etc., since it is based on the same concept as the method embodiment of the present application, the modules designed by the system and the steps executed by the device and the technical effects brought thereby can all be seen in the method embodiment part, and will not be elaborated here.

[0092] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0093] An embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented.

[0094] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the foregoing method embodiments when executed.

[0095] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above method embodiments of the present application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc.

[0096] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0097] Those of ordinary skill in the art will realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0098] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network device and method can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the apparatus or unit can be in an electrical, mechanical or other form.

[0099] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0100] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included in the protection scope of this application.

Claims

1. A maintenance assistance method based on digital twin, characterized in that: include: Receive maintenance trigger operations from users for the aircraft to be inspected; In response to the maintenance trigger operation being a maintenance trigger operation for the power system of the aircraft to be inspected, a first voxel model corresponding to the power system is constructed by digital modeling to assist maintenance; wherein the first voxel model has different colors and transparencies at different positions; the power system includes at least one of an engine and an auxiliary power unit; In response to the maintenance trigger operation being a maintenance trigger operation for the control system of the aircraft to be inspected, a second voxel model corresponding to the control system is constructed through digital modeling to assist in maintenance; wherein the second voxel model has different colors and transparencies at different positions; and the control system includes at least one of a mechanical control system, an electric control system, and a high-lift device.

2. The digital twin-based maintenance assistance method according to claim 1 is characterized in that: The step of constructing a first voxel model corresponding to the power system by digital modeling includes: Acquire image data of the power system at multiple angles; Convert the image data from the multiple angles into a sparse point cloud model; Converting the sparse point cloud model into an initial voxel model; Inputting the feature of each voxel in the initial voxel model into a first multi-layer perceptron to obtain the color of the voxel, and inputting the feature of each voxel in the initial voxel model into a second multi-layer perceptron to obtain the transparency of the voxel; The first voxel model is obtained based on the initial voxel model, the output result of the first multi-layer perceptron and the output result of the second multi-layer perceptron.

3. The digital twin-based maintenance assistance method according to claim 2 is characterized in that: The step of inputting the feature of each voxel in the initial voxel model into a first multi-layer perceptron to obtain the color of the voxel, and inputting the feature of each voxel in the initial voxel model into a second multi-layer perceptron to obtain the transparency of the voxel comprises: Create high-dimensional features, medium-dimensional features, and low-dimensional features for each voxel; Based on the third multi-layer perceptron, the weights of the three features of each voxel are obtained by combining the relative distance and direction between each voxel and the image acquisition device; The high-dimensional features, medium-dimensional features, low-dimensional features and their corresponding weights of each voxel are fused to obtain the target features; The target feature corresponding to each voxel is input into the first multi-layer perceptron to obtain the color of each voxel, and the target feature corresponding to each voxel is input into the second multi-layer perceptron to obtain the transparency of each voxel.

4. The digital twin-based maintenance assistance method according to claim 1 is characterized in that: The method of constructing a second voxel model corresponding to the control system by digital modeling includes: Acquiring image data of the control system at multiple angles; Convert the image data from the multiple angles into a sparse point cloud model; Converting the sparse point cloud model into an initial voxel model; Inputting the feature of each voxel in the initial voxel model into a fourth multi-layer perceptron to obtain the color of the voxel, and inputting the feature of each voxel in the initial voxel model into a fifth multi-layer perceptron to obtain the transparency of the voxel; The second voxel model is obtained based on the initial voxel model, the output result of the fourth multi-layer perceptron and the output result of the fifth multi-layer perceptron.

5. The digital twin-based locomotive maintenance assistance method according to claim 4 is characterized in that: Inputting the feature of each voxel in the initial voxel model to a fourth multi-layer perceptron to obtain the color of the voxel, and inputting the feature of each voxel in the initial voxel model to a fifth multi-layer perceptron to obtain the transparency of the voxel, including: Create high-dimensional features, medium-dimensional features, and low-dimensional features for each voxel; Based on the sixth multi-layer perceptron, the weights of the three features of each voxel are obtained by combining the relative distance and direction of each voxel to the image acquisition device; The high-dimensional features, medium-dimensional features, low-dimensional features and their corresponding weights of each voxel are fused to obtain the target features; The target feature corresponding to each voxel is input into the fourth multi-layer perceptron to obtain the color of each voxel, and the target feature corresponding to each voxel is input into the fifth multi-layer perceptron to obtain the transparency of each voxel.

6. The digital twin-based locomotive maintenance assistance method according to claim 5 is characterized in that: The method further comprises: Constructing voxel models of other components of the aircraft to be detected; Based on the mutual control logic relationship of other components of the aircraft to be inspected, construct a component association map; In response to the first control instruction being unable to be transmitted to the target component to complete the control task, based on the component association map, voxel models of other components on the transmission line of the first control instruction are determined and displayed.

7. The digital twin-based maintenance assistance method according to claim 1 is characterized in that: The method further comprises: In response to the maintenance trigger operation being a maintenance trigger operation for the fuselage structure of the aircraft to be inspected, image information of the fuselage structure is acquired; the fuselage structure includes at least one of a fuselage, a wing, a tail, and a landing gear; Inputting the image information into a pre-trained image segmentation and recognition model to obtain a defect mask and an aircraft structure mask; The damage degree of the aircraft to be inspected is calculated based on the defect mask, the airframe structure mask, the size and shape of the aircraft to be inspected, and the proportion of the airframe structure captured in the image information to the complete airframe.

8. The digital twin-based locomotive maintenance assistance method according to claim 7 is characterized in that: When the maintenance trigger operation is a maintenance trigger operation for a fuselage and a wing in the fuselage structure of the aircraft to be inspected, the method further includes: The lift of the aircraft to be inspected after the defect is identified is outputted based on the damage degree of the aircraft to be inspected, the wing projection area, and the fuselage structure mask.

9. The digital twin-based locomotive maintenance assistance method according to claim 1, characterized in that: Before receiving the maintenance trigger operation of the user for the aircraft to be inspected, the method further includes: Get the number of the aircraft to be detected input by the user; Based on the serial number of the aircraft to be inspected, acquiring basic aircraft information corresponding to the serial number of the aircraft to be inspected from a maintenance database; Based on the basic aircraft information corresponding to the serial number of the aircraft to be detected, the maintenance process of the aircraft to be detected is determined; wherein different maintenance processes correspond to failed maintenance trigger operations.

10. A maintenance auxiliary system based on digital twin, characterized in that: include: A receiving module, used for receiving a user's maintenance trigger operation for the aircraft to be inspected; A first construction module is used for constructing a first voxel model corresponding to the power system to assist in the maintenance in response to the maintenance trigger operation being a maintenance trigger operation for the power system of the aircraft to be inspected, by means of digital modeling; wherein the first voxel model has different colors and transparencies at different positions; and the power system includes at least one of an engine and an auxiliary power unit; The second construction module is used to respond to the maintenance trigger operation being a maintenance trigger operation for the control system of the aircraft to be inspected, and to construct a second voxel model corresponding to the control system through digital modeling to assist in maintenance; wherein the second voxel model has different colors and transparencies at different positions; and the control system includes at least one of a mechanical control system, an electric control system, and a high-lift device.

Images