Aviation pipeline-oriented augmented reality auxiliary maintenance method and related device

Abstract

The invention belongs to the technical field of augmented reality auxiliary maintenance, and discloses an augmented reality auxiliary maintenance method for aviation pipelines and a related device. The aviation pipeline-oriented augmented reality auxiliary maintenance system comprises a target identification data set acquisition and expansion module, a fault target identification model construction and pre-training module, a virtual and real information matching and fusion module, a maintenance process database construction module, a maintenance process text generation module and an AR visual interface design module. According to the invention, technologies such as defective pipeline identification, maintenance target space positioning, maintenance guide information visualization and the like are combined with the augmented reality technology, so that the pipeline maintenance efficiency and accuracy of workers are improved through the augmented reality technology in a complex structure scene, the maintenance efficiency and quality can be improved, and the maintenance cost can be reduced.

Description

Technical Field

[0001] This invention relates to the field of augmented reality-assisted maintenance technology, and more specifically to an augmented reality-assisted maintenance method and related apparatus for aviation pipelines. Background Technology

[0002] As a crucial component of aircraft engines, the aviation piping system connects various aircraft parts to transport fluids such as fuel, hydraulic fluid, and air, serving as a critical infrastructure for the aircraft's various functions. A survey by General Electric (GE) on in-flight engine failures revealed that approximately 50% of failures were caused by external piping damage, about 27% by improper maintenance, and only about 6% were due to major engine components. Therefore, any piping failure can affect the normal operation of the aircraft and even lead to serious safety incidents, making its maintenance and support extremely demanding.

[0003] Meanwhile, modern aircraft have increasingly complex air piping systems and confined maintenance spaces. Maintenance personnel need extensive professional knowledge and experience to accurately locate faults and perform repairs, resulting in high training costs. The maintenance process also requires consulting numerous technical documents and maintenance manuals, making the procedures cumbersome and increasing the difficulty of maintenance while also increasing the risk of human error. Due to the high demands for maintenance efficiency and quality, traditional maintenance methods are inefficient and can no longer meet the needs of modern aviation operations.

[0004] Therefore, a new and effective auxiliary maintenance method is needed to improve the efficiency and accuracy of pipeline maintenance for workers in complex structural scenarios by using augmented reality technology, and to solve the problems of difficulty in identifying pipeline problems and difficulty in confirming their location. Summary of the Invention

[0005] The purpose of this invention is to provide an augmented reality-assisted maintenance method and related device for aviation pipelines to overcome the problems existing in the prior art. This invention combines technologies such as problem pipeline identification, spatial positioning of maintenance targets, and visualization of maintenance guidance information with augmented reality technology to improve the efficiency and accuracy of pipeline maintenance for workers in complex structural scenarios. This not only improves maintenance efficiency and quality but also reduces maintenance costs.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides an augmented reality-assisted maintenance method for aviation pipelines, comprising the following steps: Obtain and construct a target identification dataset for faulty air pipelines, and expand the target identification dataset; Construct a fault target identification model, pre-train the fault target identification model, and obtain aviation pipeline fault information through the pre-trained fault target identification model; A virtual model library is built to obtain the geometric and texture information of components in aviation pipeline fault information. The geometric and texture information is matched with the features in the virtual model library to obtain virtual and real information. The virtual and real information is registered and fused during registration to obtain virtual and real fused information. Establish a 3D model of maintenance process information based on MBD, and construct a maintenance process database based on the 3D model of maintenance process information using MySQL; The maintenance process database is clustered using a vector clustering algorithm, and maintenance process text is generated from the clustered maintenance process database. Design an AR visualization interface based on the maintenance process text to achieve augmented reality-assisted maintenance; Furthermore, the acquisition and construction of the target identification dataset for the faulty air pipeline, and the expansion of the target identification dataset, specifically include: Images of air pipelines under different fault conditions are collected and labeled to obtain image label data of air pipelines under different fault conditions. A target recognition dataset of faulty air pipelines is constructed using the image label data of air pipelines under different fault conditions. The target recognition dataset is expanded using the Mosaic data augmentation method to obtain the expanded target recognition dataset. Furthermore, the construction of the fault target identification model, the pre-training of the fault target identification model, and the acquisition of aviation pipeline fault information through the pre-trained fault target identification model specifically include: A YOLOv9 target recognition deep learning neural network was built, and a lightweight network architecture GELAN based on gradient path planning and an ECA attention mechanism module were introduced to obtain a fault target recognition model. The fault target recognition model was pre-trained, and the expanded faulty air pipeline target recognition dataset was trained and evaluated on the pre-trained weights using transfer learning to obtain a pre-trained fault target recognition model. Air pipeline fault information was obtained in real time online through the pre-trained fault target recognition model. Furthermore, the process of building a virtual model library, acquiring geometric and texture information of components from aviation pipeline fault information, matching the geometric and texture information of the components with features in the virtual model library to obtain virtual and real information, registering the virtual and real information, and fusing the virtual and real information during registration to obtain virtual and real fused information, specifically includes: A virtual model library is built to acquire features from images and point cloud data of aviation pipelines. Based on these features, geometric and texture information of components in aviation pipeline fault information is obtained. The geometric and texture information is parameterized and matched with features in the virtual model library to obtain virtual and real information. The virtual and real information is registered, and during registration, occlusion is detected. Simultaneously, the background and foreground of the virtual and real information are segmented to obtain boundary information for occlusion processing. Based on the boundary information, compensation processing is performed on the occluded parts to obtain virtual and real fusion information. Furthermore, the establishment of a 3D model of maintenance process information based on MBD, and the construction of a maintenance process database based on the 3D model of maintenance process information using MySQL, specifically includes: A 3D model of maintenance process information based on MBD is established. The 3D model of maintenance process information includes several maintenance process information. The virtual and real information is visualized through the 3D model of maintenance process information based on MBD. Based on MySQL, the several maintenance process information are set as table structures, and the specific attributes corresponding to each maintenance process information are set as columns of the table structure. Foreign keys are used to realize the association between several maintenance process information, thus obtaining the maintenance process database. Furthermore, the step of clustering the maintenance process database using a vector clustering algorithm and generating maintenance process text from the clustered maintenance process database specifically includes: The maintenance process information in the maintenance process database is decomposed into semantic vectors. The semantic vectors expressing the same maintenance process are clustered together using a vector clustering algorithm to obtain a clustered maintenance process database, which is used for filtering and classification. Then, maintenance process text is generated from the clustered maintenance process database. Furthermore, the step of designing an AR visualization interface based on the maintenance process text to achieve augmented reality-assisted maintenance specifically includes: Based on the maintenance process text, 3D models of maintenance scenarios and parts are created, and AR visualization interfaces are designed based on the 3D models to achieve augmented reality-assisted maintenance.

[0007] Secondly, the present invention provides an augmented reality-assisted maintenance system for aviation pipelines, comprising: The target recognition dataset acquisition and expansion module is used to acquire and construct a target recognition dataset for faulty air pipelines and to expand the target recognition dataset. The fault target identification model construction and pre-training module is used to construct a fault target identification model, pre-train the fault target identification model, and obtain aviation pipeline fault information through the pre-trained fault target identification model. The virtual-real information matching and fusion module is used to build a virtual model library, obtain the geometric and texture information of components in the aviation pipeline fault information, match the geometric and texture information with the features in the virtual model library to obtain virtual-real information, register the virtual-real information, and fuse the virtual-real information during registration to obtain virtual-real fused information. The maintenance process database construction module is used to establish a 3D model of maintenance process information based on MBD, and to construct a maintenance process database based on the 3D model of maintenance process information using MySQL. The maintenance process text generation module is used to cluster the maintenance process database using a vector clustering algorithm, and then generate maintenance process text from the clustered maintenance process database. The AR visualization interface design module is used to design AR visualization interfaces based on maintenance process texts, thereby enabling augmented reality-assisted maintenance.

[0008] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0009] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0010] The above technical solution has the following advantages or beneficial effects: Firstly, this invention provides an augmented reality-assisted maintenance method for aviation pipelines. By constructing and expanding a target recognition dataset, it can accurately acquire aviation pipeline fault information and improve the accuracy of fault identification. By building a virtual model library and matching and fusing virtual and real information, it allows maintenance personnel to understand the fault situation more intuitively. By establishing a 3D model and database of maintenance process information based on MBD and combining it with vector clustering algorithm to generate maintenance process text, it provides scientific guidance for maintenance. Finally, based on the maintenance process text, it designs an AR visualization interface to realize augmented reality-assisted maintenance, reduce maintenance difficulty and error rate, improve maintenance efficiency and quality, and ensure the safe operation of aviation pipelines.

[0011] Furthermore, collecting and labeling images of air pipelines under different fault conditions can comprehensively cover various fault characteristics, making the constructed target recognition dataset more representative and complete, laying the foundation for accurate fault identification. Expanding the dataset using Mosaic data augmentation can enrich information such as background, angle, and scale of the images, increasing data diversity and effectively improving the model's generalization ability. This not only helps the model accurately identify air pipeline faults in complex and ever-changing real-world scenarios but also reduces overfitting problems caused by insufficient or limited data, improving the accuracy and reliability of fault diagnosis, and thus ensuring the efficient and precise conduct of air pipeline maintenance work.

[0012] Furthermore, by building a YOLOv9 target recognition deep learning neural network and introducing a lightweight GELAN architecture, the model complexity and computational load are reduced, while the running speed is improved, facilitating real-time detection. The addition of the ECA attention mechanism module enables the model to focus on key features, enhancing its ability to capture the characteristics of aviation pipeline faults. By adopting transfer learning and leveraging pre-trained weights, the model can quickly adapt to new data, reducing training time and data requirements. Real-time online acquisition of fault information allows for timely feedback on pipeline fault conditions, enabling maintenance personnel to respond quickly, effectively shortening fault diagnosis time and improving aviation pipeline maintenance efficiency. The optimized and improved algorithm can more accurately and effectively identify different problematic pipelines.

[0013] Furthermore, by building a virtual model library and extracting features from aviation pipeline images and point cloud data, the geometric and texture information of components can be accurately obtained, providing a reliable basis for subsequent matching. Parametric processing makes the information more standardized and universal, improving matching efficiency and accuracy. After obtaining the virtual and real information through matching, registration and fusion are performed. During the process, occlusion is detected and the background and foreground are segmented. Boundary information is obtained and compensation processing is implemented, effectively solving the problem of information loss caused by occlusion during virtual-real fusion. This makes the fused information more complete and accurate, helping maintenance personnel to understand the aviation pipeline fault status more clearly and comprehensively, providing them with more accurate maintenance guidance, and thus improving maintenance quality and efficiency.

[0014] Furthermore, a 3D model of maintenance process information based on MBD is established, which can integrate rich maintenance process information and present it intuitively in a 3D form, providing maintenance personnel with a clear and comprehensive reference, which helps them quickly understand the maintenance process and key points. The virtual and real information visualization display through this model further enhances the intuitiveness and comprehensibility of the information. The maintenance process database is built based on MySQL, and the maintenance process information is reasonably set as a table structure and columns. Foreign keys are used to realize information association, making the data storage standardized and orderly, and facilitating efficient querying and management.

[0015] Furthermore, decomposing maintenance process information into semantic vectors can accurately capture the semantic features in the information, providing a reliable foundation for subsequent clustering. Using vector clustering algorithms to cluster semantic vectors expressing the same maintenance process can efficiently achieve the screening and classification of maintenance process information, making the database structure clearer and more organized, greatly improving the efficiency of information retrieval and management. Generating maintenance process text based on the clustered database can provide maintenance personnel with more targeted and systematic maintenance guidance, reducing information search time, lowering maintenance difficulty, helping to improve the accuracy and efficiency of aviation pipeline maintenance, ensuring the safety and stability of aviation operations, and reducing maintenance costs.

[0016] Furthermore, 3D modeling based on maintenance process documents can accurately recreate maintenance scenarios and parts, allowing maintenance personnel to have an intuitive and clear understanding of the objects to be maintained in advance and plan maintenance steps accordingly. AR visualization interface design based on 3D modeling seamlessly integrates virtual information with real-world scenes. Maintenance personnel can see overlaid maintenance guidance information in a realistic environment, which not only enhances the interactivity and immersion of the maintenance process but also allows them to obtain maintenance information more conveniently and accurately, effectively avoiding operational errors, significantly shortening maintenance time, improving maintenance quality, and providing strong support for the safe and stable operation of aviation pipelines.

[0017] Secondly, this invention provides an augmented reality-assisted maintenance system for aviation pipelines. The system includes a target recognition dataset acquisition and expansion module, providing rich and comprehensive data support for accurate fault identification; a fault target recognition model construction and pre-training module, enabling rapid and accurate acquisition of aviation pipeline fault information; a virtual-real information matching and fusion module, allowing maintenance personnel to intuitively understand fault details; a maintenance process database construction module, achieving standardized storage and efficient management of maintenance process information; a maintenance process text generation module, providing targeted guidance for maintenance; and an AR visualization interface design module, enhancing the interactivity and immersion of maintenance. These modules work collaboratively to improve the efficiency and quality of aviation pipeline maintenance, reduce maintenance costs and risks, and ensure aviation operational safety.

[0018] Thirdly, the present invention provides a computer device that, through a processor executing a specific computer program, can efficiently implement the steps of the method of the present invention. When performing data processing tasks, the computer device can accurately perform numerical calculations and logical judgments, avoiding errors caused by human factors. At the same time, since the computer program has high stability and reliability, it can ensure the accuracy and consistency of the data processing results.

[0019] Fourthly, the present invention provides a computer-readable storage medium. By programming the steps of the method of the present invention into a computer program and storing it on the computer-readable storage medium, users can easily load these programs onto any compatible computer device and execute them without rewriting or converting the code, which greatly improves the convenience and flexibility of program execution. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating an augmented reality-assisted maintenance method for aviation pipelines according to the present invention. Figure 2 This is a flowchart illustrating an embodiment of the present invention; Figure 3 This is a framework diagram of the YOLOv9 pipeline identification algorithm of the present invention; Figure 4 This is a 3D modeling diagram of the present invention; Figure 5 This is a schematic diagram illustrating the NLP template text generation method of the present invention. Figure 6 This is a flowchart of the QEM algorithm of the present invention; Figure 7 This is a three-dimensional registration diagram of the present invention; Figure 8 This is a schematic diagram of the spatial positioning of the present invention; Figure 9 This is a schematic diagram of the structure of the computer device of the present invention. Detailed Implementation

[0021] The present invention will be further described in detail below with reference to specific embodiments. These descriptions are for explanation purposes only and are not intended to limit the scope of the invention. To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention. It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] Example: See Figure 1 and Figure 2 This invention provides an augmented reality-assisted maintenance method for aviation pipelines, comprising the following steps: Step 1: Obtain and construct a target identification dataset for faulty air pipelines, and expand the target identification dataset; Specifically, images of air pipelines under different fault conditions are collected and labeled to obtain image label data for air pipelines under different fault conditions. A target recognition dataset for faulty air pipelines is constructed using the image label data for air pipelines under different fault conditions. The target recognition dataset for faulty air pipelines is then expanded using Mosaic (Mosaic Data Augmentation) data augmentation to obtain an expanded target recognition dataset. This directly increases the number of training samples several times in size, improving the generalization of the training model. Furthermore, the robustness of the neural network can be improved by adding random noise and other methods.

[0023] Step 2, Problem Pipeline Identification Based on Multi-Attention Mechanism Deep Learning Neural Network: Construct a fault target identification model, pre-train the fault target identification model, and obtain aviation pipeline fault information through the pre-trained fault target identification model. The specific steps are as follows: Step 2.1, see Figure 3We built a YOLOv9 (You Only Look Once version 9, YOLO ninth generation object detection algorithm) deep learning neural network for object recognition, and introduced a lightweight network architecture GELAN (General Efficient Layer Aggregation Network) based on gradient path planning. By combining two gradient path planning neural networks, CSPNet (Cross Stage Partial Network) and ELAN (Efficient Layer Aggregation Network Module / Stacked Convolutional Feature Fusion Module), we still used traditional convolution techniques and achieved higher parameter utilization than depthwise convolution. Step 2.2: Introduce the ECA attention mechanism module and add it to the backbone module of the original YOLOv9 network to maximize the role of the attention mechanism and enhance the ability of the fault target recognition model (original algorithm) to extract important features, thus obtaining the fault target recognition model. Specifically, the mathematical principles of the ECA attention mechanism are as follows: (1) Global Average Pooling (GAP): Global average pooling is performed on each channel of the input feature map to obtain a channel description vector. The specific formula is as follows: ; In the formula: S c This represents the output of the pooling operation; This indicates that the c-th channel of the input feature map is at position The value of ; c represents the number of channels; i The horizontal coordinate representing the position; j The vertical coordinate representing the position; H , indicates the height of the feature map; W Indicates the width of the feature map; (2) One-dimensional convolution (1D Conv): Perform one-dimensional convolution on the channel description vector to generate channel attention weights. The specific formula is as follows:

[0024] in, w Indicates channel attention weights; The sigmoid activation function is represented by Conv1D; Conv1D represents a one-dimensional convolution operation. r Represents the channel description vector; Step 2.3: Pre-train the fault target recognition model. Use transfer learning to train and evaluate the expanded faulty airway target recognition dataset on the pre-trained weights to obtain the pre-trained fault target recognition model. Use the pre-trained fault target recognition model to obtain airway fault information in real time online.

[0025] Step 3: Based on 3D registration technology, virtual-real occlusion handling methods, and human-computer interaction interface design in virtual-real fusion, address the matching problem of virtual information of aviation components to real-world scenes: Build a virtual model library, acquire geometric and texture information of components from aviation pipeline fault information, match the geometric and texture information with features in the virtual model library to obtain virtual-real information, register the virtual-real information, and fuse the virtual-real information during registration to obtain fused virtual-real information. The specific steps are as follows: Step 3.1: Build a virtual model library using Unity. Utilize the depth sensor and camera of HoloLens (Microsoft Holographic Glasses) to acquire features of real-world aviation pipeline images and point cloud data. Based on these features, obtain the geometric and texture information of components in the aviation pipeline fault information. Parameterize the geometric and texture information, and match the parameterized geometric and texture information with features in the virtual model library to obtain virtual-real information. Based on this, research 3D registration technology to match real-world aviation components with features in the virtual model library, achieving virtual-real registration. Step 3.2: Register the virtual and real information. During registration, the occlusion of the virtual and real information is detected. At the same time, the background and foreground of the virtual and real information are segmented to obtain the boundary information of the occlusion processing. Based on the boundary information, compensation processing is performed on the occluded part to obtain the virtual and real fusion information, ensuring the smooth integration of virtual information and real environment, so as to achieve a more natural virtual and real fusion effect. Specifically, occlusion during the virtual-real fusion process is obtained using HoloLens, and occlusion detection and segmentation methods of HoloLens during registration are studied. Virtual-real fusion-assisted installation uses QR (Quick Response) codes for 3D registration. QR code registration is a type of marker-based registration, where artificially generated patterns are placed in the scene, and the camera identifies the patterns to determine the pose and content. The QR code mainly consists of two areas: a functional area and an encoding area. The image in the functional area contains position detection graphics, separators, positioning graphics, and correction graphics; the image in the encoding area contains format information, data, and error correction codewords. See Figure 7 With the front as z Positive axis direction, directly above is y The positive direction of the axis, and the right side is x Establish a spatial rectangular coordinate system along the positive axis, where the origin of the coordinate system is... O Let the origin of the camera be the point on the screen. The cone-shaped area radiating outward from this origin is the field of view that the camera can observe. The observable external field of view is defined by the angle... θWithin this area, the vertical line on the right side of the field of view represents the image captured by the camera, including the QR code; it is parallel to... x The axis; the solid trapezoid represents the shape of the QR code in the image when viewed from the camera's perspective, currently tilted to the right; if we assume z From the axis to the point O The distance to a point is z Then the camera at this point is perpendicular to zx View height on a plane h for: h = ; Let the actual side length of the QR code (square) be... L The captured image is high h Simultaneously, the program can measure the coordinates of the four vertices of the QR code in the captured image, thereby obtaining the height of the QR code on the left and right sides in that view. L 1 and L 2, then we can conclude: Li (i=1,2)= ; The left and right edges of the QR code correspond to the distance. y The axis distance refers to the distance between the left and right sides of the QR code and the camera. z Axis coordinates z 1 and z 2 is: ; ; The center point of the QR code z Axis coordinates z 0 is: ; In the top view, the thick solid line represents the tilted view of the QR code, based on its original side length. L and left and right z Axis coordinate difference z 2- z 1. Obtain the tilt angle of the QR code. β for: β = ; In addition to the above, to locate the QR code's position... z 0 and βIn addition, we need to know a parameter of the QR code in space. When the QR code changes within the captured image, the relative position of its midpoint remains unchanged. Therefore, if we know the coordinates of the QR code's center point, we can determine its spatial location. The intersection of the diagonals of a square remains the center of the square after the change, and we can obtain the coordinates of the four intersection points programmatically. Thus, we can analyze and determine the coordinates of the QR code's center position. Let the four vertices of the QR code in the image be... A 1. A 2. A 3 and A 4. Since its coordinates on the image plane are known, the intersection of the two diagonals can be determined, which is the center point of the QR code. Q of x The axis coordinates, their positions are as follows: Figure 8 As shown, the ratio of line segment CQ to line segment CD is equal to that shown in the actual image. Q of x Axis coordinates xc Distance to the center line of the image xc - Distance from half the width of the image The ratio, that is: ; Meanwhile, the ratio of the tangent of the horizontal viewpoint to the tangent of the vertical viewpoint is equal to the ratio of the image's width to its height, i.e.: ; From this, we can derive the angle. : ; At this point, the location information of the QR code can be obtained. α , β and z By using three parameters, we can determine the position and orientation of the placed QR code in space. Since the QR code is placed on the object that needs to be located, we can also know the position and orientation of the object in actual space.

[0026] Step 4, Research on Augmented Reality-Based Guided Perception and Virtual-Real Fusion Maintenance Methods for Complex Processes (Research on virtual-real fusion visualization of maintenance processes, real-time perception and guidance of maintenance processes, and multi-mode high-comfort natural interaction methods): Establish a 3D model of maintenance process information based on Augmented Reality (MBD), and construct a maintenance process database based on the 3D model of maintenance process information using MySQL. The specific steps are as follows: Step 4.1: Establish a 3D model of maintenance process information based on MBD (Model Based Define, digital product definition technology). The 3D model of maintenance process information includes several maintenance process information items. The virtual and real information is visualized through the 3D model of maintenance process information based on MBD. Preferably, all process information that may be involved in the assembly and maintenance environment is defined, and a process information ontology model based on the MBD concept is designed. By integrating it into the 3D solid model, it can easily generate visualized maintenance manuals and operation guides, displaying maintenance steps and methods in the form of intuitive 3D animations, pictures, etc. Specific process information includes: tooling involved in the current maintenance process, including tooling number, tooling name, and specific instruction document number; problematic parts involved in the maintenance process, including their part number, name, 3D model number, quantity, etc.; electronic maintenance process documents, including their process name, process number, or process code; 3D process paths for augmented reality-assisted maintenance, including the coordinates of the start node and end node in 3D space; maintenance personnel information; safety requirements information; auxiliary material information, etc. Preferably, the visualization of virtual and real information through the 3D model of maintenance process information based on MBD can be achieved by establishing an AR (Augmented Reality) model of the entire maintenance process of the scene, building a process knowledge base based on semantic analysis for different maintenance process steps, and intelligently pushing process information. Step 4.2: Based on MySQL (MySQL Database Management System / My Structured Query Language), several maintenance process information items are set as table structures, and the specific attributes corresponding to each maintenance process information item are set as columns of the table structure. Foreign keys are used to realize the association between several maintenance process information items. At the same time, an appropriate primary key is selected for each table to ensure the uniqueness of the data. Then, an appropriate storage engine is selected, such as InnoDB (InnoDB Storage Engine), and table storage space is allocated to obtain the maintenance process database. Preferably, MySQL is an open-source relational database management system that can be used to store structured data such as maintenance process data, fault case data, and component information. Through SQL (Structured Query Language) statements, data can be easily queried, inserted, updated, and deleted.

[0027] Step 5: Cluster the maintenance process database using a vector clustering algorithm, and generate maintenance process text from the clustered maintenance process database. The specific steps are as follows: Step 5.1, see Figure 5 By using Natural Language Processing (NLP) technology, the maintenance process information described in natural language in the maintenance process database is decomposed into semantic vectors. Then, the semantic vectors expressing the same maintenance process are clustered together using a vector clustering algorithm to obtain a clustered maintenance process database. This database is used to achieve automatic semantic filtering and classification of information describing the same maintenance process, and supports the rapid querying and entry of assembly data. Preferably, the data is first preprocessed by calling Python libraries such as Spacy and Spark to perform word segmentation and cleaning. Next, feature extraction and vectorization are performed on the preprocessed data, with each text or word corresponding to a point in a vector space. Specifically, the Word2Vec (Word to Vector) algorithm is used to convert words into numerical form through word embedding. The algorithm includes two types: one is to infer the center word based on the background word of CBOW, and the other is to infer the background word based on the center word of skip-gram. (1) CBOW (Continuous Bag of Words) model: The basic principle is to use predict Establish the objective function: ; In the formula, A Indicator; P Represents a conditional probability function; T This indicates the total number of words in the corpus; t This indicates the total number of words in the corpus; m Indicates the size of the context window (the number of context words used to predict the target word); Establish the cross-entropy loss function: ; Output layer: ; In the formula, o Indicates the target word index; v Represents the vector of the center word; u Represents context word vectors; Training gradient (stochastic gradient ascent): ; In the formula, Indicates partial derivative; (2) Skip-Gram (Continuous Skip-Gram Model) Basic principle: In contrast to the CBOW model, it uses... predict Establish the objective function: ; Establish the loss function (estimate model parameters by maximizing the likelihood function): ; Output layer (using the Softmax function to convert the output vectors of the hidden layers into a probability distribution for each word in the vocabulary): ; Training gradient (stochastic gradient ascent): ; Step 5.2: Generate maintenance process text using the template-based NLG algorithm from the clustered maintenance process database; Preferably, the [Workpiece Name] needs to undergo [Machining Process] repair, and the machining accuracy requirement is [Accuracy Requirement]. Based on the repair process database and the contextual understanding of the clustering steps, the repair process text is further generated.

[0028] Step 6: Design an AR visualization interface based on the repair process text to achieve augmented reality-assisted repair. The specific steps are as follows: Based on the maintenance process text, 3D models of maintenance scenarios and parts are created, and AR visualization interfaces are designed based on the 3D models to achieve augmented reality-assisted maintenance. Preferably, high-precision 3D modeling and AR content design are used for 3D modeling of the scene and its related parts, as well as AR guided animation interface design. See [reference needed] for details. Figure 4 This process involves high-precision 3D modeling, reasonable simplification and optimization to improve model rendering efficiency, and the construction of a virtual maintenance scene. Tools such as 3ds Max and Maya are used to create AR animations, text, images, and other multimedia content. The MVC design pattern is adopted to design the presentation and interaction logic of the AR content, providing clear and intuitive operation guidance. LOD (Level of Detail) technology for AR content is researched to dynamically adjust model details based on viewing distance, balancing rendering performance and visual quality. The spatial layout and interaction methods of the AR content are designed to ensure clear information presentation and accurate operation guidance. The performance of the AR guidance screen is optimized to improve the system's real-time response capability, frame rate, and stability. The continuous, local mesh simplification algorithm QEM (Quadric Error Metrics) is used for implementation. (See [link to relevant documentation]). Figure 6QEM shrinks the model mesh based on the local operation of edge folding, thereby achieving dynamic adjustment of model details; The basic principle is as follows: By defining a quadratic error metric function, we measure which edge in the mesh will be folded (i.e., the two endpoints of the edge will be merged into a new vertex), and then calculate the geometric error between the new mesh and the original mesh. Through dynamic programming, the algorithm will continuously select the edge with the smallest error to fold until the preset simplification goal is achieved. Establishing the quadratic error matrix: In three-dimensional space, a plane can be represented by the point normal form equation as follows: ax + by + ez + d =0, where n =( a , b , e ) is the normal vector of the plane. d It is the distance from the plane to the origin, which can be represented as a four-dimensional vector. p =( a , b , e , d For a point in space v =( x , y , z ), it to the plane p =( a , b , e , d Square distance It can be represented as: ; In the formula, V Represents a vertex set; Q Represents the quadratic error matrix; V T express V transpose; Q Formation: ; For a vertex v The sum of the squares of its distances to all its adjacent planes can be expressed as: ; In the formula, k Represents vertices v The number of adjacent planes; Q Represents the quadratic error matrix. Represents vertices v The total quadratic error matrix; (2) Using dynamic programming, initialize the quadratic error matrix. For each vertex in the grid, calculate the quadratic error matrix of its adjacent planes and sum these matrices to obtain the total quadratic error matrix of the vertex. Calculate the edge folding cost. For each edge in the grid, consider folding the two endpoints of the edge. v 1 and v 2 merge into a new vertex v New Vertex v The location can be determined by minimizing the error function: Once determined, the edge with the smallest error cost is selected for the folding operation, and the quadratic error matrix of the new vertex and its adjacent vertices is updated. Preferably, the transformation mechanism of different maintenance AR visualization scenarios is studied, taking the following scenarios as examples: temporary sampling of flared catheter, disassembly and assembly of sealed ear plate nut, and replacement of plug and rear accessory: For temporary sampling scenarios using flared tubing, this study investigates the spatial mapping and positioning of virtual pipeline objects to achieve real-time synchronization between AR content and the pipeline. It can capture and locate problematic pipeline objects, display the pipeline cut-off location, and visualize maintenance information and process steps using AR, guiding workers to perform the cut-off operation. After cut-off, three possible scenarios arise, each with corresponding different operation steps and AR guidance information. Workers manually select the current pipeline status on the AR virtual interface, and AR switches to the corresponding interface content, displaying relevant operation animations, process information, and text prompts. For the disassembly and assembly of nuts without lugs in sealed environments, this study investigates the optimal implementation of AR visualization for body protection and screw removal to improve disassembly and assembly efficiency and safety. It also develops AR interactive auxiliary tools for nut drilling and wire threading, improving operational accuracy through virtual guides and real-time feedback. Furthermore, it designs vivid AR process prompts for key processes such as gluing and gun installation to standardize operating procedures. For the replacement of plug accessories, an AR-based dynamic presentation mechanism for disassembly and assembly sequences was developed to ensure the correct disassembly and assembly order. The visualization method intuitively displays potential problems to assist in quick location and handling. In summary, this research focuses on automated AR training content generation technology to improve content production efficiency and reusability. It also explores combining AR technology with intelligent content optimization algorithms to achieve real-time evaluation and feedback of maintenance operations, thereby enhancing training effectiveness. Specific optimizations are tailored to each scenario.

[0029] Step 7, Develop AR process monitoring and evidence traceability functions: Design an AR on-site data acquisition scheme, utilize multi-source data such as depth cameras, sensors, and manual input to comprehensively record the maintenance training process and key maintenance training data, realize data acquisition and storage of key links, and achieve intelligent management and optimization of the maintenance training process; Preferably, an AR-based key node annotation and data retention mechanism is developed, enabling rapid information entry through natural interaction methods such as voice and gestures; spatiotemporal AR visualization data management technology is researched to support the visual playback and traceability analysis of the maintenance process; AR data visualization tools are developed to transform the collected maintenance data into an intuitive visualization form; based on the maintenance process text, data storage and management modules are further developed to achieve structured storage and efficient retrieval of maintenance data; and AR data visualization interfaces for different roles are designed to meet the needs of different users such as managers and quality control personnel.

[0030] In one embodiment of the present invention, an augmented reality-assisted maintenance system for aviation pipelines is provided, comprising: The target recognition dataset acquisition and expansion module is used to acquire and construct a target recognition dataset for faulty air pipelines and to expand the target recognition dataset. The fault target identification model construction and pre-training module is used to construct a fault target identification model, pre-train the fault target identification model, and obtain aviation pipeline fault information through the pre-trained fault target identification model. The virtual-real information matching and fusion module is used to build a virtual model library, obtain the geometric and texture information of components in the aviation pipeline fault information, match the geometric and texture information with the features in the virtual model library to obtain virtual-real information, register the virtual-real information, and fuse the virtual-real information during registration to obtain virtual-real fused information. The maintenance process database construction module is used to establish a 3D model of maintenance process information based on MBD, and to construct a maintenance process database based on the 3D model of maintenance process information using MySQL. The maintenance process text generation module is used to cluster the maintenance process database using a vector clustering algorithm, and then generate maintenance process text from the clustered maintenance process database. The AR visualization interface design module is used to design AR visualization interfaces based on maintenance process texts, thereby enabling augmented reality-assisted maintenance.

[0031] The following is a summary of the principles and steps of the augmented reality-assisted maintenance method for aviation pipelines according to the present invention: Problem pipeline identification based on a deep learning neural network with a multi-attention mechanism: Problem pipeline identification mainly relies on the YOLOv9 target detection algorithm. Different problems and faults under different pipelines are collected as datasets and labeled. The cross-layer connections of the YOLOv9 network are optimized. Attention modules such as ECA are introduced into key positions of the YOLOv9 backbone network to improve the attention to target areas. At the same time, the multi-scale feature fusion strategy is optimized to make the network more advantageous in detecting small targets and occluded targets. The dataset is expanded using the Mosaic data augmentation method. The images captured by the camera are identified in real time online through YOLOv9, and the identified pipeline problem and fault information is output. This paper addresses the matching problem of virtual information of aviation components to real-world scenes based on 3D registration technology, virtual-real occlusion handling methods, and human-computer interaction interface design in virtual-real fusion. Specifically, it extracts features from images and point cloud data of aviation pipelines in the real world, parameterizes the geometry and texture information of components, and matches them with features in the virtual model to ensure consistency between the virtual model and the actual scene, achieving accurate alignment of virtual and real objects. It also performs occlusion detection and segmentation of virtual and real information during registration, obtains the boundaries of occlusion handling, and compensates for occluded parts to ensure smooth integration of virtual information and the real environment, achieving a more natural virtual-real fusion effect. Finally, it establishes a reasonable component information architecture and operation interaction method to ensure that users can quickly find the information they need through relevant operations, and saves key information for future detection and traceability, improving the usability of the subsequent system development. Research on Augmented Reality-Based Guided Perception and Virtual-Real Fusion Maintenance Methods for Complex Processes: Addressing the issues of limited intuitiveness in maintenance process documents, poor real-time guidance methods, and reliance on paper documents in aero-engine equipment maintenance, this study introduces an augmented reality (AR)-based guided perception and virtual-real fusion maintenance method for complex processes. The research focuses on virtual-real fusion visualization of maintenance processes, real-time perception and guidance of maintenance processes, and multi-modal, highly comfortable, and natural interaction methods. A process information ontology model based on the MBD (Modular Model Decomposition) concept is designed. By integrating it into a 3D entity model, it can easily generate visualized maintenance manuals and operation guides, displaying maintenance steps and methods in intuitive 3D animations and images. A maintenance process database is developed based on MySQL. For maintenance processes in typical scenarios, a natural language processing-based morpheme decomposition algorithm for process documents is studied to extract morpheme features from similar process documents. A semantic vector clustering algorithm is also investigated, enabling automatic screening and classification of morphemes describing the same process. A template-based NLG algorithm is designed for process text generation. Addressing typical aviation pipeline faults, this paper presents an augmented reality (AR) maintenance system based on Unity3D and Vuforia. The overall framework of the AR maintenance system is designed, and the system modules are planned, including AR content presentation, scene management, and data storage. Suitable AR display devices (such as HoloLens and Meta) are selected to provide a smooth user experience, and device performance is evaluated and optimized. The system employs MVC or MVVM design patterns to achieve low coupling and high cohesion. The user interface and scene switching logic are designed, along with a flexible scene management mechanism that supports multiple scene switching and dynamic loading of scene content. Specifically, the augmented reality-assisted maintenance (AR) system for typical aircraft scenarios includes: A temporary sampling AR module for flared conduits: Developing spatial positioning capabilities for pipeline objects to achieve accurate spatial overlay of virtual information with the real world; exploring object collision and motion relationships to obtain a more realistic AR visualization experience; researching AR visualization of maintenance information and procedures for pipelines currently under maintenance, and providing AR display and interactive guidance for cutting positions based on spatial location mapping, offering real-time AR animation guidance to ensure correct cutting operations on problematic pipelines; developing corresponding AR operation guidance screens for three post-cutting scenarios: pipeline overlap grinding guidance, direct welding guidance, and pipeline addition welding guidance to ensure welding quality; developing AR on-site recording functionality to achieve full-process status traceability; and a sealing nut disassembly and assembly AR module: opening... The system includes an AR-guided visualization interface for machine body protection and screw removal. Based on real-time process information from AR process monitoring, it features AR animations for drill bit selection and drilling operations to guide workers in correct operation. AR interactive functions for wire threading and tray nut removal are developed to improve operational accuracy. AR process prompts for cleaning, gluing, and gun installation are designed to standardize the installation process. An AR record traceability function is developed to achieve intelligent management and optimization of the maintenance process. For plug and accessory replacement, an AR module is included: AR disassembly guidance for plugs and accessories is developed, demonstrating the correct disassembly sequence and method; AR operation instructions for plug and accessory replacement are designed to reduce errors; an AR quality inspection function is developed to highlight potential quality problems; AR assembly guidance for plugs and accessories is designed to ensure correct assembly; and AR process monitoring and evidence retention traceability functions are developed to achieve data collection and storage for key processes.

[0032] This invention proposes a problem pipeline identification algorithm based on a YOLOv9 target recognition deep learning neural network. The optimized and improved algorithm can more accurately and effectively identify different problem pipelines. It utilizes 3D registration technology based on virtual-real fusion, virtual-real occlusion handling methods, and human-computer interaction interface design to match virtual information of aerospace components with real-world scenarios. Furthermore, it studies a virtual-real fusion visualization method for maintenance processes, real-time perception and guidance of maintenance processes, and multi-mode, highly comfortable, and natural interaction methods based on augmented reality-based guidance and virtual-real fusion maintenance for complex processes. Finally, it provides a realistic maintenance operation experience and immersive training through an augmented reality-assisted maintenance AR system for typical aircraft scenarios, including corresponding AR animation guidance, process content display, step documentation, process traceability, and AR process prompts, thereby improving maintenance efficiency and quality.

[0033] See Figure 7 In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, and is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to realize a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of augmented reality-assisted maintenance methods for aviation pipelines.

[0034] In one embodiment of the present invention, a computer-readable storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space containing the operating system of a terminal; and the storage space also contains one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the augmented reality-assisted maintenance method for aviation pipelines in the embodiment.

[0035] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0036] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0037] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0038] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An augmented reality assist maintenance method for an aircraft tube, characterized in that, The method comprises the following steps: acquiring and constructing a target recognition data set of a fault aviation pipeline, expanding the target recognition data set; constructing a fault target recognition model, pre-training the fault target recognition model, and obtaining aviation pipeline fault information through the pre-trained fault target recognition model; building a virtual model library, acquiring geometric information and texture information of components in the aviation pipeline fault information, matching the geometric information and texture information with features in the virtual model library to obtain virtual-real information, registering the virtual-real information, fusing the virtual-real information during registration, and obtaining virtual-real fusion information; establishing an MBD-based maintenance process information three-dimensional model, and constructing a maintenance process database based on the maintenance process information three-dimensional model and Mysql; clustering the maintenance process database based on a vector clustering algorithm, generating maintenance process text based on the clustered maintenance process database; designing an AR visualization interface based on the maintenance process text to realize augmented reality assisted maintenance.

2. The augmented reality method for aircraft tubing according to claim 1, wherein, The acquiring and constructing a target recognition data set of a fault aviation pipeline, and expanding the target recognition data set, specifically comprises: collecting aviation pipeline images under different fault conditions, labeling the aviation pipeline images to obtain aviation pipeline image label data under different fault conditions, constructing a target recognition data set of a fault aviation pipeline through the aviation pipeline image label data under different fault conditions, expanding the target recognition data set by using a Mosaic data enhancement method, and obtaining an expanded target recognition data set.

3. The augmented reality method for aviation tube maintenance of claim 1, wherein, The constructing a fault target recognition model, pre-training the fault target recognition model, and obtaining aviation pipeline fault information through the pre-trained fault target recognition model specifically comprises: building a YOLOv9 target recognition deep learning neural network, introducing a lightweight network architecture GELAN based on gradient path planning design and an ECA attention mechanism module, obtaining a fault target recognition model, pre-training the fault target recognition model, training and evaluating the expanded fault aviation pipeline target recognition data set on the pre-trained weights by using a transfer learning method, obtaining a pre-trained fault target recognition model, and obtaining aviation pipeline fault information through the pre-trained fault target recognition model in a real-time online manner.

4. The augmented reality method for aviation tube maintenance of claim 1, wherein, The building a virtual model library, acquiring geometric information and texture information of components in the aviation pipeline fault information, matching the geometric information and texture information of the components with features in the virtual model library to obtain virtual-real information, registering the virtual-real information, fusing the virtual-real information during registration, and obtaining virtual-real fusion information specifically comprises: The virtual model library is built, features of image and point cloud data of the aviation pipeline are acquired, geometric information and texture information of parts in the aviation pipeline fault information are obtained according to the features of the image and the point cloud data of the aviation pipeline, the geometric information and the texture information are parameterized, the parameterized geometric information and texture information are matched with features in the virtual model library, virtual-real information is obtained, the virtual-real information is registered, occlusion of the virtual-real information is detected during the registration, and the virtual-real information is segmented into background and foreground, boundary information of the occlusion processing is obtained, the occluded part is compensated according to the boundary information, and virtual-real fusion information is obtained.

5. The augmented reality method for aviation tube maintenance of claim 1, wherein, The MBD-based maintenance process information three-dimensional model is established, and a maintenance process database is constructed based on Mysql according to the maintenance process information three-dimensional model, and specifically includes: The MBD-based maintenance process information three-dimensional model is established, the maintenance process information three-dimensional model includes a plurality of maintenance process information, the virtual-real fusion information is visually displayed through the MBD-based maintenance process information three-dimensional model, the plurality of maintenance process information is set as a table structure based on Mysql, the specific attributes corresponding to each maintenance process information are set as columns of the table structure, the association between the plurality of maintenance process information is realized through a foreign key, and a maintenance process database is obtained.

6. The augmented reality method for aviation tube maintenance of claim 1, wherein, The vector clustering algorithm is used to cluster the maintenance process database, and a maintenance process text is generated based on the clustered maintenance process database, and specifically includes: The maintenance process information in the maintenance process database is decomposed into semantic vectors, the semantic vectors expressing the same maintenance process are clustered together through the vector clustering algorithm, a clustered maintenance process database is obtained, which is used to realize screening and classification, and then a maintenance process text is generated based on the clustered maintenance process database.

7. The augmented reality method for aviation tube maintenance of claim 1, wherein, The AR visualization interface is designed according to the maintenance process text, and the augmented reality assisted maintenance is realized, and specifically includes: The maintenance scene and the parts are 3D modeled according to the maintenance process text, the AR visualization interface is designed according to the 3D modeling, and the augmented reality assisted maintenance is realized.

8. An augmented reality assist maintenance system for aviation tubing, characterized by, The augmented reality assisted maintenance method for the aviation pipeline according to any one of the above claims 1-7, comprising: A target recognition dataset acquisition and expansion module is configured to acquire and construct a target recognition dataset of the fault aviation pipeline, and expand the target recognition dataset. A fault target recognition model construction and pre-training module is configured to construct a fault target recognition model, pre-train the fault target recognition model, and obtain aviation pipeline fault information through the pre-trained fault target recognition model. A virtual-real information matching and fusion module is configured to build a virtual model library, acquire geometric information and texture information of parts in aviation pipeline fault information, match the geometric information and the texture information with features in the virtual model library, obtain virtual-real information, register the virtual-real information, and fuse the virtual-real information during the registration to obtain virtual-real fusion information. A maintenance process database construction module is configured to establish an MBD-based maintenance process information three-dimensional model, and construct a maintenance process database based on Mysql according to the maintenance process information three-dimensional model. The maintenance process text generation module is configured to perform clustering on the maintenance process database by a vector clustering algorithm, and generate a maintenance process text based on the clustered maintenance process database. The AR visualization interface design module is configured to design an AR visualization interface based on the maintenance process text, and realize augmented reality assisted maintenance.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the method according to any one of claims 1-7.

Images