A mixed reality based aero-engine auxiliary assembly system and method

Abstract

The application provides an aero-engine auxiliary assembly system and method based on mixed reality, which is divided into three levels of data layer, service layer and application layer; the data layer comprises a virtual assembly scene initialization module and an MR auxiliary assembly process generation module; the service layer comprises a communication module, an instruction conversion module and an extension function module; and the application layer comprises a positioning and tracking module, an identification module, an interaction module and a visualization module. The application improves the intuitiveness and interaction efficiency of the assembly process instruction, avoids the distraction of paper files during high-intensity work of workers, and completes the assembly process in a more convenient, simple and efficient form to ensure the quality requirements.

Description

Technical Field

[0001] This invention relates to mixed reality technology and assisted assembly processes, specifically to a mixed reality-based aero-engine assisted assembly system and method. Background Technology

[0002] The turbine casing is one of the most important components of an aero-engine. It serves as the base of the entire engine and is the main load-bearing component. During aero-engine assembly, the high-pressure turbine casing and low-pressure vortex shear system involve numerous components, complex assembly spaces, and many repetitive steps, posing a challenge to the high assembly quality required for aero-engine assembly. Because different engine models have different structures and component layouts, the corresponding assembly processes vary significantly. Currently, the assembly of the high-pressure turbine casing and low-pressure vortex shear system largely relies on paper and computer-generated process drawings and the experience of on-site assembly workers. Using paper and electronic text drawings as information carriers often results in information redundancy and difficulty in expressing non-quantitative factors beyond the text during downward transmission. Therefore, traditional manual assembly tasks are affected by factors such as the portability of materials, the intuitiveness of process information, and the skill level of personnel. The assembly process is labor-intensive, inefficient, and prone to errors and omissions.

[0003] Mixed Reality (MR) technology is a technology that enables real and virtual scenes to be displayed and interacted with in the same visual space. Utilizing advanced image processing technology and display devices, it merges the real and virtual worlds, creating a new visual environment with integrated virtual and real characteristics. Physical and digital objects coexist, embedding disparate spatiotemporal scenes. It establishes an interactive feedback loop between the real world, the virtual world, and the user, enhancing the user's sense of realism and featuring characteristics such as authenticity, real-time interactivity, and imaginative possibilities. Using MR technology to assist in guiding the aero-engine assembly process can visualize the required process information in a more natural and diverse way, enhancing assembly workers' information perception capabilities and understanding of the assembly scenario and tasks. Through virtual-real integrated assembly guidance instructions, it improves the intuitiveness of guidance and demonstration behaviors during multi-person collaboration, enabling human-machine collaborative assembly, ultimately reducing workload, improving assembly efficiency, and enhancing assembly quality.

[0004] MR-based assisted assembly systems provide additional auxiliary functions for existing assembly tasks. They require generating MR-assisted assembly processes tightly linked to the task flow based on traditional processes, integrating related manufacturing entities such as assembly parts, tools, and fixtures, along with MR virtual guidance information and assembly steps. In most existing MR-assisted assembly system development processes, developers work on existing frameworks and development engines, writing application logic from low-level script code. Therefore, developers need process understanding, programming skills, MR development expertise, and engine usage experience to generate MR-assisted assembly processes corresponding to traditional processes, raising the development threshold. Furthermore, MR-assisted assembly processes are pre-designed, and the functional structure and flow of systems and software applications developed based on them are fixed. For specific tasks and application scenarios, significant preparation time is required, and they lack the ability to be quickly and robustly modified when tasks change or when process engineers and assembly personnel disagree. Summary of the Invention

[0005] To address the aforementioned issues in generating MR-assisted assembly processes, this invention deploys a novel MR-assisted assembly process generation module in a mixed reality-based aero-engine assisted assembly system. Users can generate MR-assisted assembly processes intuitively, conveniently, and instantly in an immersive virtual environment, tailored to the current environment and assembly task, without needing to prepare complex prior knowledge.

[0006] The technical solution of this invention is as follows:

[0007] A mixed reality-based aero-engine auxiliary assembly system is divided into three layers: data layer, service layer, and application layer.

[0008] The data layer includes a virtual assembly scene initialization module and an MR-assisted assembly process generation module.

[0009] The service layer includes a communication module, an instruction conversion module, and an extended function module;

[0010] The application layer includes a positioning and tracking module, an identification module, an interaction module, and a visualization module;

[0011] In the data layer, the virtual environment corresponding to the assembly site is restored through the virtual assembly scene initialization module, and the user's operation data on the virtual model is collected to complete the virtual assembly; the complete assembly process is recorded through the MR-assisted assembly process generation module, MR-assisted assembly instructions are interactively constructed, and then an instruction set is formed, which finally automatically generates the MR-assisted assembly process specification; the virtual scene and MR-assisted assembly process are finally stored on the server in the form of data including 3D models, TXT files, pictures, and CSV files.

[0012] The service layer stores and manages all data in the server; the communication module receives and processes data submissions and access requests from various hardware, clients, and modules, and provides information transmission functions between them; the instruction conversion module parses the MR-assisted assembly process; the extended function module stores function implementation interfaces, including error and omission detection and surface quality detection, for users to selectively call, and can extend the interfaces.

[0013] The application layer is the basic component of the deployed client applications, which are divided into AR clients for users and VR clients for remote experts. The recognition module initializes the spatial coordinate system synchronization by scanning the identification code, and the positioning and tracking module continuously runs to locate the local AR client and the target object. The interaction module receives commands sent by the user to the system and passes them to the service layer. The service layer receives the commands and initiates an assembly instruction retrieval request. The instruction conversion module determines the various resources and information required by the current client for the instruction, and then retrieves and instantiates it from the server into the AR client device. Through the visualization module, it is mapped to the correct position in the real field of view, ultimately forming a complete offline MR-assisted assembly process.

[0014] During the assembly process, the AR client initiates a multi-person collaboration request command, requesting the VR client of a remote expert to work collaboratively through the communication module of the service layer. The VR client receives the initial virtual assembly scene stored in the server and the real-time spatial layout information provided by the AR client. In this way, the VR client restores a virtual scene that is consistent with the assembly site space and synchronized with the assembly status. The VR client user can issue guidance commands in an immersive way and transmit them to the AR client. The AR client visualizes the guidance information and realizes real-time online collaborative assisted assembly.

[0015] Furthermore, the virtual scene initialization module is connected to the external data management module PDM. Based on the target scene applied in the MR-assisted assembly process, the virtual scene initialization module exports solid CAD models from the PDM, including parts, tools, workbenches, and fixtures, while preserving complete spatial layout information and geometric relationships between the assemblies. After model lightweighting, these models are loaded into a pre-prepared Unity-based virtual space, and functional components including colliders and hand interactions are added. The virtual scene initialization module also exports text and images from traditional assembly process documents from the PDM and converts them into Unity-based virtual assets, which are then provided to the interactive user interface along with markers in the MR collaboration, including virtual pointers, arrows, and annotation boxes. The virtual scene initialization module can also adjust the positions of all virtual assets distributed in the virtual environment to maintain a consistent spatial layout with the assembly site. Finally, a virtual scene corresponding to the current assembly site is generated, the virtual assets are stored on the server, and the scene layout information is recorded in a Scene.CSV file via the instruction conversion module.

[0016] Furthermore, the MR-assisted assembly process generation module can record the complete operation process of a user directly manipulating the virtual model of parts and performing virtual assembly based on their personal assembly experience and traditional processes displayed in the virtual environment in the form of text and pictures.

[0017] For each step a user completes, the MR-assisted assembly process generation module records which parts the user manipulated, their initial and final positions, and their movement trajectories. Simultaneously, during the completion of each step, the user selects the necessary tools from the interactive UI, chooses the most helpful guidance information and markers, and determines their optimal visualization method. The tool types, the poses of the guidance information and markers, and their trajectories are also recorded by the MR-assisted assembly process generation module. This completes the creation of single-step assembly instructions in the MR-assisted assembly process, and the relevant information is recorded in the Step.CSV file via the instruction conversion module.

[0018] All assembly instructions for each step, along with additional user-created guidance information, together form the MR-assisted assembly process and are recorded in the Task.CSV file.

[0019] Furthermore, the instruction conversion module converts the MR-assisted assembly process summarized in the complete virtual assembly process in the data layer into a series of unified CSV files, and parses the CSV files into an information format that can be read by each client application layer. The Scene.CSV file stores the virtual model IDs of each workstation's corresponding virtual scene, including work platforms, tooling, parts, tools, virtual markers, text, and images, as well as their respective spatial positions and relative relationships, in a structured form, for initializing the virtual assembly environment. The Step.CSV file includes assembly instruction information involved in each step, including step ID, part model ID, initial / final spatial coordinates of the part, process guidance information, and required tools. The Task.CSV file organizes the steps according to the assembly task and adds additional guidance information and quality inspection steps.

[0020] Furthermore, the communication module serves as an information transmission relay station in the MR-assisted assembly system. It receives and stores text, images, 3D models, and CSV file data generated by the data layer. It also receives and responds to data call commands issued by the client and various modules, and outputs the data. The module records the control model ID, model pose, and client pose information obtained by the AR / VR client positioning and tracking module, enabling data communication between workers and experts and achieving spatial layout consistency to achieve the effect of multi-person collaborative operation synchronization.

[0021] Furthermore, the extended function module is an expandable cloud workstation, on which algorithm implementations or application software for quality inspection functions during the assembly process are deployed, and an interface is provided to the client for invocation.

[0022] Furthermore, the positioning and tracking module relies on the visual sensors, laser sensors, inertial sensors, and / or spatial locators mounted on the client device to perform real-time self-localization using SLAM algorithms and optical positioning technology, determining the spatial position of the head-mounted device in the real environment. This enables the acquisition of spatial layout information, allowing MR-assisted assembly instructions and guidance information to be registered with high precision in the AR terminal, superimposed onto the correct position in the user's real field of vision, achieving good realism and intuitiveness through virtual-real fusion. Simultaneously, it provides a pose estimation function based on library model matching to determine the poses of key location points and components identified by the recognition module, maintaining synchronous pose estimation during subsequent movements, while ensuring that the spatial layout of the VR virtual environment remains consistent.

[0023] Furthermore, the recognition module is used to recognize specified QR codes, barcodes, components, gestures, and voice; the positioning and tracking module uses the recognition module to recognize QR codes to initialize the initial position of the head-mounted MR device and register the virtual space and real space for subsequent positioning and tracking; specific parts are identified by affixing barcodes or natural features to the components for tracking the position and pose of the parts and for taking pictures and measuring parts that require quality inspection.

[0024] The method for assisted assembly of aero-engines based on the above system includes the following steps:

[0025] Step 1: Create a virtual scene:

[0026] Step 1.1: Using a virtual reality headset worn by the user, enter a pre-prepared standard virtual construction space application. The virtual reality headset provides a real-time 3D point cloud stream captured by a camera and an interactive UI interface. Based on user-specified information or an interface provided by the recognition module, automatically determine the assembly scenario for this construction and assembly process. Instantiate the 3D model corresponding to the assembly scenario and the virtual resources provided in the interactive UI interface into the virtual space through the interface provided by the PDM, and manually or using the interface provided by the positioning and tracking module to adjust their spatial poses and relationships. Add pre-prepared functional components to these virtual resources.

[0027] Step 1.2: Deploy the scene: Use the interface provided by the instruction conversion module to describe and record the current scene as a Scene.CSV file and store it on the server;

[0028] Step 2: Interactively generate MR-assisted assembly process:

[0029] Step 2.1: Using the virtual reality headset worn by the user, enter the pre-prepared assembly process generation application, select the target assembly scene from the virtual interactive UI interface; The assembly process generation application submits a request to the server through the communication module to obtain the Scene.CSV description file corresponding to the assembly scene. After approval, the instruction conversion module parses this CSV file, and according to the client type that made the request, transmits the corresponding virtual assets in the server to the client and restores the spatial layout.

[0030] Step 2.2: In the virtual assembly scenario, user operations are collected through human-computer interaction. The virtual model is assembled successfully part by part and step by step on the virtual workbench. The initial and final positions and movement trajectories of the parts in the assembly process are recorded. Based on user operations, tools and virtual annotations for marking the current action are selected and instantiated from the interactive interface. The most helpful parts for guiding the current task are marked in the text and images of traditional processes. Key control points in the assembly process are selected and recorded using voice, gaze, and gesture interaction methods, and additional guidance information is added through function buttons provided by the interactive interface. Based on user operations, the current operation sequence number is assigned to divide the steps of the MR-assisted assembly process, and it is determined whether quality inspection is required at each position between steps. A single step operation is an assembly instruction, and the resulting instruction set and additional information are summarized into the final MR-assisted assembly process.

[0031] Step 2.3: The assembly instructions representing the work steps are converted into Step.CSV files using the instruction conversion module; the MR-assisted assembly process is converted into Task.CSV files; the newly generated files and virtual assets are stored on the server;

[0032] Step 3: Launch the MR-assisted assembly application on the AR client:

[0033] Step 3.1: The AR device identifies the identification code in the real scene, completes pose initialization and spatial registration, and then performs continuous and stable self-localization and target tracking.

[0034] Step 3.2: The system identifies the main assembly unit at the assembly site to determine its current workstation, and sends an application to the server to obtain the corresponding MR auxiliary assembly process. Once approved, the system obtains the Task.CSV file.

[0035] Step 4: The system parses the acquired Task.CSV file to determine the relevant assembly scene layout, virtual assembly model, guidance information, pose, key point positions, visualization mode information, and downloads the relevant resources;

[0036] Step 5: Using self-localization and target tracking functions, combined with pose information, visualize the virtual model, guidance information, and interactive interface resources into the correct area of ​​the AR device's real field of view;

[0037] Step 6: Under the assembly assistance guidance process, the user completes the assisted assembly process, including offline MR assisted assembly and real-time online collaborative assisted assembly.

[0038] Beneficial effects

[0039] (1) This invention proposes an aero-engine assisted assembly system based on mixed reality, which can utilize the advantages of MR technology to improve the intuitiveness and interactive efficiency of assembly process instructions, avoid distracting workers from manipulating paper documents while performing high-intensity work, and complete the assembly process in a more convenient, simple and efficient manner, thus ensuring quality requirements.

[0040] (2) This invention proposes a novel interactive MR-assisted assembly process generation method. Based on virtual reality, users can generate instruction sets and process operation manuals suitable for MR-assisted assembly in a virtual scene in a highly free and efficient manner. During the assembly process in the virtual space, users have a real immersive experience and have the ability to repeatedly try and fail to assemble in reality, realizing the automatic conversion from traditional processes to MR-assisted assembly processes.

[0041] (3) By converting the MR-assisted assembly process specifications into a CSV file, this invention can efficiently link information relationships such as instructions, operation models, guidance information, and model poses, thereby reducing the storage space burden of head-mounted MR devices and improving real-time performance.

[0042] (4) By storing the application software with additional functions such as quality detection on the server, and relying on the interaction module and TCP protocol, the system can directly obtain the returned results by transmitting only the input data required by the application, which improves the system's scalability. When adding functions in the future, there is no need to redeploy the AR terminal program; only the application and the corresponding program need to be configured on the server.

[0043] (5) Through the positioning and tracking module, the interactive virtual model can be accurately superimposed on the real world. Compared with paper and electronic two-dimensional pictures, users can more intuitively observe the assembly status and understand the assembly task.

[0044] (6) The present invention enables multi-person collaboration through MR technology, TCP communication technology, etc., allowing remote experts to guide local workers in assembly operations in real time and obtain a more realistic, accurate and intuitive guidance effect than video streams.

[0045] (7) The quality inspection achieved by the present invention through hardware sensors and software programs is more efficient than manual measurement.

[0046] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0047] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0048] Figure 1Framework for an aero-engine assisted assembly system based on mixed reality;

[0049] Figure 2 Workflow diagram of MR-assisted assembly process generation module;

[0050] Figure 3 Instruction conversion module workflow diagram;

[0051] Figure 4 Workflow diagram of the location tracking module;

[0052] Figure 5 Workflow diagram of the recognition module;

[0053] Figure 6 Interactive module workflow diagram;

[0054] Figure 7 Visualized module workflow diagram. Detailed Implementation

[0055] This invention addresses the characteristics of the assembly process of the high-pressure turbine casing and low-pressure vortex shear of aero-engines and the shortcomings of existing technologies. By combining mixed reality technology, it provides an aero-engine auxiliary assembly system and method based on mixed reality.

[0056] This invention establishes a mixed reality-based assisted assembly system, namely the SUC (Scene Understanding Cooperation) system. This system uses assembly scenes as the basis for implementing assisted assembly, helps users understand assembly tasks, and completes the assembly process and ensures assembly quality requirements under the support of MR for multi-person collaboration and human-computer interaction.

[0057] To address the lack of intuitive and efficient auxiliary effects in traditional assembly process specifications, this system proposes and utilizes a MR-assisted assembly process generation module. In the virtual environment reconstructed by the system for different workstations (assembly scenes), users, guided by information prompts, directly manipulate virtual models (parts, tools) to complete the assembly process. This, combined with an interactive UI, visualizes the process ideas, allowing users to select additional auxiliary guidance information as needed. The entire assembly process and user operations are recorded and further analyzed to convert traditional process specifications into MR-assisted assembly process specifications. Therefore, each assembly scene corresponds to its own unique MR-assisted assembly process, including the current scene layout, the parts and tools used, and the worker's assembly steps. The assembly scene serves as the basic unit of the system application, establishing a correspondence between scene, assembly, and operation.

[0058] The system automatically identifies key assemblies to determine the current assembly scenario and corresponding assembly tasks. It utilizes virtual-real fusion technology to accurately overlay virtual information onto the real scenario, including corresponding part attribute information, MR-assisted operation guidance information, process information, etc., to enhance local users' understanding of the assembly scenario and assembly tasks. By implementing a visualization strategy for MR-assisted information that reflects the scenario-task-understanding, the system enhances intuitiveness and convenience.

[0059] MR collaborative technology enables remote collaboration between workers and experts, as well as human-machine collaboration between humans and systems. Remote experts on the VR end monitor the site conditions in a consistent virtual environment and issue virtual instructions for guidance, while local workers on the AR end receive visual guidance and perform actual assembly. Through human-machine interaction, workers can independently control the offline assisted guidance process and methods, enabling quality checks including measuring key part dimensions, bolt force limiting and pre-tightening, and blade sealing. Combining the high autonomy of human workers with the precision and efficiency of machines ensures orderly assembly, guarantees quality requirements, and improves assembly efficiency.

[0060] Based on the aforementioned MR-assisted assembly concept, this invention also proposes a mixed reality-based aero-engine assisted assembly method, which is applied to the assisted assembly of high-pressure turbine casings and low-pressure vortex-guided engines.

[0061] The following details the aircraft engine assisted assembly system and method based on mixed reality:

[0062] A mixed reality-based aircraft engine assisted assembly system:

[0063] The aircraft engine auxiliary assembly system based on mixed reality is divided into three levels:

[0064] The data layer (i.e., the scene, process, and virtual resource construction layer) includes the virtual assembly scene initialization module and the MR-assisted assembly process generation module;

[0065] The service layer includes a communication module, an instruction conversion module, and an extended function module;

[0066] The application layer includes a location tracking module, a recognition module, an interaction module, and a visualization module.

[0067] In the data layer, the virtual assembly scene initialization module first restores the virtual environment corresponding to the assembly site. Users operate the virtual model in it to complete virtual assembly. The complete assembly process is recorded by the MR-assisted assembly process generation module. MR-assisted assembly instructions are interactively constructed, forming an instruction set, and finally, MR-assisted assembly process specifications are automatically generated. The virtual scene and MR-assisted assembly process are ultimately stored on the server in a series of 3D model representations, TXT files, images, CSV files, and other data formats.

[0068] The service layer stores and manages all data on the server. The communication module receives and processes data submissions and access requests from various hardware, clients, and modules, and provides information transmission functions between them. The instruction conversion module helps different clients parse MR-assisted assembly processes. The extended function module stores the specific implementation interfaces of functions such as error and omission detection and surface quality inspection, and can be continuously expanded for users to selectively call.

[0069] The application layer is the basic component of the deployed client applications, which are divided into AR clients for users and VR clients for remote experts. The identification module scans the identification code to initialize the spatial coordinate system synchronization, and the positioning and tracking module runs continuously to locate the local worker's AR client and the target object. Users send commands such as start, next / previous step, display guidance information, and quality inspection to the system through the interaction module. The service layer receives the commands and initiates an assembly instruction acquisition request. The instruction conversion module determines the various resources and information required by the current client for the instruction, obtains them from the server and instantiates them into the AR client device. Through the visualization module, it maps them to the correct position in the real field of view, ultimately forming a complete offline MR-assisted assembly process.

[0070] During the assembly process, the AR client initiates a multi-person collaboration request command, requesting the VR client of a remote expert to work collaboratively through the communication module of the service layer. The VR client receives the initial virtual assembly scene stored in the server and the real-time spatial layout information provided by the AR client. In this way, the VR client restores a virtual scene that is consistent with the assembly site space and synchronized with the assembly status. The VR client user can issue guidance commands in an immersive way and transmit them to the AR client. The AR client visualizes the guidance information and realizes real-time online collaborative assisted assembly.

[0071] The virtual scene initialization module first determines the target scene for the MR-assisted assembly process. This can be done manually or by using the target detection function provided by the recognition module, which detects key objects (assembly bodies, special tooling, etc.) in the field to determine which workstation the current scene belongs to. Then, based on the target scene, it exports CAD models of relevant components, tools, workbenches, fixtures, and other entities from the external data management module (PDM) connected to this module. The assemblies retain complete spatial layout information and geometric relationships. After lightweight model processing, these models are loaded into a pre-prepared Unity-based virtual space, with added colliders, hand interaction components, and other functional components. Next, it exports text and images from traditional assembly process documents from the PDM, converting them into Unity-based virtual assets. These assets, along with virtual pointers, arrows, annotation boxes, and other markers commonly used in MR collaboration, are provided to the interactive user interface. Users can freely select and move their coordinates and poses in the virtual space. Finally, it adjusts the positions of all virtual assets distributed in the virtual environment to maintain a consistent spatial layout with the assembly site. Users can manually adjust these assets or use the pose matching function in the positioning and tracking module to achieve automatic alignment of individual objects. Finally, a virtual scene corresponding to the current assembly site is generated, and virtual assets such as 3D models, text, images, and markers are stored on the server in the form of FBX, TXT, PNG, etc. The scene layout information (what virtual assets are and their positions) is recorded in the Sense.CSV file using the instruction conversion module.

[0072] The MR-assisted assembly process generation module can define the assembly process in the generated virtual scene: 1) Users can directly manipulate the virtual models of parts based on their personal assembly experience and traditional processes displayed in the virtual environment in the form of text and images to perform virtual assembly. This module automatically records the complete operation process. 2) For each step completed by the user, all actions will be recorded, including which parts were manipulated, their initial and final positions, and movement trajectories. 3) During the completion of the step task, the user can select the necessary tools from the interactive UI interface, choose the most helpful guidance information (text, images) and markers, and determine their optimal visualization method. Their type, pose, and trajectory will also be recorded. 4) Determine the assembly sequence. Users can independently decide the assembly sequence in the virtual assembly process based on their personal habits and understanding of the assembly task, and specify the position of the current step in the entire process. If no specification is made, the default is the ascending sequence. 5) Optimize auxiliary guidance information. The interactive UI interface provides users with additional guidance information carriers beyond existing virtual assets to intuitively express their understanding of a certain step or operation. For a certain stage in the assembly process, users can add quality inspection operations based on control function modules. Steps 2) and 3) complete the creation of single-step assembly instructions in the MR-assisted assembly process, and their relevant information is recorded in the Step.CSV file through the instruction conversion module. All assembly instructions for each step, together with the auxiliary guidance information added in step 5), constitute the MR-assisted assembly process and are recorded in the Task.CSV file.

[0073] The instruction conversion module is responsible for converting the MR-assisted assembly process summarized in the complete virtual assembly process in the data layer into a series of unified CSV files, and parsing the CSV files into an information format that can be read by various client application layers. The Scene.CSV file stores, in a structured form, the virtual model IDs (pointing to the corresponding virtual assets on the server) of the work platform, tooling, parts, tools, virtual markers, text, and images in the virtual scene corresponding to each workstation, as well as their respective spatial positions and relative relationships, used to initialize the virtual assembly environment. The Step.CSV file includes assembly instruction information involved in each step: step ID (identifying the step), part model ID, initial / final spatial coordinates of the part, process guidance information, required tools, etc. The Task.CSV file organizes the steps according to the assembly task and adds additional auxiliary guidance information, quality inspection steps, etc. The CSV files generated by the instruction conversion module do not involve the virtual assets themselves but only use their numbers, therefore they are small in size and have a clear structure, facilitating fast communication between the client and the server, reducing file memory usage, and promoting cross-platform information exchange between multiple clients.

[0074] The communication module, acting as an information transmission relay station in the MR-assisted assembly system, receives and stores data such as text, images, 3D models, and CSV files generated by the data layer. It also receives and responds to data call commands from clients and other modules, and outputs data. Furthermore, it records information such as the control model ID, model pose, and client pose obtained by the AR / VR client positioning and tracking module, facilitating data communication between workers and experts and ensuring spatial layout consistency for synchronized multi-person collaborative operation. This module utilizes a high-performance computer as hardware support and employs the TCP protocol for network communication between the server and clients.

[0075] The extended functionality module is a scalable cloud workstation. Algorithms or application software for quality inspection functions that may be used during assembly, such as curve radius measurement, surface quality inspection, misassembly / missing part detection, and spacing measurement, are deployed on the cloud workstation and provided to client interfaces. Data is collected through sensors such as cameras, or user-input data; the aforementioned interfaces are called, calculations are performed on the cloud workstation, and the results are returned to the client. Within the limits of the cloud workstation's memory and performance, the module's functionality can be continuously expanded, providing rapid response, additional sensing and computing capabilities for the user, and avoiding the limitations of limited computing power and storage capacity of mobile hardware devices used by the client.

[0076] The positioning and tracking module relies on hardware devices such as visual sensors, laser sensors, inertial sensors, and spatial locators mounted on the VR / AR client device. Through SLAM algorithms and optical positioning technology, it performs real-time self-localization to determine the spatial position of the head-mounted device in the real environment. This enables the acquisition of spatial layout information, allowing MR-assisted assembly commands and guidance information to be registered accurately in the AR terminal and superimposed onto the correct position in the user's real field of vision. This virtual-real fusion achieves good realism and intuitiveness. It also provides a pose estimation function based on library model matching to determine the poses of key locations and components identified by the recognition module. This pose estimation is maintained synchronously during subsequent movements, while ensuring the spatial layout of the VR virtual environment remains consistent. AR devices can include Nreal Glasses, HoloLens, and other hardware supporting developer mode; VR devices can include HTC VIVE PRO, Oculus, etc.

[0077] The recognition module is used to recognize specified QR codes, barcodes, components, gestures, voice, etc. The positioning and tracking module uses the recognition module to recognize QR codes to initialize the initial position of the head-mounted MR device and register the virtual and real spaces for subsequent positioning and tracking; it identifies specific parts by affixing barcodes or natural features to components, enabling tracking of part poses and taking pictures and measurements of parts requiring quality inspection.

[0078] The interactive module enables human-computer interaction between the user and the system. Through a recognition module, it identifies user commands expressed via QR codes, gestures, voice, hand rays, etc., and the system provides feedback accordingly. This includes actions such as moving to the next / previous step in the assembly process, showing / hiding assembly guidance information, performing quality checks, and manipulating virtual models. The interactive module also enables multi-user collaboration supported by MR technology by recognizing, parsing, transmitting, and receiving collaborative commands between VR / AR clients. This enhanced interactivity improves the user experience and increases assembly efficiency.

[0079] The visualization module is responsible for obtaining the corresponding 3D model and guidance information from the server, and displaying them in the user's real field of vision in combination with the corresponding pose and visualization form.

[0080] Auxiliary assembly method based on the above system:

[0081] In the assembly process of the high-pressure turbine casing and low-pressure vortex guide of aero-engines, there are numerous components, the assembly space is highly complex, and multiple worktables need to be switched. The guide sleeve requires an interference fit and needs to be frozen before assembly. Each blade requires the installation of three sealing plates, emphasizing the assembly sequence of individual blades. The shaft diameters at both ends of the locating pins and their mating dimensions with the inner ring, front ring, and middle ring holes all have strict requirements. To avoid incorrect assembly, omissions, and non-compliance with assembly quality requirements, and to reduce the cognitive load on assembly workers, a mixed reality-based assisted assembly system is adopted to provide a mixed reality-based aero-engine assisted assembly method to guide operators in assembly, thereby improving assembly efficiency and ensuring process requirements.

[0082] The specific steps are as follows:

[0083] Step 1: Create a virtual scene.

[0084] Step 1.1: Using the HTC Vive Pro virtual reality headset (the kit includes a locator and controllers), the user enters a pre-prepared standard virtual construction space application. The headset provides real-time 3D point cloud data captured by a camera, as well as an interactive UI (including virtual buttons to respond to script functions, and interface functions provided by the virtual asset library, PDM, and other modules). Based on user specifications (i.e., user-defined) or automatic identification by the recognition module, the target workstation (assembly scene) for this assembly process is determined. Through the PDM interface, the 3D models of the workbench, tooling, and components corresponding to the assembly scene, along with traditional process specification text, images, virtual assembly tool models, virtual markers, and other virtual resources provided in the interactive UI, are instantiated into the virtual space. Their spatial poses and relationships are then adjusted manually or using the interface provided by the positioning and tracking module. Pre-prepared functional components are added to these virtual resources, such as adding colliders to components to prevent interference and adding hand interaction components for user operation.

[0085] Step 1.2: Click the Scene Publish button. Through the interface provided by the instruction conversion module, describe and record the current scene as a Scene.CSV file, which records the workstation name (assembly scene ID), virtual assets and their functional components, and spatial coordinates. Also, store the corresponding virtual asset number on the server.

[0086] Step 2: Interactively generate MR-assisted assembly process

[0087] Step 2.1: Using the user's virtual reality headset, enter the pre-prepared assembly process generation application and select the target assembly scene from the virtual interactive UI interface. The assembly process generation application submits a request to the server via the communication module to obtain the Scene.CSV description file corresponding to the assembly scene. After approval, the instruction conversion module parses this CSV file and, based on the client type that made the request, transmits the corresponding virtual assets from the server to the client and restores the spatial layout.

[0088] Step 2.2: In the virtual assembly scenario, user operations are collected through human-computer interaction. The virtual model is assembled step-by-step on the virtual workbench, and the initial and final positions and movement trajectories of the components are recorded. Users operate based on experience and the image and text prompts of traditional assembly processes. Based on user operations, tools and virtual annotations for marking the current action are selected and instantiated from the interactive interface. The most helpful parts for guiding the current task are marked in the text and images of traditional processes. Key control points in the assembly process are selected and recorded using voice, gaze, and gesture interactions, and additional guidance information (text, 3D annotations, etc.) is added through function buttons provided by the interactive interface. Based on user operations, the current operation is numbered to divide the steps of the MR-assisted assembly process, and it is determined whether quality checks are needed at each position between steps. A single step operation is an assembly instruction, and the instruction set and additional information are summarized into the final MR-assisted assembly process.

[0089] Step 2.3: The assembly instructions representing each work step are converted into a Step.CSV file using the instruction conversion module. This file contains the step ID, assembly scene ID, component ID, tool ID, process guidance information, coordinate information, and trajectory information. The MR-assisted assembly process is converted into a Task.CSV file, containing attributes such as assembly scene ID, assembly instruction set, assembly key point locations, additional guidance information, and quality inspection function interface ID, along with their inter-attribute relationships. The newly generated files and virtual assets are then stored on the server.

[0090] Step 3: Launch the MR-assisted assembly application on the AR client.

[0091] Step 3.1: The AR device (Hololens2) identifies the identification code in the real scene, completes pose initialization and spatial registration, and then performs continuous and stable self-localization and target tracking.

[0092] Step 3.2: The system identifies the assembly subject at the assembly site to determine the current workstation, and sends an application to the server to obtain the corresponding MR auxiliary assembly process. After approval, the system obtains the Task.CSV file.

[0093] Step 4: The system parses the obtained Task.CSV file to determine the relevant assembly scene layout, virtual assembly model, guidance information, pose, key point positions, visualization mode, and other information, and downloads the relevant resources.

[0094] Step 5: Utilize self-localization and target tracking functions, combined with pose information, to visualize virtual models, guidance information, interactive interfaces, and other resources in the correct area of ​​the AR device's real field of view.

[0095] Step 6: Under the correct and complete assembly assistance guidance process, the user completes the assisted assembly process:

[0096] AR recognizes and parses user intent commands expressed through gestures, voice, barcodes, and pressing virtual buttons: obtaining the ID of the model selected by the gesture and its corresponding CSV file representing the assembly step; executing the assembly sequence from when a button is pressed (obtaining the Task.CSV file of the first step); executing the previous / next assembly command; selecting the next / next workstation; and showing / hiding guidance information. This is an offline MR-assisted assembly process.

[0097] On the AR end, local workers select a multi-person collaborative virtual button. Information such as the current assembly scene layout, displayed model ID, model pose, client pose, and step ID is uploaded to the server, converted into information usable on the VR end, and transmitted to the VR end. The VR client uses this information to recreate a virtual scene consistent with the AR end in terms of space and operation synchronization, enabling remote experts on the VR end to monitor the AR end in real time. The VR end can manipulate the virtual model, issue virtual guidance commands, and relay the corresponding model ID, model pose, and command content to the AR end via the server for synchronized visualization. This step enables real-time online collaborative assisted assembly.

[0098] With the help of AR-enabled local workers, including guidance information such as images and text provided offline or through multi-person collaboration, as well as the accurate virtual-real registration of virtual models in their field of vision, they complete the assembly process.

[0099] Further quality testing can be conducted.

[0100] During the parts inventory process, the AR device selects the virtual parts inventory button by gesture. The system recognizes the gesture, calls the corresponding CSV file from the server, parses it, obtains and registers the virtual models of all the required parts into the real field of view. The user then compares them one by one to check if they are complete.

[0101] After the positioning pin is installed, select the virtual button for gap detection. The system identifies the pin and the inner and front rings, measures them using the HoloLens2's built-in camera, and transmits the measurement data to the interface of the corresponding detection application software on the server. After obtaining the calculation results, the data is sent back to the HoloLens2 and visualized in the real field of view.

[0102] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A mixed reality-based aero-engine auxiliary assembly system, characterized in that: It is divided into three layers: data layer, service layer, and application layer. The data layer includes a virtual assembly scene initialization module and an MR-assisted assembly process generation module. The service layer includes a communication module, an instruction conversion module, and an extended function module; The application layer includes a positioning and tracking module, an identification module, an interaction module, and a visualization module; In the data layer, the virtual environment corresponding to the assembly site is restored through the virtual assembly scene initialization module, and the user's operation data on the virtual model is collected to complete the virtual assembly; the complete assembly process is recorded through the MR-assisted assembly process generation module, MR-assisted assembly instructions are interactively constructed, and then an instruction set is formed, which finally automatically generates the MR-assisted assembly process specification; the virtual scene and MR-assisted assembly process are finally stored on the server in the form of data including 3D models, TXT files, pictures, and CSV files. The virtual assembly scene initialization module is connected to the external data management module PDM. Based on the target scene applied to the MR-assisted assembly process, the virtual assembly scene initialization module exports solid CAD models from the PDM, including parts, tools, workbenches, and fixtures. Complete spatial layout information and geometric relationships are preserved between the assemblies. After lightweight model processing, the models are loaded into a pre-prepared Unity-based virtual space, and functional components including colliders and hand interactions are added. The virtual assembly scene initialization module also exports text and images from traditional assembly process documents from the PDM and converts them into Unity-based virtual assets, which are then provided to the interactive user interface along with markers in the MR collaboration, including virtual pointers, arrows, and annotation boxes. The virtual assembly scene initialization module can also adjust the positions of all virtual assets distributed in the virtual environment to maintain a consistent spatial layout with the assembly site. Finally, a virtual scene corresponding to the current assembly site is generated, the virtual assets are stored on the server, and the scene layout information is recorded in a Sense.CSV file via the instruction conversion module. The service layer stores and manages all data in the server; the communication module receives and processes data submissions and access requests from various hardware, clients, and modules, and provides information transmission functions between them; the instruction conversion module parses the MR-assisted assembly process. The extended function module stores the function implementation interfaces, including error and omission detection and surface quality detection, for users to selectively call, and the interfaces can be extended; The application layer is the basic component of the deployed client applications, which are divided into AR clients used by users and VR clients used by remote experts. The recognition module initializes the spatial coordinate system synchronization by scanning the identification code, and the positioning and tracking module runs continuously to locate the local AR client and the target object. The interaction module receives commands sent by the user to the system and passes them to the service layer. The service layer receives the commands and initiates an assembly instruction acquisition request. The instruction conversion module determines the various resources and information required by the instruction for the current client, and then obtains and instantiates it from the server into the AR client device. Through the visualization module, it is mapped to the correct position in the real field of view, thus forming a complete offline MR-assisted assembly process; During the assembly process, the AR client initiates a multi-person collaboration request command, requesting the VR client of a remote expert to work collaboratively through the communication module of the service layer. The VR client receives the initial virtual assembly scene stored in the server and the real-time spatial layout information provided by the AR client. In this way, the VR client restores a virtual scene that is consistent with the assembly site space and synchronized with the assembly status. The VR client user can issue guidance commands in an immersive way and transmit them to the AR client. The AR client visualizes the guidance information and realizes real-time online collaborative assisted assembly.

2. The aero-engine assisted assembly system based on mixed reality according to claim 1, characterized in that: The MR-assisted assembly process generation module can record the complete operation process of a user directly manipulating the virtual model of parts and performing virtual assembly based on their personal assembly experience and traditional processes displayed in the virtual environment in the form of text and pictures. For each step a user completes, the MR-assisted assembly process generation module will record which parts the user manipulated, the initial and final positions of each part, and their movement trajectories. At the same time, during the process of completing the step task, the user selects the necessary tools from the interactive UI interface, selects the most helpful guidance information and markers for operation, and determines their best visualization method. The poses and trajectories of these tool types, guidance information, and markers are also recorded by the MR-assisted assembly process generation module. This completes the creation of single-step assembly instructions in the MR-assisted assembly process, and the relevant information is recorded in the Step.CSV file through the instruction conversion module; All assembly instructions for each step, along with additional user-created guidance information, together form the MR-assisted assembly process and are recorded in the Task.CSV file.

3. The aero-engine assisted assembly system based on mixed reality according to claim 2, characterized in that: The instruction conversion module converts the MR-assisted assembly process summarized in the complete virtual assembly process in the data layer into a series of unified CSV files, and parses the CSV files into an information format that can be read by each client application layer. The Scene.CSV file stores the virtual model IDs of each workstation in a structured format, including work platforms, tooling, parts, tools, virtual markers, text, and images, as well as their spatial positions and relative relationships, for initializing the virtual assembly environment. The Step.CSV file includes assembly instruction information for each step, including step ID, part model ID, initial / final spatial coordinates of the part, process guidance information, and required tools. The Task.CSV file organizes the steps according to the assembly task and includes additional guidance information and quality inspection steps.

4. The aero-engine assisted assembly system based on mixed reality according to claim 2, characterized in that: The communication module serves as an information transmission relay station in the MR-assisted assembly system. It receives and stores text, images, 3D models, and CSV file data generated by the data layer. It also receives and responds to data call commands issued by the client and various modules, and outputs the data. The module records the control model ID, model pose, and client pose information obtained by the AR / VR client positioning and tracking module, enabling data communication between workers and experts and achieving a consistent spatial layout to achieve the effect of synchronized multi-person collaborative operation.

5. The aero-engine auxiliary assembly system based on mixed reality according to claim 1, characterized in that: The extended function module is an expandable cloud workstation, on which the algorithm implementation or application software for quality inspection during the assembly process is deployed, and an interface is provided to the client for calling.

6. The aero-engine assisted assembly system based on mixed reality according to claim 1, characterized in that: The positioning and tracking module relies on the visual sensors, laser sensors, inertial sensors, and / or spatial locators on the client device to perform real-time self-localization using SLAM algorithms and optical positioning technology, determining the spatial position of the head-mounted device in the real environment. This enables the acquisition of spatial layout information, allowing MR-assisted assembly instructions and guidance information to be registered with high precision in the AR terminal, superimposed onto the correct position in the user's real field of vision. This virtual-real fusion achieves excellent realism and intuitiveness. Simultaneously, it provides a pose estimation function based on library model matching, determining the poses of key location points and components identified by the recognition module, maintaining synchronous pose estimation during subsequent movements, and ensuring the spatial layout of the VR virtual environment remains consistent.

7. The aero-engine assisted assembly system based on mixed reality according to claim 1, characterized in that: The identification module is used to identify specified QR codes, barcodes, components, gestures, and voice; the positioning and tracking module uses the identification module to identify QR codes to initialize the initial position of the head-mounted MR device and register the virtual space and real space for subsequent positioning and tracking; specific parts are identified by affixing barcodes or natural features to the components for tracking the position and pose of the parts and for taking pictures and measuring parts that require quality inspection.

8. A method for assisted assembly of an aero-engine based on the system described in any one of claims 1 to 7, characterized in that: Includes the following steps: Step 1: Create a virtual scene: Step 1.1: Using a virtual reality headset worn by the user, enter a pre-prepared standard virtual construction space application. The virtual reality headset provides a real-time 3D point cloud stream captured by a camera and an interactive UI interface. Based on user-specified information or an interface provided by the recognition module, automatically determine the assembly scenario for this construction and assembly process. Instantiate the 3D model corresponding to the assembly scenario and the virtual resources provided in the interactive UI interface into the virtual space through the interface provided by the PDM, and manually or using the interface provided by the positioning and tracking module to adjust their spatial poses and relationships. Add pre-prepared functional components to these virtual resources. Step 1.2: Deploy the scene: Use the interface provided by the instruction conversion module to describe and record the current scene as a Scene.CSV file and store it on the server; Step 2: Interactively generate MR-assisted assembly process: Step 2.1: Using the virtual reality headset worn by the user, enter the pre-prepared assembly process generation application, select the target assembly scene from the virtual interactive UI interface; The assembly process generation application submits a request to the server through the communication module to obtain the Scene.CSV description file corresponding to the assembly scene. After approval, the instruction conversion module parses this CSV file, and according to the client type that made the request, transmits the corresponding virtual assets in the server to the client and restores the spatial layout. Step 2.2: In the virtual assembly scenario, user operations are collected through human-computer interaction. The virtual model is assembled successfully by directly operating it on the virtual workbench, part by part and step by step. The starting and ending positions and movement trajectories of the parts in the assembly process are recorded. Based on the user operations, the tools needed for the current action and the virtual annotations used for marking are selected and instantiated from the interactive interface. In the text and images of traditional processes, the parts that are most helpful for guiding the current task are marked. Use voice, gaze, and gesture interaction to select and record key control points in the assembly process, and add additional guidance information through function buttons provided by the interactive interface; based on user operation, write the sequence number of the current operation to divide the steps of the MR-assisted assembly process, and determine whether quality inspection is required at this position between each step; a single step operation is an assembly instruction, and the resulting instruction set and additional information between them are summarized into the final MR-assisted assembly process. Step 2.3: Convert the assembly instructions representing the work steps into a Step.CSV file using the instruction conversion module; The MR-assisted assembly process is converted into a Task.CSV file; the newly generated files and virtual assets are stored on the server; Step 3: Launch the MR-assisted assembly application on the AR client: Step 3.1: The AR device identifies the identification code in the real scene, completes pose initialization and spatial registration, and then performs continuous and stable self-localization and target tracking. Step 3.2: The system identifies the main assembly unit at the assembly site to determine its current workstation, and sends an application to the server to obtain the corresponding MR auxiliary assembly process. Once approved, the system obtains the Task.CSV file. Step 4: The system parses the acquired Task.CSV file to determine the relevant assembly scene layout, virtual assembly model, guidance information, pose, key point positions, visualization mode information, and downloads the relevant resources; Step 5: Using self-localization and target tracking functions, combined with pose information, visualize the virtual model, guidance information, and interactive interface resources into the correct area of ​​the AR device's real field of view; Step 6: Under the assembly assistance guidance process, the user completes the assisted assembly process, including offline MR assisted assembly and real-time online collaborative assisted assembly.

Images