Augmented reality online inspection system and inspection method for aircraft assembly

Abstract

The present application relates to an augmented reality online inspection system and inspection method for aircraft assembly, including: a real-time data acquisition module, an augmented reality module, which adopts a client-server architecture, wherein the client part runs on a head-mounted display, and the server part is hosted in the cloud to achieve cross-platform support; the augmented reality module presents real-time information through augmented reality technology based on the collected and analyzed data, combines the data with the physical world, and presents augmented reality scenes, including virtual wing-body docking and measurement tools; the augmented reality module can overlay virtual labels, graphics, and warnings in the real environment, and provide a corresponding visual interface; the user configuration and customization module provides a user interface to allow users to customize the augmented reality view and data display, and set alarm and notification conditions. The present application relates to the field of intelligent manufacturing.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing of aviation equipment, and in particular to an augmented reality online detection system and a detection method for aircraft assembly. Background Art

[0002] In modern industrial and production environments, real-time monitoring and data visualization are essential to maintaining the efficiency and safety of operations. Augmented reality technology has shown great potential in this field, and can improve the efficiency of monitoring and maintenance through real-time information presentation, data analysis and visualization. The existing aircraft docking process has weak perception means and narrow range, and the virtual data has poor correspondence with the real entity and little interaction. In the field of traditional aircraft assembly, augmented reality technology has not received enough attention. With the increasing number of flexible parts in aircraft structures and the increasing complexity of structures, more and more data needs to be monitored in real time during the aircraft assembly process. The traditional weak perception and small range can no longer meet the mass production needs of large aircraft.

[0003] In addition, the existing technical solutions do not integrate the industrial Internet to study multi-source perception methods for the docking process, dynamic docking monitoring based on augmented reality, and virtual-reality interactive early warning and guidance. Real-time communication of data throughout the entire process of aircraft assembly and docking and immersive interaction in virtual reality are not yet possible. Summary of the invention

[0004] In view of the above problems in the prior art, the present application proposes an augmented reality online inspection system for aircraft assembly, comprising:

[0005] Real-time data acquisition module, which collects and analyzes various types of data, including image data, text data, and sensor readings;

[0006] The augmented reality module adopts a client-server architecture. The client part runs on the head-mounted display and the server part is hosted in the cloud to achieve cross-platform support. The augmented reality module presents real-time information through augmented reality technology based on the collected and analyzed data, combines data with the physical world, and presents augmented reality scenes, including virtual wing-body docking and measurement tools. The augmented reality module can overlay virtual labels, graphics and warnings in the real environment and provide a corresponding visual interface.

[0007] A user configuration and customization module provides a user interface to allow users to customize the augmented reality view and data display and set conditions for alerts and notifications.

[0008] Preferably, the augmented reality online detection system includes four levels:

[0009] Interface layer: responsible for presenting augmented reality scenes;

[0010] Functional layer: including augmented reality scene rendering, positioning and tracking of virtual objects; the functional layer is integrated with the augmented reality engine to ensure that virtual objects are correctly aligned with the actual scene;

[0011] Support layer: handles user interactions, including control panels, settings, and user configuration;

[0012] Data layer: receives view and sensor data, performs distance and clearance measurements, and feeds the results back to the user.

[0013] The present application also relates to a detection method of an augmented reality online detection system for aircraft assembly, which is characterized by comprising:

[0014] The augmented reality online detection system will first collect images of the target object, and then extract various features from the image, including the edge, color and texture of the object; these extracted features will serve as the basis for the system to identify the target object; the system will use the pre-captured template image to provide a reference for the recognition process; at the same time, the camera of the augmented reality device will capture images of the real world in real time, and the augmented reality online detection system will detect and describe feature points in these real-time images to ensure the real-time and accuracy of detection and recognition;

[0015] The augmented reality online detection system captures the user's field of view in real time and tries to find objects that match the matching template in the actual scene. Based on the matched objects, the position and direction of the target object in the real world are determined;

[0016] Collect image data of the wing in different postures in advance, use computer vision algorithms to detect feature points of the wing in the image data; generate descriptors for each feature point, select a portion of the feature points for offline training of the template; save the coordinates and descriptors of the selected feature points in a file to create a feature point set for the template;

[0017] When the feature points in the two images are matched, a transformation matrix needs to be determined to map the coordinates of one image to the other image;

[0018] Based on the estimated object pose, the virtual object is superimposed onto the real-time image to achieve an augmented reality effect; the position and orientation of the virtual object will be adjusted to align with the object based on the estimated pose;

[0019] The augmented reality view includes real-time rendering of the virtual wing-body docking scenario, as well as real-time feedback of measurement results.

[0020] Preferably, the target object image collected by the augmented reality online detection system is a specific part of the wing or fuselage.

[0021] Preferably, the image data of the wing in different postures include images of the wing in various rotational and translational positions, and images under different light and background conditions.

[0022] Preferably, the characteristic points of the wing are corner points or textures of the wing.

[0023] Preferably, the feature point set of the template is used for pose estimation and virtual overlay.

[0024] Preferably, the transformation matrix is ​​capable of translation, rotation, scaling, and perspective transformation.

[0025] Preferably, estimating the object pose includes estimating the position and orientation of the template, using saved template feature points and feature points of the real-time image for matching to determine the position and orientation of the wing in the real world.

[0026] Preferably, the control panel provides buttons and gesture controls that enable the user to select different measurement tools and toggle display options; the settings screen allows the user to customize application preferences.

[0027] The augmented reality online inspection system for aircraft assembly mainly implements the following contents:

[0028] (1) Real-time data acquisition: Collect real-time data from various sensors and data sources (e.g., cameras, temperature sensors, humidity sensors, etc.); the data acquisition module is able to process multiple types of data, including images, text, and sensor readings. (2) Augmented reality presentation: Based on the collected and analyzed data, real-time information is presented through augmented reality technology, combining data with the physical world; AR views can overlay virtual labels, graphics, and warnings in the real environment to provide a user-friendly visualization interface. (3) User configuration and customization: Provide a user interface to allow users to customize AR views and data displays; users can set conditions for alerts and notifications to suit their specific needs.

[0029] The design method of augmented reality online inspection system for aircraft assembly involves:

[0030] Architecture design: The AR application adopts a client-server architecture, where the client part runs on the AR headset and the server part is hosted in the cloud. This enables cross-platform support and covers more AR devices.

[0031] Furthermore, the technology stack is divided into client and server, where the client is developed using the Unity engine, providing rich augmented reality support and cross-platform capabilities, using C# as the main programming language. On the server side, Node.js is used as the backend framework, combined with the MongoDB database to store user configuration and measurement data.

[0032] Module design: The module design includes: (1) AR view module, which is responsible for presenting the AR scene, including virtual wing-body docking and measurement tools; (2) User interface module: handles user interaction, including control panel, settings and user configuration; (3) Measurement module: responsible for actual distance and gap measurement, using sensor data and AR scene information.

[0033] Data flow and data structure design: The data flow diagram shows the flow of data between the user and the application; the data structure involves: (1) scene description, which uses the JSON format to describe the virtual scene, including the location, size, and material of the objects; (2) user configuration, user settings and preferences are stored in JSON format, including unit systems, display options, and voice control settings; (3) measurement data, the data record of each measurement includes distance, gap position, and timestamp.

[0034] Algorithm design: The algorithm design includes: (1) Measurement algorithm, the application uses computer vision and sensor data to measure the distance and gap between objects. The input is the virtual object position and sensor data (such as depth sensor and camera data) provided for the augmented reality view; the distance and gap between objects are calculated using a three-dimensional space geometry algorithm; finally, the measurement results are presented on the augmented reality view in a user-friendly manner, including virtual lines and digital labels; (2) Performance optimization: In order to meet real-time performance requirements, GPU acceleration is used to speed up the calculation, and local caching of data is implemented to reduce memory usage.

[0035] User interface design: The user interface is divided into three main parts: augmented reality view, control panel, and settings. The augmented reality view is the main interface for users to interact with the virtual scene, the control panel provides selection and control of measurement tools, and the settings interface allows users to customize the appearance and behavior of the application.

[0036] Security and Privacy: Use OAuth 2.0 to ensure the security of user identities; all data is encrypted using SSL / TLS during transmission; only authorized users can access sensitive data and settings; comply with relevant privacy regulations, including GDPR, to ensure that users' personal data is properly protected; user data is only used for the normal operation of the application and will not be shared with third parties.

[0037] The above technical features can be combined in various suitable ways or replaced by equivalent technical features as long as the purpose of the present invention can be achieved.

[0038] The augmented reality online inspection system design method for aircraft assembly provided by the present invention has at least the following beneficial effects compared with the prior art:

[0039] It solves the problems of data collection, reading, enhanced data display and giving users a high degree of freedom to customize augmented reality views and data display to meet the needs of different working conditions, and improves aircraft assembly performance through the augmented reality online detection system during the aircraft assembly process. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention will be described in more detail below based on embodiments and with reference to the accompanying drawings, wherein:

[0041] Figure 1 This is a schematic diagram for overall description of the system functions of an embodiment of the present invention;

[0042] Figure 2 A tracking and registration module according to an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of the process of model matching and augmented reality display according to an embodiment of the present invention;

[0044] Figure 4 The embodiment of the present invention is Figure 3 The steps in the process being described;

[0045] Figure 5 For KepWare architecture;

[0046] Figure 6 The edge node cluster structure defined in the embodiment of the present invention;

[0047] Among them, OPC stands for OLE for process control, VM stands for virtual machine, and POD stands for data structure. DETAILED DESCRIPTION

[0048] The present invention will be further described below in conjunction with the accompanying drawings.

[0049] like Figure 1 As shown in the figure, it is a schematic diagram of the overall function of the system in the embodiment of the present invention, and the architecture of the online monitoring system of the augmented reality docking process is established. The system realizes functions through four levels, and its functions are divided as follows:

[0050] Layer 101, interface layer: responsible for presenting augmented reality scenes, including virtual wing-body docking and measurement tools.

[0051] Level 102, functional layer: This module includes augmented reality scene rendering, positioning and tracking of virtual objects. It integrates with the AR SDK to ensure that virtual objects are correctly aligned with the actual scene.

[0052] Layer 103, Support Layer: handles user interactions, including control panels, settings, and user configuration.

[0053] Layer 104, data layer: This module receives view and sensor data, performs distance and clearance measurements, and feeds the results back to the user.

[0054] like Figure 2 The following figure shows the tracking and registration module. The tracking and registration technology based on natural features and SLAM is used. The camera pose is estimated based on the natural features in the scene extracted by the algorithm. The SLAM technology framework is used to calculate the pose changes at different times. The extracted feature points and the camera pose are placed in the same world coordinate system to construct a virtual 3D scene to meet the requirements of large component docking scenarios.

[0055] like Figure 3 The figure shows a flow chart of model matching and augmented reality display, that is, the program processing flow and functions. Template matching is widely used to track and identify specific objects, patterns or signs in the real world to achieve the superposition and interaction of virtual content.

[0056] Step 301: The augmented reality system collects an image of a target object, which may be a specific part of a wing or fuselage.

[0057] Step 302, extract the features of this image, which can be edges, colors, textures, etc. These features will help the system identify the target object. The template image is usually a pre-captured clear image that contains the object or sign to be tracked. The real-time image is a real-world image captured by the camera of the augmented reality device. In the real-time image, feature point detection and description are also performed. In Unity, you can use the API in the Windows Mixed Reality SDK to access the camera of the augmented reality glasses. Set up real-time image acquisition through the classes in the Windows.Media.Capture namespace.

[0058] Step 303, create a matching template, which includes the feature descriptors mentioned in the previous section, and contains key information about the target object. The descriptors will be used to find matching targets in the real world.

[0059] In particular, when workers use an augmented reality system, the system captures their field of view in real time and tries to find objects in the real scene that match the matching template. This involves extracting features in the real scene and matching them with features in the matching template. If a match is found, the system can determine the position and orientation of the target object in the real world.

[0060] Step 304, collect image data of the wing in different postures in advance (these data may include images of the wing in various rotational and translational positions, and images under different light and background conditions). For each collected image, use a computer vision algorithm to detect feature points on the wing. These feature points may be corner points, textures, or other unique marks of the wing.

[0061] Step 305, generate descriptors for each feature point for subsequent matching. Select a portion of feature points from all acquired images, which will be used for offline training of the template. Save the coordinates and descriptors of the selected feature points in a file to create a feature point set for the template. This file will be used for subsequent pose estimation and virtual overlay.

[0062] Step 306, when the feature points in the two images are matched, a transformation matrix needs to be determined to map the coordinates of one image to the other image. This transformation matrix is ​​usually represented as a matrix H(3×3). It can perform translation, rotation, scaling, and perspective transformations, and H may be more complex. The calculation of the transformation matrix relies on the least squares method to minimize the reprojection error between the feature points.

[0063] Step 307, based on the estimated object pose, the virtual object is superimposed on the real-time image to achieve an augmented reality effect. The position and orientation of the virtual object will be adjusted based on the estimated pose to align with the object. In augmented reality applications, template matching can be used to continuously track objects to enable user interaction with virtual content. Users can view and interact with virtual objects by moving the augmented reality device.

[0064] like Figure 4 As shown, Figure 3 The steps of the process described. Includes:

[0065] Step 401, data collection, collects wing images in a laboratory or collection environment to obtain data in different postures.

[0066] Step 402, feature point detection and description, uses SIFT algorithm to detect feature points in the image and generate descriptors. Feature points are unique marks on the wings and fuselage. Descriptors contain information about the area around the feature points.

[0067] Step 403, saving the template feature points, saving the coordinates and descriptors of the selected feature points into a file for subsequent posture estimation and virtual superposition.

[0068] Step 404, the pose of the template is estimated by matching the saved template feature points (descriptors) with the feature points of the real-time image. The position and orientation of the wing in the real world are determined. This is achieved by using the OpenCV library function cv2.BFMatcher or cv2.FlannBasedMatcher to estimate the translation and rotation information of the wing using the matched feature points.

[0069] At the assembly site, the equipment is mainly divided into two categories, one is the automation equipment mainly based on tooling equipment, and the other is the measurement equipment mainly based on optical measuring instruments. The on-site tooling automation equipment is mainly composed of positioners, all of which are connected through a general PLC controller. The general controller is equipped with a digital I / O unit and a network communication unit, and transmits data through industrial Ethernet to complete the data collection of the automation equipment itself and the process. The measurement equipment is mainly laser trackers, laser rangefinders and visual cameras, all of which are connected through the professional measurement software Spatial Analyzer, and then the measurement data is directly transmitted to the module that needs to use the data through the Spatial Analyzer software interface.

[0070] In addition, for a large number of multi-source heterogeneous equipment and systems on site, there may be multiple industrial protocol standards coexisting, and different protocol standards are incompatible with each other, resulting in difficulties in protocol parsing, data format conversion and interconnection.

[0071] Based on the above, KepWare software is used to implement the protocol conversion function. Its internal framework is as follows: Figure 5 As shown in the figure, it uses technologies such as protocol analysis, data conversion, and middleware to be compatible with various industrial protocol standards and realize data format conversion and unification. On the other hand, it uses standard Internet protocols such as TCP to transmit the converted data to downstream data receivers.

[0072] The software acts as an industrial gateway, connecting the devices on site to the KepWare server through a hub. The KepWare server also builds an OPC server locally to establish the OPC UA information model of the device.

[0073] Figure 6 It is an edge node cluster structure. In the embodiment of the present invention, the data acquisition module together with the data preprocessing algorithm is classified as the workstation edge node, and they jointly play the role of edge computing in the entire system. At the assembly site, the large amount of data is one aspect. If all the equipment data is transferred to the cloud for processing, it will not only be costly, but also take more time. Due to the instability of the network, the availability of the system will deteriorate. On the other hand, some data at the assembly site has a short shelf life, such as the status information of the locator. If it is not processed in time, it will quickly deteriorate, and the value of the data will be lost.

[0074] The edge nodes are divided into individual workstations. The structure of each edge node is as follows: Figure 6 As shown, the characteristics are: (1) the data acquisition modules of each workstation are decoupled, and the data acquisition of each workstation works independently. Even if the data acquisition server of a workstation fails, the data acquisition process of other workstations is still in normal progress, and the data of this assembly process will not be lost due to the failure of the data acquisition server; (2) the assembly data of each workstation is similar. Therefore, designing a data analysis algorithm suitable for each workstation based on the data characteristics of the workstation can not only process data more quickly, but also improve the data analysis performance due to the targeted optimization of the algorithm; (3) the edge nodes of each workstation are combined to form a distributed server cluster of the assembly production line, which can realize unified automated operation and maintenance. In addition, when the assembly process is optimized and the assembly station is changed, the distributed cluster can achieve elastic scaling.

[0075] The user interface module is the last part of the augmented reality online monitoring system, which plays the role of interacting with the user and provides control, visualization and personalization options. Its design method is:

[0076] Step 701, the augmented reality view includes real-time rendering of a virtual wing-body docking scene and real-time feedback of measurement results;

[0077] Step 702, the control panel provides buttons and gesture controls, and the user can select different measurement tools and switch display options;

[0078] Step 703, the settings screen allows the user to customize the unit system, voice control and other application preferences.

[0079] The above disclosure is only a preferred embodiment of the present invention, but the present invention is not limited thereto. Any non-creative changes that can be thought of by a person skilled in the art, as well as several improvements and modifications made without departing from the principle of the present invention, should be within the scope of protection of the present invention.

Claims

1. An augmented reality online detection method for aircraft assembly, characterized in that: The following steps are involved: Step S1, the augmented reality online detection system will first collect the image of the target object through the edge node cluster structure. The edge nodes are divided according to the workstations, and the data acquisition modules of each workstation are decoupled. The data acquisition of each workstation works independently, and the edge nodes of each workstation are combined to form a distributed server cluster of the assembly production line; then various features are extracted from the image, including the corners, edges, colors and textures of the object; these extracted features will serve as the basis for the system to identify the target object; the system will use the pre-captured template image to provide a reference for the recognition process, and the template image contains the object or mark to be tracked; at the same time, the camera of the augmented reality device will capture the image of the real world in real time, and the augmented reality online detection system will detect and describe the feature points in these real-time images to ensure the real-time and accuracy of detection and recognition; Step S2: The augmented reality online detection system captures the user's field of view in real time and tries to find an object matching the matching template in the actual scene. The matching template includes a feature descriptor, which is used to find a matching target in the real world and determine the position and direction of the target object in the real world. Step S3, collecting image data of the wing in different postures in advance, including images of the wing in various rotational and translational positions, and images under different light and background conditions, and using a computer vision algorithm to detect feature points of the wing in the image data; feature point detection and description using a SIFT algorithm to detect feature points in the image and generate descriptors; Feature points are unique marks on a wing or fuselage; descriptors contain information about the area around the feature points; feature descriptors are generated for each feature point; Save the coordinates and descriptors of the selected feature points in a file to create a feature point set for the template; Step S4: When the feature points in the two images are matched, a transformation matrix is ​​determined to map the coordinates of the image to the other image; The transformation matrix can perform translation, rotation, scaling, and perspective transformations; Step S5: Based on the estimated object posture, the virtual object is superimposed on the real-time image to achieve an augmented reality effect; the position and orientation of the virtual object are adjusted based on the estimated posture to align with the object; Step S6: The augmented reality view includes real-time rendering of the virtual wing-body docking scene and real-time feedback of the measurement results.

2. The detection method according to claim 1, characterized in that: The image data of the wing in different postures includes images of the wing in various rotational and translational positions, as well as images under different lighting and background conditions.

3. The detection method according to claim 1, characterized in that: The feature point set of the template is used for pose estimation and virtual overlay.

4. The detection method according to claim 1, characterized in that: The control panel offers buttons and gesture controls that allow users to select different measurement tools and switch display options; the settings screen allows users to customize application preferences.

5. The detection method according to claim 1, characterized in that: The positioners at the assembly site are all connected through the PLC controller. The controller is equipped with a digital I / O unit and a network communication unit. Data is transmitted through industrial Ethernet to complete the data collection of the automation equipment itself and the process.

6. The detection method according to claim 1, characterized in that: The laser tracker, laser rangefinder and visual camera at the assembly site are all connected through the professional measurement software Spatial Analyzer, and then the measurement data is directly transmitted to the module that needs to use the data through the SpatialAnalyzer software interface.

7. An augmented reality online inspection system for aircraft assembly, characterized in that: Used to perform the detection method according to any one of claims 1 to 6, comprising: Real-time data acquisition module, which collects and analyzes various types of data, including image data, text data, and sensor readings; The augmented reality module adopts a client-server architecture. The client part runs on the head-mounted display and the server part is hosted in the cloud to achieve cross-platform support. The augmented reality module presents real-time information through augmented reality technology based on the collected and analyzed data, combines data with the physical world, and presents augmented reality scenes, including virtual wing-body docking and measurement tools. The augmented reality module can overlay virtual labels, graphics and warnings in the real environment and provide a corresponding visual interface. A user configuration and customization module provides a user interface to allow users to customize the augmented reality view and data display and set conditions for alerts and notifications.

8. The augmented reality online inspection system for aircraft assembly according to claim 7, characterized in that: The augmented reality online detection system includes four levels: Interface layer: responsible for presenting augmented reality scenes; Functional layer: including augmented reality scene rendering, virtual object positioning and tracking; The functional layer integrates with the augmented reality engine to ensure that virtual objects are correctly aligned with the actual scene; Support layer: handles user interactions, including control panels, settings, and user configuration; Data layer: receives view and sensor data, performs distance and clearance measurements, and feeds the results back to the user.

Images