A system and method for augmented reality based inspection of manual assembly processes

Abstract

The application provides a manual assembly process interference checking system and method based on augmented reality, which can superimpose a virtual assembly prototype on a real assembly environment to check and verify assembly process interference in a virtual-real combined form, and is efficient and low in cost.

Description

Technical Field

[0001] This invention belongs to the field of assembly process simulation and testing technology, specifically relating to an interference inspection system and method for manual assembly processes based on augmented reality. Background Technology

[0002] In modern manufacturing, assembly accounts for 20%–70% of the total workload and 40%–60% of the total manufacturing time. It involves significant manual labor and is very costly. Therefore, simulating and verifying the assembly process before actual production is essential. This helps identify potential problems during assembly, optimize component design and assembly process planning, and avoid downtime costs and losses due to adjustments during production. In assembly process simulation and verification, especially for large and complex products such as automobiles and aircraft, interference checks during manual assembly are a crucial step. The main focus is on verifying whether workers have sufficient operating space to complete the designed assembly tasks during actual assembly. For example, in automobile final assembly, there are numerous wiring harness installation steps where workers' hands need to smoothly pass through holes in areas such as the dashboard to complete the wiring harness connection. Due to the large number of components involved and the complex hand movements, simulation and calculation using currently widely used industrial software are difficult to perform.

[0003] In recent years, with the development of virtual reality (VR) technology, many assembly simulation systems based on virtual reality environments have been researched and developed. Chinese Patent Publication No. CN110299138A discloses a virtual assembly system for a reducer. The system includes modules such as an assembly information module, an assembly training module, and a navigation module. Users can understand the structure of the reducer and the working principle of each component by using virtual three-dimensional models and disassembly / assembly scenarios. Such systems are based on virtual reality technology and integrate natural manual operation processes, intuitive feedback, and three-dimensional models into the assembly process simulation process. However, there are still shortcomings: (1) There is still a gap between the pure virtual assembly scenario and the real assembly environment, and it cannot truly simulate the impact of the assembly environment; (2) In a pure virtual environment, the user's perception of the real environment will be closed and limited, and it will cause a bad user experience such as "motion sickness". Summary of the Invention

[0004] In view of this, the present invention provides an augmented reality-based system and method for inspecting the interference of manual assembly processes. It can overlay virtual assembly prototypes onto real assembly environments to inspect and verify the interference of assembly processes in a virtual-real combination, which is highly efficient and low-cost.

[0005] This invention is achieved through the following technical solution:

[0006] An augmented reality-based manual assembly process interference inspection system includes: HoloLens augmented reality glasses and embedded in the HoloLens augmented reality glasses a 3D model management module, a 3D model spatial positioning module, a UI interaction module, a gesture control module, a hand virtual mapping module, and a collision detection module;

[0007] HoloLens augmented reality glasses are used to provide the relevant hardware and software support for virtual assembly.

[0008] The UI interaction module provides an interactive interface for users;

[0009] The 3D model management module is used to load 3D model files of assembly components and assembly parts into the program, generating assembly component models and part models;

[0010] The 3D model spatial positioning module positions the assembly component model within the real scene;

[0011] The virtual hand mapping module uses a virtual hand to simulate a real hand, realizing the mapping of a real hand in virtual space;

[0012] The collision detection module has a built-in hierarchical collision detection algorithm. The gesture control module combines the hierarchical collision detection algorithm in the collision detection module to determine the assembly part model that the user is manipulating with gestures and the interaction intention with the assembly part model, and adjust the pose of the assembly part model.

[0013] The collision detection module is used to calculate in real time the collisions between moving objects and surrounding stationary objects during the assembly process of assembly components and parts, in order to check whether there are collisions and interferences during the virtual manual assembly process and to prompt the user.

[0014] Furthermore, the 3D model spatial positioning module positions the assembly component model within the real scene. Specifically, the 3D model spatial positioning module embeds an image recognition algorithm from the Vuforia augmented reality engine. The 3D model spatial positioning module obtains a marker image in the real scene by calling the RGB camera in the HoloLens augmented reality glasses. Based on the image recognition algorithm of the Vuforia augmented reality engine, the module recognizes the image and identifies the pose information of the marker image relative to the RGB camera. According to the preset positional relationship between the marker image and the assembly component model, the module positions the assembly component model at a specific spatial location in the real scene.

[0015] Furthermore, the virtual hand mapping module simulates a real hand using a virtual hand. Specifically, the software in the HoloLens augmented reality glasses is a gesture tracking API. The virtual hand mapping module calls the gesture tracking API to obtain the pose of key nodes of the real hand in real time, and overlays a cylindrical virtual joint model on each key node. The pose of the virtual joint model changes with the pose information of the key nodes of the real hand, simulating the movement and deformation of the real hand.

[0016] Furthermore, the hierarchical collision detection algorithm is divided into three stages, and different stages correspond to different hierarchical divisions, as detailed below:

[0017] Phase S1, the spatial segmentation phase, corresponds to the division of the space surrounding the part model or virtual hand. It calculates whether the space surrounding the part model or virtual hand collides with existing virtual objects. When the part model is located on the assembly component model, the surrounding space is the component's bounding box. When the part model is stored on a virtual shelf or inside a box, the surrounding space is the shelf's or box's bounding box. When the part model or hand is in a suspended state, it directly proceeds to phase S2.

[0018] Phase S2: Coarse detection phase. The corresponding layer for this phase is either the part bounding box layer or the hand bounding box layer. It calculates whether the part bounding box layer or the hand bounding box layer collides with the surrounding virtual objects.

[0019] Phase S3: Fine detection phase. The corresponding layer for this phase is the patch layer. The specific collision points are calculated based on the collisions between the patch layers.

[0020] Furthermore, the gesture control module is based on a gesture interaction intent recognition algorithm. The specific steps of the gesture interaction intent recognition algorithm are as follows:

[0021] Step S1: Combining the hierarchical collision detection algorithm within the collision detection module, determine that the virtual hand and the manipulated part model are in a grasping state;

[0022] Step S2: In the grasping state, calculate the grasping center of the virtual hand; the grasping center is the geometric center of the figure formed by connecting the contact points of the virtual hand and the manipulated part model;

[0023] Step S3: Determine the pose of the manipulated part model being grasped based on the pose change of the grasping center between the current frame and the previous frame.

[0024] Step S4: Determine whether all key nodes on the virtual hand are separated from the manipulated part model; if it is detected that all key nodes on the virtual hand are separated from the manipulated part model, it is determined that the manipulated part model has been released, the position and pose of the manipulated part model will no longer be updated, and the operation on the manipulated part model will end; if it is determined that the manipulated part model has not been released, then step S3 will be repeated continuously, and the position and pose of the 3D model will be refreshed in each frame to realize the manipulation of the 3D model.

[0025] Furthermore, step S1 is specifically implemented as follows: based on the virtual joint model of the hand, combined with the hierarchical collision detection algorithm in the collision detection module, it is determined whether the key nodes of the virtual hand are in contact with the manipulated part model; if all the key nodes of the virtual hand are in contact with the manipulated part model, it is determined that the manipulated part model has been successfully grasped and the state is changed to grasping; if not all the key nodes of the virtual hand are in contact with the manipulated part model, the manipulated part model is grasped again and the judgment is made again until the state is grasping.

[0026] Furthermore, the specific method of step S3 is as follows:

[0027] On the one hand, the displacement change ΔT of the capture center in the current frame relative to the previous frame is calculated. n :

[0028] ΔT n =P n -P n-1

[0029] Among them, P n Let P be the position vector of the capture center in the world coordinate system for the nth frame. n-1 This is the position vector of the capture center in the world coordinate system for the (n-1)th frame.

[0030] On the other hand, first calculate the pose change ΔQ of the grab center in the current frame relative to the previous frame. n :

[0031] ΔQ n =q n(-1) ·q n-1

[0032] Where, q n Let q be the pose vector of the capture center in the world coordinate system for the nth frame. n-1 Let q be the position vector of the center of the captured image in the world coordinate system in the (n-1)th frame. n and q n-1 All are represented using quaternions;

[0033] Then the attitude change ΔQ i Represented as quaternions: ΔQ i=w+xi+yj+zk

[0034] Where w, x, y and z are all real numbers, i·i=-1, j·j=-1, k·k=-1;

[0035] Then the attitude change ΔQ n Convert to rotation matrix ΔR n :

[0036]

[0037] Based on displacement change ΔT n and rotation matrix ΔR n Calculate the pose transformation matrix ΔM of the manipulated assembly part model in frame n. n ,

[0038]

[0039] Based on the pose transformation matrix ΔM of the nth frame n Solve for the pose of the manipulated assembly part model in frame n.

[0040] Furthermore, the collision detection module is based on the hierarchical collision detection algorithm, specifically as follows:

[0041] The hierarchical division corresponding to stage S1 is the component bounding box layer;

[0042] Determine if the moving object collides with the bounding box of a component in the assembly model. If a collision occurs, add all parts within that bounding box to the interference part list; otherwise, continue assembly.

[0043] The partitioning level corresponding to stage S2 is the part bounding box layer;

[0044] Calculate whether the moving object collides with the bounding box of each part in the interference part list. If a collision occurs, the part is retained in the interference part list; if no collision occurs, the part is removed from the interference part list to further narrow down the range of interference parts.

[0045] The partitioning level corresponding to stage S3 is the patch layer;

[0046] In the traversal phase S2, the list of interfering parts is obtained. For each part in the list of interfering parts, it is calculated whether there is a collision between the triangular facets of each part and the triangular facets of the moving object.

[0047] In summary, after collision calculations in stages S1-S3, the assembly part models and positions on the final assembly component models where collisions occur are determined; the collision detection module will set markers near the location where the collisions occur to indicate the assembly part models and positions where collisions occur.

[0048] Furthermore, in stage S1, the separation axis algorithm is used to determine whether the moving object collides with the bounding box of another component.

[0049] In stage S2, the separation axis algorithm is used to calculate whether the moving object collides with the bounding box of each part in the interference part list;

[0050] In stage S3, the GJK algorithm is used to calculate whether there is a collision between all the triangular facets in each part and the triangular facets of the moving object.

[0051] An augmented reality-based method for interferometry inspection of manual assembly processes, based on an augmented reality-based system for interferometry inspection of manual assembly processes, includes the following steps:

[0052] Step 1: The user places the 3D model file to be tested in the storage location specified by the augmented reality device, starts the program on the augmented reality device, uses the 3D model management module to load the 3D model file into the program, and generates the assembly component model and assembly part model; at the same time, the UI interface generated by the UI interaction module is launched.

[0053] Step 2: The user enables the 3D model space positioning module through the UI interaction module, calls the RGB camera of the HoloLens augmented reality device, identifies the pose of the marker map in the current environment, and renders the assembly component model at the corresponding real scene space position based on the identified pose of the marker map.

[0054] Step 3: The user selects to start assembly through the UI interaction module. According to the assembly plan, the user selects the corresponding manipulated part model through gesture interaction, grasps the manipulated part model and manipulates its movement and adjusts its posture to perform assembly.

[0055] During the assembly process, the program simultaneously calls the hand virtual mapping module, the gesture control module, and the collision detection module;

[0056] The virtual hand mapping module captures the pose data of key nodes of the user's hand in real time and maintains the pose of the virtual hand based on this data.

[0057] The gesture control module uses the pose of the user's hand key nodes in each frame, combined with gesture intent recognition algorithm and hierarchical collision algorithm, to determine the user's interaction intent, and then grabs and adjusts the pose of the manipulated part model according to the intent to respond to the user's control intent.

[0058] The collision detection module calculates the collision interference between the user's hand and the part models within the assembly component model, the manipulated part model, and the part models within the assembly component model in each frame.

[0059] Step 4: If a collision or interference occurs during the assembly process, the collision detection module uses virtual markers to display the location of the collision and prompts the user. The user will then readjust the assembly path of the parts based on the collision or interference information.

[0060] If the user is still unable to complete the assembly after adjusting the path, based on the user's subjective judgment, confirm that there is a problem with the structural design or assembly scheme of the current assembly, and record it.

[0061] Step 5: The user repeats steps 3 and 4 until all part models are assembled according to the assembly plan. Based on the analysis results during the virtual assembly process, the user determines the parts that need further adjustment and optimization of the design.

[0062] Beneficial effects:

[0063] (1) This invention integrates a 3D model management module, a 3D model spatial positioning module, a UI interaction module, a gesture control module, a hand virtual mapping module, and a collision detection module into HoloLens augmented reality glasses. Based on augmented reality technology, it constructs a virtual assembly space that blends the virtual and real worlds, enabling positioning and assembly in different real assembly scenarios using only one HoloLens augmented reality glasses. It can efficiently and conveniently perform interference checks during the assembly process in a virtual-real fusion space that perceives the real environment, thereby verifying the feasibility of the assembly process. No additional input devices are required, making it convenient and simple to use, providing a good user experience, and preventing dizziness.

[0064] (2) This invention uses the image recognition algorithm of the Vuforia augmented reality engine to recognize marker images in a real scene, identify the pose information of the marker images relative to the RGB camera, and locate the assembly component model at a specific spatial position in the real scene according to the preset positional relationship between the marker images and the assembly component model. The marker images are images that facilitate feature recognition and can be placed at any position in the real scene as needed, serving as a reference for positioning. Therefore, for the positioning of the assembly component model in the real scene, the HoloLens augmented reality glasses only need to recognize the pose of the marker images and locate the model according to the preset positional relationship between the marker images and the assembly component model. When changing the position of the assembly component model, only the marker images need to be moved, thus enabling simple and flexible positioning.

[0065] (3) The hand virtual mapping module of this invention uses a virtual hand to simulate a real hand. By utilizing the change in pose of the virtual joint model with the pose information of the key nodes of the real hand, it simulates the movement and deformation of the real hand. The setting of the hand virtual mapping module of this invention can realize the mapping of the real hand in virtual space in conjunction with the gesture tracking API embedded in HoloLens augmented reality glasses, providing a basis for subsequent calculations of contact and collision between the virtual hand and the virtual part model.

[0066] (4) The collision detection module of the present invention is embedded with a hierarchical collision detection algorithm. The hierarchical collision algorithm has three stages: spatial segmentation stage, coarse detection stage, and fine detection stage, which correspond to three hierarchical divisions from large to small. Compared with directly performing collision calculations on the surface layer, it can reduce the amount of calculation, improve the calculation efficiency, and provide timely feedback on collisions.

[0067] (5) This invention uses a gesture interaction algorithm to identify the user's gestures and interaction intentions on the manipulated part model, such as grasping the virtual model, moving the virtual model, and rotating the virtual model, and adjusts the pose of the virtual manipulated part model based on the intention. This realizes natural manipulation interaction of the manipulated part model in the virtual-real fusion scene. Users can directly manipulate the virtual manipulated part model with their hands. The verification process is intuitive and accurate.

[0068] (6) The collision detection module of the present invention examines the collision between parts from coarse to fine, and finally calculates the interference collision between the hand and the part model on the assembly component model, the manipulated part model and the part model on the assembly component model by calculating the collision between the facets that make up the virtual model. It can achieve accurate calculation and ensure the efficiency of calculation as much as possible.

[0069] (7) This invention provides an augmented reality-based method for interferometry testing in manual assembly processes. Specifically, it includes the following steps: program initialization; positioning the assembly component model in real space using a 3D model space positioning module; virtual assembly via a UI interaction module; during assembly, the program simultaneously calls a hand virtual mapping module, a gesture control module, and a collision detection module for technical support; if a collision or interference occurs during assembly, the user is prompted; if a collision still occurs after the user adjusts the assembly path, the user determines the parts requiring further adjustment and optimization based on the analysis results from the virtual assembly process. Through this method, users can utilize an augmented reality-based system for interferometry testing in manual assembly processes. In different real-world assembly scenarios, users can directly manipulate a virtual model to simulate the assembly process. Based on the hand virtual mapping module, the user's key hand node poses are identified. Combined with the gesture control module and the collision detection module, the parts requiring adjustment and optimization are determined, thus realizing virtual assembly and the assembly interference testing process. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of an augmented reality-based manual assembly process interference inspection system;

[0071] Figure 2 It is a diagram illustrating a virtual hand mapping and grasping a virtual object;

[0072] Figure 3 This is a flowchart of the algorithm used by the gesture recognition module to identify the intention of a gesture.

[0073] Figure 4 This is a flowchart of a collision detection algorithm between virtual objects;

[0074] Figure 5 This is a flowchart of an interference inspection method for manual assembly processes based on augmented reality. Detailed Implementation

[0075] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0076] Example 1:

[0077] This embodiment provides an augmented reality-based interference inspection system for manual assembly processes. See Appendix. Figure 1 It includes: HoloLens augmented reality glasses and embedded in HoloLens augmented reality glasses a 3D model management module, a 3D model spatial positioning module, a UI interaction module, a gesture control module, a hand virtual mapping module and a collision detection module;

[0078] HoloLens augmented reality glasses provide related software and hardware support for virtual assembly. The software is a gesture tracking API, and the hardware includes sensors, an RGB camera, and a network communication chip.

[0079] The UI interaction module provides an interactive interface for users;

[0080] The 3D model management module is used to load 3D model files from a specified storage location into the program, generating assembly component models and assembly part models;

[0081] The 3D model spatial positioning module positions the assembly component model within the real scene;

[0082] The virtual hand mapping module uses a virtual hand to simulate a real hand, realizing the mapping of a real hand in virtual space;

[0083] The collision detection module has a built-in hierarchical collision detection algorithm. The gesture control module combines the hierarchical collision detection algorithm in the collision detection module to determine the assembly part model (i.e. the manipulated part model) manipulated by the user's gesture and the interaction intention with the assembly part model, and adjust the pose of the assembly part model.

[0084] The collision detection module is used to calculate collisions between moving objects and surrounding stationary objects in real time during the assembly process. This is to check for collisions and interferences during the virtual manual assembly process and to alert the user. Moving objects include the virtual hand and the manipulated part model, while stationary objects include the assembly component model and the assembly scene.

[0085] This embodiment integrates the 3D model management module, 3D model spatial positioning module, UI interaction module, gesture control module, hand virtual mapping module, and collision detection module into the HoloLens augmented reality glasses. Based on augmented reality technology, it constructs a virtual assembly space that blends the virtual and real worlds, seamlessly connecting virtual 3D information with the real environment. This provides users with a method to directly manipulate virtual 3D models in the real environment to complete assembly process simulation and interference verification without limiting their perception of the real environment. It also enables positioning and assembly in different real assembly scenarios, making it simple, efficient, and convenient.

[0086] Furthermore, the 3D model spatial positioning module locates the assembly component model within the real-world scene. Specifically, this module incorporates an image recognition algorithm from the Vuforia augmented reality engine. It acquires a marker image from the real-world scene using the RGB camera in the HoloLens augmented reality glasses. Based on the Vuforia augmented reality engine's image recognition algorithm, it identifies the pose information of the marker image relative to the RGB camera. According to a preset positional relationship between the marker image and the assembly component model, the module positions the assembly component model at a specific spatial location within the real-world scene, achieving fusion between the assembly component model and the real-world environment. In a specific example, the preset positional relationship between the marker image and the assembly component model is that the assembly component model is located 30cm directly above the marker image. Users can also modify this positional value through the UI interaction module.

[0087] Among them, the marker image is an image that facilitates the identification of features. It can be placed at any location in the real scene as needed to serve as a reference for positioning.

[0088] Furthermore, the hand virtual mapping module uses a virtual hand to simulate a real hand, specifically as follows: see appendix. Figure 2The HoloLens augmented reality glasses use the gesture tracking API (Application Programming Interface) to obtain the pose of key nodes of the real hand in real time. A cylindrical virtual joint model is superimposed on each key node. The pose of the virtual joint model changes with the pose information of the key nodes of the real hand, simulating the movement and deformation of the real hand, and realizing the mapping of the real hand in virtual space.

[0089] Furthermore, the gesture control module is based on a gesture interaction intent recognition algorithm, see Appendix Figure 3 The specific steps of the gesture interaction intent recognition algorithm are as follows:

[0090] Step S1: Determine that the virtual hand and the manipulated part model are in a grasping state, specifically:

[0091] Based on the virtual joint model of the hand, combined with the hierarchical collision detection algorithm, it is determined whether the key nodes of the virtual hand are in contact with the manipulated part model. If all the key nodes of the virtual hand are in contact with the manipulated part model, it is determined that the manipulated part model has been successfully grasped and the state is changed to grasping. If not all the key nodes of the virtual hand are in contact with the manipulated part model, the manipulated part model is grasped again and the judgment is made again until the state is grasped, and then step S2 is performed.

[0092] Step S2: In the grasping state, calculate the grasping center of the virtual hand, which is the geometric center of the figure formed by connecting the contact points of the virtual hand and the manipulated part model.

[0093] Step S3: Determine the pose of the manipulated part model based on the pose change of the grab center between the current frame and the previous frame. Specifically:

[0094] On the one hand, the displacement change ΔT of the capture center in the current frame (i.e., the nth frame) relative to the previous frame (i.e., the (n-1)th frame) is calculated. n :

[0095] ΔT n =P n -P n-1 Formula (1)

[0096] Among them, P n Let P be the position vector of the capture center in the world coordinate system for the nth frame. n-1 The position vector of the capture center in the world coordinate system for the (n-1)th frame.

[0097] On the other hand, first calculate the pose change ΔQ of the grab center in the current frame (i.e., the nth frame) relative to the previous frame (i.e., the (n-1th frame). n:

[0098] ΔQ n =q n(-1) ·q n-1 Formula (2)

[0099] Where, q n Let q be the pose vector of the capture center in the world coordinate system for the nth frame. n-1 Let q be the position vector of the center of the captured image in the world coordinate system in the (n-1)th frame. n and q n-1 All are represented using quaternions;

[0100] Then the attitude change ΔQ i Represented as quaternions: ΔQ i =w+xi+yj+zk

[0101] Where w, x, y and z are all real numbers, i·i=-1, j·j=-1, k·k=-1;

[0102] Then the attitude change ΔQ n Convert to rotation matrix ΔR n :

[0103]

[0104] In summary, based on the displacement change ΔT n and rotation matrix ΔR n Calculate the pose transformation matrix ΔM of the manipulated part model in frame n. n ,

[0105]

[0106] Based on the pose transformation matrix ΔM of the nth frame n Solve for the pose of the manipulated part model in frame n.

[0107] Step S4: Determine whether all key nodes on the virtual hand are separated from the manipulated part model; if it is detected that all key nodes on the virtual hand are separated from the manipulated part model, it is determined that the manipulated part model has been released, the position and pose of the manipulated part model will no longer be updated, and the operation on the manipulated part model will end; if it is determined that the manipulated part model has not been released, then step S3 will be repeated continuously, and the position and pose of the 3D model will be refreshed in each frame to realize the manipulation of the 3D model.

[0108] Furthermore, the collision detection module is based on a hierarchical collision detection algorithm, see Appendix Figure 4 The hierarchical collision detection algorithm consists of three stages, and each stage corresponds to a different hierarchical division, as detailed below:

[0109] Phase S1, the spatial segmentation phase, corresponds to the division of the space surrounding the part model or virtual hand. It calculates whether the space surrounding the part model or virtual hand collides with existing virtual objects. When the part model is located on the assembly component model, the surrounding space is the component's bounding box. When the part model is stored on a virtual shelf or inside a box, the surrounding space is the shelf's or box's bounding box. When the part model or hand is in a suspended state, it directly proceeds to phase S2.

[0110] Phase S2: Coarse detection phase. The corresponding layer for this phase is either the part bounding box layer or the hand bounding box layer. It calculates whether the part bounding box layer or the hand bounding box layer collides with the surrounding virtual objects.

[0111] Phase S3: Fine detection phase. The corresponding layer for this phase is the patch layer. The specific collision points are calculated based on the collisions between the patch layers.

[0112] Furthermore, in the collision detection module, the division level corresponding to stage S1 is the component bounding box layer. That is, around each component, according to the boundaries of the parts that make up the component, a corresponding component bounding box is generated. Each component bounding box divides all the parts on the assembly into different regions.

[0113] The separation axis algorithm is used to determine whether the moving object collides with the bounding box of a component in the assembly model. The moving object is the hand and the manipulated part model. If a collision occurs, all parts within the bounding box of that component are added to the interference part list. If no collision occurs, the assembly continues.

[0114] The partitioning level corresponding to stage S2 is the part bounding box layer, that is, the corresponding part bounding box is generated around each static part model. In this embodiment, the OBB bounding box is used.

[0115] The separation axis algorithm is used to calculate whether the moving object collides with the bounding box of each part in the interference part list. If a collision occurs, the part is kept in the interference part list; if no collision occurs, the part is removed from the interference part list to further narrow down the range of interference parts.

[0116] Phase S3 corresponds to the facet layer; in augmented reality programs, virtual models are all composed of triangular faces. The interference part list obtained in Phase S2 is traversed, and the GJK (Gilbert–Johnson–Keerthi, GJK) algorithm is used to calculate whether any collision occurs between the triangular faces of each stationary part model and the triangular faces of the moving object.

[0117] In summary, after collision calculations in stages S1-S3, the part models and positions on the final assembly component model where the collision occurred are determined. The collision detection module will set a marker near the location of the collision to indicate to the user the part model and position where the collision occurred; in a specific example, a prominent virtual marker is rendered 10cm directly above the location of the collision to indicate to the user the stationary part model and position where the collision occurred.

[0118] Example 2:

[0119] This embodiment provides an augmented reality-based method for interferometry testing in manual assembly processes, based on an augmented reality-based system for interferometry testing in manual assembly processes described in Embodiment 1. (See attached document.) Figure 5 The specific steps are as follows:

[0120] Step 1: The user places the 3D model file to be tested in the storage location specified by the augmented reality device, starts the program on the augmented reality device, uses the 3D model management module to load the assembly component model into the program, generates the assembly component model and assembly part model, and starts the UI interface generated by the UI interaction module at the same time.

[0121] Step 2: The user enables the 3D model space positioning module through the UI interaction module, calls the RGB camera of the HoloLens augmented reality device, identifies the pose of the marker map in the current environment, and renders the assembly component model at the corresponding real scene space position according to the identified pose of the marker map, realizing the fusion of virtual model and real environment, and establishing a virtual assembly verification space that integrates virtual and real.

[0122] Step 3: The user selects to start assembly through the UI interaction module. According to the assembly plan, the user selects the corresponding manipulated part model through gesture interaction, grasps the manipulated part model and manipulates its movement and adjusts its posture to perform assembly.

[0123] During the assembly process, the program simultaneously calls the hand virtual mapping module, the gesture control module, and the collision detection module;

[0124] The virtual hand mapping module calls the gesture tracking API of HoloLens augmented reality glasses to capture the pose data of key nodes of the user's hand in real time, and maintains the pose of the virtual hand based on this data;

[0125] The gesture control module uses the pose of the user's hand key nodes in each frame, combined with gesture intent recognition algorithm and hierarchical collision detection algorithm, to determine the user's interaction intent, and then grabs and adjusts the pose of the manipulated part model according to the intent to respond to the user's control intent.

[0126] The collision detection module calculates the collision interference between the user's hand and the part models on the assembly component model, the manipulated part model, and the part models on the assembly component model in each frame.

[0127] Step 4: If a collision or interference occurs during the assembly process, the collision detection module uses virtual markers to display the location of the collision and prompts the user. The user will then readjust the assembly path of the parts based on the collision or interference information.

[0128] If the user is still unable to complete the assembly after adjusting the path, based on the user's subjective judgment, confirm that there is a problem with the structural design or assembly scheme of the current assembly, and record it.

[0129] Step 5: The user repeats steps 3 and 4 until all part models are assembled according to the assembly plan. Based on the analysis results during the virtual assembly process, the user determines the parts that need further adjustment and optimization of the design.

[0130] Through this embodiment, users can use an augmented reality-based manual assembly process interference inspection system. In different real assembly scenarios, users can directly use hand-manipulated virtual models to simulate the assembly process. Based on the hand virtual mapping module, the system identifies the key node poses of the user's hand. Combined with the gesture control module and collision detection module, the system realizes virtual assembly and assembly interference inspection.

[0131] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An augmented reality-based interference inspection system for manual assembly processes, characterized in that, include: HoloLens augmented reality glasses and the 3D model management module, 3D model spatial positioning module, UI interaction module, gesture control module, hand virtual mapping module and collision detection module embedded in HoloLens augmented reality glasses; HoloLens augmented reality glasses are used to provide the relevant hardware and software support for virtual assembly. The UI interaction module provides an interactive interface for users; The 3D model management module is used to load 3D model files of assembly components and assembly parts into the program, generating assembly component models and part models. The 3D model spatial positioning module positions the assembly component model within the real scene; The virtual hand mapping module uses a virtual hand to simulate a real hand, realizing the mapping of a real hand in virtual space; The collision detection module has a built-in hierarchical collision detection algorithm. The gesture control module combines the hierarchical collision detection algorithm in the collision detection module to determine the assembly part model that the user is manipulating with gestures and the interaction intention with the assembly part model, and adjust the pose of the assembly part model. The collision detection module is used to calculate in real time the collisions between moving objects and surrounding stationary objects during the assembly process of assembly components and parts, in order to check whether there are collisions and interferences during the virtual manual assembly process and to prompt the user. The hierarchical collision detection algorithm is as follows: Phase S1 is divided into layers corresponding to the space surrounding the part model or virtual hand. It calculates whether the space surrounding the part model or virtual hand collides with existing virtual objects. When the part model is located on the assembly component model, the surrounding space is the component's bounding box. When the part model is stored on a virtual shelf or inside a box, the surrounding space is the shelf's or box's bounding box. When the part model or hand is in a suspended state, it directly enters Phase S2. Phase S2: The corresponding layer for this phase is either the part bounding box layer or the hand bounding box layer; calculate whether the part bounding box layer or the hand bounding box layer collides with the surrounding virtual objects. Stage S3: The corresponding layer for this stage is the patch layer, and the specific collision points are calculated based on the collisions between the patch layers. The gesture control module is based on a gesture interaction intent recognition algorithm. The specific steps are as follows: Step S1: Combining the hierarchical collision detection algorithm within the collision detection module, determine that the virtual hand and the manipulated part model are in a grasping state; Step S2: In the grasping state, calculate the grasping center of the virtual hand; the grasping center is the geometric center of the figure formed by connecting the contact points of the virtual hand and the manipulated part model; Step S3: Determine the pose of the manipulated part model being grasped based on the pose change of the grasping center between the current frame and the previous frame. Step S4: Determine whether all key nodes on the virtual hand are separated from the manipulated part model; If it is detected that the key nodes on the virtual hand are separated from the manipulated part model, it is determined that the manipulated part model has been released, the position and orientation of the manipulated part model will no longer be updated, and the operation on the manipulated part model will end. If it is determined that the manipulated part model has not been released, step S3 is repeated continuously, and the position and pose of the 3D model are refreshed in each frame to achieve control of the 3D model.

2. The augmented reality-based manual assembly process interference inspection system as described in claim 1, characterized in that, The 3D model spatial positioning module locates the assembly component model in the real scene. Specifically, the 3D model spatial positioning module embeds the image recognition algorithm of the Vuforia augmented reality engine. The 3D model spatial positioning module obtains the marker image in the real scene by calling the RGB camera in the HoloLens augmented reality glasses, and recognizes the image based on the image recognition algorithm of the Vuforia augmented reality engine. It identifies the pose information of the marker image relative to the RGB camera, and locates the assembly component model at a specific spatial position in the real scene according to the preset positional relationship between the marker image and the assembly component model.

3. The augmented reality-based manual assembly process interference inspection system as described in claim 2, characterized in that, The virtual hand mapping module simulates a real hand using a virtual hand. Specifically, the software in the HoloLens augmented reality glasses is a gesture tracking API. The virtual hand mapping module calls the gesture tracking API to obtain the pose of key nodes of the real hand in real time, and overlays a cylindrical virtual joint model on each key node. The pose of the virtual joint model changes with the pose information of the key nodes of the real hand, simulating the movement and deformation of the real hand.

4. The augmented reality-based manual assembly process interference inspection system as described in any one of claims 1-3, characterized in that, The specific method of step S1 is as follows: Based on the virtual joint model of the hand, combined with the hierarchical collision detection algorithm in the collision detection module, it is determined whether the key nodes of the virtual hand are in contact with the manipulated part model; if all the key nodes of the virtual hand are in contact with the manipulated part model, it is determined that the manipulated part model has been successfully grasped and the state is changed to grasping; if not all the key nodes of the virtual hand are in contact with the manipulated part model, the manipulated part model is grasped again and the judgment is made again until the state is grasping.

5. The augmented reality-based manual assembly process interference inspection system as described in claim 4, characterized in that, The specific method of step S3 is as follows: On the one hand, it calculates the change in displacement of the capture center relative to the previous frame in the current frame. : in, For the first The frame captures the position vector of the center in the world coordinate system. For the first The position vector of the frame capture center in the world coordinate system; On the other hand, first calculate the change in pose of the grab center in the current frame relative to the previous frame. : in, For the first The pose vector of the frame capture center in the world coordinate system. For the first The frame captures the pose vector of the center in the world coordinate system. and All are represented using quaternions; Then attitude change Represented using quaternions: in, , , and All are real numbers. , , ; Then the attitude change amount Convert to rotation matrix : Based on displacement change and rotation matrix Calculate the manipulated assembly part model in the first... Frame pose transformation matrix , According to the Frame pose transformation matrix Solve the manipulated assembly part model in the first... The pose of the frame.

6. The augmented reality-based manual assembly process interference inspection system as described in claim 1, characterized in that, The collision detection module is based on the hierarchical collision detection algorithm, as follows: The hierarchical division corresponding to stage S1 is the component bounding box layer; Determine if the moving object collides with the bounding box of a component in the assembly model. If a collision occurs, add all parts within that bounding box to the interference part list; otherwise, continue assembly. The partitioning level corresponding to stage S2 is the part bounding box layer; Calculate whether the moving object collides with the bounding box of each part in the interference part list. If a collision occurs, the part is retained in the interference part list; if no collision occurs, the part is removed from the interference part list to further narrow down the range of interference parts. The partitioning level corresponding to stage S3 is the patch layer; In the traversal phase S2, the list of interfering parts is obtained. For each part in the list of interfering parts, it is calculated whether there is a collision between the triangular facets of each part and the triangular facets of the moving object. In summary, after the collision calculations in stages S1-S3, the assembly part models and positions on the assembly component model where the final collision occurs are determined. The collision detection module will set markers near the location where a collision occurs, indicating the assembly part model and location where the collision occurred.

7. The augmented reality-based manual assembly process interference inspection system as described in claim 6, characterized in that, In stage S1, the separation axis algorithm is used to determine whether the moving object collides with the bounding box of another component. In stage S2, the separation axis algorithm is used to calculate whether the moving object collides with the bounding box of each part in the interference part list; In stage S3, the GJK algorithm is used to calculate whether there is a collision between all the triangular facets in each part and the triangular facets of the moving object.

8. An augmented reality-based method for interferometry testing of manual assembly processes, based on the augmented reality-based system for interferometry testing of manual assembly processes as described in claim 2, characterized in that, Includes the following steps: Step 1: The user places the 3D model file to be tested in the storage location specified by the augmented reality device, starts the program on the augmented reality device, uses the 3D model management module to load the 3D model file into the program, and generates the assembly component model and assembly part model; at the same time, the UI interface generated by the UI interaction module is launched. Step 2: The user enables the 3D model space positioning module through the UI interaction module, calls the RGB camera of the HoloLens augmented reality device, identifies the pose of the marker map in the current environment, and renders the assembly component model at the corresponding real scene space position based on the identified pose of the marker map. Step 3: The user selects to start assembly through the UI interaction module. According to the assembly plan, the user selects the corresponding manipulated part model through gesture interaction, grasps the manipulated part model and manipulates its movement and adjusts its posture to perform assembly. During the assembly process, the program simultaneously calls the hand virtual mapping module, the gesture control module, and the collision detection module; The virtual hand mapping module captures the pose data of key nodes of the user's hand in real time and maintains the pose of the virtual hand based on this data. The gesture control module uses the pose of the user's hand key nodes in each frame, combined with gesture intent recognition algorithm and hierarchical collision algorithm, to determine the user's interaction intent, and then grabs and adjusts the pose of the manipulated part model according to the intent to respond to the user's control intent. The collision detection module calculates the collision interference between the user's hand and the part models within the assembly component model, the manipulated part model, and the part models within the assembly component model in each frame. Step 4: If a collision or interference occurs during the assembly process, the collision detection module uses virtual markers to display the location of the collision and prompts the user. The user will then readjust the assembly path of the parts based on the collision or interference information. If the user is still unable to complete the assembly after adjusting the path, based on the user's subjective judgment, confirm that there is a problem with the structural design or assembly scheme of the current assembly, and record it. Step 5: The user repeats steps 3 and 4 until all part models are assembled according to the assembly plan. Based on the analysis results during the virtual assembly process, the user determines the parts that need further adjustment and optimization of the design.

Images