A handheld tool suitability verification method and system based on augmented reality technology

Abstract

This invention discloses a method and system for verifying the applicability of handheld tooling based on augmented reality (AR) technology, belonging to the field of handheld tooling. Based on the analysis of existing 3D real-time tracking registration results and experimental effects, this invention systematically constructs scoring indicators for the object to be identified. It identifies the object using a natural feature point matching method, overlays the digital model of the product to be tested onto the verification environment, and achieves virtual-real fusion and synchronous operation of the virtual and physical models of the handheld tooling, thus realizing the applicability verification of the handheld tooling based on AR technology. This invention can improve user interactivity, reduce production costs, improve production quality, and shorten the production cycle. Furthermore, when constructing the handheld tooling applicability verification environment, the STL model compression method is used, which significantly reduces the number of vertices in the digital model of the product to be tested and the virtual model of the handheld tooling after compression, saving storage space occupied in AR devices and further improving verification efficiency.

Description

Technical Field

[0001] This invention relates to a method and system for verifying the suitability of handheld tooling based on augmented reality technology, belonging to the field of handheld tooling. Background Technology

[0002] With the development of Industry 4.0, the manufacturing industry needs new technologies to reduce product development time and costs. Augmented Reality (AR), as one of the most popular and cutting-edge technologies in the field of information technology, also has broad development prospects in industry, such as in the assembly process of large industrial products like aircraft and automobiles. Assembly, as the most important and time-consuming part of mechanical product production, often faces problems such as high skill thresholds, high employee training costs, overly cumbersome assembly steps, and a wide variety of similar parts required for assembly. Therefore, it requires highly skilled, capable, and proficient assembly workers, and AR technology can solve this problem. Traditional process verification techniques are cumbersome and easily affected by external factors, such as changes in tooling and fixtures, missing parts, and changes in the working environment. They also require steps such as assembly BOM, assembly order tracking, and signing and follow-up of process documents. AR technology can greatly simplify these cumbersome steps, replacing traditional paper BOMs with digitized electronic lists, and reducing assembly worker training costs through highly realistic virtual environments and reasonable human-computer interaction methods.

[0003] In aircraft manufacturing, the assembly of components such as cables, environmental control systems, hydraulic systems, and large parts is completed during the final aircraft assembly stage. Aircraft are characterized by high assembly quality requirements and relatively small production quantities; therefore, the vast majority of component assembly work is done manually by workers during final aircraft assembly. Thus, the analysis of the applicability of handheld tooling is particularly important in the product verification phase. Augmented reality (AR) technology allows manufacturers to quickly create virtual prototypes of products and verify their feasibility, obtaining rapid and direct design feedback. This enables the early identification of shortcomings in process or product design schemes, significantly improving the rationality of product design and the feasibility of process solutions, saving time and greatly reducing R&D costs. Furthermore, compared to virtual reality (VR), AR's fusion of virtual and real elements allows for a more realistic and direct recreation of production scenarios, providing operators with the most realistic operational feedback while saving costs. This fusion effect is achieved through high-precision collision detection technology, high-precision 3D tracking and registration technology, and highly realistic virtual scenes. Summary of the Invention

[0004] To address the issues of reduced production quality in traditional manufacturing processes due to unreasonable design and changes in the working environment, and the problems of difficult virtual simulation modeling, unrealistic human body modeling, and poor system interactivity in existing tooling suitability analysis processes, the main objective of this invention is to provide a handheld tooling suitability verification method and system based on augmented reality (AR) technology. This method constructs a handheld tooling suitability verification environment using AR devices and a physics engine, establishes scoring indicators for the objects to be identified, selects objects with high scores, and identifies the objects using natural feature point matching in a 3D real-time tracking registration module. The digital model of the product to be tested is then overlaid on the verification environment, achieving virtual-real fusion and synchronous operation of the virtual and physical models of the handheld tooling, thereby realizing handheld tooling suitability verification based on AR technology. This invention can improve user interactivity, reduce production costs, improve production quality, and shorten production time. Furthermore, when constructing the handheld tooling suitability verification environment, the STL model compression method is used to greatly reduce the number of vertices in the digital model of the product to be tested and the virtual model of the handheld tooling after compression, thereby shortening the import time of the digital model of the product to be tested and the virtual model of the handheld tooling and saving the storage space occupied by the digital model of the product to be tested and the virtual model of the handheld tooling in the AR device.

[0005] The scoring criteria include the image content, contrast, and actual physical size of the object to be identified collected by the AR device, the material of the object to be identified, and the intensity and uniformity of scattered light in the test environment.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] This invention discloses a handheld tooling suitability verification method based on augmented reality (AR) technology. The method constructs a handheld tooling suitability verification environment based on an AR device and a physics engine. This environment includes a digital model of the product to be tested and a virtual model of the handheld tooling. A scoring index for the object to be identified is constructed, and objects with higher scores are selected. The object is then identified using a natural feature point matching method in a 3D real-time tracking registration module. The digital model of the product to be tested is overlaid on the verification environment. Within the verification environment, the virtual model of the handheld tooling and a human hand are fused and synchronized. The implementation method is selected based on the presence or absence of a physical model of the handheld tooling. A box-type collider attribute is added to the virtual model of the handheld tooling to reduce the load on the AR device while ensuring collision accuracy. The collision body construction method is selected based on the characteristics of the digital model of the product to be tested and the verification environment. If the collision detection accuracy requirement of the digital model of the product to be tested is low, a mesh collider is used to fit the digital model of the product to be tested. Collision body attributes are added to the sub-models to obtain the collision body of the digital model of the product to be tested, thereby improving the collision detection accuracy. If the collision detection accuracy requirement of the digital model of the product to be tested is high, virtual mesh points of the digital model of the product to be tested are constructed according to the characteristics of the digital model of the product to be tested. A virtual mesh fitted to the surface of the digital model of the product to be tested is generated based on the graphics algorithm. The mesh point layout information is corrected by integrating the visual perception information of the AR device with the information of the digital model of the product to be tested, thereby improving the construction accuracy and perception accuracy of the collision body of the digital model of the product to be tested. A trigger detection method is used to perform collision detection between the virtual mesh of the digital model of the product to be tested and the box-type collision body of the virtual model of the handheld tooling. Then, the applicability verification of the handheld tooling is realized based on augmented reality technology. This invention can improve user interactivity, reduce production costs, improve production quality, and shorten the production cycle. Furthermore, when constructing the handheld tooling suitability verification environment, the STL model compression method is used to greatly reduce the number of vertices in the digital model of the product to be tested and the virtual model of the handheld tooling after compression, thereby shortening the import time of the digital model of the product to be tested and the virtual model of the handheld tooling and saving the storage space occupied by the digital model of the product to be tested and the virtual model of the handheld tooling in the AR device.

[0008] This invention discloses a method for verifying the applicability of handheld tooling based on augmented reality technology, comprising the following steps:

[0009] Step 1: Construct a handheld tooling suitability verification environment based on augmented reality devices and a physics engine. This environment includes a digital model of the product to be tested and a virtual model of the handheld tooling. During construction, an STL model compression method is used to significantly reduce the number of vertices in both the digital model and the virtual model, shortening the import time and saving storage space on the AR device.

[0010] Step 1.1: Lightweight processing is performed on the STL models of the digital model of the product to be inspected and the virtual model of the handheld tooling. Since the physics engine does not support the direct import of STL format files, we need to split a single STL format model into multiple objects, remove vertices with the same position, and then import them into the physics engine.

[0011] Step 1.1.1: Read all the vertex information of the STL model and divide these vertices into multiple sub-networks in a certain number.

[0012] Step 1.1.2: Select all vertices with the same position in each sub-network and set them as the same vertex, and store only the information of one of the vertices to achieve lightweight model processing.

[0013] Read the vertex information of each model triangle face one by one and create an array to store vertices and an array to store triangle faces. Then, determine whether the vertex positions of each face are the same. If they are the same, only store the information of one vertex and do not add information to the vertex array, but add the duplicate triangle face to the triangle face array.

[0014] Step 1.2: Import the Mixed Reality Toolkit (MRTK) into the physics engine to develop cross-platform AR applications, and use MRTK to perform operations such as rotating, grabbing, and scaling objects.

[0015] To achieve more accurate verification results, the Unity3D physics engine was selected.

[0016] Step 2: Construct a scoring index for the object to be identified, select objects with higher scores, and overlay the digital model of the product to be detected onto the verification environment using the natural feature point matching method in the 3D real-time tracking registration module.

[0017] Step 2.1: First, the format and size of the image to be recognized need to be modified. Second, the image is imported into the cloud platform for recognition. After importing into the platform, the platform scores the uploaded images based on image content richness, contrast, and physical size, and selects images with a score of four stars or above as the images to be recognized. Finally, the obtained data package is downloaded and imported into the physical engine to complete the image recognition.

[0018] Step 2.2: The server processes the uploaded image into grayscale, converting it to a black and white image. Feature points are then extracted from the black and white image using a natural feature point matching method.

[0019] Step 2.3: Identify the object to be identified by overlaying a predefined digital model from the physics engine onto the verification environment, thus enabling the digital model of the product to be tested to be overlaid onto the verification environment.

[0020] Step 3: Verify the virtual fusion and synchronous operation of the handheld tool virtual model and the human hand in the environment.

[0021] The implementation method is selected based on whether a physical model of the handheld tool is available. Step three involves the virtual-real fusion and synchronous operation of the handheld tool virtual model and the human hand, which includes the following two implementation methods.

[0022] Method 1: When there is a physical model of the handheld tooling, repeat the natural feature point matching method in the 3D real-time tracking registration module in step 2 to identify the object to be identified, realize the superposition of the virtual model of the handheld tooling onto the physical model of the handheld tooling, and realize the virtual-real fusion and synchronous operation of the virtual model and the physical model of the handheld tooling.

[0023] Method 2: When there is no virtual model of the handheld tool, construct the "grip pair" condition based on the characteristics of the physical process of the handheld tool's grasp and the augmented reality environment. Based on the "grip pair", construct the grasping intention recognition algorithm to determine the grasping situation between the human hand and the virtual model of the handheld tool, and realize the virtual-real fusion and synchronous operation of the virtual model of the handheld tool and the human hand.

[0024] The aforementioned "grip pair" consists of two contact points. If there is more than one "grip pair", the gripped virtual model is determined to be in a gripping state.

[0025] Step 3.1: Acquire an image of the human hand in the current frame, determine the position and pose data of the key nodes of the human hand relative to the AR device based on the convolutional neural network hand pose estimation algorithm, and overlay a virtual hand model at the key nodes of the hands identified in the current frame.

[0026] The virtual hand model consists of several virtual joint models, each of which is a cylinder, approximately simulating the finger joints of a real hand. There are topological relationships between the virtual hand joint models; that is, higher-level virtual joint models contain lower-level virtual joint models, and movement of a higher-level virtual joint model will cause movement of the lower-level virtual joint models. The virtual hand model is represented by the following parameterization:

[0027]

[0028] Where Jointti is the i-th virtual joint model, pi is the position of the virtual joint model, represented by a set of vectors xi, yi and zi in the augmented reality environment coordinate system, ei is the pose of the virtual joint model, represented by a set of vectors wi, ri and li in the augmented reality environment coordinate system, Size is the parameter of the virtual joint model, li represents the length of the cylinder, di represents the diameter of the cylinder, children represents the child joint models driven by this virtual joint model, and Jk represents the k-th virtual joint model.

[0029] Each virtual joint model in the virtual hand model corresponds to a key node in the gesture tracking and recognition process. The position and pose data of each recognized key hand node are used to update the position and pose of the virtual joint model in the current frame. The position and pose of the virtual hand model are determined based on the position and pose of the key nodes, thus realizing the mapping of real hands in virtual space. The formula is as follows:

[0030]

[0031] r i =R z (w i )R y (r i )R x (l i (5)

[0032] Where pi is the position vector of the i-th virtual joint model, ri is the rotation matrix of the virtual joint model, and the transformation relationship between this rotation matrix and Euler angles is shown in the equation, where Rz(wi) represents the rotation around the z-axis by wi degrees; T is the transformation matrix between the augmented reality environment coordinate system and the camera coordinate system in which the virtual joint model is located. Pi is the position vector of the key node corresponding to the i-th virtual joint model, and ri is the rotation matrix of the key node corresponding to the i-th virtual joint model.

[0033] Step 3.2: Based on the virtual joint model of the human hand, combined with the collision detection algorithm, determine whether the human hand and the virtual model of the handheld tool are in contact. If the collision detection algorithm detects that the human hand and the virtual model of the handheld tool are in contact, according to the grasping intention recognition algorithm, calculate whether a "grasping pair" can be formed between multiple contact points of the human hand and the virtual model of the handheld tool, and determine whether there is a grasping situation between the human hand and the virtual model of the handheld tool. If the human hand is in contact with different fingers, it is determined that the grasping is successful, and the grasping state is changed. The virtual and real fusion and synchronous operation of the virtual model of the handheld tool and the human hand are realized.

[0034] A "grip pair" consists of two contact points. If there is more than one "grip pair", the grasped virtual model is determined to be in a grasping state. There is no need to calculate the contact based on multiple contact points to determine whether the grasp is complete. This makes the grasping intention judgment more flexible, closer to the real three-dimensional gesture manipulation situation, more adaptable to complex gesture interaction scenarios, and more in line with the user's intuitive interaction experience. At the same time, if there are multiple pairs of "grip pairs", the contact points that make up the grip pairs will participate in the interaction intention recognition, improving the robustness, flexibility, efficiency and immersion of gesture interaction intention recognition.

[0035] The physical process of grasping a real object is characterized by applying the basic laws of Newtonian rigid body mechanics. It determines whether the object can be grasped based on whether it is in force balance and the friction between the contact surfaces of the virtual hand model and the manipulated model. The principle behind this is to analyze the force state of the object using a simplified Coulomb friction model.

[0036] The aforementioned "grip pair" consists of two contact points of a virtual hand model that meets certain conditions and the model being gripped. The conditions for the "grip pair" are as follows: the angle between the line connecting the two contact points and the normal to their respective contact surfaces does not exceed a fixed angle α; in this case, the two contact points will form a stable grip pair g(a,b). The fixed angle α is the friction angle.

[0037] The grasping intent recognition algorithm is established based on the "grasping pair" condition, iteratively determining whether all current contact points can form a "grasping pair" with another contact point. For any two contact points a and b between the virtual hand and the virtual object in one iteration, if the angle between the line connecting the two contact points and the normal to their respective contact surfaces does not exceed a fixed angle α, then the two contact points will form a stable grasping pair g(a,b). This fixed angle α is the friction angle, meaning the grasping pair g(a,b) should satisfy...

[0038]

[0039] Where, n a and n bLet l be the normal vector for contact points a and b, which is the normal vector of the cylindrical surface of the joint virtual model at the contact point; ab Let α be the line connecting contact points a and b; α is the friction angle, the value of which needs to be set through testing for the specific manipulated model to ensure stable and natural gripping of the virtual component.

[0040] Step 4: Add box collider attributes to the handheld tooling virtual model to reduce the load on the AR device while ensuring collision accuracy.

[0041] Add a box-type collider attribute to the virtual model of the handheld fixture, presenting it as a cuboid. The length, width, and height of this cuboid are all equal to the maximum values ​​of the entire handheld fixture in the three dimensions, which can reduce the hardware load while ensuring collision accuracy.

[0042] Step 5: Construct a virtual mesh for the digital model of the product to be tested based on the characteristics and accuracy requirements of the verification environment. If the collision detection accuracy requirement of the digital model of the product to be tested is low, a mesh collider is used to fit the digital model of the product to be tested. Collision attributes are added to the sub-models to obtain the collision bodies of the digital model of the product to be tested, thereby improving the collision detection accuracy. If the collision detection accuracy requirement of the digital model of the product to be tested is high, virtual mesh points are constructed based on the characteristics of the digital model of the product to be tested. A virtual mesh that fits the surface of the digital model of the product to be tested is generated based on a graphics algorithm. The mesh point layout information is corrected by integrating the visual perception information of the AR device with the information of the digital model of the product to be tested, thereby improving the construction accuracy and perception accuracy of the collision bodies of the digital model of the product to be tested.

[0043] Step 5.1: Select the collision body construction method for the digital model of the product to be tested based on the characteristics and accuracy requirements of the digital model of the product to be tested and the verification environment.

[0044] Step 5.2: If the collision detection accuracy requirement for the digital model of the product to be inspected is low, a mesh collider is used to fit the digital model of the product to be inspected. Collision attributes are added to each sub-model to obtain the collision body of the digital model of the product to be inspected, thus improving the collision detection accuracy. The entire digital model of the product to be inspected is divided into several sub-models, and collision attributes are added to each sub-model. The resulting collider can fit the product model with high accuracy, resulting in good collision detection accuracy and performance.

[0045] Step 5.3: If the collision detection accuracy requirement of the digital model of the product to be inspected is high, construct virtual grid points of the digital model of the product to be inspected according to the characteristics of the digital model of the product to be inspected, manually arrange the grid vertex set, generate a virtual grid that fits the surface of the digital model of the product to be inspected based on the graphics algorithm, and integrate the information of the digital model of the product to be inspected with the visual perception information of the AR device to correct the grid point layout information, thereby improving the construction accuracy and perception accuracy of the collision objects of the digital model of the product to be inspected.

[0046] After triangulation of the virtual mesh fitted to the surface of the digital model of the product to be inspected, a set of triangle vertex indices is obtained. The triangulation process decomposes the polygon into several triangles, with these vertices forming the polygon. Since the manually defined vertex sequence is the required direct modeling sequence, the set of triangle vertex indices is established based on the adjacency principle. Then, a combination of triangles is generated using a computer graphics algorithm to create a virtual mesh adapted to the surface of the digital model of the product to be inspected. The set of mesh vertices and the set of triangle vertex indices are represented by the following parameterization:

[0047] {V0, V1, V2, ..., V i , ...}, V i ∈polygon (8)

[0048]

[0049] Where polygon represents the set of vertices of the virtual mesh, V i Represents the i-th grid point; △i represents the i-th triangle forming the virtual grid, (V i 0 V i 1 V i 2 ) represents the set of indices of the three vertices of this triangle.

[0050] Step Six: Using a trigger detection method, collision detection is performed between the virtual mesh of the digital model of the product to be tested obtained in Step Five and the box-shaped collider of the virtual model of the handheld tooling obtained in Step Four, thereby realizing the applicability verification of the handheld tooling based on augmented reality technology.

[0051] The mathematical principle of the trigger detection algorithm is shown in the following formula:

[0052] A+B={a+b|a∈A, b∈B} (10)

[0053] A and B refer to the set of points on the collision body (virtual mesh of the digital model of the product to be tested, and box-type collision body of the virtual model of the handheld tooling), and a and b are points in A and B.

[0054] AB = {ab|a∈A,b∈B} (11)

[0055] Equation (11) is called the Minkowski difference, which means that when collision bodies A and B overlap or intersect, their difference set {ab} must contain the origin. For the trigger detection of collision bodies between the virtual mesh of the digital model of the product to be detected and the box-shaped collision body of the virtual model of the handheld tool, the difference set between the point set of the virtual mesh of the digital model of the product to be detected and the point set of the box-shaped collision body of the virtual model of the handheld tool is judged to verify whether interference has occurred. Containing the origin indicates that a collision has occurred.

[0056] The scoring criteria include the image content, contrast, and actual physical size of the object to be identified collected by the AR device, the material of the object to be identified, and the intensity and uniformity of scattered light in the test environment.

[0057] The present invention also discloses a handheld tooling applicability verification system based on augmented reality technology, used to implement the handheld tooling applicability verification method based on augmented reality technology. The handheld tooling applicability verification system based on augmented reality technology includes a data acquisition and processing module, a handheld tooling applicability verification environment construction module, a three-dimensional real-time tracking and registration module, and a collision detection module.

[0058] The data acquisition and processing module is used to acquire the RGB image and depth image of the current frame, and obtain the natural feature point information of the object to be identified based on the natural feature point matching method in the 3D real-time tracking registration module; and obtain the position and pose information of the key hand nodes in the current frame based on the convolutional neural network hand pose estimation algorithm.

[0059] The handheld tooling suitability verification environment construction module constructs a handheld tooling suitability verification environment based on augmented reality devices and a physics engine. The handheld tooling suitability verification environment includes a digital model of the product to be tested and a virtual model of the handheld tooling.

[0060] The 3D real-time tracking and registration module includes a digital model overlay submodule for the product to be detected and a virtual model overlay submodule for the handheld fixture. The digital model overlay submodule identifies the object to be identified using a natural feature point matching method and overlays the digital model of the product onto the verification environment. The virtual model overlay submodule for the handheld fixture refers to the virtual-real fusion and synchronous operation of the virtual model of the handheld fixture and the human hand in the verification environment. The implementation method is selected based on the presence or absence of a physical handheld fixture model. When a physical handheld fixture model is present, the object to be identified is identified using a natural feature point matching method, and the virtual model of the handheld fixture is overlaid onto the physical handheld fixture model. When no virtual handheld fixture model is present, a "grasping pair" condition is constructed based on the characteristics of the physical grasping process of the physical handheld fixture model and the augmented reality environment. A grasping intent recognition algorithm is then constructed based on the "grasping pair" to determine the grasping situation between the human hand and the virtual handheld fixture model, and to achieve the virtual-real fusion and synchronous operation of the virtual handheld fixture model and the human hand.

[0061] The collision detection module includes a handheld fixture virtual model collider construction submodule, a product-to-be-detected digital model collider construction submodule, and a trigger detection submodule. The handheld fixture virtual model collider construction submodule adds box-type collider attributes to the handheld fixture, presenting it as a cuboid. The length, width, and height of this cuboid are all equal to the maximum values ​​of the entire handheld fixture in the three dimensions, ensuring collision accuracy while reducing hardware load. The collider construction submodule of the digital model of the product under test refers to constructing a virtual mesh of the digital model of the product under test based on the characteristics and accuracy requirements of the digital model of the product under test and the verification environment. If the collision detection accuracy requirement of the digital model of the product under test is low, a mesh collider is used to fit the digital model of the product under test. Collision attributes are added to the sub-models to obtain the collider of the digital model of the product under test, thereby improving the collision detection accuracy. If the collision detection accuracy requirement of the digital model of the product under test is high, virtual mesh points of the digital model of the product under test are constructed according to the characteristics of the digital model of the product under test. A virtual mesh that fits the surface of the digital model of the product under test is generated based on a graphics algorithm. The mesh point layout information is corrected by integrating the visual perception information of the AR device with the information of the digital model of the product under test, thereby improving the construction accuracy and perception accuracy of the collider of the digital model of the product under test. The trigger detection submodule performs collision detection between the virtual mesh of the digital model of the product under test and the box-shaped collider of the handheld tooling virtual model, and then realizes the applicability verification of the handheld tooling based on augmented reality technology.

[0062] Beneficial effects:

[0063] 1. This invention discloses a handheld tooling applicability verification method and system based on augmented reality technology. Based on the analysis of existing 3D real-time tracking registration results and experimental effects, it is the first to systematically construct a scoring index standard for the object to be identified. The image of the object to be identified should be rich in content, have high contrast, appropriate real physical size, be made of non-reflective, relatively hard materials, be moderately bright under diffuse lighting, and have uniform illumination on the image surface. This ensures effective image collection and guarantees the effectiveness, accuracy, and efficiency of 3D real-time tracking registration. Objects with higher scores are selected, and the natural feature point matching method in the 3D real-time tracking registration module is used to identify the object. A predefined digital model is superimposed on the verification environment, achieving virtual-real fusion and synchronous operation of the handheld tooling's virtual and physical models.

[0064] 2. The handheld tooling applicability verification method and system disclosed in this invention, based on augmented reality technology, does not simply use one method, but rather selects a collider construction method according to the characteristics of the digital model of the product to be tested, based on specific working conditions, to meet the needs of more models and environments. If the collision detection accuracy requirement of the digital model of the product to be tested is low, a mesh collider is used to fit the digital model of the product to be tested, and the collider attributes are added to the sub-models to obtain the collider of the digital model of the product to be tested, thereby improving the collision detection accuracy. If the collision detection accuracy requirement of the digital model of the product to be tested is high, a virtual mesh point of the digital model of the product to be tested is constructed according to the characteristics of the digital model of the product to be tested, the mesh vertex set is manually arranged, a virtual mesh that fits the surface of the digital model of the product to be tested is generated based on the graphics algorithm, and the mesh point layout information is corrected by integrating the visual perception information of the AR device with the information of the digital model of the product to be tested, thereby improving the construction accuracy and perception accuracy of the collider of the digital model of the product to be tested.

[0065] 3. This invention discloses a handheld tooling applicability verification method and system based on augmented reality technology. In the verification environment, a virtual model of the handheld tooling and a human hand are fused and synchronized. Depending on whether a physical model of the handheld tooling is available, an implementation method is selected. Besides the traditional natural feature point matching method in the 3D real-time tracking registration module, handheld tooling applicability verification can also be performed when the physical model of the handheld tooling has not yet been produced or is inconvenient to obtain. The method employs a "grip pair" condition constructed based on the physical characteristics of real object grasping and the augmented reality environment. Based on the "grip pair," a grasping intent recognition algorithm is built to identify the user's grasping intent. If the collision detection algorithm detects that the hands are in contact with other virtual models, the grasping intent recognition algorithm calculates whether a "grasping pair" can be formed between multiple contact points between the hands and the manipulated model to determine whether there is a grasping situation between the hands and the virtual model. A "grasping pair" consists of two contact points. If there is more than one "grasping pair", the grasped virtual model is determined to be in a grasping state. There is no need to calculate the contact based on multiple contact points to determine whether the grasping is completed, making the grasping intent judgment more flexible, closer to the real three-dimensional gesture manipulation situation, more adaptable to complex gesture interaction scenarios, and more in line with the user's intuitive interaction experience. Attached Figure Description

[0066] Figure 1 This is a flowchart of a handheld tooling applicability verification method based on augmented reality technology disclosed in this invention.

[0067] Figure 2 This is a block diagram of a handheld tooling applicability verification system based on augmented reality technology disclosed in this invention.

[0068] Figure 3 This is a schematic diagram demonstrating the collision detection results. Detailed Implementation

[0069] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. The technical problems solved by the present invention and its beneficial effects are also described. It should be noted that the described embodiments are only intended to facilitate understanding of the present invention and do not constitute any limitation thereof.

[0070] This embodiment discloses a handheld tooling suitability verification system and method based on augmented reality technology, which is applied to the product verification process in manufacturing industries such as aircraft and automobiles. Taking the manual glue application process of the white body in automobile manufacturing as an example, the operator can use a real glue gun to apply glue to the virtual model of the white body, detect whether the handheld glue gun collides with the virtual model of the white body, and complete the white body design verification and handheld glue gun suitability analysis process.

[0071] like Figure 1As shown in the figure, this embodiment discloses a method for verifying the applicability of handheld tooling based on augmented reality technology. The specific implementation steps are as follows:

[0072] Step 1: Construct a handheld tooling suitability verification environment based on augmented reality devices and a physics engine. This environment includes a digital model of the vehicle body-in-white and a digital model of the handheld glue gun. During construction, an STL model compression method is used to significantly reduce the number of vertices in both the vehicle body-in-white and handheld glue gun models, shortening the import time and saving storage space on the AR device.

[0073] Step 1.1: Lightweight the STL models of the white body digital model and the handheld glue gun digital model. Since the physics engine does not support direct import of STL format files, we need to split the single STL format model into 17 sub-models, remove vertices with the same position, and then import them into the Unity3D physics engine.

[0074] Step 1.1.1: Read all the vertex information of the STL model and divide these vertices into multiple sub-networks in a certain number.

[0075] Step 1.1.2: Select all vertices with the same position in each sub-network and set them as the same vertex, and store only the information of one of the vertices to achieve lightweight model processing.

[0076] Read the vertex information of each model triangle face one by one and create an array to store vertices and an array to store triangle faces. Then, determine whether the vertex positions of each face are the same. If they are the same, only store the information of one vertex and do not add information to the vertex array, but add the duplicate triangle face to the triangle face array.

[0077] Step 1.2: Import the Mixed Reality Toolkit (MRTK) into the Unity3D physics engine to develop cross-platform AR applications, and use MRTK to perform rotation, grabbing, and scaling operations on objects.

[0078] Step 2: Using the natural feature point matching method in the 3D real-time tracking registration module, the digital model of the white body is superimposed on the verification environment.

[0079] Step 2.1: First, the format and size of the image to be recognized need to be modified. Second, the image is imported into the cloud platform for recognition. After importing into the platform, the platform scores the uploaded images based on image content richness, contrast, and physical size, and selects images with a score of four stars or higher as the images to be recognized. Finally, the obtained data package is downloaded and imported into the Unity3D physics engine to complete the image recognition.

[0080] Step 2.2: The server processes the uploaded image into grayscale, converting it to a black and white image. Feature points are then extracted from the black and white image using a natural feature point matching method.

[0081] Step 2.3: Recognize the image and overlay the white body digital model onto the verification environment in the Unity3D physics engine to achieve the overlay of the white body digital model onto the verification environment.

[0082] Step 3: Perform virtual-real fusion and synchronous operation of the handheld glue gun virtual model and the human hand in the verification environment.

[0083] The implementation method is selected based on whether or not a physical model of the handheld glue gun is available. Step three involves the virtual-real fusion and synchronous operation of the handheld glue gun model and the human hand, which includes the following two implementation methods.

[0084] Method 1: When there is a physical model of a handheld glue gun, repeat the natural feature point matching method in the 3D real-time tracking registration module in step 2 to recognize the image to be identified, so as to superimpose the virtual model of the handheld glue gun onto the physical model of the handheld glue gun, and realize the virtual-real fusion and synchronous operation of the virtual model and the physical model of the handheld glue gun.

[0085] Method 2: When there is no virtual model of the handheld glue gun, construct the "grip pair" condition based on the characteristics of the physical process of the handheld glue gun physical model and the augmented reality environment. Based on the "grip pair", construct the grip intention recognition algorithm to determine the grip between the human hand and the virtual model of the handheld glue gun, and realize the virtual-real fusion and synchronous operation of the virtual model of the handheld glue gun and the human hand.

[0086] The aforementioned "grip pair" consists of two contact points. If there is more than one "grip pair", the gripped virtual model is determined to be in a gripping state.

[0087] Step 3.1: Acquire an image of the human hand in the current frame, determine the position and pose data of the key nodes of the human hand relative to the AR device based on the convolutional neural network hand pose estimation algorithm, and overlay a virtual hand model at the key nodes of the hands identified in the current frame.

[0088] The virtual hand model consists of 19 virtual joint models, each a cylinder, approximating the finger joints of a real hand. There are topological relationships between the virtual hand joint models; higher-level virtual joint models contain lower-level ones. When a higher-level virtual joint model moves, it will cause the lower-level virtual joint models to move. For example, the movement of the root joint of a finger will cause the movement of the middle and top joints. The parametric representation of this virtual hand model is shown below:

[0089]

[0090] Where Jointti is the i-th virtual joint model, pi is the position of the virtual joint model, represented by a set of vectors xi, yi and zi in the augmented reality environment coordinate system, ei is the pose of the virtual joint model, represented by a set of vectors wi, ri and li in the augmented reality environment coordinate system, Size is the parameter of the virtual joint model, li represents the length of the cylinder, di represents the diameter of the cylinder, children represents the child joint models driven by this virtual joint model, and Jk represents the k-th virtual joint model.

[0091] Each virtual joint model in the virtual hand model corresponds to a key node in the gesture tracking and recognition process. The position and pose data of each recognized key hand node are used to update the position and pose of the virtual joint model in the current frame, thus mapping real hands in virtual space. The formula is shown below:

[0092]

[0093] r i =R z (w i )R y (r i )R x (l i (5)

[0094] Where pi is the position vector of the i-th virtual joint model, ri is the rotation matrix of the virtual joint model, and the transformation relationship between this rotation matrix and Euler angles is shown in the equation, where Rz(wi) represents the rotation around the z-axis by wi degrees; T is the transformation matrix between the augmented reality environment coordinate system and the camera coordinate system in which the virtual joint model is located. Pi is the position vector of the key node corresponding to the i-th virtual joint model, and ri is the rotation matrix of the key node corresponding to the i-th virtual joint model.

[0095] Step 3.2: Based on the virtual joint model of the human hand, combined with the collision detection algorithm, determine whether the human hand and the virtual model of the handheld glue gun are in contact. If the collision detection algorithm detects that the human hand and the virtual model of the handheld glue gun are in contact, according to the grasping intention recognition algorithm, calculate whether a "grasping pair" can be formed between multiple contact points of the human hand and the virtual model of the handheld glue gun, and determine whether there is a grasping situation between the human hand and the virtual model of the handheld glue gun. If the human hand makes contact on different fingers, it is determined that the grasp is successful, and the grasping state is changed. The virtual and real fusion and synchronous operation of the virtual model of the handheld glue gun and the human hand are realized.

[0096] A "grip pair" consists of two contact points. If there is more than one "grip pair", the grasped virtual model is determined to be in a grasping state. There is no need to calculate the contact based on multiple contact points to determine whether the grasp is complete. This makes the grasping intention judgment more flexible, closer to the real three-dimensional gesture manipulation situation, more adaptable to complex gesture interaction scenarios, and more in line with the user's intuitive interaction experience. At the same time, if there are multiple pairs of "grip pairs", the contact points that make up the grip pairs will participate in the interaction intention recognition, improving the robustness, flexibility, efficiency and immersion of gesture interaction intention recognition.

[0097] The physical process of grasping a real object is characterized by applying the basic laws of Newtonian rigid body mechanics. It determines whether the object can be grasped based on whether it is in force balance and the friction between the contact surfaces of the virtual hand model and the manipulated model. The principle behind this is to analyze the force state of the object using a simplified Coulomb friction model.

[0098] The aforementioned "grip pair" consists of two contact points of a virtual hand model that meets certain conditions and the model being gripped. The conditions for the "grip pair" are as follows: the angle between the line connecting the two contact points and the normal to their respective contact surfaces does not exceed a fixed angle α; in this case, the two contact points will form a stable grip pair g(a,b). The fixed angle α is the friction angle.

[0099] The grasping intent recognition algorithm is established based on the "grasping pair" condition, iteratively determining whether all current contact points can form a "grasping pair" with another contact point. For any two contact points a and b between the virtual hand and the virtual object in one iteration, if the angle between the line connecting the two contact points and the normal to their respective contact surfaces does not exceed a fixed angle α, then the two contact points will form a stable grasping pair g(a,b). This fixed angle α is the friction angle, meaning the grasping pair g(a,b) should satisfy...

[0100]

[0101] Where, n a and n bLet l be the normal vector for contact points a and b, which is the normal vector of the cylindrical surface of the joint virtual model at the contact point; ab Let α be the line connecting contact points a and b; α is the friction angle, the value of which needs to be set through testing for the specific manipulated model to ensure stable and natural gripping of the virtual component.

[0102] Step 4: Add box-type collision properties to the hand glue gun to reduce the load on the AR device while ensuring collision accuracy.

[0103] Add a box-type collision body attribute to the handheld glue gun, making it a cuboid. The length, width, and height of this cuboid are all equal to the maximum values ​​of the entire handheld glue gun in the three dimensions. This ensures collision accuracy while reducing the hardware load.

[0104] Step 5: Construct a virtual mesh for the digital body-in-white model based on its characteristics and accuracy requirements, considering the features and precision requirements of the digital body-in-white model and the verification environment. If the collision detection accuracy requirement for the digital body-in-white model is low, a mesh collider is fitted to the model, and collision attributes are added to the sub-models to obtain the collision bodies, thus improving collision detection accuracy. If the collision detection accuracy requirement for the digital body-in-white model is high, virtual mesh points are generated based on the characteristics of the model, and a virtual mesh fitted to the surface of the model is created using graphics algorithms. The mesh point layout information is then corrected by integrating the visual perception information from the AR device with the information from the digital body-in-white model, improving the construction and perception accuracy of the collision bodies.

[0105] Step 5.1: Select the collision body construction method for the digital model of the body-in-white based on the characteristics and accuracy requirements of the digital model of the body-in-white and the verification environment.

[0106] Step 5.2: If the required collision detection accuracy for the body-in-white digital model is low, a mesh collider is used to fit the body-in-white digital model. Collision body attributes are added to each sub-model to obtain the body-in-white digital model's collider, thus improving collision detection accuracy. The entire body-in-white digital model is divided into 17 sub-models, and collision body attributes are added to each sub-model. The resulting collider can fit the body-in-white digital model with high accuracy, resulting in good collision detection accuracy and performance.

[0107] Step 5.3: If the collision detection accuracy requirement of the digital body-in-white model is high, construct virtual grid points of the digital body-in-white model according to its characteristics, manually arrange the grid vertex set, generate virtual grids on the surface of the digital body-in-white model based on graphics algorithms, and modify the grid point layout information by integrating the visual perception information of the AR device with the information of the digital body-in-white model, thereby improving the construction accuracy and perception accuracy of the collision object of the digital body-in-white model.

[0108] After triangulation of the virtual mesh fitted to the surface of the digital model of the body-in-white, a set of triangle vertex indices is obtained. Triangulation decomposes the polygon into several triangles, with these vertices forming the polygon. Since the manually defined vertex sequence is the required direct modeling sequence, the set of triangle vertex indices is established based on the adjacency principle. Then, a combination of triangles is generated using a computer graphics algorithm to create a virtual mesh adapted to the surface of the digital model of the body-in-white. The set of mesh vertices and the set of triangle vertex indices are represented by the following parameterization:

[0109] {V0, V1, V2, ..., V i , ...}, V i ∈polygon (8)

[0110]

[0111] Where polygon represents the set of vertices of the virtual mesh, V i Represents the i-th grid point; △i represents the i-th triangle forming the virtual grid, (V i 0 V i 1 V i 2 ) represents the set of indices of the three vertices of this triangle.

[0112] Step Six: Using a trigger detection method, collision detection is performed between the virtual mesh of the white body digital model obtained in Step Five and the box-shaped collision body of the handheld glue gun virtual model obtained in Step Four, thereby realizing the applicability verification of the handheld tooling based on augmented reality technology.

[0113] If the digital model of the car body collides with the virtual model of the handheld glue gun, the virtual model of the handheld glue gun will turn red. For example... Figure 3 As shown, the virtual model of the handheld glue gun turns red, indicating that a collision has occurred between the virtual model of the handheld glue gun and the digital model of the white body.

[0114] The mathematical principle of the trigger detection algorithm is shown in the following formula:

[0115] A+B={a+b|a∈A, b∈B} (10)

[0116] A and B refer to the set of points on the collision body (virtual mesh of the white body digital model, box-shaped collision body of the handheld glue gun virtual model), and a and b are points in A and B.

[0117] AB = {ab|a∈A,b∈B} (11)

[0118] Equation (11) is called the Minkowski difference, which means that when collision bodies A and B overlap or intersect, their difference set {ab} must contain the origin. For the trigger detection of the collision body between the virtual mesh of the white body digital model and the box-shaped collision body of the handheld glue gun virtual model, the difference set between the virtual mesh point set of the white body digital model and the box-shaped collision body point set of the handheld glue gun virtual model is used to verify whether interference has occurred. Containing the origin indicates that a collision has occurred.

[0119] like Figure 2 As shown, this embodiment also discloses a handheld tooling applicability verification system based on augmented reality technology, used to implement the handheld tooling applicability verification method based on augmented reality technology. The handheld tooling applicability verification system based on augmented reality technology includes a data acquisition and processing module, a handheld tooling applicability verification environment construction module, a three-dimensional real-time tracking and registration module, and a collision detection module.

[0120] The data acquisition and processing module is used to acquire the RGB image and depth image of the current frame. Based on the natural feature point matching method in the 3D real-time tracking registration module, it obtains the natural feature point information of the image to be identified from the RGB image. Based on the convolutional neural network hand pose estimation algorithm, it obtains the position and pose information of the key hand nodes in the current frame from the RGB image and depth image.

[0121] The handheld tooling suitability verification environment construction module constructs a handheld tooling suitability verification environment based on augmented reality devices and the Unity3D physics engine. The handheld tooling suitability verification environment includes a digital model of the white body and a virtual model of a handheld glue gun.

[0122] The 3D real-time tracking and registration module includes a white body digital model overlay submodule and a handheld glue gun virtual model overlay submodule. The white body digital model overlay submodule uses a natural feature point matching method to identify the image to be recognized and overlays the white body digital model onto the verification environment. The handheld glue gun virtual model overlay submodule refers to the virtual-real fusion and synchronization operation of the handheld glue gun virtual model and the human hand in the verification environment. The implementation method is selected based on whether a physical handheld glue gun model is present. When a physical handheld glue gun model is present, the image to be recognized is identified using natural feature point matching to overlay the handheld glue gun virtual model onto the physical handheld glue gun model. When no virtual handheld glue gun model is present, a "grip pair" condition is constructed based on the physical characteristics of the handheld glue gun model's gripping process and the augmented reality environment. A gripping intention recognition algorithm is then built based on the "grip pair" to determine the gripping situation between the human hand and the handheld glue gun virtual model, achieving virtual-real fusion and synchronization between the handheld glue gun virtual model and the human hand.

[0123] The collision detection module includes a handheld glue gun virtual model collision body construction submodule, a body-in-white digital model collision body construction submodule, and a trigger detection submodule. The handheld glue gun virtual model collision body construction submodule adds box-shaped collision body attributes to the handheld glue gun, presenting it as a cuboid. The length, width, and height of this cuboid are all equal to the maximum values ​​of the entire handheld fixture in the three dimensions, ensuring collision accuracy while reducing hardware load. The body-in-white digital model collision body construction submodule constructs a virtual mesh of the body-in-white digital model based on the characteristics and accuracy requirements of the body-in-white digital model and the verification environment. If the collision detection accuracy requirement of the body-in-white digital model is low, a mesh collision body is used to fit the body-in-white digital model, and collision body attributes are added to the sub-models separately to obtain the collision body of the body-in-white digital model, improving collision detection accuracy. If the collision detection accuracy requirement of the body-in-white digital model is high, virtual mesh points of the body-in-white digital model are constructed based on the characteristics of the body-in-white digital model. A virtual mesh fitted to the surface of the body-in-white digital model is generated based on a graphics algorithm. The mesh point layout information is corrected by integrating the visual perception information of the AR device with the information of the body-in-white digital model, improving the construction accuracy and perception accuracy of the collision body of the body-in-white digital model. The trigger detection submodule performs collision detection on the virtual grid points of the white body digital model and the box-shaped collision body of the handheld glue gun virtual model, thereby realizing the applicability verification of the handheld tooling based on augmented reality technology.

[0124] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for verifying the applicability of handheld tooling based on augmented reality technology, characterized in that: Includes the following steps: Step 1: Construct a handheld tooling suitability verification environment based on augmented reality devices and a physics engine. The handheld tooling suitability verification environment includes a digital model of the product to be tested and a virtual model of the handheld tooling. When constructing the handheld tooling suitability verification environment, the STL model compression method is used to greatly reduce the number of vertices in the digital model of the product to be tested and the virtual model of the handheld tooling after compression, thereby shortening the import time of the digital model of the product to be tested and the virtual model of the handheld tooling and saving the storage space occupied by the digital model of the product to be tested and the virtual model of the handheld tooling in the AR device. Step 2: Construct scoring indicators for the objects to be identified, select objects with higher scores, and identify the objects with higher scores using the natural feature point matching method in the 3D real-time tracking registration module. Then, overlay the digital model of the product to be tested onto the verification environment. Step 3: Perform virtual-real fusion and synchronous operation of the handheld tooling virtual model and the human hand in the verification environment; Step 4: Add box collider attributes to the handheld tooling virtual model to reduce the load on the AR device while ensuring collision accuracy; Add a box-type collider attribute to the virtual model of the handheld fixture, presenting it as a cuboid. The length, width, and height of this cuboid are all equal to the maximum values ​​of the entire handheld fixture in the three dimensions, which can reduce the hardware load while ensuring collision accuracy. Step 5: Construct a virtual mesh for the digital model of the product to be tested based on the characteristics and accuracy requirements of the digital model and the verification environment. If the collision detection accuracy requirement of the digital model of the product to be tested is low, a mesh collider is used to fit the digital model of the product to be tested. Collision attributes are added to the sub-models to obtain the collision bodies of the digital model of the product to be tested, thereby improving the collision detection accuracy. If the collision detection accuracy requirement of the digital model of the product to be tested is high, virtual mesh points are constructed based on the characteristics of the digital model of the product to be tested. A virtual mesh is generated based on a graphics algorithm to fit the surface of the digital model of the product to be tested. The mesh point layout information is corrected by integrating the visual perception information of the AR device with the information of the digital model of the product to be tested, thereby improving the construction accuracy and perception accuracy of the collision bodies of the digital model of the product to be tested. Step Six: Using a trigger detection method, collision detection is performed between the virtual mesh of the digital model of the product to be tested obtained in Step Five and the box-shaped collider of the virtual model of the handheld tooling obtained in Step Four, thereby realizing the applicability verification of the handheld tooling based on augmented reality technology.

2. The handheld tooling applicability verification method based on augmented reality technology as described in claim 1, characterized in that: The implementation method for step one is as follows: Step 1.1: Lightweight processing is performed on the STL models of the digital model of the product to be inspected and the virtual model of the handheld tooling. Since the physics engine does not support the direct import of STL format files, we need to split a single STL format model into multiple objects, remove vertices with the same position, and then import them into the physics engine. Step 1.1.1: Read all the vertex information of the STL model and divide these vertices into multiple sub-networks according to a certain number; Step 1.1.2: Select all vertices with the same position in each sub-network and set them as the same vertex, and store only the information of one of the vertices to achieve lightweight model processing; Read the vertex information of each model triangle face one by one and create an array to store vertices and an array to store triangle faces. Then, determine whether the vertex positions of each face are the same. If they are the same, only store the information of one vertex and do not add information to the vertex array, but add the duplicate triangle face to the triangle face array. Step 1.2: Import the Mixed Reality Toolkit (MRTK) into the physics engine to develop cross-platform AR applications, and use MRTK to perform operations such as rotating, grabbing, and scaling objects.

3. The handheld tooling suitability verification method based on augmented reality technology as described in claim 2, characterized in that: The second step is implemented as follows: Step 2.1: First, the format and size of the image to be recognized need to be modified. Then, the image is imported into the cloud platform for recognition. After importing into the platform, the platform scores the uploaded images based on image content richness, contrast, and physical size. Images with a score of four stars or above are selected as the images to be recognized. Finally, the obtained data package is downloaded and imported into the physical engine to complete the image recognition. Step 2.2: The server processes the uploaded image into grayscale, converting it to a black and white image. Feature points are then extracted from the black and white image using a natural feature point matching method. Step 2.3: Identify the object to be identified by overlaying a predefined digital model from the physics engine onto the verification environment, thus enabling the digital model of the product to be tested to be overlaid onto the verification environment.

4. The handheld tooling suitability verification method based on augmented reality technology as described in claim 3, characterized in that: In step three, Based on whether there is a physical model of the handheld tooling, select the implementation method; the virtual-real fusion and synchronous operation of the handheld tooling virtual model and the human hand in step three includes the following two implementation methods; Method 1: When there is a physical model of the handheld tooling, repeat the natural feature point matching method in the 3D real-time tracking registration module in step 2 to identify the object to be identified, realize the virtual model of the handheld tooling superimposed on the physical model of the handheld tooling, and realize the virtual-real fusion and synchronous operation of the virtual model and the physical model of the handheld tooling. Method 2: When there is no virtual model of the handheld tool, construct the "grip pair" condition based on the characteristics of the physical process of the handheld tool's grasping and the augmented reality environment. Based on the "grip pair", construct the grasping intention recognition algorithm to determine the grasping situation between the human hand and the virtual model of the handheld tool, and realize the virtual-real fusion and synchronous operation of the virtual model of the handheld tool and the human hand. The aforementioned "grip pair" consists of two contact points. If there is more than one "grip pair", the gripped virtual model is determined to be in a gripping state. Step 3.1: Acquire an image of the human hand in the current frame, determine the position and pose data of the key nodes of the human hand relative to the AR device based on the convolutional neural network hand pose estimation algorithm, and overlay a virtual hand model at the key nodes of the hands identified in the current frame. The virtual hand model consists of several virtual joint models, each of which is a cylinder, approximately simulating the finger joints of a real hand. There are topological relationships between the virtual hand joint models; that is, higher-level virtual joint models contain lower-level virtual joint models, and the movement of a higher-level virtual joint model will cause the movement of the lower-level virtual joint models. The virtual hand model is represented by the following parameterization: in, For the i-th virtual joint model, The position of the virtual joint model is determined by a set of vectors in the augmented reality environment coordinate system. , and express, The pose of the virtual joint model is given by a set of vectors in the augmented reality environment coordinate system. , and express, These are the parameters of the virtual joint model. Indicates the length of the cylinder. Indicates the diameter of the cylinder. This represents the sub-joint model that is driven by the virtual joint model. This represents the k-th virtual joint model; Each virtual joint model in the virtual hand model will correspond to a key node in gesture tracking and recognition. The position and pose data of each recognized key hand node will be used to update the position and pose of the virtual joint model in the current frame. The position and pose of the virtual hand model are determined based on the position and pose of the key nodes, realizing the mapping of real hands in virtual space, as shown in the following formula: in, Let be the position vector of the i-th virtual joint model. Let be the rotation matrix of the virtual joint model. This rotation matrix has a transformation relationship with Euler angles as shown in the equation, where Indicates rotation around the z-axis Spend; This represents the transformation matrix between the augmented reality environment coordinate system and the camera coordinate system in which the virtual joint model resides; Let be the position vector of the key node corresponding to the i-th virtual joint model. Let be the rotation matrix of the key node corresponding to the i-th virtual joint model; Step 3.2: Based on the virtual joint model of the human hand, combined with the collision detection algorithm, determine whether the human hand and the virtual model of the handheld tool are in contact. If the collision detection algorithm detects that the human hand and the virtual model of the handheld tool are in contact, according to the grasping intention recognition algorithm, calculate whether a "grasping pair" can be formed between multiple contact points of the human hand and the virtual model of the handheld tool, and determine whether there is a grasping situation between the human hand and the virtual model of the handheld tool. If the human hand is in contact with different fingers, it is determined that the grasping is successful, and the grasping state is changed. The virtual and real fusion and synchronous operation of the virtual model of the handheld tool and the human hand are realized. A "grip pair" consists of two contact points. If there is more than one "grip pair", the grasped virtual model is determined to be in a grasping state. There is no need to calculate the contact based on multiple contact points to determine whether the grasp is complete. This makes the grasping intention judgment more flexible, closer to the real three-dimensional gesture manipulation situation, more adaptable to complex gesture interaction scenarios, and more in line with the user's intuitive interaction experience. At the same time, if there are multiple pairs of "grip pairs", the contact points that make up the grip pair will participate in the interaction intention recognition, improving the robustness, flexibility, efficiency and immersion of gesture interaction intention recognition. The physical process of grasping a real object is characterized by applying the basic laws of Newtonian rigid body mechanics. It judges whether the object can be grasped based on whether it is in force balance and the friction between the contact surfaces of the virtual hand model and the manipulated model. Its implementation principle is to analyze the force state of the object using a simplified Coulomb friction model. The "grip pair" consists of two contact points of a virtual hand model that meets certain conditions and the model being gripped; the conditions for the "grip pair" are as follows: the angle between the line connecting the two contact points and the normal to their respective contact surfaces does not exceed a fixed angle. Then the two contact points will form a stable gripping pair g(a,b); the fixed angle That is, the friction angle; The grasping intent recognition algorithm is established based on the "grasping pair" condition, and iteratively judges whether all current contact points can form a "grasping pair" with another contact point; for any two contact points a and b between the virtual hand and the virtual object in one iteration, the angle between the line connecting the two contact points and the normal of their respective contact surfaces does not exceed a fixed angle. Then the two contact points will form a stable gripping pair g(a,b); this fixed angle This is the friction angle, meaning the gripping joint g(a,b) should satisfy... in, and Let be the normal vectors of contact points a and b, which are the normal vectors of the cylindrical surface of the joint virtual model at the contact points; Let the line connecting contact points a and b be denoted as . The friction angle is the angle of friction. The value of the friction angle needs to be set by testing for the specific manipulated model to ensure stable and natural gripping of the virtual component.

5. The handheld tooling suitability verification method based on augmented reality technology as described in claim 4, characterized in that: Step five is implemented as follows: Step 5.1: Select the collision body construction method for the digital model of the product to be tested based on the characteristics and accuracy requirements of the digital model and the verification environment; Step 5.2: If the collision detection accuracy requirement of the digital model of the product to be tested is low, a mesh collider is used to fit the digital model of the product to be tested. Collision body attributes are added to the sub-models to obtain the collision body of the digital model of the product to be tested, thereby improving the collision detection accuracy. The entire digital model of the product to be tested is divided into several sub-models, and then collision body attributes are added to each sub-model. The collider body is then fitted to obtain the collision body of the digital model of the product to be tested. This collision body can fit the product model with high accuracy, and the collision detection accuracy and effect are good. Step 5.3: If the collision detection accuracy requirement of the digital model of the product to be tested is high, construct virtual grid points of the digital model of the product to be tested according to the characteristics of the digital model of the product to be tested, manually arrange the grid vertex set, generate a virtual grid that fits the surface of the digital model of the product to be tested based on the graphics algorithm, and integrate the information of the digital model of the product to be tested with the visual perception information of the AR device to correct the grid point layout information, thereby improving the construction accuracy and perception accuracy of the collision body of the digital model of the product to be tested. After triangulation of the virtual mesh fitted to the surface of the digital model of the product to be inspected, a set of triangle vertex indices is obtained. The triangulation process decomposes the polygon into several triangles, with these vertices forming the polygon. Since the manually defined vertex sequence is the required direct modeling sequence, the set of triangle vertex indices is established based on the adjacency principle. Then, a combination of triangles is generated using a computer graphics algorithm to create a virtual mesh adapted to the surface of the digital model of the product to be inspected. The set of mesh vertices and the set of triangle vertex indices are represented by the following parameterization: (8) (9) in V represents the set of virtual mesh vertices. i This represents the i-th grid point; Represents the i-th triangle that makes up the virtual mesh. ) represents the set of indices of the three vertices of this triangle.

6. The handheld tooling suitability verification method based on augmented reality technology as described in claim 5, characterized in that: In step six, The mathematical principle of the trigger detection algorithm is shown in the following formula: （10） , The term refers to the set of points on the collider (a virtual mesh of the digital model of the product to be tested, or a box-type collider of the handheld tooling virtual model). , yes , The point in the middle; （11） Equation (11) is called the Minkowski difference, which represents the difference when the colliding bodies... , When they overlap or intersect, their difference set { The origin must be included. For the trigger detection of the collision body between the virtual mesh of the digital model of the product to be tested and the box-shaped collision body of the handheld tooling virtual model, the difference between the point set of the virtual mesh of the digital model of the product to be tested and the point set of the box-shaped collision body of the handheld tooling virtual model is used to verify whether interference has occurred. The inclusion of the origin indicates that a collision has occurred.

7. The handheld tooling suitability verification method based on augmented reality technology as described in claim 6, characterized in that: To achieve more accurate verification results, the Unity3D physics engine was selected.

8. A handheld tooling suitability verification system based on augmented reality technology, used to implement the handheld tooling suitability verification method based on augmented reality technology as described in claims 1, 2, 3, 4, 5, 6 or 7, characterized in that: It includes a data acquisition and processing module, a handheld tooling applicability verification environment construction module, a 3D real-time tracking and registration module, and a collision detection module; The data acquisition and processing module is used to acquire the RGB image and depth image of the current frame, and obtain the natural feature point information of the object to be identified based on the natural feature point matching method in the three-dimensional real-time tracking registration module. Based on a convolutional neural network-based two-hand pose estimation algorithm, the position and pose information of key hand nodes in the current frame are obtained from RGB and depth images. The handheld tooling suitability verification environment construction module constructs a handheld tooling suitability verification environment based on augmented reality devices and a physics engine. The handheld tooling suitability verification environment includes a digital model of the product to be tested and a virtual model of the handheld tooling. The 3D real-time tracking and registration module includes a digital model overlay submodule for the product to be detected and a virtual model overlay submodule for the handheld fixture. The digital model overlay submodule identifies the object to be identified using a natural feature point matching method and overlays the digital model onto the verification environment. The virtual model overlay submodule performs virtual-real fusion and synchronous operation between the virtual model of the handheld fixture and the human hand in the verification environment. The implementation method is selected based on the presence or absence of a physical handheld fixture model. When a physical handheld fixture model is present, the object to be identified is identified using a natural feature point matching method, and the virtual model is overlaid onto the physical handheld fixture model. When no virtual handheld fixture model is present, a "grip pair" condition is constructed based on the physical characteristics of the gripping process of the physical handheld fixture model and the augmented reality environment. A gripping intention recognition algorithm is then built based on the "grip pair" to determine the gripping situation between the human hand and the virtual handheld fixture model, and to achieve virtual-real fusion and synchronous operation between the virtual handheld fixture model and the human hand. The collision detection module includes a handheld fixture virtual model collider construction submodule, a product-to-be-detected digital model collider construction submodule, and a trigger detection submodule. The handheld fixture virtual model collider construction submodule adds box-type collider attributes to the handheld fixture, presenting it as a cuboid. The length, width, and height of this cuboid are all equal to the maximum values ​​of the entire handheld fixture in the three dimensions, ensuring collision accuracy while reducing hardware load. The product-to-be-detected digital model collider construction submodule constructs a virtual mesh of the product-to-be-detected digital model based on the characteristics and accuracy requirements of the product-to-be-detected digital model and the verification environment. If the product-to-be-detected digital model... For models with low collision detection accuracy requirements, a mesh collider is used to fit the digital model of the product to be detected. Collision attributes are added to the sub-models to obtain the collision bodies of the digital model of the product to be detected, thereby improving the collision detection accuracy. If the collision detection accuracy requirements of the digital model of the product to be detected are high, virtual mesh points of the digital model of the product to be detected are constructed according to the characteristics of the digital model of the product to be detected. A virtual mesh is generated based on the graphics algorithm to fit the surface of the digital model of the product to be detected. The mesh point layout information is corrected by integrating the visual perception information of the AR device with the information of the digital model of the product to be detected, thereby improving the construction accuracy and perception accuracy of the collision bodies of the digital model of the product to be detected. The trigger detection submodule performs collision detection on the virtual mesh of the digital model of the product to be tested and the box-shaped collider of the virtual model of the handheld tooling, thereby realizing the applicability verification of the handheld tooling based on augmented reality technology.

Images