Event-driven ar visualization assembly system and its construction and editing method

Abstract

The application provides an event-driven AR visualization assembly system and a construction and editing method thereof, comprising the following steps: S1, constructing an assembly process knowledge graph based on a Neo4j graph database; S2, establishing an augmented reality assembly AR material model, extending and developing an augmented reality construction engine based on a C# language, and realizing a model conversion module; S3, defining augmented reality visualization assembly visual guidance content, extending and developing the augmented reality construction engine based on the C# language, and realizing a visualization node editor; S4, editing and associating node elements in a visualization node editing mode through the visualization node editor, defining AR visualization guidance elements, trigger events, demonstration animations and control logic; and S5, generating an augmented reality assembly guidance program. The application can improve the construction efficiency of the augmented reality auxiliary assembly program and reduce the construction period of the augmented reality guidance program.

Description

Technical Field

[0001] This invention relates to the fields of augmented reality and augmented reality program construction, and more specifically, to an event-driven AR visual assembly system and its construction and editing method. Background Technology

[0002] Augmented reality assembly is a technology that integrates stereoscopic display, intelligent interaction, scene positioning, real-time registration and tracking to overlay and blend virtual images used for assembly guidance onto the real assembly scene, providing assembly operators with a virtual-real fusion assembly guidance visual environment.

[0003] Traditional assembly processes are primarily presented to frontline assembly operators in the form of paper-based process cards. Given the diverse needs of assembled products, operators are required to understand and translate a large amount of assembly process information, undoubtedly increasing their workload. Augmented reality (AR) technology offers a novel solution to this problem. AR-assisted assembly systems can effectively improve the work efficiency of assembly personnel and reduce risks. Currently, AR technology has achieved widespread application in the assembly field; however, related research and applications focus on the design and management of process models for AR-assisted assembly systems. For different assembly product conditions, it is often necessary to customize and develop corresponding AR-assisted assembly programs, resulting in low efficiency in building AR-assisted assembly programs. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an event-driven AR visualization assembly system and its construction and editing methods.

[0005] According to the present invention, an event-driven AR visualization assembly system and its construction and editing method are provided as follows:

[0006] Firstly, an event-driven AR visualization assembly system construction and editing method is provided, the method comprising:

[0007] Step S1: Construct an assembly process knowledge graph based on the Neo4j graph database;

[0008] Step S2: Extend the augmented reality building engine based on the C# language to implement the model conversion module; take the CAD model of the assembled product as input, perform data parsing, topology reconstruction and model lightweighting operations on the CAD model, and convert the CAD model into AR material models required to build AR guiding elements;

[0009] Step S3: Define the augmented reality visualization assembly visual guidance content, extend the augmented reality building engine based on the C# language to implement the visual node editor; by accessing the Neo4j graph database, process knowledge entities are transformed into editable visual node information and associated with the AR material model;

[0010] Step S4: Edit and associate node elements in a visual node editor, and define AR visual guide elements, trigger events, demonstration animations and control logic.

[0011] Step S5: Generate the augmented reality assembly bootloader.

[0012] Preferably, the AR material model is obtained by converting a CAD model. An automatic model conversion module is implemented through secondary development of the augmented reality building engine using the C# language. This module can automatically convert the CAD model into an AR material model for building AR guiding elements. The automatic conversion module provides functions such as model parsing, topology reconstruction, model lightweighting, and product processing data (PMI) extraction. Specifically, it includes the following steps:

[0013] Step S2.1: Extract the hierarchical structure data of the CAD model, and reconstruct the model hierarchy tree in the augmented reality building engine based on the hierarchical structure of the CAD model;

[0014] Step S2.2: Through the model conversion module, perform data parsing and topology reconstruction on the CAD model to build augmented reality visualization model data, and populate the model hierarchy tree with the reconstructed model data;

[0015] Step S2.3: Perform a model lightweighting operation on the reconstructed augmented reality visualization model data by deleting overlapping primitives, small objects, and invisible mesh data to reduce the performance overhead of the augmented reality bootstrap program;

[0016] Step S2.4: Extract the PMI data of the CAD model and reproduce it by annotation in the augmented reality building engine;

[0017] Step S2.5: Associate and map the visualization model tree with the process nodes.

[0018] Preferably, the visual node editor obtains assembly process knowledge by accessing the Neo4j graph database, quickly converts the assembly process knowledge into process flow nodes, and constructs augmented reality guidance content based on the process flow nodes in a visual editing manner.

[0019] Preferably, the visual node editor can define node content including process knowledge, guidance content, triggering events, behavior control, and logic control definitions;

[0020] 1) Process Knowledge: The visual node editor supports the editing and mapping function of the process knowledge graph database. By accessing the knowledge graph database, the process knowledge graph is mapped to the node elements of the visual node editor, and the process knowledge graph is presented visually. The node editor can be used to visually edit the presented node elements, realizing the dynamic visual editing and updating of the process knowledge graph.

[0021] 2) Guiding Content: Augmented reality visualization elements presented to users, including images, text, videos, 3D models, and demonstration animations. The parameters of the guiding content can be configured through corresponding attribute definitions.

[0022] 3) Triggering events: The system's response when it enters a certain state or when a user performs an operation. Triggering events are the entry points for presenting visual guidance content. Triggering events include: initialization events, target recognition events, target loss events, human-computer interaction events, process loading events, process end events, and process completion events.

[0023] The initialization event is triggered when the system initializes.

[0024] Target recognition event: Triggered when the system detects an AR registered target;

[0025] Target loss event: Triggered when a target registered with AR is lost during tracking;

[0026] Human-computer interaction events: triggered when a user interacts with a gesture;

[0027] Process loading / ending / completion events: triggered when the assembly process reaches a specific stage;

[0028] 4) Behavior control: Used to control the onboarding content. Behavior control includes: video playback / pause, onboarding element display / hide, and demo animation playback / pause;

[0029] 5) Logical Control: Logical control nodes define the logical execution order between different behavior control nodes. Logical control includes: conditional execution, sequential execution, and synchronous execution.

[0030] The conditional execution means that the system executes the functional nodes that meet the specified conditions.

[0031] The sequential execution means that all associated functional nodes are executed in the order they are arranged.

[0032] Synchronous execution: All functional nodes associated with this node execute synchronously.

[0033] Preferably, the human-computer interaction event is triggered by user gesture interaction. A virtual hand model is constructed in virtual space by acquiring the user's hand pose and joint posture in real time. Collision interference detection is performed on UI elements based on the coordinate mapping relationship of virtual and real integration to determine the gesture interaction state. The collision interference detection of UI elements includes collision interference detection between UI elements and the virtual hand model, and collision detection between UI elements and rays emitted from the virtual finger positions, wherein:

[0034] Collision interference detection between UI elements and the virtual hand model simplifies the contact point between the virtual hand and the UI element to a sphere, and the UI element to an OBB bounding box. Then, UI triggering is performed by calculating the collision between the sphere and the OBB bounding box. The construction process of the OBB bounding box is as follows:

[0035] First, calculate the mean center of the UI element in the local coordinate system. For an object with n vertices, the mean center point P of the OBB bounding box is... m It can be calculated using the following formula:

[0036]

[0037] Among them, P i Let M represent the i-th vertex of the UI element, and then calculate the covariance matrix M of the object relative to the center point. c :

[0038]

[0039] For the covariance matrix M c Perform singular value decomposition to obtain a matrix U containing the principal axis directions:

[0040] M c =U∑U T

[0041] Where ∑ is the eigenvalue matrix, and U = {u1, u2, u3} represents the three principal axis directions of the OBB bounding box in the local coordinate system;

[0042] The half-axis width vector h of the OBB bounding box is:

[0043] h = {h1, h2, h3}

[0044] For each element h of vector h j ,

[0045] h j =r j u j j = 1, 2, 3

[0046] Where, r jh is half the average of the maximum and minimum vertex distances of the j-th basis vector. j Let be the width vector of the semi-axis along the j-th principal axis direction, i.e.:

[0047]

[0048] The center point P of the OBB bounding box obb for:

[0049] P obb =P m +h

[0050] After constructing the OBB bounding box of the UI element, the collision detection between the sphere and the OBB bounding box is used as the trigger for the UI event. The collision calculation process between the sphere and the OBB bounding box is as follows:

[0051] 1) Transform the sphere's center coordinates from the world coordinate system to the local coordinate system of the OBB bounding box:

[0052] P sl =R obb T (P sw -P obb )

[0053] Among them, P sl P obb These are the coordinates of the sphere's center and the center of the OBB bounding box in the world coordinate system, respectively. sw R represents the coordinates of the sphere's center in the world coordinate system. obb T It is the transpose of the OBB bounding box rotation matrix;

[0054] 2) For the sphere center coordinates P in the local coordinate system of the OBB bounding box sl If the absolute values ​​of its components on the x, y, and z axes are all less than the length of the semi-axis |E i If i = 1, 2, 3, then return a collision; otherwise, proceed to the next step.

[0055] 3) Find the nearest point P from the center of the sphere to the bounding box of the OBB using the following formula. near :

[0056] P near =(max(-h1,min(P)) x ,h1),max(-h2,min(P y ,h2),max(-h3),min(P z h3))

[0057] Among them, P x P y P zP respectively sl Components on the x, y, and z axes;

[0058] If the point is inside the sphere, return to the collision; otherwise, proceed to the next step.

[0059] 4) Calculate the distance d from the center of the sphere to each face of the OBB bounding box. If the shortest distance d is... min If r < r, then the sphere collides with the OBB bounding box, where r is the radius of the sphere, and the distance d from the center of the sphere to each face is calculated using the following formula:

[0060]

[0061] Among them, P plane Let N be any point in the plane, and let N be the plane normal vector.

[0062] Collision detection between UI elements and the ray emitted from the virtual finger position is simplified to collision detection between the ray and the OBB bounding box. If the ray collides with the OBB bounding box, it means that the current UI element is selected by the user. The collision detection calculation process between the ray and the OBB bounding box is as follows:

[0063] Transform the ray from the world coordinate system to the local coordinate system. Given the ray's origin P0, ray direction D, and the center coordinates P of the OBB bounding box in world space. obb The rotation matrix R of the OBB bounding box, and the formulas for calculating the coordinates P′0 and direction D′ of the ray in the local space of the OBB bounding box are as follows:

[0064] P′0=R T (P0-P obb )

[0065] D′=R T D

[0066] In the local coordinate system of the OBB bounding box, the ray intersects the six SLAB surfaces of the OBB bounding box at point t. minx t miny t minz t maxx t maxy t maxz They are respectively:

[0067]

[0068]

[0069]

[0070] Among them, X min Y min Z min X max Ymax Z max Let P' be the minimum and maximum values ​​of the six faces of the OBB bounding box in the local coordinate system. x 、P′ y 、P′ z Let P′0 be the component of P′0 on the X, Y, and Z coordinate axes, and D′ be the component of P′0 on the X, Y, and Z coordinate axes, respectively. x 、D′ y 、D′ z Let D′ represent the components of the X, Y, and Z coordinate axes, respectively; then calculate the intersection point t of the near and far planes of the OBB bounding box in the local coordinate system. near and t far :

[0071] t near =max(X min Y min Z min )

[0072] t far =min(X) max Y max Z max )

[0073] If t near <t far If the ray intersects the bounding box of the OBB, then the ray will intersect the bounding box of the OBB; otherwise, it will not intersect.

[0074] Preferably, the visual node editor supports keyframe animation editing, and defines animation types including translation, rotation, scaling, color change, and show / hide, and defines animation parameters including animation callback, animation speed, and delay time;

[0075] The animation properties of the translation, rotation, and scaling animations include the starting coordinate value, the target coordinate value, the reference coordinate system, and the animation duration;

[0076] The translation, scaling, and color-changing animations define the starting coordinate value and the target coordinate value as keyframes. The intermediate process uses a linear interpolation method for interpolation calculation. For two vertices v1 and v2 on the keyframe, the interpolation formula for the vertex v(t) value at time t is v(t) = (1-t)v1 + tv2.

[0077] The rotation animation is represented in Euler angles by the visual editor. However, due to the ambiguity and gimbal lock issues associated with Euler angles, they are converted to quaternions for interpolation. The quaternion q = [wxyz] is defined as follows. T The formula for converting Euler angles to quaternions is:

[0078]

[0079] Where α, β, and γ are the rotational components about the y-axis, x-axis, and z-axis, respectively;

[0080] For two vertices q0 and q1 in the keyframe, the rotation quaternion interpolation formula at time t is:

[0081]

[0082] Where θ = arccos(q0·q1);

[0083] The color-changing animation attributes include: starting color, target color, and support for defining transparency channel attributes;

[0084] The animation callbacks include: Start, Update, and Stop callbacks. Start and Stop are callbacks for starting and stopping the animation, while Update is a callback performed once for each frame during the animation playback. Different animation segments can be organized together through the animation callback parameters to form complex mechanical simulation animations.

[0085] Secondly, an event-driven AR visualization assembly system is provided, the system comprising:

[0086] Module M1: Constructing an assembly process knowledge graph based on the Neo4j graph database;

[0087] Module M2: Based on the C# language, this module extends the augmented reality building engine to realize the model conversion module. Taking the CAD model of the assembled product as input, it performs data parsing, topology reconstruction, and model lightweighting operations on the CAD model to convert it into AR material models required for building AR guiding elements.

[0088] Module M3: Defines the visual guidance content for augmented reality assembly visualization. Based on the C# language, it extends the augmented reality building engine to implement a visual node editor. By accessing the Neo4j graph database, it transforms process knowledge entities into editable visual node information and associates and maps them with the AR material model.

[0089] Module M4: Through the visual node editor, node elements can be edited and associated in a visual node editing manner, defining AR visual guide elements, trigger events, demonstration animations and control logic;

[0090] Module M5: Generates the augmented reality assembly bootloader;

[0091] Preferably, the AR material model is obtained by converting a CAD model. An automatic model conversion module is implemented through secondary development of the augmented reality building engine using the C# language. This module can automatically convert the CAD model into an AR material model for building AR guiding elements. The automatic conversion module provides functions such as model parsing, topology reconstruction, model lightweighting, and product processing data (PMI) extraction. Specifically, it includes the following steps:

[0092] Module M2.1: Extracts hierarchical structure data from the CAD model and reconstructs the model hierarchy tree in the augmented reality building engine based on the hierarchical structure of the CAD model;

[0093] Module M2.2: Through the model conversion module, data parsing and topology reconstruction of CAD models are performed to build augmented reality visualization model data, and the reconstructed model data is populated into the model hierarchy tree;

[0094] Module M2.3: Performs model lightweighting operations on the reconstructed augmented reality visualization model data by deleting overlapping primitives, small objects, and invisible mesh data, thereby reducing the performance overhead of the augmented reality bootstrap program;

[0095] Module M2.4: Extracts PMI data from CAD models and reproduces it in the augmented reality building engine;

[0096] Module M2.5: Associates and maps the visual model tree with process nodes;

[0097] The visual node editor obtains assembly process knowledge by accessing the Neo4j graph database, quickly transforms the assembly process knowledge into process flow nodes, and builds augmented reality guidance content based on the process flow nodes in a visual editing manner.

[0098] The visual node editor can define node content including process knowledge, guidance content, trigger events, behavior control, and logic control.

[0099] 1) Process Knowledge: The visual node editor supports the editing and mapping function of the process knowledge graph database. By accessing the knowledge graph database, the process knowledge graph is mapped to the node elements of the visual node editor, and the process knowledge graph is presented visually. The node editor can be used to visually edit the presented node elements, realizing the dynamic visual editing and updating of the process knowledge graph.

[0100] 2) Guiding Content: Augmented reality visualization elements presented to users, including images, text, videos, 3D models, and demonstration animations. The parameters of the guiding content can be configured through corresponding attribute definitions.

[0101] 3) Triggering events: The system's response when it enters a certain state or when a user performs an operation. Triggering events are the entry points for presenting visual guidance content. Triggering events include: initialization events, target recognition events, target loss events, human-computer interaction events, process loading events, process end events, and process completion events.

[0102] The initialization event is triggered when the system initializes.

[0103] Target recognition event: Triggered when the system detects an AR registered target;

[0104] Target loss event: Triggered when a target registered with AR is lost during tracking;

[0105] Human-computer interaction events: triggered when a user interacts with a gesture;

[0106] Process loading / ending / completion events: triggered when the assembly process reaches a specific stage;

[0107] 4) Behavior control: Used to control the onboarding content. Behavior control includes: video playback / pause, onboarding element display / hide, and demo animation playback / pause;

[0108] 5) Logical Control: Logical control nodes define the logical execution order between different behavior control nodes. Logical control includes: conditional execution, sequential execution, and synchronous execution.

[0109] The conditional execution means that the system executes the functional nodes that meet the specified conditions.

[0110] The sequential execution means that all associated functional nodes are executed in the order they are arranged.

[0111] The synchronous execution means that all functional nodes associated with this node execute synchronously.

[0112] The human-computer interaction event is triggered by user gesture interaction. A virtual hand model is constructed in virtual space by acquiring the user's hand pose and joint posture in real time. Collision interference detection is performed on UI elements based on the coordinate mapping relationship between the virtual and real worlds to determine the gesture interaction state. The collision interference detection of UI elements includes collision interference detection between UI elements and the virtual hand model, and collision detection between UI elements and rays emitted from the virtual finger positions.

[0113] Collision interference detection between UI elements and the virtual hand model simplifies the contact point between the virtual hand and the UI element to a sphere, and the UI element to an OBB bounding box. Then, UI triggering is performed by calculating the collision between the sphere and the OBB bounding box. The construction process of the OBB bounding box is as follows:

[0114] First, calculate the mean center of the UI element in the local coordinate system. For an object with n vertices, the mean center point P of the OBB bounding box is... m It can be calculated using the following formula:

[0115]

[0116] Among them, P i Let M represent the i-th vertex of the UI element, and then calculate the covariance matrix M of the object relative to the center point. c :

[0117]

[0118] For the covariance matrix M c Perform singular value decomposition to obtain a matrix U containing the principal axis directions:

[0119] M c =U∑U T

[0120] Where ∑ is the eigenvalue matrix, and U = {u1, u2, u3} represents the three principal axis directions of the OBB bounding box in the local coordinate system;

[0121] The half-axis width vector h of the OBB bounding box is:

[0122] h = {h1, h2, h3}

[0123] For each element h of vector h j ,

[0124] h j =r j u j j = 1, 2, 3

[0125] Where, r j h is half the average of the maximum and minimum vertex distances of the j-th basis vector. j Let be the width vector of the semi-axis along the j-th principal axis direction, i.e.:

[0126]

[0127] The center point P of the OBB bounding box obb for:

[0128] P obb =P m +h

[0129] After constructing the OBB bounding box of the UI element, the collision detection between the sphere and the OBB bounding box is used as the trigger for the UI event. The collision calculation process between the sphere and the OBB bounding box is as follows:

[0130] 1) Transform the sphere's center coordinates from the world coordinate system to the local coordinate system of the OBB bounding box:

[0131] P sl =R obb T (P sw -P obb )

[0132] Among them, P sl P obb These are the coordinates of the sphere's center and the center of the OBB bounding box in the world coordinate system, respectively. sw R represents the coordinates of the sphere's center in the world coordinate system. obb T It is the transpose of the OBB bounding box rotation matrix;

[0133] 2) For the sphere center coordinates P in the local coordinate system of the OBB bounding box sl If the absolute values ​​of its components on the x, y, and z axes are all less than the length of the semi-axis |E i If i = 1, 2, 3, then return a collision; otherwise, proceed to the next step.

[0134] 3) Find the nearest point P from the center of the sphere to the bounding box of the OBB using the following formula. near :

[0135] P near =(max(-h1,min(P)) x ,h1),max(-h2,min(P y ,h2),max(-h3),min(P z h3))

[0136] Among them, P x P y P z P respectively sl Components on the x, y, and z axes;

[0137] If the point is inside the sphere, return to the collision; otherwise, proceed to the next step.

[0138] 4) Calculate the distance d from the center of the sphere to each face of the OBB bounding box. If the shortest distance d is... min If r < r, then the sphere collides with the OBB bounding box, where r is the radius of the sphere, and the distance d from the center of the sphere to each face is calculated using the following formula:

[0139]

[0140] Among them, P plane Let N be any point in the plane, and let N be the plane normal vector.

[0141] Collision detection between UI elements and the ray emitted from the virtual finger position is simplified to collision detection between the ray and the OBB bounding box. If the ray collides with the OBB bounding box, it means that the current UI element is selected by the user. The collision detection calculation process between the ray and the OBB bounding box is as follows:

[0142] Transform the ray from the world coordinate system to the local coordinate system. Given the ray's origin P0, ray direction D, and the center coordinates P of the OBB bounding box in world space. obb The rotation matrix R of the OBB bounding box, and the formulas for calculating the coordinates P′0 and direction D′ of the ray in the local space of the OBB bounding box are as follows:

[0143] P′0=R T (P0-P obb )

[0144] D′=R T D

[0145] In the local coordinate system of the OBB bounding box, the ray intersects the six SLAB surfaces of the OBB bounding box at point t. minx t miny t minz t maxx t maxy t maxz They are respectively:

[0146]

[0147] Among them, X min Y min Z min X max Y max Z max Let P' be the minimum and maximum values ​​of the six faces of the OBB bounding box in the local coordinate system. x 、P′ y 、P′ z Let P′0 be the component of P′0 on the X, Y, and Z coordinate axes, and D′ be the component of P′0 on the X, Y, and Z coordinate axes, respectively. x 、D′ y 、D′ z Let D′ represent the components of the X, Y, and Z coordinate axes, respectively; then calculate the intersection point t of the near and far planes of the OBB bounding box in the local coordinate system. near and t far :

[0148] t near =max(X min Y min Z min )

[0149] t far =min(X)max Y max Z max )

[0150] If t near <t far If the ray intersects the OBB bounding box, then the ray will intersect the OBB bounding box; otherwise, it will not intersect.

[0151] The visual node editor supports keyframe animation editing, and defines animation types including translation, rotation, scaling, color change, and show / hide. It also defines animation parameters including animation callback, animation speed, and delay time.

[0152] The animation properties of the translation, rotation, and scaling animations include the starting coordinate value, the target coordinate value, the reference coordinate system, and the animation duration;

[0153] The translation, scaling, and color-changing animations define the starting coordinate value and the target coordinate value as keyframes. The intermediate process uses a linear interpolation method for interpolation calculation. For two vertices v1 and v2 on the keyframe, the interpolation formula for the vertex v(t) value at time t is v(t) = (1-t)v1 + tv2.

[0154] The rotation animation is represented in Euler angles by the visual editor. However, due to the ambiguity and gimbal lock issues associated with Euler angles, they are converted to quaternions for interpolation. The quaternion q = [wxyz] is defined as follows. T The formula for converting Euler angles to quaternions is:

[0155]

[0156] Where α, β, and γ are the rotational components about the y-axis, x-axis, and z-axis, respectively;

[0157] For two vertices q0 and q1 in the keyframe, the rotation quaternion interpolation formula at time t is:

[0158]

[0159] Where θ = arccos(q0·q1);

[0160] The color-changing animation attributes include: starting color, target color, and support for defining transparency channel attributes;

[0161] The animation callbacks include: Start, Update, and Stop callbacks. Start and Stop are callbacks for starting and stopping the animation, while Update is a callback performed once for each frame during the animation playback. Different animation segments can be organized together through the animation callback parameters to form complex mechanical simulation animations.

[0162] Thirdly, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements the steps of the event-driven AR visualization assembly system construction and editing method.

[0163] Fourthly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the event-driven AR visualization assembly system construction and editing method.

[0164] Compared with the prior art, the present invention has the following beneficial effects:

[0165] 1. This invention extends the augmented reality building engine to realize an automatic model conversion module. Through model parsing, topology reconstruction, and lightweight operations, CAD models are quickly converted into AR material models for building AR visualization guidance elements. This solves the problem of difficulty in building model materials due to format incompatibility in the traditional AR program building process and improves the efficiency of AR material model building.

[0166] 2. This invention extends the augmented reality building engine to realize a visual node editor for building assembly guidance content. By accessing graph data, it automatically maps process knowledge into process node information. Developers can define AR visual guidance content based on this, which can improve the construction efficiency of AR guidance programs in customized AR assembly guidance development.

[0167] 3. The visual node editor of this invention provides a code-free augmented reality visual editing method, which can define assembly guidance animations and guidance behavior logic through visual node editing, realize the rapid visual construction of demonstration animations and AR visual guidance content, reduce the construction threshold of AR assembly guidance programs, improve the construction efficiency of AR assembly guidance programs, and adapt to the ever-changing product auxiliary assembly needs.

[0168] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0169] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0170] Figure 1 Architecture diagram for building and editing methods of AR visualization assembly system;

[0171] Figure 2 This is a flowchart of the automatic model conversion module. Detailed Implementation

[0172] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0173] This invention provides an event-driven AR visualization assembly system construction and editing method. It stores structured assembly process information through a knowledge graph and extends the augmented reality building engine to achieve an automatic model conversion module and a visualization node editor for assembly guidance content. Events serve as the trigger entry point for visualization, enabling rapid conversion from CAD models to AR material models and the visualization definition of AR assembly guidance content. (Refer to...) Figure 1 As shown, the method specifically includes:

[0174] Step S1: Construct an assembly process knowledge graph based on the Neo4j graph database;

[0175] Step S2: Extend the augmented reality building engine using C# to implement a model conversion module. Using the CAD model of the assembled product as input, perform data parsing, topology reconstruction, and model lightweighting operations on the CAD model to convert it into AR material models needed to build AR guiding elements. The CAD model is the input model data, typically created during the product design phase. CAD models cannot be directly used by the AR building engine; therefore, the model conversion module is developed to transform the CAD model into AR material models that the AR building engine can use.

[0176] Step S3: Define the augmented reality visualization assembly visual guidance content, extend the augmented reality building engine based on the C# language to implement the visual node editor; by accessing the Neo4j graph database, process knowledge entities are transformed into editable visual node information and associated with the AR material model;

[0177] Step S4: Edit and associate node elements in a visual node editor, and define AR visual guide elements, trigger events, demonstration animations and control logic.

[0178] Step S5: Generate the augmented reality assembly bootloader.

[0179] Specifically, the AR material models are obtained by converting CAD models. An automatic model conversion module is implemented through secondary development of the augmented reality building engine using the C# language. This module can automatically convert CAD models into AR material models used to build AR guiding elements. The automatic conversion module provides functions such as model parsing, topology reconstruction, model lightweighting, and product processing data (PMI) extraction. The specific steps include:

[0180] Step S2.1: Extract the hierarchical structure data of the CAD model, and reconstruct the model hierarchy tree in the augmented reality building engine based on the hierarchical structure of the CAD model;

[0181] Step S2.2: Through the model conversion module, perform data parsing and topology reconstruction on the CAD model to build augmented reality visualization model data, and populate the model hierarchy tree with the reconstructed model data;

[0182] Step S2.3: Perform a model lightweighting operation on the reconstructed augmented reality visualization model data by deleting overlapping primitives, small objects, and invisible mesh data to reduce the performance overhead of the augmented reality bootstrap program;

[0183] Step S2.4: Extract the PMI data of the CAD model and reproduce it by annotation in the augmented reality building engine;

[0184] Step S2.5: Associate and map the visualization model tree with the process nodes.

[0185] The visual node editor accesses the Neo4j graph database to obtain assembly process knowledge, quickly transforms the assembly process knowledge into process flow nodes, and builds augmented reality guided content based on the process flow nodes in a visual editing manner.

[0186] The visual node editor allows you to define node content including process knowledge, guidance content, trigger events, behavior control, and logic control.

[0187] 1) Process Knowledge: The visual node editor supports the editing and mapping function of the process knowledge graph database. By accessing the knowledge graph database, the process knowledge graph is mapped to the node elements of the visual node editor, and the process knowledge graph is presented visually. The node editor can be used to visually edit the presented node elements, realizing the dynamic visual editing and updating of the process knowledge graph.

[0188] 2) Guiding Content: Augmented reality visualization elements presented to users, including images, text, videos, 3D models, and demonstration animations. The parameters of the guiding content can be configured through corresponding attribute definitions.

[0189] 3) Triggering events: The system's response when it enters a certain state or when a user performs an operation. Triggering events are the entry points for presenting visual guidance content. Triggering events include: initialization events, target recognition events, target loss events, human-computer interaction events, process loading events, process end events, and process completion events.

[0190] The initialization event is triggered when the system initializes.

[0191] Target recognition event: Triggered when the system detects an AR registered target;

[0192] Target loss event: Triggered when a target registered with AR is lost during tracking;

[0193] Human-computer interaction events: triggered when a user interacts with a gesture;

[0194] Process loading / ending / completion events: triggered when the assembly process reaches a specific stage;

[0195] 4) Behavior control: Used to control the onboarding content. Behavior control includes: video playback / pause, onboarding element display / hide, and demo animation playback / pause;

[0196] 5) Logical Control: Logical control nodes define the logical execution order between different behavior control nodes. Logical control includes: conditional execution, sequential execution, and synchronous execution.

[0197] Conditional execution: The system executes function nodes that meet specified conditions;

[0198] Sequential execution: Executes all associated functional nodes sequentially according to their arrangement order;

[0199] Synchronous execution: All functional nodes associated with this node execute synchronously.

[0200] Human-computer interaction events are triggered by user gestures. A virtual hand model is constructed in virtual space by acquiring the user's hand pose and joint posture in real time. Collision interference detection is performed on UI elements based on the coordinate mapping relationship between the virtual and real worlds to determine the gesture interaction state. The collision interference detection of UI elements includes collision interference detection between the UI element and the virtual hand model, and collision detection between the UI element and rays emitted from the virtual finger positions.

[0201] Collision interference detection between UI elements and the virtual hand model simplifies the contact point between the virtual hand and the UI element to a sphere, and the UI element to an OBB bounding box. Then, UI triggering is performed by calculating the collision between the sphere and the OBB bounding box. The construction process of the OBB bounding box is as follows:

[0202] First, calculate the mean center of the UI element in the local coordinate system. For an object with n vertices, the mean center point P of the OBB bounding box is... m It can be calculated using the following formula:

[0203]

[0204] Among them, P i Let M represent the i-th vertex of the UI element, and then calculate the covariance matrix M of the object relative to the center point. c :

[0205]

[0206] For the covariance matrix M c Perform singular value decomposition to obtain a matrix U containing the principal axis directions:

[0207] M c =U∑U T

[0208] Where ∑ is the eigenvalue matrix, and U = {u1, u2, u3} represents the three principal axis directions of the OBB bounding box in the local coordinate system;

[0209] The half-axis width vector h of the OBB bounding box is:

[0210] h = {h1, h2, h3}

[0211] For each element h of vector h j ,

[0212] h j =r j u j j = 1, 2, 3

[0213] Where, r j h is half the average of the maximum and minimum vertex distances of the j-th basis vector. j Let be the width vector of the semi-axis along the j-th principal axis direction, i.e.:

[0214]

[0215] The center point P of the OBB bounding box obb for:

[0216] P obb =P m +h

[0217] After constructing the OBB bounding box of the UI element, the collision detection between the sphere and the OBB bounding box is used as the trigger for the UI event. The collision calculation process between the sphere and the OBB bounding box is as follows:

[0218] 1) Transform the sphere's center coordinates from the world coordinate system to the local coordinate system of the OBB bounding box:

[0219] P sl =R obb T (P sw -P obb )

[0220] Among them, P sl P obb These are the coordinates of the sphere's center and the center of the OBB bounding box in the world coordinate system, respectively. sw R represents the coordinates of the sphere's center in the world coordinate system. obb T It is the transpose of the OBB bounding box rotation matrix;

[0221] 2) For the sphere center coordinates P in the local coordinate system of the OBB bounding box sl If the absolute values ​​of its components on the x, y, and z axes are all less than the length of the semi-axis |E i If i = 1, 2, 3, then return a collision; otherwise, proceed to the next step.

[0222] 3) Find the nearest point P from the center of the sphere to the bounding box of the OBB using the following formula. near :

[0223] P near =(max(-h1,min(P)) x ,h1),max(-h2,min(P y ,h2),max(-h3),min(P z h3))

[0224] Among them, P x P y P z P respectively sl Components on the x, y, and z axes;

[0225] If the point is inside the sphere, return to the collision; otherwise, proceed to the next step.

[0226] 4) Calculate the distance d from the center of the sphere to each face of the OBB bounding box. If the shortest distance d is... min If r < r, then the sphere collides with the OBB bounding box, where r is the radius of the sphere, and the distance d from the center of the sphere to each face is calculated using the following formula:

[0227]

[0228] Among them, P plane Let N be any point in the plane, and let N be the plane normal vector.

[0229] Collision detection between UI elements and the ray emitted from the virtual finger position is simplified to collision detection between the ray and the OBB bounding box. If the ray collides with the OBB bounding box, it means that the current UI element is selected by the user. The collision detection calculation process between the ray and the OBB bounding box is as follows:

[0230] Transform the ray from the world coordinate system to the local coordinate system. Given the ray's origin P0, ray direction D, and the center coordinates P of the OBB bounding box in world space. obb The rotation matrix R of the OBB bounding box, and the formulas for calculating the coordinates P′0 and direction D′ of the ray in the local space of the OBB bounding box are as follows:

[0231] P′0=R T (P0-P obb )

[0232] D′=R T D

[0233] In the local coordinate system of the OBB bounding box, the ray intersects the six SLAB surfaces of the OBB bounding box at point t. minx t miny t minz t maxx t maxy t maxz They are respectively:

[0234]

[0235] Among them, X min Y min Z min X max Y max Z max Let P' be the minimum and maximum values ​​of the six faces of the OBB bounding box in the local coordinate system. x 、P′ y 、P′ z Let P′0 be the component of P′0 on the X, Y, and Z coordinate axes, and D′ be the component of P′0 on the X, Y, and Z coordinate axes, respectively. x 、D′ y 、D′ z Let D′ represent the components of the X, Y, and Z coordinate axes; then calculate the local coordinates of the OBB bounding box.

[0236] The intersection point t of the near plane and the far plane under the system near and t far :

[0237] t near =max(X min Y min Z min )

[0238] t far =min(X) max Y max Z max )

[0239] If t near <t far If the ray intersects the bounding box of the OBB, then the ray will intersect the bounding box of the OBB; otherwise, it will not intersect.

[0240] The visual node editor supports keyframe animation editing, allowing you to define animation types including translation, rotation, scaling, color change, and show / hide, and define animation parameters including animation callback, animation speed, and delay time.

[0241] The animation properties for translation, rotation, and scaling animations include the starting coordinate value, the target coordinate value, the reference coordinate system, and the animation duration;

[0242] Translation, scaling, and color-changing animations define the starting and target coordinate values ​​as keyframes. The intermediate process uses linear interpolation to perform interpolation calculations. For two vertices v1 and v2 on the keyframe, the interpolation formula for the vertex v(t) value at time t is v(t) = (1-t)v1 + tv2.

[0243] The rotation animation is represented in Euler angles by the visual editor. However, due to the ambiguity and gimbal lock issues associated with Euler angles, they are converted to quaternions for interpolation. The quaternion is defined as q = [wxyz]. T The formula for converting Euler angles to quaternions is:

[0244]

[0245] Where α, β, and γ are the rotational components about the y-axis, x-axis, and z-axis, respectively;

[0246] For two vertices q0 and q1 in the keyframe, the rotation quaternion interpolation formula at time t is:

[0247]

[0248] Where θ = arccos(q0·q1);

[0249] The color-changing animation properties include: starting color, target color, and support for defining transparency channel properties;

[0250] Animation callbacks include: Start, Update, and Stop callbacks. Start and Stop are callbacks for starting and stopping the animation, while Update is a callback that occurs once for each frame during the animation playback. Different animation clips can be organized together through animation callback parameters to form complex mechanical simulation animations.

[0251] The present invention will now be described in more detail.

[0252] This invention provides an event-driven AR visualization assembly system and its construction and editing method. By extending and developing an augmented reality (AR) construction engine, it implements an automatic model conversion module and a visualization node editor for assembly guidance content, enabling rapid conversion from CAD models to AR material models and the visualization definition of guidance content. This invention effectively simplifies the construction process of AR-assisted assembly programs, enabling rapid conversion from process knowledge to AR-assisted assembly programs to adapt to changing product assembly needs. Figure 1 As shown, the content is as follows:

[0253] S1. Construct an assembly process knowledge graph based on the Neo4j graph database;

[0254] S2. Based on the C# language, the augmented reality building engine is extended and developed to realize the model conversion module. Taking the CAD model of the assembled product as input, the CAD model is subjected to data parsing, topology reconstruction and model lightweighting operations to convert the CAD model into AR material model required to build AR guiding elements.

[0255] S3. Define the augmented reality visualization assembly visual guidance content, extend the augmented reality building engine based on the C# language, implement the visual node editor, and convert the process knowledge entities into editable visual node information by accessing the Neo4j graph database, and associate and map them with the AR material model described in S2.

[0256] S4. Through the visual node editor, edit and associate node elements in a visual node editing way, and define AR visual guide elements, trigger events, demonstration animations and control logic;

[0257] S5. Generate the augmented reality assembly bootloader.

[0258] The visual model resources are created within the augmented reality building engine. The engine is further developed using C# to implement an automatic model conversion module. This module automatically converts CAD models into AR material models required for building AR guide elements. The automatic conversion module provides model parsing, topology reconstruction, model lightweighting, and Product Processing Data (PMI) extraction functions, specifically including the following steps:

[0259] S1. Extract the hierarchical structure data of the CAD model and reconstruct the model hierarchy tree in the augmented reality building engine based on the hierarchical structure of the CAD model.

[0260] S2. Through the model conversion module, perform data parsing and topology reconstruction on the CAD model to build augmented reality visualization model data, and fill the reconstructed model data into the model hierarchy tree;

[0261] S3. Perform model lightweighting operations on the reconstructed augmented reality visualization model data, such as deleting overlapping primitives, small objects, and invisible mesh data, to reduce the performance overhead of the augmented reality bootstrap program.

[0262] S4. Extract PMI data from the CAD model, such as model geometry, tolerances, surface roughness, and common manufacturing parameters, and then annotate and reproduce them in the augmented reality building engine.

[0263] S5. Map the visualization model tree to the AM layer data of the process knowledge graph.

[0264] like Figure 2 As shown, the model hierarchy data and PMI data in the CAD model are output internally by the Creo modeling software. The Creo Toolkit API is used to extract the CAD model's hierarchy data and PMI data, outputting it as structured XML data. This data is then parsed and reconstructed in the augmented reality building engine by the model conversion module. The model hierarchy data is mapped into a folded tree, and the PMI data is reconstructed into visual model annotation information in the augmented reality building engine through line and text rendering. Model import and lightweighting operations are completed within the augmented reality building engine, using the Pixyz Plugin to implement these functions, converting the NURBS surface model into a mesh model as a visual resource for augmented reality guidance.

[0265] The visual node editor is an extension of the augmented reality building engine based on the C# language. It enables the rapid construction of augmented reality process guidance content. The node editor supports access to process knowledge graphs and can transform process knowledge entities into node elements in the node editor. Then, different nodes can be connected together in a node-based visual programming manner, thereby quickly and intuitively building process visualization guidance resources.

[0266] Furthermore, the visual node editor allows for the definition of node content, including process knowledge, guiding content, triggering events, behavior control, and logic control.

[0267] (1) Process Knowledge: The visual node editor supports reading and updating the process knowledge graph database. Based on the Neo4j.NET Drive API, a graph database access submodule is developed to realize the addition, deletion, modification and query of the graph database. By accessing the graph database, the process knowledge graph is mapped to the node elements of the visual node editor, and the process knowledge graph is visualized. The points of the process knowledge network are mapped to the visual node graph in the node editor. The node elements are visualized and edited through the node editor, realizing the dynamic visual editing and updating of the process knowledge graph.

[0268] (2) Guiding content: Augmented reality visualization elements presented to users, including images, text, videos, 3D models, and demonstration animations. The parameters of the visualization guiding content can be configured through the corresponding attribute definitions.

[0269] (3) Triggering events: The system's response when it enters a certain state or when the user performs an operation. Triggering events are the entry points for presenting visual guidance content. Triggering events include: initialization events, target recognition events, target loss events, human-computer interaction events, process loading events, process end events, and process completion events.

[0270] Initialization event: Triggered when the system initializes;

[0271] Target recognition event: Triggered when the system detects an AR registered target;

[0272] Target loss event: Triggered when a target registered with AR is lost during tracking;

[0273] Human-computer interaction events: triggered when a user interacts with a gesture;

[0274] Process loading / ending / completion events: triggered when the assembly process reaches a specific stage;

[0275] Initialization event, target identification event, target loss event, process loading event, process end event, and process completion event are automatic triggering events during the assembly process execution, which are automatically triggered when the system runs to a certain stage;

[0276] Human-computer interaction events are triggered by user gestures. A virtual hand model is constructed in virtual space by acquiring the user's hand pose and joint posture in real time. Collision interference detection is performed on UI elements based on the coordinate mapping relationship between the virtual and real worlds to determine the gesture interaction state. The collision interference detection of UI elements includes collision interference detection between the UI element and the virtual hand model, and collision detection between the UI element and rays emitted from the virtual finger positions.

[0277] Collision interference detection between UI elements and the virtual hand model can be simplified by representing the contact point between the virtual hand and the UI element as a sphere, and the UI element as an OBB bounding box. Then, UI triggering is performed by calculating the collision between the sphere and the OBB bounding box. The construction process of the OBB bounding box is as follows:

[0278] First, calculate the mean center of the UI element in the local coordinate system. For an object with n vertices, the mean center point P of the OBB bounding box is... m It can be calculated using the following formula:

[0279]

[0280] Among them, P i Let M represent the i-th vertex of the UI element, and then calculate the covariance matrix M of the object relative to the center point. c :

[0281]

[0282] For the covariance matrix M c Perform singular value decomposition to obtain a matrix U containing the principal axis directions:

[0283] M c =U∑U T

[0284] Where ∑ is the eigenvalue matrix, and U = {u1, u2, u3} represents the three principal axis directions of the OBB bounding box in the local coordinate system;

[0285] The half-axis width vector h of the OBB bounding box is:

[0286] h = {h1, h2, h3}

[0287] For each element h of vector h j ,

[0288] h j =r j u j j = 1, 2, 3

[0289] Where, r j h is half the average of the maximum and minimum vertex distances of the j-th basis vector. j Let be the width vector of the semi-axis along the j-th principal axis direction, i.e.:

[0290]

[0291] The center point P of the OBB bounding box obb for:

[0292] P obb =Pm +h

[0293] After constructing the OBB bounding box of the UI element, the collision detection between the sphere and the OBB bounding box is used as the trigger for the UI event. The collision calculation process between the sphere and the OBB bounding box is as follows:

[0294] S1. Transform the sphere's center coordinates from the world coordinate system to the local coordinate system of the OBB bounding box:

[0295] P sl =R obb T (P sw -P obb )

[0296] Among them, P sl P obb These are the coordinates of the sphere's center and the center of the OBB bounding box in the world coordinate system, respectively. sw R represents the coordinates of the sphere's center in the world coordinate system. obb T It is the transpose of the OBB bounding box rotation matrix;

[0297] S2. For the coordinates of the sphere center P in the local coordinate system of the OBB bounding box. sl If the absolute values ​​of its components on the x, y, and z axes are all less than the length of the semi-axis |E i If i = 1, 2, 3, then return a collision; otherwise, proceed to the next step.

[0298] S3. Find the nearest point P from the center of the sphere to the bounding box of the OBB using the following formula. near :

[0299] P near =(max(-h1,min(P)) x ,h1),max(-h2,min(P y ,h2),max(-h3),min(P z h3))

[0300] Among them, P x P y P z P respectively sl Components on the x, y, and z axes;

[0301] If the point is inside the sphere, return to the collision; otherwise, proceed to the next step.

[0302] S4. Calculate the distance d from the center of the sphere to each face of the OBB bounding box. If the shortest distance d is... min If r < r, then the sphere collides with the OBB bounding box, where r is the radius of the sphere, and the distance from the center of the sphere to each face can be calculated using the following formula:

[0303]

[0304] Among them, P plane Let N be any point in the plane, and let N be the plane normal vector.

[0305] Collision detection between UI elements and rays emitted from the virtual finger position can be simplified to collision detection between the ray and the OBB bounding box. If the ray collides with the OBB bounding box, it means that the current UI element is selected by the user. The collision detection calculation process between the ray and the OBB bounding box is as follows:

[0306] First, transform the ray from the world coordinate system to the local coordinate system. Given the ray's origin P0, ray direction D, and the center coordinates P of the OBB bounding box in world space. obb The rotation matrix R of the OBB bounding box, and the formulas for calculating the coordinates P′0 and direction D′ of the ray in the local space of the OBB bounding box are as follows:

[0307] P′0=R T (P0-P obb )

[0308] D′=R T D

[0309] In the local coordinate system of the OBB bounding box, the ray intersects the six SLAB surfaces of the OBB bounding box at point t. minx t miny t minz t maxx t maxy t maxz They are respectively:

[0310]

[0311] Among them, X min Y min Z min X max Y max Z max Let P' be the minimum and maximum values ​​of the six faces of the OBB bounding box in the local coordinate system. x 、P′ y 、P′ z Let P′0 be the component of P′0 on the X, Y, and Z coordinate axes, and D′ be the component of P′0 on the X, Y, and Z coordinate axes, respectively. x 、D′ y 、D′ z Let D′ represent the components of the X, Y, and Z coordinate axes, respectively; then calculate the intersection point t of the near and far planes of the OBB bounding box in the local coordinate system. near and t far :

[0312] t near =max(X min Y min Z min )

[0313] t far =min(X) max Y max Z max )

[0314] If t near <t far If the ray intersects the bounding box of the OBB, then the ray will intersect the bounding box of the OBB; otherwise, it will not intersect.

[0315] (4) Behavior control: Used to control the onboarding content. Behavior control includes: video playback / pause, onboarding element display / hide, and demo animation playback / pause.

[0316] (5) Logical Control: The logical control node defines the logical execution order between different behavior control nodes. Logical control includes: conditional execution, sequential execution, and synchronous execution, among which:

[0317] Conditional execution: The system executes function nodes that meet specified conditions;

[0318] Sequential execution: Executes all associated functional nodes sequentially according to their arrangement order;

[0319] Synchronous execution: All functional nodes associated with this node execute synchronously.

[0320] Synchronous execution: All functional nodes associated with this node execute synchronously.

[0321] The visual node editor supports keyframe animation editing, and allows defining animation types including translation, rotation, scaling, color change, and show / hide. It also allows defining animation parameters including animation callback, animation speed, and delay time.

[0322] The animation properties for translation, rotation, and scaling animations include the starting coordinate value, the target coordinate value, the reference coordinate system, and the animation duration;

[0323] Translation, scaling, and color-changing animations define the starting and target coordinate values ​​as keyframes. The intermediate process uses linear interpolation to perform interpolation calculations. For two vertices v1 and v2 on the keyframe, the interpolation formula for the vertex v(t) value at time t is v(t) = (1-t)v1 + tv2.

[0324] The rotation animation is represented in Euler angles by the visual editor. However, due to the ambiguity and gimbal lock issues associated with Euler angles, they are converted to quaternions for interpolation. The quaternion is defined as q = [wxyz].T The formula for converting Euler angles to quaternions is:

[0325]

[0326] Where α, β, and γ are the rotational components about the y-axis, x-axis, and z-axis, respectively;

[0327] For two vertices q0 and q1 in the keyframe, the rotation quaternion interpolation formula at time t is:

[0328]

[0329] Where θ = arccos(q0·q1);

[0330] The color-changing animation properties include: starting color, target color, and support for defining transparency channel properties;

[0331] Animation callbacks include: Start, Update, and Stop callbacks. Start and Stop are callbacks for starting and stopping the animation, while Update is a callback that occurs once for each frame during the animation playback. Different animation clips can be organized together through animation callback parameters to form complex mechanical simulation animations.

[0332] In summary, by combining 3D modeling and knowledge graph technology, the AR assembly guidance system constructed by the method described in this invention possesses assembly error correction and real-time feedback functions. The system can match the model resources associated with the current assembly process status and progress. Once an assembly error is detected, the system marks the error through prompts and guidance graphics and triggers real-time feedback adjustment functions to help the operator make real-time adjustments during error correction, ensuring the correct alignment and adjustment of the assembled parts.

[0333] Among them, model matching uses a model conversion module to convert CAD models into material models for AR guide programs. This includes extracting geometric and texture information from CAD model resources. During the assembly process, users identify assembly parts through sensors and cameras and obtain information such as position, posture, and size in real time. The model matching algorithm compares the geometric features and relative positional relationships of the assembly parts, identifies the erroneous position and posture of the assembly parts according to the error type, and provides corresponding adjustment methods and steps.

[0334] Feedback and adjustments are made by monitoring the operator's real-time assembly status through sensors and cameras. Based on the assembler's adjustment of the assembly parts position, the display content and position of the guidance information are updated to ensure correct alignment and adjustment.

[0335] This invention provides an event-driven AR visualization assembly system and its construction and editing method. It stores assembly process information based on knowledge graph technology; by extending the augmented reality construction engine, it implements an automatic model conversion module, quickly converting CAD design models into AR material models required for constructing AR guidance elements; and it develops a visualization node editor for assembly guidance content, mapping process knowledge to process nodes by accessing the knowledge graph, and enabling the visualization definition and editing of AR guidance content through node editing. This invention can effectively improve the construction efficiency of augmented reality-assisted assembly programs and reduce the construction cycle of augmented reality guidance programs to adapt to the ever-changing product-assisted assembly needs.

[0336] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0337] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for constructing and editing an event-driven AR visualization assembly system, characterized in that, include: Step S1: Construct an assembly process knowledge graph based on the Neo4j graph database; Step S2: Extend the augmented reality building engine based on the C# language to implement the model conversion module; take the CAD model of the assembled product as input, perform data parsing, topology reconstruction and model lightweighting operations on the CAD model, and convert the CAD model into AR material models required to build AR guiding elements; Step S3: Define the augmented reality visualization assembly visual guidance content, extend the augmented reality building engine based on the C# language to implement the visual node editor; by accessing the Neo4j graph database, process knowledge entities are transformed into editable visual node information and associated with the AR material model; Step S4: Edit and associate node elements in a visual node editor, and define AR visual guide elements, trigger events, demonstration animations and control logic. Step S5: Generate the augmented reality assembly bootloader.

2. The method for constructing and editing an event-driven AR visualization assembly system according to claim 1, characterized in that, The AR material model is obtained by converting a CAD model. An automatic model conversion module is implemented through secondary development of the augmented reality building engine using the C# language. This module can automatically convert CAD models into AR material models for building AR guiding elements. The automatic conversion module provides model parsing, topology reconstruction, model lightweighting, and product processing data (PMI) extraction functions, specifically including the following steps: Step S2.1: Extract the hierarchical structure data of the CAD model, and reconstruct the model hierarchy tree in the augmented reality building engine based on the hierarchical structure of the CAD model; Step S2.2: Through the model conversion module, perform data parsing and topology reconstruction on the CAD model to build augmented reality visualization model data, and populate the model hierarchy tree with the reconstructed model data; Step S2.3: Perform a model lightweighting operation on the reconstructed augmented reality visualization model data by deleting overlapping primitives, small objects, and invisible mesh data to reduce the performance overhead of the augmented reality bootstrap program; Step S2.4: Extract the PMI data of the CAD model and reproduce it by annotation in the augmented reality building engine; Step S2.5: Associate and map the visualization model tree with the process nodes.

3. The method for constructing and editing an event-driven AR visualization assembly system according to claim 1, characterized in that, The visual node editor accesses the Neo4j graph database to obtain assembly process knowledge, quickly transforms the assembly process knowledge into process flow nodes, and builds augmented reality guidance content based on the process flow nodes in a visual editing manner.

4. The method for constructing and editing an event-driven AR visualization assembly system according to claim 1, characterized in that, The visual node editor can define node content including process knowledge, guidance content, trigger events, behavior control, and logic control. 1) Process Knowledge: The visual node editor supports the editing and mapping function of the process knowledge graph database. By accessing the knowledge graph database, the process knowledge graph is mapped to the node elements of the visual node editor, and the process knowledge graph is presented visually. The node editor can be used to visually edit the presented node elements, realizing the dynamic visual editing and updating of the process knowledge graph. 2) Guiding Content: Augmented reality visualization elements presented to users, including images, text, videos, 3D models, and demonstration animations. The parameters of the guiding content can be configured through corresponding attribute definitions. 3) Triggering events: The system's response when it enters a certain state or when a user performs an operation. Triggering events are the entry points for presenting visual guidance content. Triggering events include: initialization event, target recognition event, target loss event, human-computer interaction event, process loading event, process end event, and process completion event; Initialization event: Triggered when the system initializes; Target recognition event: Triggered when the system detects an AR registered target; Target loss event: Triggered when a target registered with AR is lost during tracking; Human-computer interaction events: triggered when a user interacts with a gesture; Process loading / ending / completion events: triggered when the assembly process reaches a specific stage; 4) Behavior control: Used to control the onboarding content. Behavior control includes: video playback / pause, onboarding element display / hide, and demo animation playback / pause; 5) Logical Control: Logical control nodes define the logical execution order between different behavior control nodes. Logical control includes: conditional execution, sequential execution, and synchronous execution. The conditional execution means that the system executes the functional nodes that meet the specified conditions. The sequential execution means that all associated functional nodes are executed in the order they are arranged. Synchronous execution: All functional nodes associated with this node execute synchronously.

5. The method for constructing and editing an event-driven AR visualization assembly system according to claim 4, characterized in that, The human-computer interaction event is triggered by user gesture interaction. A virtual hand model is constructed in virtual space by acquiring the user's hand pose and joint posture in real time. Collision interference detection is performed on UI elements based on the coordinate mapping relationship between the virtual and real worlds to determine the gesture interaction state. The collision interference detection of UI elements includes collision interference detection between UI elements and the virtual hand model, and collision detection between UI elements and rays emitted from the virtual finger positions. Collision interference detection between UI elements and the virtual hand model simplifies the contact point between the virtual hand and the UI element to a sphere, and the UI element to an OBB bounding box. Then, UI triggering is performed by calculating the collision between the sphere and the OBB bounding box. The construction process of the OBB bounding box is as follows: First, calculate the mean center of the UI element in the local coordinate system. For an object with n vertices, the mean center point P of the OBB bounding box is... m It can be calculated using the following formula: Among them, P i Let M represent the i-th vertex of the UI element, and then calculate the covariance matrix M of the object relative to the center point. c : For the covariance matrix M c Perform singular value decomposition to obtain a matrix U containing the principal axis directions: M c =U∑U T Where ∑ is the eigenvalue matrix, and U = {u1, u2, u3} represents the three principal axis directions of the OBB bounding box in the local coordinate system; The half-axis width vector h of the OBB bounding box is: h = {h1, h2, h3} For each element h of vector h j , h j =r j u j ,j=1,2,3 Where, r j h is half the average of the maximum and minimum vertex distances of the j-th basis vector. j Let be the width vector of the semi-axis along the j-th principal axis direction, i.e.: The center point P of the OBB bounding box obb for: P obb =P m +h After constructing the OBB bounding box of the UI element, the collision detection between the sphere and the OBB bounding box is used as the trigger for the UI event. The collision calculation process between the sphere and the OBB bounding box is as follows: 1) Transform the sphere's center coordinates from the world coordinate system to the local coordinate system of the OBB bounding box: P sl =R obb T (P sw -P obb ) Among them, P sl P obb These are the coordinates of the sphere's center and the center of the OBB bounding box in the world coordinate system, respectively. sw R represents the coordinates of the sphere's center in the world coordinate system. obb T It is the transpose of the OBB bounding box rotation matrix; 2) For the sphere center coordinates P in the local coordinate system of the OBB bounding box sl If the absolute values ​​of its components on the x, y, and z axes are all less than the length of the semi-axis | i If i = 1, 2, 3, then return a collision; otherwise, proceed to the next step. 3) Find the nearest point P from the center of the sphere to the bounding box of the OBB using the following formula. near : P near <(max(-h1, min(P x ,h1) ,max(-h2 ,min(P y ,h2) ,max(-h3 ,min(Pz ， h3)) Among them, P x P y P z P respectively sl Components on the x, y, and z axes; If the point is inside the sphere, return to the collision; otherwise, proceed to the next step. 4) Calculate the distance d from the center of the sphere to each face of the OBB bounding box. If the shortest distance d is... min If r < r, then the sphere collides with the OBB bounding box, where r is the radius of the sphere, and the distance d from the center of the sphere to each face is calculated using the following formula: Among them, P plane Let N be any point in the plane, and let N be the plane normal vector. Collision detection between UI elements and the ray emitted from the virtual finger position is simplified to collision detection between the ray and the OBB bounding box. If the ray collides with the OBB bounding box, it means that the current UI element is selected by the user. The collision detection calculation process between the ray and the OBB bounding box is as follows: Transform the ray from the world coordinate system to the local coordinate system. Given the ray's origin P0, ray direction D, and the center coordinates P of the OBB bounding box in world space. obb The rotation matrix R of the OBB bounding box, and the formulas for calculating the coordinates P′0 and direction D′ of the ray in the local space of the OBB bounding box are as follows: P′0=R T (P0-P obb ) D′=R T D In the local coordinate system of the OBB bounding box, the ray intersects the six SLAB surfaces of the OBB bounding box at point t. minx t miny t minz t maxx t maxy t maxz They are respectively: Among them, X min Y min Z min X max Y max Z max Let P' be the minimum and maximum values ​​of the six faces of the OBB bounding box in the local coordinate system. x 、P′ y 、P′ z Let P′0 be the component of P′0 on the X, Y, and Z coordinate axes, and D′ be the component of P′0 on the X, Y, and Z coordinate axes, respectively. x 、D′ y 、D′ z Let D′ represent the components of the X, Y, and Z coordinate axes, respectively; then calculate the intersection point t of the near and far planes of the OBB bounding box in the local coordinate system. near and t far : t near =max(X min ,Y min ,Z min ) t far =min(X max ,Y max ,Z max ) If t near <t far If the ray intersects the bounding box of the OBB, then the ray will intersect the bounding box of the OBB; otherwise, it will not intersect.

6. The method for constructing and editing an event-driven AR visualization assembly system according to claim 1, characterized in that, The visual node editor supports keyframe animation editing, and defines animation types including translation, rotation, scaling, color change, and show / hide. It also defines animation parameters including animation callback, animation speed, and delay time. The animation properties of the translation, rotation, and scaling animations include the starting coordinate value, the target coordinate value, the reference coordinate system, and the animation duration; The translation, scaling, and color-changing animations define the starting coordinate value and the target coordinate value as keyframes. The intermediate process uses a linear interpolation method for interpolation calculation. For two vertices v1 and v2 on the keyframe, the interpolation formula for the vertex v(t) value at time t is v(t) = (1-t)v1 + tv2. The rotation animation is represented in Euler angles by the visual editor. However, due to the ambiguity and gimbal lock issues associated with Euler angles, they are converted to quaternions for interpolation. The quaternion q = [wxyz] is defined as follows. T The formula for converting Euler angles to quaternions is: Where α, β, and γ are the rotational components about the y-axis, x-axis, and z-axis, respectively; For two vertices q0 and q1 in the keyframe, the rotation quaternion interpolation formula at time t is: Where θ = arccos(q0·q1); The color-changing animation attributes include: starting color, target color, and support for defining transparency channel attributes; The animation callbacks include: Start, Update, and Stop callbacks. Start and Stop are callbacks for starting and stopping the animation, while Update is a callback performed once for each frame during the animation playback. Different animation segments can be organized together through the animation callback parameters to form complex mechanical simulation animations.

7. An event-driven AR visualization assembly system, characterized in that, include: Module M1: Constructing an assembly process knowledge graph based on the Neo4j graph database; Module M2: Based on the C# language, this module extends the augmented reality building engine to realize the model conversion module. Taking the CAD model of the assembled product as input, it performs data parsing, topology reconstruction, and model lightweighting operations on the CAD model to convert it into AR material models required for building AR guiding elements. Module M3: Defines the visual guidance content for augmented reality assembly visualization. Based on the C# language, it extends the augmented reality building engine to implement a visual node editor. By accessing the Neo4j graph database, it transforms process knowledge entities into editable visual node information and associates and maps them with the AR material model. Module M4: Through the visual node editor, node elements can be edited and associated in a visual node editing manner, defining AR visual guide elements, trigger events, demonstration animations and control logic; Module M5: Generates augmented reality assembly bootloader.

8. The event-driven AR visualization assembly system according to claim 7, characterized in that, The AR material model is obtained by converting a CAD model. An automatic model conversion module is implemented through secondary development of the augmented reality building engine using the C# language. This module can automatically convert CAD models into AR material models for building AR guiding elements. The automatic conversion module provides model parsing, topology reconstruction, model lightweighting, and product processing data (PMI) extraction functions, specifically including the following steps: Module M2.1: Extracts hierarchical structure data from the CAD model and reconstructs the model hierarchy tree in the augmented reality building engine based on the hierarchical structure of the CAD model; Module M2.2: Through the model conversion module, data parsing and topology reconstruction of CAD models are performed to build augmented reality visualization model data, and the reconstructed model data is populated into the model hierarchy tree; Module M2.3: Performs model lightweighting operations on the reconstructed augmented reality visualization model data by deleting overlapping primitives, small objects, and invisible mesh data, thereby reducing the performance overhead of the augmented reality bootstrap program; Module M2.4: Extracts PMI data from CAD models and reproduces it in the augmented reality building engine; Module M2.5: Associates and maps the visual model tree with process nodes; The visual node editor obtains assembly process knowledge by accessing the Neo4j graph database, quickly transforms the assembly process knowledge into process flow nodes, and builds augmented reality guidance content based on the process flow nodes in a visual editing manner. The visual node editor can define node content including process knowledge, guidance content, trigger events, behavior control, and logic control. 1) Process Knowledge: The visual node editor supports the editing and mapping function of the process knowledge graph database. By accessing the knowledge graph database, the process knowledge graph is mapped to the node elements of the visual node editor, and the process knowledge graph is presented visually. The node editor can be used to visually edit the presented node elements, realizing the dynamic visual editing and updating of the process knowledge graph. 2) Guiding Content: Augmented reality visualization elements presented to users, including images, text, videos, 3D models, and demonstration animations. The parameters of the guiding content can be configured through corresponding attribute definitions. 3) Triggering events: The system's response when it enters a certain state or when a user performs an operation. Triggering events are the entry points for presenting visual guidance content. Triggering events include: initialization events, target recognition events, target loss events, human-computer interaction events, process loading events, process end events, and process completion events. Initialization event: Triggered when the system initializes; Target recognition event: Triggered when the system detects an AR registered target; Target loss event: Triggered when a target registered with AR is lost during tracking; Human-computer interaction events: triggered when a user interacts with a gesture; Process loading / ending / completion events: triggered when the assembly process reaches a specific stage; 4) Behavior control: Used to control the onboarding content. Behavior control includes: video playback / pause, onboarding element display / hide, and demo animation playback / pause; 5) Logical Control: Logical control nodes define the logical execution order between different behavior control nodes. Logical control includes: conditional execution, sequential execution, and synchronous execution. The conditional execution means that the system executes the functional nodes that meet the specified conditions. The sequential execution means that all associated functional nodes are executed in the order they are arranged. The synchronous execution means that all functional nodes associated with this node execute synchronously. The human-computer interaction event is triggered by user gesture interaction. A virtual hand model is constructed in virtual space by acquiring the user's hand pose and joint posture in real time. Collision interference detection is performed on UI elements based on the coordinate mapping relationship between the virtual and real worlds to determine the gesture interaction state. The collision interference detection of UI elements includes collision interference detection between UI elements and the virtual hand model, and collision detection between UI elements and rays emitted from the virtual finger positions. Collision interference detection between UI elements and the virtual hand model simplifies the contact point between the virtual hand and the UI element to a sphere, and the UI element to an OBB bounding box. Then, UI triggering is performed by calculating the collision between the sphere and the OBB bounding box. The construction process of the OBB bounding box is as follows: First, calculate the mean center of the UI element in the local coordinate system. For an object with n vertices, the mean center point P of the OBB bounding box is... m It can be calculated using the following formula: Among them, P i Let M represent the i-th vertex of the UI element, and then calculate the covariance matrix M of the object relative to the center point. c : For the covariance matrix M c Perform singular value decomposition to obtain a matrix U containing the principal axis directions: M c =U∑U T Where ∑ is the eigenvalue matrix, and U = {u1, u2, u3} represents the three principal axis directions of the OBB bounding box in the local coordinate system; The half-axis width vector h of the OBB bounding box is: h = {h1, h2, h3} For each element h of vector h j , h j =r j u j ,j=1,2,3 Where, r j h is half the average of the maximum and minimum vertex distances of the j-th basis vector. j Let be the width vector of the semi-axis along the j-th principal axis direction, i.e.: The center point P of the OBB bounding box obb for: P obb =P m +h After constructing the OBB bounding box of the UI element, the collision detection between the sphere and the OBB bounding box is used as the trigger for the UI event. The collision calculation process between the sphere and the OBB bounding box is as follows: 1) Transform the sphere's center coordinates from the world coordinate system to the local coordinate system of the OBB bounding box: P sl =R obb T (P sw -P obb ) Among them, P sl P obb These are the coordinates of the sphere's center and the center of the OBB bounding box in the world coordinate system, respectively. sw R represents the coordinates of the sphere's center in the world coordinate system. obb T It is the transpose of the OBB bounding box rotation matrix; 2) For the sphere center coordinates P in the local coordinate system of the OBB bounding box sl If the absolute values ​​of its components on the x, y, and z axes are all less than the length of the semi-axis | i If i = 1, 2, 3, then return a collision; otherwise, proceed to the next step. 3) Find the nearest point P from the center of the sphere to the bounding box of the OBB using the following formula. near : P near <(max(-h1, min(P x ,h1) ,max(-h2 ,min(P y ,h2) ,max(-h3 ,min(P z ,h3)) Among them, P x P y P z P respectively sl Components on the x, y, and z axes; If the point is inside the sphere, return to the collision; otherwise, proceed to the next step. 4) Calculate the distance d from the center of the sphere to each face of the OBB bounding box. If the shortest distance d is... min If r < r, then the sphere collides with the OBB bounding box, where r is the radius of the sphere, and the distance d from the center of the sphere to each face is calculated using the following formula: Among them, P plane Let N be any point in the plane, and let N be the plane normal vector. Collision detection between UI elements and the ray emitted from the virtual finger position is simplified to collision detection between the ray and the OBB bounding box. If the ray collides with the OBB bounding box, it means that the current UI element is selected by the user. The collision detection calculation process between the ray and the OBB bounding box is as follows: Transform the ray from the world coordinate system to the local coordinate system. Given the ray's origin P0, ray direction D, and the center coordinates P of the OBB bounding box in world space. obb The rotation matrix R of the OBB bounding box, and the formulas for calculating the coordinates P′0 and direction D′ of the ray in the local space of the OBB bounding box are as follows: P′0=R T (P0-P obb ) D′=R T D In the local coordinate system of the OBB bounding box, the ray intersects the six SLAB surfaces of the OBB bounding box at point t. minx t miny t minz t maxx t maxy t maxz They are respectively: Among them, X min Y min Z min X max Y max Z max Let P' be the minimum and maximum values ​​of the six faces of the OBB bounding box in the local coordinate system. x 、P′ y 、P′ z Let P′0 be the component of P′0 on the X, Y, and Z coordinate axes, and D′ be the component of P′0 on the X, Y, and Z coordinate axes, respectively. x 、D′ y 、D′ z Let D′ represent the components of the X, Y, and Z coordinate axes, respectively; then calculate the intersection point t of the near and far planes of the OBB bounding box in the local coordinate system. near and t far : t near =max(X min ,Y min ,Z min ) t far =min(X max ,Y max ,Z max ) If t near <t far If the ray intersects the OBB bounding box, then the ray will intersect the OBB bounding box; otherwise, it will not intersect. The visual node editor supports keyframe animation editing, and defines animation types including translation, rotation, scaling, color change, and show / hide. It also defines animation parameters including animation callback, animation speed, and delay time. The animation properties of the translation, rotation, and scaling animations include the starting coordinate value, the target coordinate value, the reference coordinate system, and the animation duration; The translation, scaling, and color-changing animations define the starting coordinate value and the target coordinate value as keyframes. The intermediate process uses a linear interpolation method for interpolation calculation. For two vertices v1 and v2 on the keyframe, the interpolation formula for the vertex v(t) value at time t is v(t) = (1-t)v1 + tv2. The rotation animation is represented in Euler angles by the visual editor. However, due to the ambiguity and gimbal lock issues associated with Euler angles, they are converted to quaternions for interpolation. The quaternion q = [wxyz] is defined as follows. T The formula for converting Euler angles to quaternions is: Where α, β, and γ are the rotational components about the y-axis, x-axis, and z-axis, respectively; For two vertices q0 and q1 in the keyframe, the rotation quaternion interpolation formula at time t is: Where θ = arccos(q0·q1); The color-changing animation attributes include: starting color, target color, and support for defining transparency channel attributes; The animation callbacks include: Start, Update, and Stop callbacks. Start and Stop are callbacks for starting and stopping the animation, while Update is a callback performed once for each frame during the animation playback. Different animation segments can be organized together through the animation callback parameters to form complex mechanical simulation animations.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the event-driven AR visualization assembly system construction and editing method according to any one of claims 1 to 6.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of the event-driven AR visualization assembly system construction and editing method according to any one of claims 1 to 6.

Images