A mixed reality-based engineering equipment real-time interaction system and motion simulation method

Abstract

This invention belongs to the field of human-computer interaction technology and discloses a real-time interactive system and motion simulation method for engineering equipment based on mixed reality. The system includes mixed reality hardware, a control handle, a simulation control module, a spatial mapping and collision detection module, and an information storage module. By establishing complex mechanical constraints in the simulation of engineering equipment operation and introducing simple constraints from a physics engine, the system realistically represents the motion process of various components of the engineering equipment. This solves the problem of balancing real-time performance and simulation effect in motion simulation. Operators can virtually operate the engineering equipment in a mixed virtual environment with a realistic sense of immersion and interactivity, simulating the construction process. Therefore, this invention provides a platform-independent virtual interactive simulation method that satisfies both interactivity and real-time performance.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction technology, specifically relating to a real-time interactive system and motion simulation method for engineering equipment based on mixed reality. Background Technology

[0002] Engineering equipment such as aerial work platforms, pump trucks, and rock drilling rigs are complex in structure, difficult to operate, and have a high risk factor, requiring a high level of technical skill from operators. In order to reduce construction risks, improve work quality and efficiency, and ensure construction safety, simulation training systems for engineering machinery have emerged to simulate the corresponding movement behaviors generated by operators controlling engineering equipment. However, most of these systems use computer 3D imaging and 3D animation technologies, which cannot meet the requirements for interactivity. Mixed reality is a computer virtual technology that allows the real world and virtual objects to be displayed and interacted in the same visual space, seamlessly connecting the real world and the virtual world to create a completely new visual environment. It has gradually developed and been applied rapidly in fields such as industry, education and training, entertainment, and medicine.

[0003] In the development of engineering equipment, motion simulation is mostly conducted in professional software environments such as PRO / E and ADAMS. Such simulations are highly dependent on the development platform and lack interactivity, making training or field verification impossible. To make the development process independent of the software platform, the traditional method is to calculate the position coordinates of different parts of the working device using a transformation matrix and then display them. While the calculation results are relatively accurate, the computation is complex and difficult to meet real-time requirements. Existing technologies include simulations of the motion of emergency rescue machinery based on virtual reality modeling languages, which simplify motion control through the setting of DOF nodes; however, the development of the corresponding control interface with the platform is not yet perfect. However, in the field of virtual simulation control of engineering equipment, there is currently no technology that can simultaneously meet the requirements of real-time construction process verification and training at the construction site, real-time simulation of equipment motion operation, and intuitive display through virtual visualization. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a real-time interactive system and motion simulation method for engineering equipment based on mixed reality.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] In a first aspect, the present invention proposes a real-time interactive system for engineering equipment based on mixed reality, including mixed reality hardware, a control handle, a simulation control module, a space mapping and collision detection module, and an information storage module.

[0007] The mixed reality hardware is communicatively connected to the simulation control module, the spatial mapping and collision detection module, and is used to collect and display the three-dimensional model of the engineering equipment, identify and collect construction site environmental information, and obtain the location information of the operators.

[0008] The control handle is communicatively connected to the simulation control module and is used to transmit digital signals of operation to the simulation control module via the handle drive.

[0009] The simulation control module is communicatively connected to the mixed reality hardware, the control handle, the spatial mapping and collision detection module, and the information storage module. It is used to receive virtual interactive information and parse the virtual interactive information into instruction information corresponding to the movement of the virtual 3D model. The virtual 3D model moves and simulates in real time according to the instructions. The virtual interactive information includes digital signals from the control handle, the handle operation of the mixed reality hardware, the operator's gestures, the operator's voice interaction instructions, the location information of the operator wearing the mixed reality hardware, and the real construction site environment information.

[0010] The spatial mapping and collision detection modules are respectively connected to the mixed reality hardware and the simulation control module. They are used to display the construction site environment information collected by the mixed reality hardware and the virtual 3D model in the simulation control module through the display screen of the mixed reality hardware in the real construction site environment in a virtual-real fusion manner. At the same time, they perform accurate collision detection and real-time motion interference simulation between virtual objects and real scene objects.

[0011] The information storage module is communicatively connected to the simulation control module, the space mapping module, and the collision detection module, respectively, and is used to save motion state information and collision detection information records during virtual training and virtual verification in order to reproduce the training and verification results.

[0012] In conjunction with the first aspect, the real-time interactive system of the present invention further includes a server, which is a hardware carrier for the simulation control module, the space mapping and collision detection module, and the information storage module.

[0013] In conjunction with the first aspect, the simulation control module further includes an interactive command acquisition module, a dynamic simulation calculation module, and a scene management and real-time graphics rendering module;

[0014] The interactive instruction acquisition module is used to receive operation information from operators wearing mixed reality hardware, and the operation information is virtual interactive information;

[0015] The dynamics simulation calculation module is the core control module of the entire simulation system. It is used to perform real-time dynamics simulation calculations based on the operator's operation information and output the simulation calculation results to the scene management and real-time graphics rendering module.

[0016] The scene management and real-time graphics rendering module updates the motion state of the 3D model of engineering equipment in the virtual scene in real time based on the simulation calculation results of the dynamics simulation calculation module, and uses the graphics rendering engine to visualize the operation results and display them to the operator on the mixed reality hardware. The simulation control module parses the received virtual interaction information into command information corresponding to the motion of the virtual 3D model through the dynamics simulation calculation module. The virtual 3D model moves and simulates in real time according to specific commands, and the image information of the virtual 3D model is synchronized to the display screen of the mixed reality hardware through the scene management and real-time graphics rendering module, spatial mapping and collision detection module.

[0017] In conjunction with the first aspect, the control handle further includes a handle driver, which transmits the digital operation signal to the simulation control module. The handle driver parses the operation control flow input by the control handle into an operation digital signal and transmits the operation digital signal to the simulation control module, specifically the interactive instruction acquisition module in the simulation control module.

[0018] In conjunction with the first aspect, the server where the handle driver and simulation control module are located is connected via Bluetooth or USB.

[0019] Secondly, this invention proposes a motion simulation method for engineering equipment based on mixed reality. Based on the aforementioned real-time pseudo-interactive system, the dynamic simulation calculation module performs multi-rigid-body dynamics modeling on the engineering equipment and uses a physics engine to establish corresponding kinematic constraints between the rigid body components of the engineering equipment. The kinematic constraints include a multi-rigid-body dynamics system and a motion constraint system.

[0020] In conjunction with the second aspect, the motion simulation method further includes the following steps:

[0021] Step 1: Configure the mixed reality hardware, controller, and server; that is, establish communication connections between the mixed reality hardware and the server, and between the controller and the server.

[0022] Step 2: Load the virtual model into the mixed reality hardware; that is, the server transmits the stored virtual scene data to the mixed reality hardware. After the mixed reality hardware recognizes the actual environment information, it merges the virtual scene with the real scene to present a virtual-real fusion scene.

[0023] Step 3: The user inputs commands by manipulating the control handle; if the user inputs incorrectly, an error message is displayed; if the user inputs correctly, dynamic simulation calculations are performed, and the simulation results are output to the scene management and real-time graphics rendering module. The scene management and real-time graphics rendering module updates the motion state of the 3D model of the engineering equipment in the virtual scene in real time based on the simulation results of the dynamic simulation calculation module, and outputs the operation results to the mixed reality hardware for display to the operator.

[0024] Step 4: Display the 3D model of the engineering equipment in the virtual scene updated in real time in Step 3 in the real construction site environment, and display it in a virtual-real fusion manner. At the same time, collision detection and real-time motion interference simulation are performed between virtual objects and real scene objects. If interference with the real environment is detected, interference information is sent to the operator wearing the mixed reality hardware. If no interference with the real environment is detected, the next verification and training task is continued.

[0025] Step 5: Write the operation process into the information storage module, record and store the operation process information during the process, and complete the task.

[0026] In conjunction with the second aspect, the multi-rigid-body dynamics system further includes a physical model of the multi-rigid-body system and a mathematical model of the multi-rigid-body system; the physical model of the multi-rigid-body system is formed by physically modeling the geometric model of the engineering equipment, thereby creating a physical model that expresses the mechanical properties of the system, and the physical modeling includes the following steps:

[0027] Step S1: Assemble the geometric model of the engineering equipment according to the kinematic constraints and the initial position conditions of the virtual 3D model. By analyzing the motion dependency relationship between each component of the virtual 3D model, set the parent-child nesting relationship between the components of the virtual 3D model, and finally complete the construction of the entire engineering equipment model tree.

[0028] Step S2: Treat each geometric model component as a rigid component and set its Cartesian generalized coordinate vector. Where l = 1, 2, ..., n, n represents a multi-rigid-body system with n points, and any adjacent rigid bodies connected by hinges are considered as one element, i.e., n represents the number of elements in the multi-rigid-body system; q l Taking a single rigid body as a reference, and describing the position of another rigid body relative to this rigid body using generalized coordinates, denoted by the Cartesian generalized coordinate vector, the system can be described using a Lagrange coordinate matrix q; let r l (x, y, z) are vectors representing the center of mass of each rigid body component in the absolute coordinate system, where x represents the x-axis, y represents the y-axis, and z represents the z-axis. γ = (ψ, θ, φ). lLet ψ represent the precession angle, θ represent the nutation angle, and φ represent the rotation angle of the rigid body relative to its own coordinate base. Then, the configuration vector matrix of the entire multi-rigid-body dynamics system is:

[0029] q = (q1 q2 … q) n ) T (1)

[0030] The kinematic constraint equations of the entire multi-rigid-body dynamic system are expressed as follows:

[0031]

[0032] In equation (2), m represents the number of constraint pairs, and Γ represents the number of constraint pairs. V As a whole, it represents the set of motion constraint equations for the entire multi-rigid-body dynamics system. This represents the individual motion constraint equations for the corresponding components, where w = 1, 2, 3, ..., m, and V is an arbitrarily chosen variable used to distinguish it from the driving constraint equations;

[0033] When the total degrees of freedom of the entire multi-rigid-body dynamics system is zero, it has definite motion. Therefore, the equation for the number of driving constraints required for the multi-rigid-body dynamics system is expressed as follows:

[0034] Γ H (q,t)=0 (3)

[0035] Where Γ(q,t)=0 is the equation for the number of driving constraints, and the driving constraints are time functions of the generalized coordinates; Γ H (q,t)=0 is the vector form of the driving constraint equation; H is an arbitrarily chosen variable used to distinguish it from the motion constraint equation; t represents the motion time of the multi-rigid-body system; for a system with n coordinates q=(q1 q2 … q n ) T In a multi-rigid-body system with *m* constraint pairs, only *nm* coordinates are independent, meaning *nm* is the number of degrees of freedom. From a kinematic perspective, for example, a planar linkage system, the system only has definite motion when the total number of degrees of freedom is zero. Therefore, the number of definite driving constraints required for the system is *nm*, and its driving constraint equation is expressed as Γ. H (q,t)=0.

[0036] The kinematic constraints of equation (2), the driving constraints of equation (3), and the Euler parameter constraints constitute all the constraints on the multi-rigid-body dynamics system:

[0037]

[0038] Equation (4) constitutes a set of n nonlinear position equations for the entire multi-rigid-body dynamics system in generalized coordinates; Γ(q,t)=0 is the driving constraint equation for the entire system, and the driving constraint is a time function of the generalized coordinates; This is the expression for the Euler parameter constraint equation.

[0039] Step S3: Analyze the geometric model of the engineering equipment. Based on the constraint equation types between the components of the geometric model, use the physics engine to establish corresponding constraint types for the components of the geometric model, and complete the physical modeling from the geometric model of the engineering equipment to the physical model of the multi-rigid-body system.

[0040] In conjunction with the second aspect, the mathematical model of the multi-rigid-body system is further described as a mathematical modeling performed after obtaining the physical model of the multi-rigid-body system, resulting in mathematical models of the velocity and acceleration of the multi-rigid-body system, and thus obtaining the dynamic model of the multi-rigid-body system.

[0041] In conjunction with the second aspect, the mathematical model of the multi-rigid-body system further includes a mathematical model of the motion of rigid-body components and a mathematical model of the rotation of rigid-body components.

[0042] In conjunction with the second aspect, the modeling process of the mathematical model of the rigid body component's motion further includes the following steps:

[0043] Step A: Differentiate equation (4), then the velocity constraint equation for the multi-rigid-body system is:

[0044]

[0045] Equation (5) is the derivative of equation (4), where Γ q The Jacobian matrix of the system is, i.e. Γ t The time derivative of the constraint equation is denoted as , i.e. For the system's generalized velocity;

[0046] Step B: Taking the second derivative of equation (4), the acceleration constraint equation for the multi-rigid-body system is:

[0047]

[0048] In the formula Γ q Γ is the Jacobian matrix; qt Let be the derivative of the Jacobian matrix with respect to time, i.e. Γ tt To find the second derivative of the constraint equation with respect to time, i.e. For the system's generalized acceleration;

[0049] Step C: From the position constraint equation of the multi-rigid-body system in equation (4), the velocity constraint equation of the multi-rigid-body system in equation (5), and the acceleration constraint equation of the multi-rigid-body system in equation (6), we obtain the first kind of Laplace multiplier form of the motion equation of the multi-rigid-body system:

[0050]

[0051] Where q, v, η∈R n These represent the system's generalized coordinates, velocity, and acceleration vectors, respectively. λ∈R m M(q,t)∈R is a column vector of Lagrange multipliers, representing the internal forces and moments between rigid body members connected by a kinematic pair, where t∈R is time, and M(q,t)∈R. n×n Let f(q,v,t)∈R represent the mass matrix of the system. n R is a generalized column vector of external forces, including external forces and external moments. n In this context, n represents an n-dimensional real vector space, numerically corresponding to the number of elements in a multi-rigid-body system; R m In this context, m represents an m-dimensional real vector space, which is numerically the same as the number of constraint pairs.

[0052] In conjunction with the second aspect, further, the modeling of the mathematical model of the rigid body component rotation requires the use of a body coordinate system fixed on the rigid body component to determine the motion of the multi-rigid body system for each mechanism of the engineering equipment in three-dimensional space. Specifically, the fixed-point rotation of the rigid body component is described by three attitude variables: direction cosine matrix, Euler angle, and Euler quaternion.

[0053] The Euler angle is: γ l =(ψ,θ,φ) l (8)

[0054] The direction cosine matrix is:

[0055]

[0056] The Euler quaternion is:

[0057]

[0058] The direction cosine matrix can be represented using Euler quaternions as follows:

[0059]

[0060] The Euler parameter variables in the direction cosine matrix and Euler quaternion must satisfy the constraint equations:

[0061]

[0062] Where a1, a2, a3 are vectors of Euler quaternions, and a4 is a scalar of Euler quaternions. The constraint vector is formed by the Euler parameters of the rigid body component.

[0063] In conjunction with the second aspect, further, the orientation parameters of the body coordinate system relative to the global coordinate system in the generalized coordinate system of the rigid body component can be represented by the direction cosine matrix, Euler angles, or Euler parameters.

[0064] In conjunction with the second aspect, the mathematical model of the multi-rigid-body system can be mathematically modeled by using Cartesian coordinates or Lagrange coordinates to model the physical model of the multi-rigid-body system.

[0065] In conjunction with the second aspect, the motion constraint system is further constructed based on the dynamics theory of multi-rigid-body systems. It calculates the Jacobian matrix of two connected rigid-body components according to the constraint type, and calculates the moment of inertia in conjunction with the shape of the rigid-body components to update the position and velocity of the interconnected rigid-body components, thereby simulating the effect of force on the connected objects. The constraint types between the rigid-body components include hinge constraints and sliding rod constraints, that is, the motion constraints of the motion constraint system include hinge constraints and sliding rod constraints.

[0066] After determining the constraint type between rigid body components, the motion constraint system uses the corresponding constraints in the physics engine to connect them, and calculates motion, rotation and collision responses by assigning realistic physical properties to rigid body objects.

[0067] In conjunction with the second aspect, further establishing the motion constraints for the dynamic model of the multi-rigid-body system of engineering equipment specifically includes the following steps:

[0068] Step 1: Simplify the multi-rigid-body system model of the actual system (i.e., the virtual three-dimensional model, which is called the multi-rigid-body system model in the dynamics of motion constraints);

[0069] Complex engineering equipment objects are theoretically abstracted and simplified into components such as bodies and hinges. A body is a component in a multi-rigid-body system; a hinge is a motion constraint between rigid bodies, without mass; the interaction between rigid bodies is limited by motion constraints.

[0070] Step 2: Construct a physical model with constrained connections;

[0071] The geometric model of the engineering equipment is assembled based on kinematic constraints and the initial position conditions of the virtual 3D model, and corresponding constraint connections are established between rigid body components based on the corresponding force analysis.

[0072] Hinge constraint: This is a constraint established at a certain node that restricts the rotation of two rigid body members to only rotate around that axis. This axis is called the hinge axis. For example, a door or wheel that can only rotate around one axis. Users can set rotation angle limits for the hinge axis.

[0073] Slider constraint: refers to a constraint relationship in which two rigid body members can only move along a certain axis.

[0074] In conjunction with the second aspect, the implementation steps of the hinge constraint are as follows:

[0075] Step 1: Obtain the centroids of the first and second rigid body components. The formula for the centroid of a rigid body component is:

[0076]

[0077] Where, r c The x vector represents the position of the center of mass of a rigid body component. c y c z c Let x, y, z represent the coordinates of the center of mass of the rigid body, respectively; i, j, and k represent the vector basis, respectively.

[0078] Step II: Calculate the position of the anchor point relative to the first rigid body component, and obtain the hinge axis of the first rigid body component;

[0079] Step 3: Calculate the position of the anchor point relative to the second rigid body component, and obtain the hinge axis of the second rigid body component;

[0080] Step IV: Apply hinge constraints to the two rigid body members.

[0081] In conjunction with the second aspect, the implementation steps of the sliding rod constraint are as follows:

[0082] Step a: Use equation (13) to obtain the centroids of the first and second rigid body components;

[0083] Step b: Calculate the position of the second rigid body member relative to the first rigid body member, and obtain the sliding axis of the first rigid body member;

[0084] Step c: Calculate the position of the first rigid body member relative to the second rigid body member, and obtain the sliding axis of the second rigid body member;

[0085] Step d: Apply sliding bar constraints to the two rigid body members.

[0086] Furthermore, in conjunction with the second aspect, the motion constraint system established through the aforementioned physics engine requires adjustments to the spatial orientation or constraint axes of the rigid body components in order to successfully use the functions within the physics engine. The adjustment method is as follows: applying the Rodriguez rotation formula, the rotation matrix around the unit vector μ(x,y,z) with a rotation angle of β is:

[0087]

[0088] After obtaining the spatial orientation of the rigid body component, the rigid body component is adjusted using formula (14).

[0089] In conjunction with the second aspect, the physics engine further includes a gravity engine, which can perform gravity tests just like in reality. If improper operation causes instability of the center of gravity, a rollover will occur.

[0090] Compared with the prior art, the present invention provides a luffing mechanism, a boom system, and a drainage robot, which have the following beneficial effects:

[0091] (1) The real-time interactive system of the present invention can effectively solve the problems of high safety risks and high training costs when traditional engineering equipment is trained by real machine operation, as well as the loss of realism due to the isolation between the real environment and the physical environment, the inability to interact with the actual construction scene, and the single virtual environment that is not suitable for process verification based on the actual construction environment. It provides a more advanced technical means for engineering equipment operation training and real-time confirmation of on-site procedures and methods, thereby achieving the purpose of training operators or verifying the rationality of procedures and methods, making timely adjustments before actual vehicle operation, and avoiding construction risks.

[0092] (2) The motion simulation method of the present invention establishes complex mechanical constraint relationships in the simulation of engineering equipment operation and introduces simple constraints in the physics engine to realistically represent the motion process of each component of the engineering equipment. This solves the problem of balancing the real-time performance and simulation effect of motion simulation. Operators can virtually operate the engineering equipment in the scene in a hybrid virtual environment with real immersion and interactivity to simulate the construction process. Thus, it provides a platform-independent virtual interactive simulation method that can meet the requirements of interactivity and real-time performance.

[0093] (3) The motion simulation method of the present invention uses mixed reality technology to simulate construction, performs precise spatial positioning mapping in complex construction sites, detects interference between virtual objects and the actual environment during the movement of virtual objects, and realizes real-time confirmation of construction process, procedures and methods, thereby completing construction verification and training tasks more efficiently. Attached Figure Description

[0094] Figure 1 This is a schematic diagram of the structure of the real-time interactive system of the present invention;

[0095] Figure 2 This is a flowchart of the dynamic modeling process for the motion simulation method of the present invention;

[0096] Figure 3 This is a flowchart illustrating the implementation of the motion simulation method of the present invention;

[0097] Figure 4 This is a schematic diagram of the control arm system of the real-time interactive system in an embodiment of the present invention, showing the control buttons related to the operation.

[0098] The meanings of the reference numerals in the figure are as follows:

[0099] 1-Mixed Reality Hardware; 2-Manipulation Control Handle; 21-Handle Driver; 3-Simulation Control Module; 31-Interactive Command Acquisition Module; 32-Dynamic Simulation Calculation Module; 33-Scene Management and Real-time Graphics Rendering Module; 4-Spatial Mapping and Collision Detection Module; 5-Information Storage Module; 6-Server. Detailed Implementation

[0100] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0101] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may include different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0102] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "upper", "lower", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only used to facilitate the description of the present invention and to simplify the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting the scope of protection of the present invention.

[0103] like Figure 1As shown, the real-time simulated interactive system of the present invention includes mixed reality hardware 1, a control handle 2, a simulation control module 3, a spatial mapping and collision detection module 4, an information storage module 5, and a server 6. In this embodiment, the engineering equipment is an aerial work platform (the method and principle are the same for other engineering equipment). The mixed reality hardware 1 uses Microsoft HoloLens 2, HoloLens 1, or Meta 2 glasses, and is connected to the simulation control module 3 and the spatial mapping and collision detection module 4 through communication. Its functions are to display the three-dimensional model of the aerial work platform, identify the construction site environment information, and obtain the operator's location information. The network communication connection follows the TCP / IP protocol. During operation, the server 6 automatically opens network listening, monitors the network in real time, and waits for the mixed reality hardware 1 client to match and connect with it. When the operator wearing HoloLens 2 glasses enters the actual work site, the operation information of the virtual aerial work platform is transmitted to the server 6 through the socket network. The control handle 2 can be a game controller. The controller driver 21 of the control handle 2 is connected to the USB port of the server 6 where the simulation control module 3 is located. The controller driver 21 parses the operation control flow input from the controller and transmits it to the interactive command acquisition module 31 of the simulation control module 3. The aerial work platform can be moved using the buttons on the control handle 2. The aerial work platform can be driven to its destination using the up, down, left, and right buttons on the control handle 2. The corresponding buttons on the control handle 2 for the boom system and turntable operations are as follows: Figure 4 As shown, press the selection button on the control handle 2. Figure 4The "PgDn" key initiates boom system operation. Keys 1-6 select the corresponding lower main boom, lower telescopic boom, upper main boom, two upper telescopic booms, and work platform, respectively, followed by the corresponding "=" and "-" operations for raising and lowering. The "K" and "D" keys rotate the turntable. The spatial mapping and collision detection module 4 communicates with the mixed reality hardware 1 and the simulation control module 3. Its function is to combine the construction scene information collected by the HoloLens 2 glasses in the mixed reality hardware 1 with the 3D model of the virtual aerial work platform in the simulation control module 3. This virtual aerial work platform 3D model is displayed on the screen of the HoloLens 2 glasses in the real construction site environment, using a virtual-real fusion display. Simultaneously, it performs precise collision detection and real-time motion interference simulation between the boom system and other moving parts of the virtual aerial work platform and objects in the real construction site environment during operation. This allows for training operators or verifying the rationality of work procedures and methods, enabling timely adjustments before actual operation and avoiding construction risks. The information storage module 4 is communicatively connected to the simulation control module 3 and the spatial mapping and collision detection module 4, respectively. Its function is to record and save the motion state information and collision detection information of the aerial work platform during virtual training and verification to reproduce the training and verification results. The simulation control module 3 is communicatively connected to the mixed reality hardware 1, the control handle 2, the spatial mapping and collision detection module 4, and the information storage module 5, respectively. Its function is to receive interactive command information from the control handle 2 and the mixed reality hardware 1, including handle operation, gestures, and voice commands, operator position information, and real construction site environment information. The dynamics simulation calculation module 32 parses the received virtual interactive information into command information corresponding to the motion of the virtual aerial work platform's 3D model. The virtual aerial work platform's 3D model moves in real-time according to specific commands. The image information of the virtual aerial work platform's 3D model is synchronized to the display screen in the mixed reality hardware 1 via the scene management and real-time graphics rendering module 33 and the spatial mapping and collision detection module 4. The server 6 is the hardware carrier for the simulation control module 3, the spatial mapping and collision detection module 4, and the information storage module 5.

[0104] In one specific embodiment of this example, the simulation control module 3 includes an interactive command acquisition module 31, a dynamics simulation calculation module 32, and a scene management and real-time graphics rendering module 33. The interactive command acquisition module 31 receives operation information from the operator wearing the mixed reality hardware 1. This operation information includes actions and voice commands from the virtual interactive operation commands of the mixed reality hardware 1, as well as operation digital signals from the controller driver 21. The dynamics simulation calculation module 32 is the core control module of the entire simulation system. Its function is to perform real-time dynamics simulation calculations based on the operator's operation information and output the simulation calculation results to the scene management and real-time graphics rendering module 33. The scene management and real-time graphics rendering module 33 updates the motion state of the aerial work platform in the virtual scene in real time based on the simulation calculation results and uses a graphics rendering engine to visualize the operation results and display them to the operator on the mixed reality hardware 1.

[0105] like Figure 2 As shown, the engineering equipment motion simulation method of the present invention addresses the fundamental issue of establishing dynamic models for each component of a complex mechanical system such as an aerial work platform in order to achieve virtual motion simulation control. From the initial geometric model to the establishment of the physical model, and through numerical solutions to the mathematical model, a dynamic model is finally obtained. The dynamic simulation calculation module 32 includes a multi-rigid-body dynamic system and a motion constraint system. The multi-rigid-body dynamic system includes a physical model and a mathematical model of the multi-rigid-body system. The physical model of the multi-rigid-body system is formed by physically modeling the geometric model of the aerial work platform, thereby creating a physical model of the aerial work platform that expresses the mechanical characteristics of the system. The motion constraint system includes hinge constraints and slider constraints. By performing multi-rigid-body dynamic modeling of the aerial work platform, corresponding motion constraints are established between the rigid body components of the aerial work platform using a physics engine.

[0106] To create a physical model of the aerial work platform that expresses the system's mechanical properties, it is necessary to perform physical modeling on the platform's geometric model. During physical modeling, the geometric model of the aerial work platform needs to be assembled according to kinematic constraints and initial position conditions. Physical modeling includes the following steps:

[0107] Treat each geometric model component as a rigid component and define its Cartesian generalized coordinate vector. Where l = 1, 2, ..., n, n represents a multi-rigid-body system with n points, and any adjacent rigid bodies connected by hinges are considered as one element, i.e., n represents the number of elements in the multi-rigid-body system; q lTaking a single rigid body as a reference, and describing the position of another rigid body relative to this rigid body using generalized coordinates, denoted by the Cartesian generalized coordinate vector, the system can be described using a Lagrange coordinate matrix q; let r l (x, y, z) are vectors representing the center of mass of each rigid body component in the absolute coordinate system, where x represents the x-axis, y represents the y-axis, and z represents the z-axis. γ = (ψ, θ, φ). l Let ψ represent the precession angle, θ represent the nutation angle, and φ represent the rotation angle of the rigid body relative to its own coordinate base. Then, the configuration vector matrix of the entire multi-rigid-body dynamics system is:

[0108] q = (q1 q2 … q) n ) T (15)

[0109] The kinematic constraint equations of the entire multi-rigid-body dynamic system are expressed as follows:

[0110]

[0111] In equation (16), m represents the number of constraint pairs, and Γ represents the number of constraint pairs. V As a whole, it represents the set of motion constraint equations for the entire multi-rigid-body dynamics system. This represents the individual motion constraint equations for the corresponding components, where w = 1, 2, 3, ..., m, and V is an arbitrarily chosen variable used to distinguish it from the driving constraint equations;

[0112] When the total degrees of freedom of the entire multi-rigid-body dynamics system is zero, it has definite motion. Therefore, the equation for the number of driving constraints required for the multi-rigid-body dynamics system is expressed as follows:

[0113] Γ H (q,t)=0 (17)

[0114] Where Γ(q,t)=0 is the equation for the number of driving constraints, and the driving constraints are time functions of the generalized coordinates; Γ H (q,t)=0 is the vector form of the driving constraint equation; H is an arbitrarily chosen variable used to distinguish it from the motion constraint equation; t represents the motion time of the multi-rigid-body system; for a system with n coordinates q=(q1 q2 … q n ) T In a multi-rigid-body system with *m* constraint pairs, only *nm* coordinates are independent, meaning *nm* is the number of degrees of freedom. From a kinematic perspective, for example, a planar linkage system, the system only has definite motion when the total number of degrees of freedom is zero. Therefore, the number of definite driving constraints required for the system is *nm*, and its driving constraint equation is expressed as Γ. H (q,t)=0.

[0115] The kinematic constraints of equation (16), the driving constraints of equation (17), and the Euler parameter constraints constitute all the constraints on the multi-rigid-body dynamics system:

[0116]

[0117] Equation (18) constitutes a set of n nonlinear position equations for the entire multi-rigid-body dynamics system in generalized coordinates; Γ(q,t)=0 is the driving constraint equation for the entire system, and the driving constraint is a time function of the generalized coordinates; This is the expression for the Euler parameter constraint equation.

[0118] Step S3: Analyze the geometric model of the engineering equipment. Based on the constraint equation types between the components of the geometric model, use the physics engine to establish corresponding constraint types for the components of the geometric model, and complete the physical modeling from the geometric model of the engineering equipment to the physical model of the multi-rigid-body system.

[0119] After analyzing the geometric model of the aerial work platform, the physics engine is used to establish corresponding constraint types for the components based on the constraint equations between the components. This successfully completes the physical modeling from the geometric model to the physical model, eliminating the need to solve the constraint equations and facilitating real-time control of the aerial work platform.

[0120] The mathematical model of a multi-rigid-body system is a mathematical modeling process performed after obtaining the physical model of the multi-rigid-body system. Using Cartesian or Lagrange coordinates, mathematical models of the system's velocity, acceleration, etc., are obtained, ultimately leading to the dynamic model of the multi-rigid-body system. This mathematical model includes both the motion mathematical model and the rotation mathematical model of the rigid body components of the aerial work platform.

[0121] The modeling process of the mathematical model of the motion of a rigid body component specifically includes the following steps:

[0122] Step A: Differentiate equation (18), then the velocity constraint equation for the multi-rigid-body system is:

[0123]

[0124] The entire equation (19) is the result of differentiating equation (18), where Γ q The Jacobian matrix of the system is, i.e. t The time derivative of the constraint equation is denoted as , i.e. For the system's generalized velocity;

[0125] Step B: Taking the second derivative of equation (18), the acceleration constraint equation for the multi-rigid-body system is:

[0126]

[0127] In the formula Γ q Γ is the Jacobian matrix; qt Let be the derivative of the Jacobian matrix with respect to time, i.e. Γ tt To find the second derivative of the constraint equation with respect to time, i.e. For the system's generalized acceleration;

[0128] Step C: From the position constraint equation of the multi-rigid-body system in equation (18), the velocity constraint equation of the multi-rigid-body system in equation (19), and the acceleration constraint equation of the multi-rigid-body system in equation (20), we obtain the first kind of Laplace multiplier form of the motion equation of the multi-rigid-body system:

[0129]

[0130] Where q, v, η∈R n These represent the system's generalized coordinates, velocity, and acceleration vectors, respectively. λ∈R m M(q,t)∈R is a column vector of Lagrange multipliers, representing the internal forces and moments between rigid body members connected by a kinematic pair, where t∈R is time, and M(q,t)∈R. n×n Let f(q,v,t)∈R represent the mass matrix of the system. n R is a generalized column vector of external forces, including external forces and external moments. n In this context, n represents an n-dimensional real vector space, numerically corresponding to the number of elements in a multi-rigid-body system; R m In this context, m represents an m-dimensional real vector space, which is numerically the same as the number of constraint pairs.

[0131] Modeling the mathematical model of rigid body component rotation requires using a body-dependent coordinate system fixed to the rigid body component to determine the motion of the multi-rigid-body system for each mechanism of the aerial work platform in three-dimensional space. The generalized coordinates of the component consist of two parts: the origin coordinates of the body-dependent coordinate system and the orientation parameters of the body-dependent coordinate system relative to the global coordinate system. Specifically, the rotation of the rigid body component at a fixed point is described by three attitude variables: direction cosine matrix, Euler angles, and Euler quaternions. In addition, the orientation parameters of the body-dependent coordinate system relative to the global coordinate system in the generalized coordinates of the rigid body component can be represented by the direction cosine matrix, Euler angles, or Euler parameters.

[0132] The Euler angle is: γ l =(ψ,θ,φ) l (twenty two)

[0133] The direction cosine matrix is:

[0134]

[0135] The Euler quaternion is:

[0136]

[0137] The direction cosine matrix can be represented using Euler quaternions as follows:

[0138]

[0139] The Euler parameter variables in the direction cosine matrix and Euler quaternion must satisfy the constraint equations:

[0140]

[0141] Where a1, a2, a3 are vectors of Euler quaternions, and a4 is a scalar of Euler quaternions. The constraint vector is formed by the Euler parameters of the rigid body component.

[0142] By performing physical and mathematical modeling of the multi-rigid-body system, a complex mechanical system was successfully simplified, resulting in a dynamic model of the multi-rigid-body system, which facilitates the simulation of the motion of the mechanical system of the aerial work platform.

[0143] The most critical issue in the virtual operation of aerial work platforms is motion control, including motion constraints between rigid body components of a multi-rigid-body system and motion control of the entire multi-rigid-body system.

[0144] The motion constraint system is constructed based on multibody system dynamics theory. It calculates the Jacobian matrix of two connected rigid body components according to the constraint type, and calculates the moment of inertia in conjunction with the shape of the rigid body components. This updates the position and velocity of the connected rigid body components, thereby simulating the force effects on the connected objects. The specific steps for establishing the motion constraints of the multibody system model of the aerial work platform include:

[0145] Step 1: Simplification of the Multi-Rigid-Body System Model of the Actual System: The working motion of the aerial work platform is simplified into the system motion composed of a finite number of rigid bodies. These rigid bodies are connected by some form of constraint, such as volumes or hinges. According to multi-rigid-body theory, the aerial work platform is simplified into several main parts, including the self-propelled chassis, turntable, and boom system. The self-propelled chassis provides traction and power to the aerial work platform. The turntable connects to the self-propelled chassis to achieve 360-degree rotation while ensuring the stability of the aerial work platform. The boom system connects to the turntable and controls the working state of the platform.

[0146] Step 2: Construct a physical model with constraint connections: Assemble the geometric model of the aerial work platform according to kinematic constraints and initial position conditions of the model. Establish corresponding constraint connections between rigid body components based on the corresponding force analysis. For the sake of calculation simplicity, the motion constraint system uses the corresponding constraints in the physics engine to connect the rigid body components after determining the constraint types between them. The physics engine is an integrated solution for real-time simulation of the physical environment. It is based on rigid body mechanics and calculates motion, rotation and collision responses by assigning real physical properties to rigid objects. Finally, it was confirmed that the aerial work platform needs to establish 21 hinge constraints and 9 sliding constraints, for a total of 30 constraints.

[0147] The types of constraints between rigid body components are specifically divided into:

[0148] Hinge constraint: This is to establish a rotation axis at a certain node that restricts the rotation of two rigid body components, so that they can only rotate around that axis. This axis is called the hinge axis. For example, the turntable and wheels of an aerial work platform that only rotate around one axis. Users can also set rotation angle limits for the hinge axis.

[0149] Slider constraint: refers to the constraint relationship between two rigid body components that can only move along a certain axis, such as the telescopic cylinder and telescopic boom of an aerial work platform.

[0150] The steps to implement a hinge constraint are as follows:

[0151] Step 1: Obtain the centroids of the first and second rigid body components. The formula for the centroid of a rigid body component is:

[0152]

[0153] Where, r c The x vector represents the position of the center of mass of a rigid body component. c y c z c Let x, y, z represent the coordinates of the center of mass of the rigid body, respectively; i, j, and k represent the vector basis, respectively.

[0154] Step II: Calculate the position of the anchor point relative to the first rigid body component, and obtain the hinge axis of the first rigid body component;

[0155] Step 3: Calculate the position of the anchor point relative to the second rigid body component, and obtain the hinge axis of the second rigid body component;

[0156] Step IV: Apply hinge constraints to the two rigid body members.

[0157] The steps to implement a slider constraint are as follows:

[0158] Step a: Use equation (27) to obtain the centroids of the first and second rigid body components;

[0159] Step b: Calculate the position of the second rigid body member relative to the first rigid body member, and obtain the sliding axis of the first rigid body member;

[0160] Step c: Calculate the position of the first rigid body member relative to the second rigid body member, and obtain the sliding axis of the second rigid body member;

[0161] Step d: Apply sliding bar constraints to the two rigid body members.

[0162] Since most constraints in the physics engine are based on a local coordinate system, the motion constraint system established through the physics engine requires adjustments to the spatial orientation or constraint axes of rigid body components in order to successfully use the functions in the physics engine. Specifically, the Rodriguez rotation formula is applied, and the rotation matrix around the unit vector μ(x,y,z) with a rotation angle of β is:

[0163]

[0164] After obtaining the spatial orientation of the rigid body component, the rigid body component is adjusted using formula (28).

[0165] In one specific implementation of this embodiment, the physics engine includes a gravity engine, which can perform gravity tests just like in reality. If improper operation causes instability of the center of gravity, a rollover will occur.

[0166] like Figure 3 As shown, the motion simulation method of the present invention includes the following steps:

[0167] Step 1: Configure the mixed reality hardware, controller, and server; that is, establish communication connections between the mixed reality hardware and the server, and between the controller and the server.

[0168] Step 2: Load the virtual model into the mixed reality hardware; that is, the server transmits the stored virtual scene data to the mixed reality hardware. After the mixed reality hardware recognizes the actual environment information, it merges the virtual scene with the real scene to present a virtual-real fusion scene.

[0169] Step 3: The user inputs commands by manipulating the control handle; if the user inputs incorrectly, an error message is displayed; if the user inputs correctly, dynamic simulation calculations are performed, and the simulation results are output to the scene management and real-time graphics rendering module. The scene management and real-time graphics rendering module updates the motion state of the 3D model of the engineering equipment in the virtual scene in real time based on the simulation results of the dynamic simulation calculation module, and outputs the operation results to the mixed reality hardware for display to the operator.

[0170] Step 4: Display the 3D model of the engineering equipment in the virtual scene updated in real time in Step 3 in the real construction site environment, and display it in a virtual-real fusion manner. At the same time, perform collision detection and real-time motion interference simulation between virtual objects and real scene objects. If interference with the real environment is detected, the interference information prompt is sent to the operator wearing the mixed reality hardware. If no interference with the real environment is detected, continue to the next verification and training task.

[0171] Step 5: Write the operation process into the information storage module, record and store the operation process information during the process, and complete the task.

[0172] The virtual control system for aerial work platforms, based on actual needs, comprises three main parts: directional control of the platform, operation of the boom system, and rotation of the turntable. This system simulates the operation of an aerial work platform. Operators use handles to control the boom system. During operation, the platform is first driven to the target location, then the boom system and turntable are operated to adjust the platform to the appropriate working position and level it. In actual operation, the operation of the aerial work platform depends on the location of the target work point. The operator can adjust the horizontal angle by rotating the turntable clockwise or counterclockwise, and adjust the height or vertical angle through the corresponding boom system operations. This system also incorporates a gravity engine, allowing the aerial work platform to undergo gravity testing similar to real-world conditions. Improper operation leading to instability can cause the platform to tip over.

[0173] This invention establishes complex mechanical constraints in the simulation of aerial work platform operation and introduces simple constraints from a physics engine to realistically represent the motion process of various components of the aerial work platform. This solves the problem of balancing real-time performance and simulation effect in motion simulation. Operators can virtually operate the aerial work platform in a mixed virtual environment with a realistic sense of immersion and interactivity, simulating the construction process. This provides a platform-independent virtual interactive simulation method that satisfies both interactivity and real-time performance. By utilizing mixed reality technology for construction simulation, precise spatial positioning mapping is performed in complex construction sites, and interference detection is conducted between the virtual aerial work platform and the actual environment during its movement. This enables real-time confirmation of construction processes, procedures, and methods, thereby more efficiently completing the construction verification and training tasks of the aerial work platform.

[0174] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0175] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A real-time interactive system for engineering equipment based on mixed reality, characterized in that: It includes mixed reality hardware, a control handle, a simulation control module, a space mapping and collision detection module, and an information storage module; The mixed reality hardware is communicatively connected to the simulation control module, the spatial mapping and collision detection module, and is used to collect and display the three-dimensional model of the engineering equipment, identify and collect construction site environmental information, and obtain the location information of the operators. The control handle is communicatively connected to the simulation control module and is used to transmit digital signals of operation to the simulation control module. The simulation control module is communicatively connected to the mixed reality hardware, the control handle, the space mapping and collision detection module, and the information storage module. It is used to receive virtual interactive information and parse the virtual interactive information into instruction information corresponding to the movement of the virtual three-dimensional model. The virtual three-dimensional model moves and simulates in real time according to the instructions. The spatial mapping and collision detection modules are respectively connected to the mixed reality hardware and the simulation control module. They are used to display the construction site environment information collected by the mixed reality hardware and the virtual 3D model in the simulation control module through the display screen of the mixed reality hardware in the real construction site environment in a virtual-real fusion manner. At the same time, collision detection and real-time motion interference simulation are performed between virtual objects and real scene objects. The information storage module is communicatively connected to the simulation control module, the space mapping module, and the collision detection module, respectively, and is used to save motion state information and collision detection information records during virtual training and virtual verification in order to reproduce the training and verification results.

2. The real-time interactive system for engineering equipment based on mixed reality according to claim 1, characterized in that: It also includes a server, which is the hardware carrier of the simulation control module, the space mapping and collision detection module, and the information storage module.

3. The real-time interactive system for engineering equipment based on mixed reality according to claim 1, characterized in that: The simulation control module includes an interactive command acquisition module, a dynamic simulation calculation module, and a scene management and real-time graphics rendering module. The interactive command acquisition module is used to receive the operation information of the operator wearing mixed reality hardware. The operation information is virtual interactive information, which includes digital signals from the control handle, the handle operation of the mixed reality hardware, the operator's gestures, the operator's voice interactive command information, the location information of the operator wearing the mixed reality hardware, and the real construction site environment information. The dynamics simulation calculation module is used to perform real-time dynamics simulation calculations based on the operator's operation information, and output the simulation calculation results to the scene management and real-time graphics rendering module. The scene management and real-time graphics rendering module is used to update the motion state of the three-dimensional model of engineering equipment in the virtual scene in real time according to the simulation calculation results of the dynamic simulation calculation module, and output the operation results to the mixed reality hardware for display to the operator.

4. The real-time interactive system for engineering equipment based on mixed reality according to claim 1, characterized in that: The control handle includes a handle driver, which transmits the digital operation signal to the simulation control module. The handle driver parses the operation control flow input by the control handle into an operation digital signal and transmits the operation digital signal to the simulation control module.

5. A method for simulating the motion of engineering equipment based on mixed reality, characterized in that: Based on the real-time interactive system described in claim 3, the dynamic simulation calculation module performs multi-rigid-body dynamics modeling on the engineering equipment and establishes corresponding kinematic constraints between the rigid body components of the engineering equipment. The kinematic constraints include a multi-rigid-body dynamics system and a motion constraint system.

6. The method for simulating the motion of engineering equipment based on mixed reality according to claim 5, characterized in that: The motion simulation method includes the following steps: Step 1: Configure the mixed reality hardware, control controller, and server; Step 2: Load the virtual model into the mixed reality hardware; Step 3: The user inputs commands by manipulating the control handle; if the user inputs incorrectly, an error message is displayed; if the user inputs correctly, dynamic simulation calculations are performed, and the simulation results are output to the scene management and real-time graphics rendering module. The scene management and real-time graphics rendering module updates the motion state of the 3D model of the engineering equipment in the virtual scene in real time based on the simulation results of the dynamic simulation calculation module, and outputs the operation results to the mixed reality hardware for display to the operator. Step 4: Display the 3D model of the engineering equipment in the virtual scene updated in real time in Step 3 in the real construction site environment, and display it in a virtual-real fusion manner. At the same time, collision detection and real-time motion interference simulation are performed between virtual objects and real scene objects. If interference with the real environment is detected, interference information is sent to the operator wearing the mixed reality hardware. If no interference with the real environment is detected, the next verification and training task is continued. Step 5: Write the operation process into the information storage module, record and store the operation process information during the process, and complete the task.

7. The method for simulating the motion of engineering equipment based on mixed reality according to claim 5, characterized in that: The multi-rigid-body dynamics system includes a physical model and a mathematical model of the multi-rigid-body system. The physical model of the multi-rigid-body system is formed by physically modeling the geometric model of the engineering equipment, thereby creating a physical model that expresses the mechanical properties of the system. The physical modeling includes the following steps: Step S1: Assemble the geometric model of the engineering equipment according to the kinematic constraints and the initial position conditions of the virtual 3D model. By analyzing the motion dependency relationship between each component of the virtual 3D model, set the parent-child nesting relationship between the components of the virtual 3D model, and finally complete the construction of the entire engineering equipment model tree. Step S2: Treat each geometric model component as a rigid component and set its Cartesian generalized coordinate vector. ,in , n represents a multi-rigid-body system with n points, and any adjacent rigid bodies connected by hinges are regarded as a unit, that is, n represents the number of units in the multi-rigid-body system; Using a single rigid body as a reference, describe the position of another rigid body relative to this rigid body using generalized coordinates; let... Let x be the vector representing the center of mass of each rigid body component in the absolute coordinate system, where x represents the x-axis, y represents the y-axis, and z represents the z-axis. These are the three Euler angles of the rigid body relative to its own coordinate base. Indicates the precession angle. Indicates the nutation angle. Let represent the rotation angle, then the configuration vector matrix of the entire multi-rigid-body dynamics system is: (1)； The kinematic constraint equations of the entire multi-rigid-body dynamic system are expressed as follows: (2)； In equation (2), —The number of constraint pairs, As a whole, it represents the set of motion constraint equations for the entire multi-rigid-body dynamics system. Represent the individual motion constraint equations for the corresponding components, where, V is an arbitrarily chosen variable used to distinguish it from the driving constraint equation; When the total degrees of freedom of the entire multi-rigid-body dynamics system is zero, it has definite motion. Therefore, the equation for the number of driving constraints required for the multi-rigid-body dynamics system is expressed as follows: (3)； in, The number of driving constraint equations is defined as follows: the driving constraints are time functions of the generalized coordinates. It is the vector form of the driving constraint equation; H is an arbitrarily chosen variable used to distinguish it from the motion constraint equation; t represents the motion time of the multi-rigid-body system; The kinematic constraints of equation (2), the driving constraints of equation (3), and the Euler parameter constraints constitute all the constraints on the multi-rigid-body dynamics system: (4)； Equation (4) constitutes a set of n nonlinear position equations for the entire multi-rigid-body dynamics system in generalized coordinates; The driving constraint equations for the entire system are time functions of the generalized coordinates. , is the expression for the Euler parameter constraint equation; Step S3: Analyze the geometric model of the engineering equipment. Based on the constraint equation types between the components of the geometric model, use the physics engine to establish corresponding constraint types for the components of the geometric model, and complete the physical modeling from the geometric model of the engineering equipment to the physical model of the multi-rigid-body system.

8. The method for simulating the motion of engineering equipment based on mixed reality according to claim 7, characterized in that: The mathematical model of the multi-rigid-body system is a mathematical modeling performed after obtaining the physical model of the multi-rigid-body system, which yields the mathematical models of the velocity and acceleration of the multi-rigid-body system, and then the dynamic model of the multi-rigid-body system.

9. The method for simulating the motion of engineering equipment based on mixed reality according to claim 8, characterized in that: The mathematical model of the multi-rigid-body system includes a mathematical model of the motion of rigid body components and a mathematical model of the rotation of rigid body components.

10. The method for simulating the motion of engineering equipment based on mixed reality according to claim 9, characterized in that: The modeling process of the rigid body component's motion mathematical model includes the following steps: Step A: Differentiate equation (4), then the velocity constraint equation for the multi-rigid-body system is: (5)； The entire equation (5) is the result of differentiating equation (4), where The Jacobian matrix of the system is, i.e. ; The time derivative of the constraint equation is denoted as , i.e. , For the system's generalized velocity; Step B: Taking the second derivative of equation (4), the acceleration constraint equation for the multi-rigid-body system is: (6)； In the formula It is a Jacobian matrix; Let be the derivative of the Jacobian matrix with respect to time, i.e. ; To find the second derivative of the constraint equation with respect to time, i.e. ; For the system's generalized acceleration; Step C: From the position constraint equation of the multi-rigid-body system in equation (4), the velocity constraint equation of the multi-rigid-body system in equation (5), and the acceleration constraint equation of the multi-rigid-body system in equation (6), we obtain the first kind of Laplace multiplier form of the motion equation of the multi-rigid-body system: (7)； in, These represent the system's generalized coordinates, velocity, and acceleration vectors, respectively. , , These are Lagrange multiplier column vectors, representing the internal forces and moments between rigid body components connected by a kinematic pair. It is time. Represents the system's quality matrix. This is a generalized column vector of external forces, including external forces and external torques. In this context, n is an n-dimensional real vector space, which numerically corresponds to the number of elements in a multi-rigid-body system. In this context, m represents an m-dimensional real vector space, which is numerically the same as the number of constraint pairs.

11. A method for simulating the motion of engineering equipment based on mixed reality according to claim 9 or 10, characterized in that: The modeling of the rigid body component rotation mathematical model requires the use of a body coordinate system fixed on the rigid body component to determine the motion of the multi-rigid body system for each mechanism of the engineering equipment in three-dimensional space. Specifically, the fixed-point rotation of the rigid body component is described by three attitude variables: direction cosine matrix, Euler angle, and Euler quaternion. The Euler angles are: (8); The direction cosine matrix is: （9）； The Euler quaternion is: (10)； The direction cosine matrix can be represented using Euler quaternions as follows: （11）； The Euler parameter variables in the direction cosine matrix and Euler quaternion must satisfy the constraint equations: (12)； Where a1, a2, a3 are vectors of Euler quaternions, and a4 is a scalar of Euler quaternions. The constraint vector is formed by the Euler parameters of the rigid body component.

12. The method for simulating the motion of engineering equipment based on mixed reality according to claim 5, characterized in that: The motion constraint system calculates the Jacobian matrix of two connected rigid body components based on the constraint type, and calculates the moment of inertia in conjunction with the shape of the rigid body components to update the position and velocity of the connected rigid body components, thereby simulating the effect of forces on the connected objects; the constraint types between the rigid body components include hinge constraints and sliding rod constraints.

13. The method for simulating the motion of engineering equipment based on mixed reality according to claim 12, characterized in that: The steps for implementing the hinge constraint are as follows: Step 1: Obtain the centroids of the first and second rigid body components. The formula for the centroid of a rigid body component is: (13)； in, This represents the position vector of the center of mass of a rigid body component. , , Let x, y, z represent the coordinates of the rigid body's center of mass, respectively; i, j, and k represent the vector basis, respectively. Step II: Calculate the position of the anchor point relative to the first rigid body component, and obtain the hinge axis of the first rigid body component; Step 3: Calculate the position of the anchor point relative to the second rigid body component, and obtain the hinge axis of the second rigid body component; Step IV: Apply hinge constraints to the two rigid body members.

14. The method for simulating the motion of engineering equipment based on mixed reality according to claim 13, characterized in that: The steps for implementing the sliding bar constraint are as follows: Step a: Use equation (13) to obtain the centroids of the first and second rigid body components; Step b: Calculate the position of the second rigid body member relative to the first rigid body member, and obtain the sliding axis of the first rigid body member; Step c: Calculate the position of the first rigid body member relative to the second rigid body member, and obtain the sliding axis of the second rigid body member; Step d: Apply sliding bar constraints to the two rigid body members.

15. The method for simulating the motion of engineering equipment based on mixed reality according to claim 7, characterized in that: The motion constraint system established by the physics engine requires adjustments to the spatial attitude or constraint axes of rigid body components in order to successfully use the functions in the physics engine. The adjustment method is as follows: applying the Rodriguez rotation formula around a unit vector... Rotate and the rotation angle is The rotation matrix is: (14）； After obtaining the spatial orientation of the rigid body component, the rigid body component is adjusted using formula (14).

Images