A method and system for simulating adsorption effect of virtual model assembly based on gesture interaction

Abstract

The present invention discloses a method and system for simulating adsorption effect of virtual model assembly based on gesture interaction, and the method includes: disassembling the virtual model to be assembled and building a mixed reality environment; realizing the adsorption effect function; realizing the delayed triggering adsorption effect function; system packaging, deployment to Hololens2 device and feedback and improvement; the system includes modules: human-computer interaction module; real-time calculation and threshold adjustment module; adsorption effect and feedback module; coroutine waiting interaction control module. The present invention provides a method and system for realizing adsorption effect of virtual model in a mixed reality environment, and provides a mechanism for delayed triggering adsorption effect, avoiding operation confusion caused by frequent triggering of adsorption effect, aiming to enhance the user's assembly operability, accuracy and feedback of virtual model, and improve the application potential in education, industrial training, games and other fields.

Description

Technical Field

[0001] The present invention relates to the fields of civil engineering and intelligent construction, and in particular to a virtual model assembly simulation method and system based on gesture interaction. Background Art

[0002] Virtual model assembly simulation technology refers to the technical means of simulating and analyzing the digital modeling and assembly process of products by using computer-aided design software, simulation tools, etc. This technology has important application value in engineering design, manufacturing and maintenance. In the process of virtual model assembly simulation, human-computer interaction technology plays a key role. In order to make the virtual model assembly simulation process more realistic, intuitive and efficient, mixed reality technology can be introduced to emphasize the connection with the real world. Mixed Reality (MR) technology is a new technology that combines Virtual Reality (VR) and Augmented Reality (AR). It creates a new hybrid environment by integrating the virtual world with the real world. Users can operate virtual objects and interact with the virtual environment in a variety of different ways of interaction. Among them, mixed reality gesture interaction technology provides users with a natural and intuitive way of interaction, enriching the experience of virtual model assembly simulation. Through the camera, sensor, etc. of the mixed reality device, the system can accurately perceive the user's position, posture and the structure of the surrounding environment, so as to realize the perception of the user's space and the depth of the object, and realize a more accurate virtual model assembly simulation experience.

[0003] In the process of assembly simulation through gesture interaction in a mixed reality environment, 1) there is a problem of insufficient assembly position accuracy, which makes it difficult to ensure that model components can be accurately assembled to the target position, and there is a lack of feedback mechanism with strong user experience; 2) there is a situation where users frequently perform gesture interactions on virtual models that have been assembled to the target position, resulting in operational confusion.

[0004] Therefore, the present invention proposes a method and system for simulating adsorption effects of virtual model assembly based on gesture interaction, aiming to enhance the user's assembly operability, accuracy and feedback of the virtual model. Summary of the invention

[0005] In view of the above problems, the present invention aims to provide a method and system for simulating adsorption effects of virtual model assembly based on gesture interaction, the method comprising the following steps:

[0006] Step 1: Building a mixed reality environment: disassemble the virtual model to be assembled and define the assembly order of each component logically; use the mixed reality toolkit MRTK to create a mixed reality scene in Unity for simulating the assembly of virtual models in a real physical environment; import the gesture recognition module Gesture Input System provided in MRTK.

[0007] Step 2, implementation of the adsorption effect function: place a semi-transparent assembly model at the target position in the mixed reality scene in Unity as an assembly prompt; use C# statements in Visual Studio to define the threshold for the assembly model to trigger the adsorption effect, and use a user interface to allow users to manually set or adjust the threshold parameters of the adsorption effect according to their own operating habits and assembly task requirements; calculate the real-time angle and distance between the assembly model and the assembly prompt position in real time to determine whether the threshold is reached; after the adsorption effect is completed, use the AudioSource component to provide a voice prompt for successful assembly.

[0008] Step 3, implementation of delayed triggering of adsorption effect function: using coroutines to delay the restoration of operability, and providing a time adjustment panel for delayed operability.

[0009] Step 4, system packaging, deployment and experience: perform packaging settings in Unity, and deploy the application to the Hololens 2 device through the Hololens 2 Developer Center using a data cable or WiFi; wear Hololens 2 and use gestures to assemble virtual models and experience the simulated adsorption effect.

[0010] Further, in step 1, the following steps may be performed in sequence:

[0011] Step 1.1: Disassemble the entire model to be assembled into its components, and set the assembly order and initial state of each component in advance, including the hierarchical relationship, initial position, size, and visibility;

[0012] Step 1.2: Create a new scene in Unity and use the mixed reality toolkit MRTK to create a mixed reality scene for human-computer interaction;

[0013] Step 1.3: Import the gesture recognition module Gesture Input System provided in MRTK.

[0014] Further, in step 2, the following steps may be performed in sequence:

[0015] Step 2.1: Place a semi-transparent assembly model at the target location in the mixed reality scene in Unity as an assembly hint.

[0016] Step 2.2: Use C# statements in Visual Studio to define the thresholds for triggering the adsorption effect of the model to be assembled, set the distance threshold and angle threshold for triggering movement to distanceThreshold and angleThreshold respectively, and associate them with the slider event. Update them as the subsequent slider value is updated, so that users can manually adjust the threshold parameters of the adsorption effect through the slider according to their own operating habits and assembly task requirements.

[0017] Step 2.3: Calculate the real-time angle and distance between the assembly model and the assembly prompt position in real time to determine whether the threshold is reached, where the orientation vector of the assembly component is , indicating that the orientation vector of the position component is , the distance between the two ; The position vector of the component to be assembled is , indicating that the position vector of the position component is , the distance between the two ; Specifically, use C# statements in Visual Studio to define the threshold for triggering the adsorption effect of the component to be assembled, including distance and angle. The position vectors of the component to be assembled and the component at the prompt position are installation.position and prompt.position respectively, and the Vector3.Distance method is used to calculate the distance between the two vectors; the orientation vectors of the component to be assembled and the component at the prompt position are installation.forward and prompt.forward respectively, and the Vector3.Angle method is used to calculate the angle between the two vectors;

[0018] Step 2.4: Declare a public AudioSource using C# statements in Visual Studio.

[0019] The variable is used to reference the AudioSource component in the scene, and a voice prompt is given to indicate successful assembly after the adsorption effect is completed.

[0020] Further, in step 3, the following steps may be performed in sequence:

[0021] Step 3.1: Use the StartCoroutine and WaitForSeconds methods in Visual Studio to restore gesture interaction control of the component after the adsorption effect is triggered and the coroutine waits for a certain period of time.

[0022] Step 3.2: Allow the user to adjust the delay length according to their own operating habits and assembly task requirements in the form of a user interface.

[0023] Further, in step 4, the following steps may be performed in sequence:

[0024] Step 4.1: Set up the package in Unity, adjust the output platform to UWP, and change the package name. Use a data cable or WiFi to deploy the APPX application to the Hololens 2 device through the Hololens 2 Developer Center.

[0025] Step 4.2: Wear Hololens 2 and use gestures to assemble the virtual model to simulate the adsorption effect, and make adjustments based on user experience feedback.

[0026] The system includes the following modules:

[0027] Module 1, Human-computer interaction module: The device’s camera detects gesture features to associate with the virtual model through natural interaction;

[0028] Module 2, real-time calculation and threshold adjustment module: real-time dynamic calculation of the distance and angle between the moving assembly component model and the prompt position component model, to trigger the assembly simulation adsorption effect module, and allow the threshold for triggering the adsorption effect to be adjusted through the visual interface;

[0029] Module 3, adsorption effect and feedback module: changes the assembly component model coordinates to the prompt position component model coordinates to achieve the adsorption effect and provide a successful assembly feedback mechanism;

[0030] Module 4, coroutine waiting interaction control module: delays the restoration of manual operation capabilities after adsorption for better user experience and interaction logic.

[0031] The present invention provides a method and system for realizing the adsorption effect of a virtual model in a mixed reality environment, thereby enhancing the user's assembly operability, accuracy and feedback of the virtual model; providing a user adaptive assembly operation mechanism so as to dynamically adjust the assembly adsorption operation according to different assembly environments and user operation habits; and providing a mechanism for delaying the triggering of the adsorption effect so as to avoid the operation confusion caused by the frequent triggering of the adsorption effect.

[0032] The specific advantages and positive effects are:

[0033] (1) The virtual model adsorption effect enables users to operate virtual models more intuitively, enhancing their sense of participation and interactive experience. Users can interact with virtual models through gestures or head movements, feeling a more realistic operation feeling, which increases their satisfaction with and desire to use the mixed reality environment.

[0034] (2) Through the adsorption effect, users can more accurately control the position, rotation and connection method of the virtual model, thereby improving the precision and accuracy of the operation. This is particularly useful for scenarios that require fine assembly or simulated operations, such as engineering design, medical surgery simulation and other fields. The virtual model adsorption effect can reduce user operation errors and mistakes. When the virtual model is adsorbed to other models or scenes, it can avoid the occurrence of model misalignment or miscutting, improving the reliability and stability of the operation. The adsorption effect makes the user's operation process more natural and intuitive. Users can achieve model adsorption and connection through gestures or head movements without complicated instructions or operation steps, which improves the convenience and fluency of operation.

[0035] (3) The virtual model adsorption effect method based on gesture interaction can help learners master operating skills and understand assembly processes more quickly. By personally experiencing the adsorption and connection process of virtual models, learners can deeply understand the key points of relevant knowledge and skills. For design, manufacturing and maintenance in fields such as industrial engineering / construction engineering, the virtual model adsorption effect can improve work efficiency and production efficiency. Operators can complete assembly or maintenance tasks faster, reducing operation time and cost, and improving work efficiency and quality.

[0036] In summary, the present invention realizes the virtual model adsorption effect in a mixed reality environment by comprehensively using virtual model assembly simulation technology and mixed reality and other technologies. This method has many advantages and positive effects, which can enhance user experience, improve operation accuracy, reduce errors, and improve operation procedures. It has important significance and value for the design, manufacturing and maintenance of industrial engineering / construction engineering and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and / or other aspects and advantages of the present invention will become clearer and easier to understand through the detailed description made in conjunction with the following drawings, which are only exemplary and do not limit the present invention, wherein:

[0038] Figure 1 This is a flow chart of a method for simulating adsorption effects of virtual model assembly based on gesture interaction.

[0039] Figure 2 This is a disassembly example of a virtual model to be assembled.

[0040] Figure 3 This is an example diagram of the gesture interaction script for the model to be assembled.

[0041] Figure 4 It is a slider

[0042] Figure 5 This is a C# statement example diagram that defines the threshold for triggering the adsorption effect of the component to be assembled.

[0043] Figure 6 This is an example diagram of C# statements using coroutines.

[0044] Figure 7 This is a sample diagram of packaging for the UWP platform in Unity.

[0045] Figure 8 This is an example of wearing Hololens2 for gesture interaction to achieve assembly adsorption effect.

[0046] Fig. 9 This is a module diagram of a virtual model assembly simulation adsorption effect system based on gesture interaction. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] like Figure 1 As shown, the present invention provides a method and system for simulating adsorption effect of virtual model assembly based on gesture interaction, and the method comprises the following steps:

[0049] Step 1: Building a mixed reality environment: disassemble the virtual model to be assembled and define the assembly order of each component logically; use the mixed reality toolkit MRTK to create a mixed reality scene in Unity for simulating the assembly of virtual models in a real physical environment; import the gesture recognition module Gesture Input System provided in MRTK.

[0050] Step 2, implementation of the adsorption effect function: place a semi-transparent assembly model at the target position in the mixed reality scene in Unity as an assembly prompt; use C# statements in Visual Studio to define the threshold for the assembly model to trigger the adsorption effect, and use a user interface to allow users to manually set or adjust the threshold parameters of the adsorption effect according to their own operating habits and assembly task requirements; calculate the real-time angle and distance between the assembly model and the assembly prompt position in real time to determine whether the threshold is reached; after the adsorption effect is completed, use the AudioSource component to provide a voice prompt for successful assembly.

[0051] Step 3, implementation of delayed triggering of adsorption effect function: using coroutines to delay the restoration of operability, and providing a time adjustment panel for delayed operability.

[0052] Step 4, system packaging, deployment and experience: perform packaging settings in Unity, and deploy the application to the Hololens 2 device through the Hololens 2 Developer Center using a data cable or WiFi; wear Hololens 2 and use gestures to assemble virtual models and experience the simulated adsorption effect.

[0053] Further, in step 1, the following steps may be performed in sequence:

[0054] Step 1.1: Disassemble the entire model to be assembled into its components, and set the assembly order and initial state of each component in advance, including the hierarchical relationship, initial position, size, and visibility state, such as Figure 2 As shown;

[0055] Step 1.2: Create a new scene in Unity and use the mixed reality toolkit MRTK to create a mixed reality scene for human-computer interaction;

[0056] Step 1.3: Import the gesture recognition module Gesture Input System provided in MRTK and add the interactive script, such as Figure 3 shown.

[0057] Further, in step 2, the following steps may be performed in sequence:

[0058] Step 2.1: Place a semi-transparent assembly model at the target location in the mixed reality scene in Unity as an assembly hint.

[0059] Step 2.2: Use C# statements in Visual Studio to define the thresholds for triggering the adsorption effect of the model to be assembled, set the distance threshold and angle threshold for triggering movement to distanceThreshold and angleThreshold respectively, and associate them with the slider event. Update them as the slider value is updated later, so that users can manually adjust the threshold parameters of the adsorption effect through the slider according to their own operating habits and assembly task requirements, such as Figure 4 shown.

[0060] Step 2.3: Calculate the real-time angle and distance between the assembly model and the assembly prompt position in real time to determine whether the threshold is reached, where the orientation vector of the assembly component is , indicating that the orientation vector of the position component is , the distance between the two ; The position vector of the component to be assembled is , indicating that the position vector of the position component is , the distance between the two ; Specifically, in Visual Studio, use C# statements to define the threshold for triggering the adsorption effect of the component to be assembled, including distance and angle. The position vectors of the component to be assembled and the component at the prompt position are installation.position and prompt.position respectively, and the Vector3.Distance method is used to calculate the distance between the two vectors; the orientation vectors of the component to be assembled and the component at the prompt position are installation.forward and prompt.forward respectively, and the Vector3.Angle method is used to calculate the angle between the two vectors, such as Figure 5 As shown;

[0061] Step 2.4: In Visual Studio, use C# statements to declare a public AudioSource variable to reference the AudioSource component in the scene, and give a voice prompt indicating successful assembly after the adsorption effect is completed.

[0062] Further, in step 3, the following steps may be performed in sequence:

[0063] Step 3.1: Use the StartCoroutine and WaitForSeconds methods in Visual Studio to restore the gesture interaction control of the component after the adsorption effect is triggered, and then wait for a certain period of time through the coroutine, such as Figure 6 shown.

[0064] Step 3.2: Allow the user to adjust the delay length according to their own operating habits and assembly task requirements in the form of a user interface, such as Figure 4 shown.

[0065] Further, in step 4, the following steps may be performed in sequence:

[0066] Step 4.1: Set up the package in Unity, adjust the output platform to UWP, and change the package name. Use a data cable or WiFi to deploy the APPX application to the Hololens 2 device through the Hololens 2 Developer Center. Figure 7 As shown;

[0067] Step 4.2: Wearing Hololens2, perform virtual model assembly simulation adsorption effect experience through gesture operation, and adjust and improve the position of the user interface, the size and length of the adsorption threshold and the delay threshold according to the user experience feedback to enhance the user experience, such as Figure 8 shown.

[0068] like Fig. 9 As shown, the system includes the following modules:

[0069] Module 1, Human-computer interaction module: The device’s camera detects gesture features to associate with the virtual model through natural interaction;

[0070] Module 2, real-time calculation and threshold adjustment module: real-time dynamic calculation of the distance and angle between the moving assembly component model and the prompt position component model, to trigger the assembly simulation adsorption effect module, and allow the threshold for triggering the adsorption effect to be adjusted through the visual interface;

[0071] Module 3, adsorption effect and feedback module: changes the assembly component model coordinates to the prompt position component model coordinates to achieve the adsorption effect and provide a successful assembly feedback mechanism;

[0072] Module 4, coroutine waiting interaction control module: delays the restoration of manual operation capabilities after adsorption for better user experience and interaction logic.

[0073] The virtual model assembly simulation adsorption effect method and system based on gesture interaction provided by the present invention is based on the following issues: 1) In the process of assembly simulation through gesture interaction in a mixed reality environment, there is a problem of insufficient assembly position accuracy, which makes it difficult to ensure that the model components can be accurately assembled to the target position, and there is a lack of feedback mechanism with strong user experience. 2) In the process of virtual model assembly, it is difficult to dynamically adjust the assembly operation according to different assembly environments and user operation habits. 3) In the process of assembly simulation, there is a situation where users frequently perform gesture interactions on the virtual model that has been assembled to the target position, resulting in confusion in operation.

[0074] The solutions are as follows: 1) A method and system for realizing the adsorption effect of virtual models in a mixed reality environment are provided, which enhances the user's assembly operability, accuracy and feedback of virtual models. 2) A user adaptive assembly operation mechanism is provided to dynamically adjust the assembly adsorption operation according to different assembly environments and user operation habits. 3) A mechanism for delayed triggering of the adsorption effect is provided to avoid operational confusion caused by frequent triggering of the adsorption effect.

[0075] In summary, the present invention realizes the virtual model adsorption effect in a mixed reality environment by comprehensively using virtual model assembly simulation technology and mixed reality and other technologies. This method has many advantages and positive effects, which can enhance user experience, improve operation accuracy, reduce errors, and improve operation procedures. It has important significance and value for the design, manufacturing and maintenance of industrial engineering / construction engineering and other fields.

[0076] The present invention is not limited to the above-mentioned embodiments. Anyone can derive other various forms of products under the inspiration of the present invention. However, no matter what changes are made in the shape or structure, any technical solution that is the same or similar to that of the present application falls within the protection scope of the present invention.

Claims

1. A method for simulating adsorption effect of virtual model assembly based on gesture interaction, characterized in that: The following steps are involved: Step 1: Building a mixed reality environment: disassemble the virtual model to be assembled and define the assembly order of each component according to logic; use the mixed reality toolkit MRTK to create a mixed reality scene in Unity for simulating the assembly of virtual models in a real physical environment; import the gesture recognition module GestureInputSystem provided in MRTK; Step 2, realization of the adsorption effect function: placing a semi-transparent assembly model at the target position in the mixed reality scene in Unity as an assembly prompt; In Visual Studio, use C# statements to define the threshold for triggering the adsorption effect of the model to be assembled, and use the user interface to allow users to manually set or adjust the threshold parameters of the adsorption effect according to their own operating habits and assembly task requirements; calculate the real-time angle and distance between the model to be assembled and the assembly prompt position in real time to determine whether the threshold is reached; after the adsorption effect is completed, use the AudioSource component to provide a voice prompt for successful assembly; Step 3, implementation of delayed triggering of adsorption effect function: using coroutine to delay recovery of operability, and providing a time adjustment panel for delayed operability; Step 4, system packaging, deployment and experience: perform packaging settings in Unity, and deploy the application to the Hololens 2 device through the Hololens 2 Developer Center using a data cable or WiFi; wear Hololens 2 and use gestures to assemble virtual models and experience the simulated adsorption effect; In step 2, perform the following steps in sequence: Step 2.1: Place a semi-transparent assembly model at the target position in the mixed reality scene in Unity as an assembly prompt; Step 2.2: Use C# statements in Visual Studio to define the threshold for triggering the adsorption effect of the model to be assembled, set the distance threshold and angle threshold for triggering movement as distanceThreshold and angleThreshold respectively, and associate them with the slider event. Update them as the slider value is updated later, so that users can manually adjust the threshold parameters of the adsorption effect through the slider according to their own operating habits and assembly task requirements; Step 2.3: Calculate the real-time angle and distance between the assembly model and the assembly prompt position to determine whether the threshold is reached. The orientation vector of the assembly component is F1, the orientation vector of the prompt position component is F2, and the distance between the two is The position vector of the component to be assembled is P1, the position vector of the component in the prompt position is P2, and the distance between the two is distance = ||P1-P2||; Use Vector3.Angle and Vector3.Distance methods in Unity to calculate the angle and distance between the two vectors; Step 2.4: In VisualStudio, use C# statements to declare a public AudioSource variable to reference the AudioSource component in the scene, and give a voice prompt for successful assembly after the adsorption effect is completed; In step 3, perform the following steps in sequence: Step 3.1: Use the StartCoroutine and WaitForSeconds methods in Visual Studio to restore gesture interaction control of the component after the adsorption effect is triggered and the coroutine waits for a certain period of time; Step 3.2: Allow the user to adjust the delay length according to their own operating habits and assembly task requirements in the form of a user interface.

2. The method for simulating adsorption effect of virtual model assembly based on gesture interaction according to claim 1, characterized in that: In step 1, perform the following steps in sequence: Step 1.1: Disassemble the entire model to be assembled into its components, and set the assembly order and initial state of each component in advance, including the hierarchical relationship, initial position, size, and visibility; Step 1.2: Create a new scene in Unity and use the mixed reality toolkit MRTK to create a mixed reality scene for human-computer interaction; Step 1.3: Import the gesture recognition module Gesture Input System provided in MRTK.

3. The method for simulating adsorption effect of virtual model assembly based on gesture interaction according to claim 1, characterized in that: In step 4, perform the following steps in sequence: Step 4.1: Set up the package in Unity, adjust the output platform to UWP, and change the package name. Use a data cable or WiFi to deploy the APPX application to the Hololens 2 device through the Hololens 2 Developer Center. Step 4.2: Wear Hololens 2 and use gestures to assemble the virtual model to simulate the adsorption effect, and make adjustments based on user experience feedback.

4. A virtual model assembly simulation adsorption effect system based on gesture interaction using the method of claim 1, characterized in that: Includes the following modules: Module 1, Human-computer interaction module: The device’s camera detects gesture features to associate with the virtual model through natural interaction; Module 2, real-time calculation and threshold adjustment module: real-time dynamic calculation of the distance and angle between the moving assembly component model and the prompt position component model, to trigger the assembly simulation adsorption effect module, and allow the threshold for triggering the adsorption effect to be adjusted through the visual interface; Module 3, adsorption effect and feedback module: changes the assembly component model coordinates to the prompt position component model coordinates to achieve the adsorption effect and provide a successful assembly feedback mechanism; Module 4, coroutine waiting interaction control module: delays the restoration of manual operation capabilities after adsorption for user experience and interaction logic.

Images