Multi-user collaborative maintenance sensing system
By designing a multi-user collaborative maintenance perception system on the virtual reality platform, the problem of fewer collaborative research on multiple people on the virtual reality platform is solved, and the synchronization and coordination of collaborative disassembly and assembly of aircraft engines is achieved, improving the efficiency of training and evaluation and effects.
Patent Information
- Application Number
- CN202411257485.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2025-02-11
AI Technical Summary
The existing technology has few research on multi-person collaborative research on virtual reality platforms, especially in aero engine disassembly and assembly training. There is a lack of network synchronization methods and multi-person collaborative systems suitable for VR, resulting in a lack of configurability and coordination between system. Spend.
A multi-user collaborative maintenance perception system was designed, and the server and client were built based on the Mirror network framework to realize the deployment and communication between the server and client. The scene resources and task sequences were managed through JSON configuration files, and the multi-person collaborative disassembly and assembly was supported. synchronization and coordination.
The synchronization and coordination of multiple people's collaborative disassembly and assembly is achieved, ensuring the correctness and consistency of the disassembly and assembly process, providing new ideas and methods for training of talent training for aircraft engine maintenance skills, and training and evaluation of equipment disassembly and assembly in other fields. A technical solution that can be learned is provided.
Smart Images
Figure CN119417659A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of virtual reality, and in particular relates to a multi-user collaborative maintenance perception system. Background Art
[0002] The disassembly and assembly of aircraft engines requires the collaboration of multiple people. The disassembly and assembly training of aircraft engines is an important training subject. Through virtual modeling, we know that disassembly and assembly training is currently an important training method.
[0003] Virtual reality technology (English name: Virtual Reality, abbreviated as VR), virtual reality technology encompasses computer, electronic information, and simulation technology. Its basic implementation method is based on computer technology, using and integrating the latest developments of various high technologies such as 3D graphics technology, multimedia technology, simulation technology, display technology, servo technology, etc., with the help of computers and other equipment to create a realistic 3D visual, tactile, olfactory and other sensory experiences in the virtual world, so that people in the virtual world have an immersive feeling At present, many systems developed for virtual multi-person collaboration are mainly based on desktop platforms, and there are fewer studies on VR platforms. However, VR platforms have higher requirements for real-time and accuracy. Therefore, it is necessary to study network synchronization methods suitable for VR multi-person collaboration. Secondly, the virtual training of VR platforms is immersive, but only a few have realized the function of editing training tasks. Most of them are customized task development, or use simple hardware to achieve unconstrained picking. When the training subjects change, they are often accompanied by a large number of code changes, and the system lacks configurability. Finally, there are few studies on VR disassembly and assembly multi-person collaboration systems for aircraft engines. On the one hand, the interaction methods of other systems are relatively simple, mostly simple picking operations, and lack the presentation of disassembly and assembly details. On the other hand, there is a lack of multi-person collaboration, which makes it difficult to cultivate the degree of cooperation between trainees in collaborative work. Summary of the invention
[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multi-user collaborative maintenance perception system to solve the above-mentioned technical problems.
[0005] The present invention provides a multi-user collaborative maintenance perception system, comprising: The architecture building module is used to build the server and client based on the Mirror network framework and deploy the server and client on multiple specified terminal devices; Configure the enumeration module to enumerate all deployed JSON configuration files and select the operation mode, which includes server mode and client mode; The first implementation module is used to broadcast the local address to the local area network and initialize the hierarchical tree in the server scene if the server mode is selected; The second implementation module is used to establish a communication connection with the server based on the server address in the local area network if the client mode is selected, and initialize the hierarchical tree in the client scene.
[0006] In an optional embodiment, the architecture building module includes: Software building unit, used to build permission-isolated servers and clients based on the Mirror network framework; A software deployment unit is used to design a spreadsheet to edit task sequences and deploy configuration files to multiple terminal devices using the PyQT5 development framework and Python language.
[0007] In an optional implementation, the terminal device includes a client device and a server device, and the client device includes a Leap Motion, an HTC VIVE Pro2, and a server.
[0008] In an optional implementation, the software construction unit includes: A functional encapsulation subunit is used to encapsulate multiple functional modules using a modular design; Multiple functional modules include: Data management module, used to select JSON parameterized files and manage scene resources required for VR multi-person collaborative disassembly and assembly training; Network module, used for the server and client to exchange object posture data, human posture data, task status data and gesture posture data; The motion simulation module is used to calculate and evaluate the physical properties of objects such as gravity and collision, and encapsulates the behavior model code of the object; The logic management module is used to receive network module data, provide JSON parameterized file parsing function, task status management function and human-computer interaction function, and return the processing result to the network module; The visualization module is used to receive data from the network module, the logic management module and the motion simulation module, and visualize the task guidance, client operation and motion simulation.
[0009] In an optional implementation, the software construction unit includes: The permission isolation subunit is used to set client permissions and server permissions. The client permissions include executing gesture recognition, local virtual avatar control and UI interface interaction, and the server permissions include executing physical computing simulation, task retrieval activation logic and network object behavior model scripts.
[0010] In an optional implementation, the software deployment unit includes: A sequence parsing subunit, for parsing a task sequence in a corresponding spreadsheet based on a selected deployment mode; A file selection subunit, used to select a directory where a JSON configuration file is located, wherein the directory includes multiple JSON configuration files, and the JSON configuration file includes a training subject of a disassembly sequence; The file deployment subunit is used to extract the target JSON configuration file corresponding to the task sequence from the directory and deploy the JSON configuration file.
[0011] In an optional implementation, the software deployment unit further includes: The file acquisition subunit is used to report the IP address of the deployment server and obtain the corresponding JSON configuration file if the deployment mode is the client mode; A file storage subunit, used to save the JSON configuration file to Application.persistentDataPath of the VR multiplayer collaboration module developed by Unity3D; The progress display sub-power supply is used to visualize the transfer process in the form of a progress bar.
[0012] In an optional implementation, the first implementation module includes: The hierarchical tree in the server scene is initialized. The multiple server scene elements after initialization include: The tool manager uses the ToolMnr script to create and destroy tools. As a child object of the tool manager, the specific tool has its own behavior model script ToolBehaviour, BoxCollider collision box component, NetworkIdentity network identity component, Rigidbody rigid body component and Highlighter highlight component. The task manager is a network object that processes the initialization, activation, and skipping of disassembly and assembly tasks through the TasksMnr script, and sends the processing results to the task manager in the client; The specific part of the aircraft engine model is a network object with a NetworkIdentity network identity component. According to the processing results of the task manager, the PartBehaviour behavior model encapsulation code of the corresponding task is dynamically loaded, and the behavior model encapsulation code is only loaded in the server. The Collider collision body type is determined by the PartBehaviour script. The specific part loads the Rigidbody rigid body component and the Highlighter highlight component. In order to ensure the consistency of the collaborative scene, the AutoSyncTransform script is designed to achieve the position and posture synchronization of the parts. The observation camera mounts the ServerCameraCtl script, which can observe the collaborative operation from any perspective on the server; The network manager manages network connections, mounts the NetworkManager script, NetworkDiscovery script, KcpTransport protocol control script, and NetworkStatics script, and implements server startup, connection establishment, broadcasting, and traffic monitoring functions; The UserMnr script is mounted on the user manager to manage the user list and facilitate user search; all users are under the user manager level, and each specific connected user is a network object, with the NetworkIdentity component mounted; the NetworkTransform and NetworkTransformChild components are responsible for the pose synchronization of the key joints of the virtual avatar; the UserCtr script is responsible for the initialization of the virtual avatar on the server and client, and provides the control method of the virtual avatar; the VRIK component is responsible for the action reproduction of the virtual avatar; the Rigidbody and BoxColider components are responsible for the collision detection of the virtual avatar; The scene manager is a network object that mounts the NetworkIdentity component. The SceneMnr script is used to manage information such as task text prompts and training time in the scene; The voice communication tool provides voice communication functions, which are implemented through the DissonanceVoice plug-in. This plug-in cooperates with the Mirror network framework to support voice transmission with location information and echo cancellation. The voice communication function is implemented through the cooperation of four components: DissonanceComms, MirrorIgnoranceCommsNetwork, VoiceBroadcastTrigger and VoiceReceiptTrigger.
[0013] In an optional implementation, the second implementation module includes: The hierarchical tree in the client scene is initialized. The multiple client scene elements after initialization include: The tool manager uses the ToolMnr script to create and destroy tools. As a child object of the tool manager, the specific tool has its own behavior model script ToolBehaviour, BoxCollider collision box component, NetworkIdentity network identity component, Rigidbody rigid body component and Highlighter highlight component. The task manager is responsible for receiving the processing results of the task manager on the server. When the task changes, it will trigger the callback function to provide visual guidance for the task and present detailed text information of the task. The specific parts of the aircraft engine model are a network object, and the client only executes the instructions issued by the server and synchronizes the position of the parts; The network manager is responsible for searching and connecting to the server; The UserMnr script is mounted on the user manager to manage the user list and facilitate user search. Local users and other users are at the user manager level. Each specific connected user is a network object. The specific connected user is the same as the server and is only responsible for synchronizing posture and playback. The local user additionally loads the Camera component, HandBinder component, GestureJudgementKNN script, and LeapXRServiceProvider component. The Camera component is responsible for rendering VR images, the HandBinder component is responsible for mapping hand movements to virtual lifting hand movements, the GestureJudgementKNN script is responsible for recognizing gestures, and the LeapXRServiceProvider component provides LeapMotion services. SceneMnr in the scene manager is responsible for receiving information such as task text prompts and training time from the server; The components in the voice communication tool are the same as those on the server side and are responsible for voice communication between users.
[0014] In an optional embodiment, the system further includes: Client virtual panel, supporting the client components include: Use the InteractionManager, InteractionHand, InteractionBehaviour, InteractionButton, and InteractionSlider components provided in the Leap Motion SDK to implement interaction; The Interaction-Manager component manages the Leap Motion interaction kernel, and its parameters include hover sensing and touch sensitivity. The InteractionHand component sets the hand interaction mode, and the parameters include hover sensing, grasping, and contact; The InteractionBehaviour component is used to set the interactive behavior of the object and implement the grabbing behavior of the virtual panel handle; The InteractionButton component is used to implement the three-dimensional button in the virtual panel; The InteractionSlider component is used to implement the slider in the virtual panel.
[0015] The beneficial effects of the present invention are that the multi-user collaborative maintenance perception system provided by the present invention can realize the synchronization and coordination of collaborative disassembly and assembly by multiple people, ensure the correctness and consistency of the disassembly and assembly process, provide new ideas and methods for the training of aircraft engine maintenance skills personnel, provide effective tools for aircraft engine maintainability assessment, provide reference for the design and optimization of aircraft engines, and also provide a technical solution that can be used as a reference for equipment disassembly and assembly training and assessment in other fields.
[0016] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0018] Figure 1 It is a schematic diagram of the architecture of a multi-user collaborative maintenance perception system according to an embodiment of the present invention.
[0019] Figure 2 It is a hardware architecture diagram of a multi-user collaborative maintenance perception system according to an embodiment of the present invention.
[0020] Figure 3 It is a framework diagram of a multi-user collaborative maintenance perception system according to an embodiment of the present invention.
[0021] Figure 4 This is a permission isolation diagram of a multi-user collaborative maintenance perception system according to an embodiment of the present invention.
[0022] Figure 5 It is a deployment flow chart of a multi-user collaborative maintenance perception system according to an embodiment of the present invention.
[0023] Figure 6 It is a schematic diagram of the server-side scene element structure of a multi-user collaborative maintenance perception system according to an embodiment of the present invention.
[0024] Figure 7 It is a schematic diagram of the client scene element structure of a multi-user collaborative maintenance perception system according to an embodiment of the present invention.
[0025] Figure 8 This is an effect diagram of a client virtual panel of a multi-user collaborative maintenance perception system according to an embodiment of the present invention. DETAILED DESCRIPTION
[0026] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.
[0028] In some embodiments, the multi-user collaborative maintenance perception system may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the multi-user collaborative maintenance perception system may be stored in a memory of a computer device and executed by at least one processor to perform (see Figure 1 Description) Functionality of a multi-user collaborative maintenance perception system.
[0029] For details, please refer to Figure 1 , providing a multi-user collaborative maintenance perception system, including: The architecture building module is used to build the server and client based on the Mirror network framework and deploy the server and client on multiple specified terminal devices; Configure the enumeration module to enumerate all deployed JSON configuration files and select the operation mode, which includes server mode and client mode; The first implementation module is used to broadcast the local address to the local area network and initialize the hierarchical tree in the server scene if the server mode is selected; The second implementation module is used to establish a communication connection with the server based on the server address in the local area network if the client mode is selected, and initialize the hierarchical tree in the client scene.
[0030] Optionally, as an embodiment of the present invention, the architecture building module includes: Software building unit, used to build permission-isolated servers and clients based on the Mirror network framework; A software deployment unit is used to design a spreadsheet to edit task sequences and deploy configuration files to multiple terminal devices using the PyQT5 development framework and Python language.
[0031] like Figure 2As shown, the terminal device includes a client device and a server device, and the client device includes LeapMotion, HTC VIVE Pro2 and a server.
[0032] The HTC VIVE Pro2 headset has a single-eye resolution of 2448×2448 and a binocular resolution of 4896×2448, with a clarity of up to 5k level; it supports a refresh rate of 90Hz or 120Hz, providing smooth and low-latency images; it has a field of view of up to 120 degrees, effectively improving the sense of immersion; it uses SteamVR positioning tracking technology to achieve precise and stable spatial positioning.
[0033] The Leap Motion somatosensory sensor can perform different operations by tracking the movements and movement trajectories of the hands. The Leap Motion has two wide-angle high-frame-rate grayscale infrared cameras and four infrared LEDs. The tracking accuracy can reach up to 0.01mm. It tracks the finger movements of people through a 150-degree viewing angle at a maximum frame rate of 290 frames per second. In this project, the Leap Motion is used for gesture recognition, and gestures are used to achieve operations such as grabbing and releasing.
[0034] Please refer to Figure 3 ,In order to facilitate the later system maintenance, it is necessary to reduce the ,coupling between the VR multiplayer collaboration module functions, therefore the ,main functions are modularly designed and encapsulated.
[0035] Software building blocks include: A functional encapsulation subunit is used to encapsulate multiple functional modules using a modular design; Multiple functional modules include: Data management module, used to select JSON parameterized files and manage scene resources required for VR multi-person collaborative disassembly and assembly training; Network module, used for the server and client to exchange object posture data, human posture data, task status data and gesture posture data; The motion simulation module is used to calculate and evaluate the physical properties of objects such as gravity and collision, and encapsulates the behavior model code of the object; The logic management module is used to receive network module data, provide JSON parameterized file parsing function, task status management function and human-computer interaction function, and return the processing result to the network module; The visualization module is used to receive data from the network module, the logic management module and the motion simulation module, and visualize the task guidance, client operation and motion simulation.
[0036] Please refer to Figure 4In order to avoid mutual influence between client and server scripts, it is necessary to use the isServerOnly and isClientOnly properties provided by the Mirror network framework to isolate the permissions of logical objects. For example, gesture recognition, local virtual avatar control, and UI interface interaction should only be executed locally on the client, and only the client has permissions; while physical calculation simulation, task retrieval activation logic, and network object behavior model scripts should only have permissions on the server, and logical operations should be performed through the server.
[0037] That is, the software construction unit includes: a permission isolation subunit, which is used to set client permissions and server permissions. The client permissions include executing gesture recognition, local virtual avatar control and UI interface interaction, and the server permissions include executing physical computing simulation, task retrieval activation logic and network object behavior model scripts.
[0038] Optionally, as an embodiment of the present invention, the software deployment unit includes: A sequence parsing subunit, for parsing a task sequence in a corresponding spreadsheet based on a selected deployment mode; A file selection subunit, used to select a directory where a JSON configuration file is located, wherein the directory includes multiple JSON configuration files, and the JSON configuration file includes a training subject of a disassembly sequence; The file deployment subunit is used to extract the target JSON configuration file corresponding to the task sequence from the directory and deploy the JSON configuration file.
[0039] The multi-terminal deployment module is developed using the PyQT5 development framework and Python language, and has designed a spreadsheet editing task sequence and deployed configuration files to multiple terminals. For ease of use, the module integrates both server and client modes, and can be started as a deployment server or client when the module is started.
[0040] like Figure 5 As shown, if started as a server, the server deployment mode provides the function of converting an Excel spreadsheet into a JSON configuration file. During the deployment process, select the directory where the JSON configuration file is located. The file directory can contain multiple JSON configuration files. A JSON configuration file represents a training subject for a disassembly and assembly sequence. After starting the deployment, all configuration files are deployed to multiple terminals.
[0041] If you choose the client mode, you only need to enter the IP address of the deployment server to receive the server's JSON configuration file and save it to the Application.persistentDataPath of the VR multiplayer collaboration module developed by Unity3D. This path is the data persistence path provided by Unity3D and is also an important path for hot updates. A progress bar is designed to visualize the transmission process in the form of a progress bar.
[0042] The VR multiplayer collaboration module interface is designed and implemented through the UGUI related components of Unity3D. The "Training Configuration" option enumerates all the deployed JSON configuration files. This article defines the file directory as Application.persistentDataPath. You can select server mode and client mode in the running mode. If you select the server mode and click Create, a server will be created and the local address will be broadcast to the local area network, allowing the client to search for the server directly. Select the client mode. Under normal circumstances, the created server will appear in the "Select Server" column, and you can establish a connection by clicking it; in addition, a server IP address input box is designed. When no server is searched in "Select Server" or the server is deployed in the public network, you can use this method to directly type in the server address and establish a connection.
[0043] Since the server mode and client mode are integrated into the same set of software, the scenarios are consistent before initialization, and different initialization strategies will be adopted when selecting different modes.
[0044] Optionally, as an embodiment of the present invention, Figure 6 As shown, the first implementation module includes: The hierarchical tree in the server scene is initialized. The multiple server scene elements after initialization include: The tool manager uses the ToolMnr script to create and destroy tools. As a child object of the tool manager, the specific tool has its own behavior model script ToolBehaviour, BoxCollider collision box component, NetworkIdentity network identity component, Rigidbody rigid body component and Highlighter highlight component. The task manager is a network object that processes the initialization, activation, and skipping of disassembly and assembly tasks through the TasksMnr script, and sends the processing results to the task manager in the client; The specific part of the aircraft engine model is a network object with a NetworkIdentity network identity component. According to the processing results of the task manager, the PartBehaviour behavior model encapsulation code of the corresponding task is dynamically loaded, and the behavior model encapsulation code is only loaded in the server. The Collider collision body type is determined by the PartBehaviour script. The specific part loads the Rigidbody rigid body component and the Highlighter highlight component. In order to ensure the consistency of the collaborative scene, the AutoSyncTransform script is designed to achieve the position and posture synchronization of the parts. The observation camera mounts the ServerCameraCtl script, which can observe the collaborative operation from any perspective on the server; The network manager manages network connections, mounts the NetworkManager script, NetworkDiscovery script, KcpTransport protocol control script, and NetworkStatics script, and implements server startup, connection establishment, broadcasting, and traffic monitoring functions; The UserMnr script is mounted on the user manager to manage the user list and facilitate user search; all users are under the user manager level, and each specific connected user is a network object, with the NetworkIdentity component mounted; the NetworkTransform and NetworkTransformChild components are responsible for the pose synchronization of the key joints of the virtual avatar; the UserCtr script is responsible for the initialization of the virtual avatar on the server and client, and provides the control method of the virtual avatar; the VRIK component is responsible for the action reproduction of the virtual avatar; the Rigidbody and BoxColider components are responsible for the collision detection of the virtual avatar; The scene manager is a network object that mounts the NetworkIdentity component. The SceneMnr script is used to manage information such as task text prompts and training time in the scene; The voice communication tool provides voice communication functions, which are implemented through the DissonanceVoice plug-in. This plug-in cooperates with the Mirror network framework to support voice transmission with location information and echo cancellation. The voice communication function is implemented through the cooperation of four components: DissonanceComms, MirrorIgnoranceCommsNetwork, VoiceBroadcastTrigger and VoiceReceiptTrigger.
[0045] Optional, such as Figure 7 As shown, the second implementation module includes: The hierarchical tree in the client scene is initialized. The multiple client scene elements after initialization include: The tool manager uses the ToolMnr script to create and destroy tools. As a child object of the tool manager, the specific tool has its own behavior model script ToolBehaviour, BoxCollider collision box component, NetworkIdentity network identity component, Rigidbody rigid body component and Highlighter highlight component. The task manager is responsible for receiving the processing results of the task manager on the server. When the task changes, it will trigger the callback function to provide visual guidance for the task and present detailed text information of the task. The specific parts of the aircraft engine model are a network object, and the client only executes the instructions issued by the server and synchronizes the position of the parts; The network manager is responsible for searching and connecting to the server; The UserMnr script is mounted on the user manager to manage the user list and facilitate user search. Local users and other users are at the user manager level. Each specific connected user is a network object. The specific connected user is the same as the server and is only responsible for synchronizing posture and playback. The local user additionally loads the Camera component, HandBinder component, GestureJudgementKNN script, and LeapXRServiceProvider component. The Camera component is responsible for rendering VR images, the HandBinder component is responsible for mapping hand movements to virtual lifting hand movements, the GestureJudgementKNN script is responsible for recognizing gestures, and the LeapXRServiceProvider component provides LeapMotion services. SceneMnr in the scene manager is responsible for receiving information such as task text prompts and training time from the server; The components in the voice communication tool are the same as those on the server side and are responsible for voice communication between users.
[0046] Optionally, as an embodiment of the present invention, the system further includes: Client virtual panel. During the VR multiplayer disassembly and assembly of aircraft engines, students need to view detailed information of the current task, skip tasks, and obtain tools at any time. To facilitate students' learning and use, a virtual panel is designed. The virtual panel is opened through interface gestures. The process of opening the virtual panel is as follows: Figure 8 shown.
[0047] The client components that support the above include: using the InteractionManager, InteractionHand, InteractionBehaviour, InteractionButton and InteractionSlider components provided in the LeapMotionSDK to achieve interaction. The Interaction-Manager component manages the interactive kernel of LeapMotion and can set parameters such as suspension sensing and touch sensitivity; the InteractionHand component sets the hand interaction mode and can support suspension sensing, grabbing and contact; the InteractionBehaviour component is used to set the interactive behavior of the object and realize the grabbing behavior of the virtual panel handle; the InteractionButton component is used to realize the three-dimensional buttons in the virtual panel, such as pressing the sleeve button to obtain the sleeve tool; the InteractionSlider component is used to realize the slider in the virtual panel, so that students can easily check multiple pages of tasks.
[0048] The multi-user collaborative maintenance perception system provided by the present invention collects the real human head posture and lower arm and hand posture in real time through the HTC VIVE Pro2 VR head display and LeapMotion, respectively, to achieve real-time motion control of the virtual avatar. And through the hybrid synchronization method based on the Mirror network communication framework, the smoothness of the collaboration is improved under the premise of ensuring the consistency of multi-terminal scenes, and the multi-user collaborative perception in the virtual environment is realized, so that users can correctly and real-time perceive the existence and operation process of others in the virtual environment, and can also realize voice communication through the network.
[0049] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.
Claims
1. A multi-user collaborative maintenance perception system, characterized in that: include: The architecture building module is used to build the server and client based on the Mirror network framework and deploy the server and client on multiple specified terminal devices; Configure the enumeration module to enumerate all deployed JSON configuration files and select the operation mode, which includes server mode and client mode; The first implementation module is used to broadcast the local address to the local area network and initialize the hierarchical tree in the server scene if the server mode is selected; The second implementation module is used to establish a communication connection with the server based on the server address in the local area network if the client mode is selected, and initialize the hierarchical tree in the client scene.
2. The system according to claim 1, characterized in that The architecture building blocks include: Software building unit, used to build permission-isolated servers and clients based on the Mirror network framework; A software deployment unit is used to design a spreadsheet to edit task sequences and deploy configuration files to multiple terminal devices using the PyQT5 development framework and Python language.
3. The system according to claim 2, characterized in that The terminal device includes a client device and a server device, and the client device includes Leap Motion, HTC VIVE Pro2 and a server.
4. The system according to claim 2, characterized in that The software building unit includes: A functional encapsulation subunit is used to encapsulate multiple functional modules using a modular design; Multiple functional modules include: Data management module, used to select JSON parameterized files and manage scene resources required for VR multi-person collaborative disassembly and assembly training; Network module, used for the server and client to exchange object posture data, human posture data, task status data and gesture posture data; The motion simulation module is used to calculate and evaluate the physical properties of objects such as gravity and collision, and encapsulates the behavior model code of the object; The logic management module is used to receive network module data, provide JSON parameterized file parsing function, task status management function and human-computer interaction function, and return the processing result to the network module; The visualization module is used to receive data from the network module, the logic management module and the motion simulation module, and visualize the task guidance, client operation and motion simulation.
5. The system according to claim 4, characterized in that The software building unit includes: The permission isolation subunit is used to set client permissions and server permissions. The client permissions include executing gesture recognition, local virtual avatar control and UI interface interaction, and the server permissions include executing physical computing simulation, task retrieval activation logic and network object behavior model scripts.
6. The system according to claim 2, characterized in that The software deployment unit includes: A sequence parsing subunit, for parsing a task sequence in a corresponding spreadsheet based on a selected deployment mode; A file selection subunit, used to select a directory where a JSON configuration file is located, wherein the directory includes multiple JSON configuration files, and the JSON configuration file includes a training subject of a disassembly sequence; The file deployment subunit is used to extract the target JSON configuration file corresponding to the task sequence from the directory and deploy the JSON configuration file.
7. The system according to claim 6, characterized in that The software deployment unit also includes: The file acquisition subunit is used to report the IP address of the deployment server and obtain the corresponding JSON configuration file if the deployment mode is the client mode; A file storage subunit, used to save the JSON configuration file to Application.persistentDataPath of the VR multiplayer collaboration module developed by Unity3D; The progress display sub-power supply is used to visualize the transfer process in the form of a progress bar.
8. The system according to claim 1, characterized in that The first implementation module includes: The hierarchical tree in the server scene is initialized. The multiple server scene elements after initialization include: The tool manager uses the ToolMnr script to create and destroy tools. As a child object of the tool manager, the specific tool has its own behavior model script ToolBehaviour, BoxCollider collision box component, NetworkIdentity network identity component, Rigidbody rigid body component and Highlighter highlight component. The task manager is a network object that processes the initialization, activation, and skipping of disassembly and assembly tasks through the TasksMnr script, and sends the processing results to the task manager in the client; The specific part of the aircraft engine model is a network object with a NetworkIdentity network identity component. According to the processing results of the task manager, the PartBehaviour behavior model encapsulation code of the corresponding task is dynamically loaded, and the behavior model encapsulation code is only loaded in the server. The Collider collision body type is determined by the PartBehaviour script. The specific part loads the Rigidbody rigid body component and the Highlighter highlight component. In order to ensure the consistency of the collaborative scene, the AutoSyncTransform script is designed to achieve the position and posture synchronization of the parts. The observation camera mounts the ServerCameraCtl script, which can observe the collaborative operation from any perspective on the server; The network manager manages network connections, mounts the NetworkManager script, NetworkDiscovery script, KcpTransport protocol control script, and NetworkStatics script, and implements server startup, connection establishment, broadcasting, and traffic monitoring functions; The UserMnr script is mounted on the user manager to manage the user list and facilitate user search; all users are under the user manager level, and each specific connected user is a network object, with the NetworkIdentity component mounted; the NetworkTransform and NetworkTransformChild components are responsible for the pose synchronization of the key joints of the virtual avatar; the UserCtr script is responsible for the initialization of the virtual avatar on the server and client, and provides the control method of the virtual avatar; the VRIK component is responsible for the action reproduction of the virtual avatar; the Rigidbody and BoxColider components are responsible for the collision detection of the virtual avatar; The scene manager is a network object that mounts the NetworkIdentity component. The SceneMnr script is used to manage information such as task text prompts and training time in the scene; The voice communication tool provides voice communication functions, which are implemented through the DissonanceVoice plug-in. This plug-in cooperates with the Mirror network framework to support voice transmission with location information and echo cancellation. The voice communication function is implemented through the cooperation of four components: DissonanceComms, MirrorIgnoranceCommsNetwork, VoiceBroadcastTrigger and VoiceReceiptTrigger.
9. The system according to claim 1, characterized in that The second implementation module includes: The hierarchical tree in the client scene is initialized. The multiple client scene elements after initialization include: The tool manager uses the ToolMnr script to create and destroy tools. As a child object of the tool manager, the specific tool has its own behavior model script ToolBehaviour, BoxCollider collision box component, NetworkIdentity network identity component, Rigidbody rigid body component and Highlighter highlight component. The task manager is responsible for receiving the processing results of the task manager on the server. When the task changes, it will trigger the callback function to provide visual guidance for the task and present detailed text information of the task. The specific parts of the aircraft engine model are a network object, and the client only executes the instructions issued by the server and synchronizes the position of the parts; The network manager is responsible for searching and connecting to the server; The UserMnr script is mounted on the user manager to manage the user list and facilitate user search. Local users and other users are at the user manager level. Each specific connected user is a network object. The specific connected user is the same as the server and is only responsible for synchronizing posture and playback. The local user additionally loads the Camera component, HandBinder component, GestureJudgementKNN script, and LeapXRServiceProvider component. The Camera component is responsible for rendering VR images, the HandBinder component is responsible for mapping hand movements to virtual lifting hand movements, the GestureJudgementKNN script is responsible for recognizing gestures, and the LeapXRServiceProvider component provides LeapMotion services. SceneMnr in the scene manager is responsible for receiving information such as task text prompts and training time from the server; The components in the voice communication tool are the same as those on the server side and are responsible for voice communication between users.
10. The system according to claim 1, characterized in that The system further comprises: Client virtual panel, supporting the client components include: Use the InteractionManager, InteractionHand, InteractionBehaviour, InteractionButton, and InteractionSlider components provided in the Leap Motion SDK to implement interaction; The Interaction-Manager component manages the Leap Motion interaction kernel, and its parameters include hover sensing and touch sensitivity. The InteractionHand component sets the hand interaction mode, and the parameters include hover sensing, grasping, and contact; The InteractionBehaviour component is used to set the interactive behavior of the object and implement the grabbing behavior of the virtual panel handle; The InteractionButton component is used to implement the three-dimensional button in the virtual panel; The InteractionSlider component is used to implement the slider in the virtual panel.