An augmented reality-based device management system and method

Abstract

This invention discloses an augmented reality-based equipment management system and method. It includes: a database module for storing and managing images, text, audio, video, and various assembly, inspection, maintenance, and equipment usage guidance information required for the augmented reality experience process; an augmented reality guidance management module for configuring and processing commands in the augmented reality equipment management system; an augmented reality equipment management operation execution environment module for users to actually perform various equipment management operations: equipment assembly, equipment inspection, equipment maintenance, equipment information viewing, and equipment operation learning; an AR support module for tracking registration and rendering of virtual objects; and a multi-mode interaction control module for providing users with multiple interaction methods. This invention uses augmented reality technology to assist and optimize the equipment planning, installation, operation, inspection, and maintenance stages of the equipment management process, thereby improving the work efficiency of each stage of equipment management.

Description

Technical Field

[0001] This invention relates to a system construction method in the field of equipment management, and more particularly to an equipment management system and method based on augmented reality. Background Technology

[0002] As the factory continues to grow, more and more equipment is being purchased. Facing anywhere from dozens to thousands of pieces of equipment, existing equipment management methods are no longer sufficient to meet the needs of equipment managers. The shortcomings of existing equipment management methods are gradually becoming apparent at every stage of equipment management. In the equipment assembly stage, current equipment management systems can only provide workers with illustrated assembly documents. The assembly learning process is lengthy, still requiring experienced technicians to guide newcomers, resulting in extremely high learning costs. After the equipment is put into use, workers cannot intuitively see real-time information on various key parameters of the equipment. When encountering malfunctions, they can only retrieve illustrated and video explanations from the equipment management system or seek help from experienced engineers. Until the malfunction is resolved, the equipment must be shut down, causing the company to incur huge economic losses. During equipment inspection, existing equipment management systems cannot provide workers with intuitive inspection routes. Furthermore, workers mostly rely on a combination of mobile equipment and drawing checklists for equipment inspection, which is inefficient. During equipment maintenance, workers cannot quickly locate the fault or find the necessary tools immediately, leading to low maintenance efficiency. Meanwhile, the existing equipment management system cannot detect the workers' work status during equipment inspection and maintenance, nor can it know whether the workers have completed the inspection tasks according to the instructions, or whether the workers have made mistakes during the maintenance process.

[0003] Existing augmented reality-based device management methods typically track and register devices using model outlines or marker recognition. However, in real-world device management environments, users encounter devices with identical or similar shapes, making model outline recognition inaccurate. Furthermore, in real production environments, affixing identification codes to devices is time-consuming and labor-intensive, and these codes are prone to detachment, wear, and eventual unusability. Replacing these codes then incurs a significant expense. Summary of the Invention

[0004] To address the aforementioned technical shortcomings, this invention provides an augmented reality-based equipment management system and method. Unlike previous augmented reality-based equipment management methods, this invention performs 3D scanning of all real-world environments involved in the equipment management process, generating 3D models. Furthermore, it uses materials such as images, text, audio, video, animation sequences, and equipment-related data as augmented reality guidance information, feeding it back to the user through the augmented reality equipment management system. This allows for rapid guidance of users in the equipment management stages—location planning, assembly, operation, inspection, and maintenance—improving work efficiency at each stage of equipment management.

[0005] The technical solution adopted by this invention to solve its technical problem is:

[0006] An augmented reality-based device management system, characterized in that it includes:

[0007] The database module is used to store and manage the files required for the augmented reality experience process, as well as augmented reality guidance information;

[0008] The multi-mode interactive control module collects user multi-mode interactive commands;

[0009] The augmented reality equipment management operation execution environment module is used to call the corresponding commands of the augmented reality guided management module according to the multi-form interactive commands. The interactive commands include: viewing equipment information, planning equipment placement, and managing each stage of equipment assembly, operation, inspection, and maintenance.

[0010] The Augmented Reality Bootstrapping Management module is used for command configuration and processing in the Augmented Reality device management system.

[0011] The AR support module is used to configure and process commands from the augmented reality guidance management module to achieve environmental perception, tracking and registration, model recognition, virtual-real fusion, rendering of virtual objects, and display of augmented reality guidance information on the AR device, thereby guiding users to manage the device according to the augmented reality guidance information.

[0012] The augmented reality guidance information includes device model files, image files, audio and video files, animation sequence files, and data files.

[0013] The model files include: equipment model files assembled according to the assembly process, flight guidance robot model files, and room environment model files;

[0014] The environmental model file is a three-dimensional model file containing room environment information obtained by laser scanning the target environment using a lidar scanning device; the target environment includes rooms, production areas, and floors.

[0015] The device model file refers to the device model to be installed in the target environment, which is constructed by 3D scanning or industrial modeling software based on CAD drawings.

[0016] The guiding robot model is used to float within the user's field of vision during augmented reality guidance, and to provide the user with operation direction guidance or directional navigation by moving to a predetermined position or orientation.

[0017] The image files include: pre-made 2D graphics, 3D graphics, and model texture files. The 2D graphics are used to display the equipment management manual, 2D equipment dashboard, and equipment instruction flow documents in a two-dimensional manner; the 3D graphics are used to present the equipment management manual, 2D equipment dashboard, and equipment instruction flow documents to users in a multi-dimensional way.

[0018] The audio and video files include: pre-recorded audio and video files related to equipment location planning, equipment assembly, equipment operation, equipment maintenance, and equipment inspection;

[0019] The animation sequence files include: pre-made animation sequence files related to equipment location planning, equipment assembly, equipment operation, equipment maintenance, and equipment inspection.

[0020] The data file includes: various operating data of the equipment and equipment status data;

[0021] Various operational data of the equipment are collected by sensors connected to the actual environment and the equipment controller during the equipment operation process, including sensor data and action execution parameter data. Sensor data includes: temperature and humidity parameters, voltage or current parameters, and communication signal parameters. Action execution parameter data includes: position parameters and speed parameters.

[0022] Equipment status data is collected by a device controller connected to the actual environment to collect equipment management status parameters. Different statuses include: equipment installation status, operating status, fault type, and the equipment management stage the equipment is in.

[0023] The various operating data and status parameters of the device are displayed in the augmented reality scene in the form of text and icons. This is used to prompt the user of the different statuses of the current device and the various operating data of the device under the corresponding status at different stages of device management.

[0024] The augmented reality guidance management module includes:

[0025] The display unit is used to load various augmented reality guidance information from the database: guidance information and equipment status for the equipment assembly stage, equipment operation stage, equipment inspection stage, and equipment maintenance stage, as well as the display location and content of the above augmented reality guidance information in the real environment, and display them in the augmented reality device environment of the smart terminal in the corresponding form.

[0026] The command mapping and parsing unit pre-sets interactive commands for voice, gesture, and eye tracking, along with their corresponding trigger events; it parses user interactive commands in real time and converts them into corresponding trigger event instructions; the trigger events include: various guidance processes and instructions for different stages of device location planning and device management, which are displayed in file format to guide users in managing the device;

[0027] The layout control unit, based on the corresponding trigger event instructions output by the command mapping and parsing unit, pre-sets the system's UI layout and the corresponding trigger operations for each model control.

[0028] The augmented reality device management operation execution environment module includes:

[0029] The device placement planning-AR guidance execution environment unit, according to the predetermined planned placement position or the user's real-time placement position instruction, calls the AR support module to render the preset device at the predetermined position in the augmented reality virtual environment, so that the user can preview the placement effect of the current virtual device in the real environment through a smart terminal; the smart terminal includes: AR glasses, mobile tablet, mobile phone, PC.

[0030] The Equipment Management - AR Guidance Execution Environment Unit includes establishing augmented reality operation execution environments corresponding to the equipment assembly stage, equipment operation stage, equipment inspection stage, and equipment maintenance stage: parts area, tool area, and operation area; parsing user interaction commands into instructions for the corresponding management stage or corresponding steps; calling the AR support module to render and simulate virtual animation scenes of equipment assembly, equipment operation, equipment inspection, and equipment maintenance in the augmented reality virtual environment; controlling the actual physical equipment to execute the management process or guided operation corresponding to each stage of the rendered simulation, and finally completing the corresponding equipment management process.

[0031] The rendering of virtual objects in the AR support module is as follows: the layout and configuration of augmented reality guidance information are performed in Unity 3D, and the rendering and display of virtual objects are performed using the HoloLens optical see-through HMD device; when actually performing device management-related operations, the user wears HoloLens glasses to obtain all AR guidance information related to the corresponding device management stage.

[0032] The multi-form interaction commands include: voice commands, gesture commands, and eye-tracking commands input by the user while wearing HoloLens glasses.

[0033] include:

[0034] 1) The AR guidance process for equipment location planning is as follows:

[0035] Step 1: Create an AR experience rendering optimization model in advance based on the target location of the device to be placed;

[0036] Step 1-1: The user scans all the real environments involved in the device management using a scanning device equipped with LiDAR, generates a room environment model, and stores it in the database module.

[0037] Steps 1-2: Generate a Unity asset package with an environment model using Vuforia's region target generator;

[0038] Steps 1-3: Import the Unity asset package containing the environment model and the equipment model of the new equipment to be acquired into Unity; at the same time, import the status icon of whether the equipment is installed into Unity.

[0039] Steps 1-4: In the Unity development environment, the user drags the device model to the corresponding target position of the environment model, and at the same time drags the device status icon above the device model to create an AR experience rendering optimization model and publish it to the smart terminal.

[0040] Step 2: In a real environment, the user wears and turns on the AR glasses. Through the environment recognition function of Area Target, the virtual device model created in Step 1 is called and superimposed on the preset real environment to realize the simulation of the placement.

[0041] 2) Device Management - AR Guided Process:

[0042] Step 1: Create an AR experience rendering optimization model in advance based on the target location of the device to be placed;

[0043] Step 2: The user approaches the device to be assembled, turns on the AR glasses, and issues device management-related commands to the AR support module through interactive instructions;

[0044] Step 3: After receiving the command, the AR support module begins to search for commands related to device management in the augmented reality guidance management module. At the same time, the augmented reality guidance management module requests related augmented reality guidance information from the database, including: augmented reality guidance-related animation sequences, audio and video files, graphics, model files and device data files related to device management.

[0045] Step 3: The augmented reality guidance management module loads the corresponding augmented reality guidance information into the AR support module. The AR support module then begins environmental perception, tracking and registration, model recognition, and rendering of virtual objects. Finally, it feeds back the corresponding augmented reality guidance scene to the user, providing corresponding AR guidance for device management within this environment.

[0046] Step 4: After receiving the augmented reality guidance scene, the user can further interact with the augmented reality guidance scene through voice command control, gesture control, and eye tracking control, and finally complete the work of equipment assembly, operation, inspection, and maintenance; according to the current equipment status, modify the equipment status icon to the status icon of the next stage.

[0047] The present invention has the following advantages and beneficial effects:

[0048] 1. By simulating all real-world scenarios involved in the equipment management process, augmented reality guidance information is embedded in the corresponding positions of the real-world scenarios, thereby solving problems such as equipment positioning and identification of similar equipment in the augmented reality-based equipment management process.

[0049] 2. At each stage of equipment management, various augmented reality guidance information can be invoked according to the scenario to provide more precise and effective guidance to on-site assembly workers. After equipment failure, workers can quickly find solutions and resolve the problem using augmented reality guidance information, minimizing unforeseen losses.

[0050] 3. During equipment inspection, AR guidance information can be used to quickly navigate workers to the expected inspection location.

[0051] 4. Workers can perform augmented reality interactive operations through voice control, gesture control, and eye-tracking control. Multi-mode interactive control significantly improves the system's adaptability in different equipment management scenarios and also increases user acceptance of the system. Attached Figure Description

[0052] Figure 1 This is a system architecture diagram of an augmented reality-based device management system and method provided in an embodiment of the present invention.

[0053] Figure 2 2D image examples for device boot information;

[0054] Figure 3 3D image examples for device boot information;

[0055] Figure 4 The icons corresponding to different states of the device provided in the embodiments of the present invention. Detailed Implementation

[0056] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0058] This invention is achieved through the following technical solution: a multi-layered parallel interactive model structure is proposed, consisting of a database, augmented reality guidance management, augmented reality device management execution environment, AR support, and multi-form interactive control. This model solves the problem of work efficiency in the actual device management process through augmented reality technology.

[0059] Example:

[0060] like Figure 1 As shown, this embodiment provides a device management system based on augmented reality. The device management system includes a database module, an augmented reality guidance management module, an augmented reality-based device management execution environment module, an AR support module, and a multi-form interaction control module.

[0061] I. Database Module

[0062] The database module contains all the augmented reality guidance information involved in device management. This includes: model files, image files, audio and video files, animation sequence files, and device data files.

[0063] Furthermore, the aforementioned model files comprise all the device model files, guide robot model files, and room environment model files required for the augmented reality experience. Device model files are used in all stages of device management. During the device placement planning stage, the device models are imported into the development environment and integrated with the room environment model files. In the development environment, device model files are dragged and dropped into the room environment model to pre-plan the specific placement of devices within the room. During the device assembly stage, the device models serve a spatial positioning function. In the development environment, AR tags are placed near their corresponding positions on the device models. Because the device models are within the overall room environment model, the relative positions of the AR tags with respect to this device and other devices are fixed. Thus, during the actual AR guidance experience, even if the user simultaneously opens AR tags on multiple devices, issues such as AR tag overlap or drifting and inaccurate positioning will not occur. During the device inspection and maintenance stage, the device models are also used to determine the placement of 2D and 3D images, documents, animation sequences, etc., and to ensure that during actual operation, the corresponding augmented reality guidance information accurately appears near the device, seamlessly integrating with the virtual reality.

[0064] Furthermore, the aforementioned image files include 2D and 3D graphics, as well as model textures. The 2D and 3D graphics refer to all image-based files involved in the augmented reality bootstrapping process, such as equipment placement plans, equipment operation manuals, equipment operation precautions, equipment inspection checklists, equipment maintenance checklists, and equipment fault response strategies. For example... Figure 2 , 3 The images shown are 2D and 3D graphics.

[0065] Furthermore, the aforementioned audio and video files refer to the guiding video and audio files required during equipment location planning, equipment assembly, equipment operation, equipment inspection, and equipment maintenance.

[0066] Furthermore, the aforementioned animation sequence files refer to the model animation sequences required for augmented reality guidance during equipment location planning, equipment assembly, equipment operation, equipment inspection, and equipment maintenance. For example, in the AR guidance process for equipment assembly, it is necessary to acquire model animation sequences of different parts of the equipment and drive the animation sequences to play step by step through different control methods.

[0067] Furthermore, the aforementioned device data files include device operation data and device status data. The device data files are used as AR tags dynamically displayed to users during device operation guidance. The device status data is used to mark the device when the device management phase changes, alerting other users to the current management phase and facilitating work handover.

[0068] like Figure 4For equipment that has been purchased but not yet installed in the room, the equipment status is set to "Not Installed" in the database; for equipment being installed, the status is set to "Installing"; for equipment operating normally, the status is set to "Operating Normally"; for equipment being inspected, the status is set to "Inspecting"; for equipment that has malfunctioned, the status is set to "Faulty"; and for equipment being repaired, the status is set to "Repairing". A unique icon is prepared for each equipment status, and the icon changes accordingly when the equipment status changes. During augmented reality guidance, the equipment status image floats above the equipment, allowing users to quickly understand the current status of the equipment.

[0069] II. Augmented Reality Guidance Management Module

[0070] The Augmented Reality (AR) Guidance Management module comprises an Information Management module and a Command Management module. The Information Management module is primarily responsible for: 1. grouping AR guidance information related to various device management functions; 2. determining the display location, content, duration, and method of AR guidance information in the real device environment, thereby organizing, managing, and processing AR guidance information. The Command Management module is primarily responsible for setting interactive commands such as voice, gestures, and eye tracking, as well as their corresponding trigger events; the system's UI layout and trigger events for various controls; and other system configurations. Specific control commands issued by users in the AR-based device management environment via gestures, voice, or eye tracking must be recognized by this module to trigger the corresponding events and provide users with specific AR-based guidance information.

[0071] III. Augmented Reality-Based Device Management Execution Environment Module

[0072] The augmented reality-based equipment management execution environment module includes AR guidance for equipment location planning, AR guidance for equipment assembly, AR guidance for equipment operation, AR guidance for equipment inspection, and AR guidance for equipment maintenance.

[0073] Device placement planning - AR guidance execution environment, that is, the real-world environment in which the user performs augmented reality-based device placement planning;

[0074] Equipment assembly - AR guidance execution environment, that is, the real-world environment in which the user performs augmented reality-based equipment assembly guidance;

[0075] Device operation - AR guidance execution environment, that is, the real-world environment in which the user provides augmented reality-based device operation guidance;

[0076] Equipment Inspection - AR Guidance Execution Environment, which is the real-world environment in which the user conducts augmented reality-based equipment inspection guidance;

[0077] Equipment maintenance - AR guidance execution environment, that is, the real-world environment in which the user provides augmented reality-based equipment maintenance guidance;

[0078] Aside from the equipment location planning environment, the remaining operational environment can be mainly divided into three areas: the parts area, the tools area, and the operation area. The parts area contains the component objects required for equipment management operations; the tools area contains the tools and fixtures needed by the user during various equipment management operations; and the operation area is where the user performs specific equipment management operations. The operation area typically includes markers to assist in aligning virtual objects with real equipment. When performing augmented reality-based equipment management operations, the user must first configure the relevant environment to ensure the system is operational, such as recalibrating the augmented reality headset and eye-tracking positioning. Then, following the prompts of various augmented reality virtual guidance messages, the user follows the steps to perform the corresponding guided operations, ultimately completing the corresponding equipment management process.

[0079] Workflow:

[0080] (1) The AR guidance process for equipment location planning is as follows:

[0081] 1. Users scan all real-world environments involved in device management using scanning devices equipped with LiDAR to create room environment models;

[0082] 2. Generate a Unity asset package with an environment model using Vuforia's Area Target Generator;

[0083] 3. Import the Unity asset package containing the environment model and the device model of the new equipment to be acquired into Unity, and plan the placement of the models. Find the best place to place the equipment in the current environment before the equipment is moved over; at the same time, place the device status icon (the icon of the inactive state) on top of the device model.

[0084] 4. After preset the best placement position or the user-specified position, the user issues an instruction through the multi-interaction command module to drag the device model to the corresponding position of the environment model, and then renders and optimizes the AR experience model in the surrounding environment. Depending on the user's device, the AR experience can be published to the iOS / Android or AR glasses.

[0085] 5. When the user enters the real environment, turns on the AR glasses, and uses Area Target's environmental recognition function to overlay the virtual device model onto the preset real environment;

[0086] (2) The equipment assembly AR guidance process is as follows:

[0087] 1. The user approaches the device to be assembled, turns on the AR glasses, and issues the command to start the assembly process;

[0088] 2. The AR support module begins retrieving device assembly-related commands from the augmented reality guidance management module. At the same time, the augmented reality guidance management module requests related augmented reality guidance information from the database, such as AR assembly guidance-related model animation sequences, assembly guidance-related audio and video files, assembly guidance-related graphics and text, etc.

[0089] 3. The Augmented Reality Guidance Management Module loads the corresponding Augmented Reality guidance into the AR Support Module. The AR glasses then begin environmental perception, tracking and registration, model recognition, and rendering of virtual objects. Finally, the corresponding Augmented Reality guidance scene is fed back to the user, and AR guidance is installed on the device in this environment.

[0090] 4. After receiving the augmented reality guidance scenario, the user can further interact with the augmented reality guidance scenario through voice command control, gesture control, and eye-tracking control to finally complete the device assembly work;

[0091] 5. Based on the current device status, change the device status icon to the status icon for the next stage.

[0092] (3) The AR guidance operation procedure for equipment operation is as follows:

[0093] 1. The user approaches the device to be assembled, turns on the AR glasses, issues a command to start operating the device, and changes the device status to "Operating";

[0094] 2. The AR support module begins to retrieve device operation-related commands from the augmented reality guidance management module. At the same time, the augmented reality guidance management module requests related augmented reality guidance information from the database, such as device operation guidance-related model animation sequences, audio and video files, device operation guidance-related graphics and text, guidance robot models, etc.

[0095] 3. The Augmented Reality Guidance Management Module loads the corresponding Augmented Reality guidance into the AR Support Module. The AR glasses then begin environmental perception, tracking and registration, model recognition, and rendering of virtual objects. Finally, the corresponding Augmented Reality guidance scene is fed back to the user, providing AR guidance for device operation within this environment.

[0096] 4. After receiving the augmented reality (AR) guidance scenario, users can further interact with it through voice commands, gestures, and eye-tracking. Once the AR guidance for device operation begins, the guidance robot model will sequentially fly to the vicinity of the next device component to be operated, according to different device operation specifications. When the user clicks on the guidance robot, the corresponding device operation guidance will pop up in the user's field of vision, such as a device operation manual, audio / video guidance files, or a sequence of device operation animations. Each time the user completes a device operation step following the AR guidance, they can use commands (voice, gestures, or eye tracking) to control the guidance robot to fly to the next device operation position and interact with it again to display the AR guidance for the next device operation. This process is repeated until the device operation guidance is complete.

[0097] 5. Based on the current device status, change the device status icon to the status icon for the next stage.

[0098] (4) The AR guidance operation procedure for equipment inspection is as follows:

[0099] 1. The user opens the AR glasses, receives the device inspection work order, accepts and confirms the execution of the inspection work order, and issues the command to start the device inspection.

[0100] 2. The AR support module begins retrieving equipment inspection-related commands from the augmented reality guidance management module. At the same time, the augmented reality guidance management module requests related augmented reality guidance information from the database, such as equipment inspection guidance-related model animation sequences, audio and video files, equipment inspection guidance-related graphics and text (such as equipment inspection checklists), guidance robot models, etc.

[0101] 3. The Augmented Reality Guidance Management Module loads the corresponding Augmented Reality guidance into the AR Support Module. The AR glasses then begin environmental perception, tracking and registration, model recognition, and rendering of virtual objects. Finally, the corresponding Augmented Reality guidance scene is fed back to the user, providing AR guidance for device inspection within this environment.

[0102] 4. After receiving the augmented reality (AR) guidance, users can further interact with the AR-guided scenario through voice commands, gestures, and eye-tracking. When the AR guidance for device inspection begins, the guiding robot, hovering within the user's field of vision, will start moving towards the device to be inspected. The robot model will maintain a fixed distance from the user; when the user stops, the robot model will also stop and prompt the user to continue following the AR guidance to the device. When the user arrives at the device, the guiding robot will provide a voice notification that they have reached the vicinity of the device. When the user clicks the "Start Inspection" button next to the device model, an inspection list will pop up in the user's field of vision, and the device status will be changed to "Inspecting." Whenever the user clicks an inspection option on the list, the guiding robot model will fly to the vicinity of the device component to be inspected, and the component of the device model to be inspected will light up to attract the user's attention. When a user clicks on the guidance robot, corresponding equipment inspection instructions will pop up in the user's field of vision. These instructions will indicate which parts of the equipment need to be checked, whether real-time data for key components exceeds thresholds, etc. If real-time data exceeds a threshold (e.g., if the temperature exceeds 120 degrees Celsius, equipment parts will burn out), the data label corresponding to that component model will turn red to alert the inspector that there is a problem. Each time the user completes an equipment inspection item following the AR guidance, they can click on the next item in the inspection list. The guidance robot will then fly to the next equipment inspection location and interact with the user again, displaying the AR instructions for the next equipment inspection. This process is repeated until the equipment inspection guidance is complete.

[0103] 5. Based on the current device status, change the device status icon to the status icon for the next stage.

[0104] (5) The AR guidance operation procedure for equipment maintenance is as follows:

[0105] 1. The user opens the AR glasses, receives a device repair work order, accepts and confirms the execution of the repair work order, and issues a command to start device repair;

[0106] 2. The AR support module begins retrieving equipment maintenance-related commands from the augmented reality guidance management module. At the same time, the augmented reality guidance management module requests related augmented reality guidance information from the database, such as equipment maintenance guidance-related model animation sequences, audio and video files, equipment inspection guidance-related graphics and text (such as equipment maintenance suggestion guides), guidance robot models, etc.

[0107] 3. The Augmented Reality Guidance Management Module loads the corresponding Augmented Reality guidance into the AR Support Module. The AR glasses then begin environmental perception, tracking and registration, model recognition, and rendering of virtual objects. Finally, the corresponding Augmented Reality guidance scene is fed back to the user, providing AR guidance for device maintenance within this environment.

[0108] 4. After receiving the augmented reality (AR) guidance, users can further interact with the AR-guided scenario through voice commands, gestures, and eye-tracking. When the AR guidance for device repair begins, the guiding robot, hovering within the user's field of vision, starts moving towards the device to be repaired. The robot model maintains a fixed distance from the user; when the user stops, the robot model also stops and prompts the user to continue following the AR guidance to the device. When the user reaches the device, the guiding robot will announce via voice that they have arrived nearby. When the user clicks the "Start Repair" button near the device model, a list of items to be repaired will pop up in the user's field of vision, and the device status will be changed to "Under Repair." Whenever the user clicks a check option on the list, the guiding robot model will fly to the vicinity of the component requiring repair, and the corresponding component will light up to attract the user's attention. When the user clicks the guiding robot, the corresponding AR repair instructions will pop up in the user's field of vision. Each time a user completes a device repair item by following the AR guidance, they can click on the next item in the list. The guidance robot will then fly to the next device repair position and interact with the guidance robot again to display the AR guidance related to the next device repair. This process is repeated until the device repair guidance is completed.

[0109] 5. Based on the current device status, change the device status icon to the status icon for the next stage.

[0110] IV. Multi-form interactive control module

[0111] The augmented reality-based device management execution environment module includes voice command control, gesture control, and eye-tracking control. After receiving augmented reality guidance feedback from the AR support module, users interact with the augmented reality guidance through the multi-mode interaction control module. Depending on the operational context involved in device management, users can freely switch interaction methods. For example, in a low-noise environment, voice command control can be used; in a noisy environment, gesture control can be used; and in a noisy environment where both hands are occupied, eye-gazing interaction can be used. For example, with HoloLens 2, pinching gestures and prolonged gaze indicate confirmation of commands on the current interface.

[0112] V. AR Support Module

[0113] The AR support module includes a computer vision algorithm module and a graphics rendering module. Through the collaborative work of related hardware and software, a perfect combination of virtual guidance information and the real device management environment is achieved. By combining with augmented reality virtual guidance information, the module of the augmented reality device management system is embedded, thereby supporting the operation of the entire system.

[0114] The computer algorithm module uses a depth camera to achieve real-time pairing and overlap between real assembled parts and corresponding virtual equipment model parts.

[0115] The graphics rendering module lays out and configures augmented reality guidance information in Unity 3D, and uses the HoloLens 2 optical see-through HMD device to render and display virtual objects. When performing actual device management operations, users only need to wear HoloLens 2 glasses to obtain all AR guidance information related to the corresponding device management module.

[0116] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Any simple modifications, alterations, or equivalent structural changes made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An augmented reality-based device management system, characterized in that, include: The database module is used to store and manage the files required for the augmented reality experience process, as well as augmented reality guidance information; The multi-mode interactive control module collects user multi-mode interactive commands; The augmented reality equipment management operation execution environment module is used to call the corresponding commands of the augmented reality guided management module according to the multi-form interactive commands. The interactive commands include: viewing equipment information, planning equipment placement, and managing each stage of equipment assembly, operation, inspection, and maintenance. The Augmented Reality Bootstrapping Management module is used for command configuration and processing in the Augmented Reality device management system. The AR support module is used to configure and process commands from the augmented reality guidance management module to achieve environmental perception, tracking and registration, model recognition, virtual-real fusion, rendering of virtual objects, and display of augmented reality guidance information on the AR device, thereby guiding users to manage the device according to the augmented reality guidance information.

2. The augmented reality-based device management system according to claim 1, characterized in that, The augmented reality guidance information includes device model files, image files, audio and video files, animation sequence files, and data files.

3. The augmented reality device management system according to claim 2, characterized in that, The model files include: equipment model files assembled according to the assembly process, flight guidance robot model files, and room environment model files; The environmental model file is a three-dimensional model file containing room environment information obtained by laser scanning the target environment using a lidar scanning device; the target environment includes rooms, production areas, and floors. The device model file refers to the device model to be installed in the target environment, which is constructed by 3D scanning or industrial modeling software based on CAD drawings. The guiding robot model is used to float within the user's field of vision during augmented reality guidance, and to provide the user with operation direction guidance or directional navigation by moving to a predetermined position or orientation.

4. The augmented reality device management system and method according to claim 2, characterized in that, The image files include: pre-made 2D graphics, 3D graphics, and model texture files. The 2D graphics are used to display the equipment management manual, 2D equipment dashboard, and equipment instruction flow documents in a two-dimensional manner; the 3D graphics are used to present the equipment management manual, 2D equipment dashboard, and equipment instruction flow documents to users in a multi-dimensional way. The audio and video files include: pre-recorded audio and video files related to equipment location planning, equipment assembly, equipment operation, equipment maintenance, and equipment inspection; The animation sequence files include: pre-made animation sequence files related to equipment location planning, equipment assembly, equipment operation, equipment maintenance, and equipment inspection.

5. An augmented reality device management system according to claim 2, characterized in that, The data file includes: various operating data of the equipment and equipment status data; Various operational data of the equipment are collected by sensors connected to the actual environment and the equipment controller during the equipment operation process, including sensor data and action execution parameter data. Sensor data includes: temperature and humidity parameters, voltage or current parameters, and communication signal parameters. Action execution parameter data includes: position parameters and speed parameters. Equipment status data is collected by a device controller connected to the actual environment to collect equipment management status parameters. Different statuses include: equipment installation status, operating status, fault type, and the equipment management stage the equipment is in. The various operating data and status parameters of the device are displayed in the augmented reality scene in the form of text and icons. This is used to prompt the user of the different statuses of the current device and the various operating data of the device under the corresponding status at different stages of device management.

6. The augmented reality-based device management system according to claim 1, characterized in that, The augmented reality guidance management module includes: The display unit is used to load various augmented reality guidance information from the database: guidance information and equipment status for the equipment assembly stage, equipment operation stage, equipment inspection stage, and equipment maintenance stage, as well as the display location and content of the above augmented reality guidance information in the real environment, and display them in the augmented reality device environment of the smart terminal in the corresponding form. The command mapping and parsing unit pre-sets interactive commands for voice, gesture, and eye tracking, along with their corresponding trigger events; it parses user interactive commands in real time and converts them into corresponding trigger event instructions; the trigger events include: various guidance processes and instructions for different stages of device location planning and device management, which are displayed in file format to guide users in managing the device; The layout control unit, based on the corresponding trigger event instructions output by the command mapping and parsing unit, pre-sets the system's UI layout and the corresponding trigger operations for each model control.

7. The augmented reality-based device management system according to claim 1, characterized in that, The augmented reality device management operation execution environment module includes: The device placement planning-AR guidance execution environment unit, according to the predetermined planned placement position or the user's real-time placement position instruction, calls the AR support module to render the preset device at the predetermined position in the augmented reality virtual environment, so that the user can preview the placement effect of the current virtual device in the real environment through a smart terminal; the smart terminal includes: AR glasses, mobile tablet, mobile phone, PC. The Equipment Management - AR Guidance Execution Environment Unit includes establishing augmented reality operation execution environments corresponding to the equipment assembly stage, equipment operation stage, equipment inspection stage, and equipment maintenance stage: parts area, tool area, and operation area; parsing user interaction commands into instructions for the corresponding management stage or corresponding steps; calling the AR support module to render and simulate virtual animation scenes of equipment assembly, equipment operation, equipment inspection, and equipment maintenance in the augmented reality virtual environment; controlling the actual physical equipment to execute the management process or guided operation corresponding to each stage of the rendered simulation, and finally completing the corresponding equipment management process.

8. The augmented reality-based device management system according to claim 1, characterized in that, The rendering of virtual objects in the AR support module is as follows: the layout and configuration of augmented reality guidance information are performed in Unity 3D, and the rendering and display of virtual objects are performed using the HoloLens optical see-through HMD device; when actually performing device management-related operations, the user wears HoloLens glasses to obtain all AR guidance information related to the corresponding device management stage.

9. The augmented reality-based device management system according to claim 1, characterized in that, The multi-form interaction commands include: voice commands, gesture commands, and eye-tracking commands input by the user while wearing HoloLens glasses.

10. The device management method based on augmented reality according to claim 1, characterized in that, include: 1) The AR guidance process for equipment location planning is as follows: Step 1: Create an AR experience rendering optimization model in advance based on the target location of the device to be placed; Step 1-1: The user scans all the real environments involved in the device management using a scanning device equipped with LiDAR, generates a room environment model, and stores it in the database module. Steps 1-2: Generate a Unity asset package with an environment model using Vuforia's region target generator; Steps 1-3: Import the Unity asset package containing the environment model and the equipment model of the new equipment to be acquired into Unity; at the same time, import the status icon of whether the equipment is installed into Unity. Steps 1-4: In the Unity development environment, the user drags the device model to the corresponding target position of the environment model, and at the same time drags the device status icon above the device model to create an AR experience rendering optimization model and publish it to the smart terminal. Step 2: In a real environment, the user wears and turns on the AR glasses. Through the environment recognition function of Area Target, the virtual device model created in Step 1 is called and superimposed on the preset real environment to realize the simulation of the placement. 2) Device Management - AR Guided Process: Step 1: Create an AR experience rendering optimization model in advance based on the target location of the device to be placed; Step 2: The user approaches the device to be assembled, turns on the AR glasses, and issues device management-related commands to the AR support module through interactive instructions; Step 3: After receiving the command, the AR support module begins to search for commands related to device management in the augmented reality guidance management module. At the same time, the augmented reality guidance management module requests related augmented reality guidance information from the database, including: augmented reality guidance-related animation sequences, audio and video files, graphics, model files and device data files related to device management. Step 3: The augmented reality guidance management module loads the corresponding augmented reality guidance information into the AR support module. The AR support module then begins environmental perception, tracking and registration, model recognition, and rendering of virtual objects. Finally, it feeds back the corresponding augmented reality guidance scene to the user, providing corresponding AR guidance for device management within this environment. Step 4: After receiving the augmented reality guidance scene, the user can further interact with the augmented reality guidance scene through voice command control, gesture control, and eye tracking control, and finally complete the work of equipment assembly, operation, inspection, and maintenance; according to the current equipment status, modify the equipment status icon to the status icon of the next stage.

Images