Intelligent workshop monitoring and decision-making auxiliary system based on augmented reality and construction method thereof
Through an intelligent workshop monitoring and decision-making assistance system based on augmented reality, using MQTT protocol and MR technology, virtual scenes are built and workshop data is collected in real time, and the comprehensive monitoring and fault warning problems of workshop monitoring and decision-making assistance are solved, realizing the intelligence of the workshop and transparent management.
Patent Information
- Application Number
- CN202410973624.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art is difficult to effectively use the data driven virtual models collected by workshop sensors and CNC equipment to complete workshop monitoring and decision-making assistance, and lacks comprehensive real-time monitoring and fault warning capabilities.
Design an intelligent workshop monitoring and decision-making assistance system based on augmented reality, collects mechanical equipment status parameter data through the equipment data module, uses the MQTT Internet of Things protocol to transmit data to the MR system server module, builds a virtual device scenario, and combines the mobile application module. Track eyeball and gesture information in real time, provide human-computer interaction, output MR images, and realize full-view monitoring and fault diagnosis in the workshop.
It realizes monitoring and early warning of faults from a full perspective in the workshop, improves the efficiency of analysis and management of production data, enhances the real-time understanding of the equipment operation status and production conditions of workshop managers, and improves product quality and enterprise management capabilities.
Smart Images

Figure CN118915661A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of augmented reality technology, and specifically relates to an augmented reality-based intelligent workshop monitoring and decision support system and a construction method thereof. Background Art
[0002] At present, the research on augmented reality at home and abroad is still focused on the technology and application of augmented reality. Augmented reality technology is used by scholars at home and abroad in various fields, such as monument restoration and digital cultural heritage protection, medicine, education and training, guide systems, etc. For example, the Phoneguide system can identify a small number of exhibits in a museum in real time. The system uses the detection of Bluetooth signals to locate the user's current area to reduce the number of objects to be identified and improve the recognition speed and accuracy. The augmented reality system can also display the enhanced layer information to provide precise guidance for doctors. Doctors complete complex surgical operations based on these prompts, which can effectively improve the accuracy of the surgery.
[0003] In 2019, a simulation method for an intelligent assembly production line based on HoloLens was proposed (2019). By mathematically modeling the production scheduling process of the assembly line, the problem of multi-objective job shop scheduling was converted into a mixed integer programming model problem. The converted problem was calculated using an improved genetic algorithm to obtain a plan for adjusting production. Finally, the device status was displayed to the user through the HoloLens headset for easy observation. This method reduced the cost of production and assembly, shortened the product assembly cycle, and enhanced the user's interactive experience.
[0004] As a closed-loop link of the intelligent manufacturing system, intelligent workshop monitoring and decision-making assistance form a data stream covering the product production and manufacturing process by combining the industrial Internet and data collection technology; and on this basis, by integrating MR augmented reality technology, data interaction technology, 5G technology, automatic control technology, etc., transparent management and control of the manufacturing process is achieved, which is an important prerequisite for realizing the intelligent manufacturing system. Virtual reality (augmented reality) is an important enabling technology for transparency of the manufacturing process. On the one hand, it provides a simple and intuitive way for third parties to participate in every link of personalized product production, and on the other hand, it provides an intuitive and effective way for workshop managers to achieve transparent monitoring of the manufacturing process. The workshop virtual monitoring and decision-making assistance system based on MR augmented reality is an important link to achieve ubiquitous situational self-perception, intelligent optimization self-decision-making, active feedback self-control and full transparency of the manufacturing process.
[0005] In summary, domestic and foreign researchers have made many studies and contributions in the field of workshop monitoring and decision support, and proposed different methods to solve simulation optimization problems in different fields, which all have certain reference significance and value. Augmented reality can realize the combination of virtual and real, real-time data reading, and three-dimensional visualization, which can realize more intuitive real-time monitoring of the entire production process and plays an irreplaceable role in monitoring and visualization.
[0006] How to use the data collected by workshop sensors and CNC equipment to drive the virtual model to complete workshop monitoring and decision-making assistance is a technical problem to be solved in this application. Summary of the invention
[0007] In view of the problems existing in the above-mentioned prior art, the present application proposes an augmented reality-based intelligent workshop monitoring and decision-making support system and a construction method thereof.
[0008] In order to achieve the above purpose, the technical solution of this application is as follows:
[0009] An augmented reality-based intelligent workshop monitoring and decision-making support system, the system includes an equipment data module, an MR system server module, a mobile application terminal module and a real equipment scene module;
[0010] The equipment data module collects and summarizes the status parameter data of mechanical equipment through the intelligent gateway, and then transmits the data to the MR system server module through the MQTT Internet of Things protocol. The model processing and data superposition work are completed in the MR system server module, and the equipment MR virtual scene is constructed. The MR virtual scene sends the corresponding scene content according to the needs of the mobile application module. The mobile application module tracks the information of eyeballs and gestures in real time through MR glasses, matches tracking identifiers, provides human-computer interaction, renders scenes, and outputs MR images.
[0011] Preferably, the equipment data module includes a workshop management system module, and the workshop management system module:
[0012] In terms of production planning in related workshops, the production status is tracked in real time and the production plan is adjusted dynamically through the collected real-time data related to production equipment and production materials;
[0013] In terms of equipment fault detection in the production workshop, based on the real-time interaction between the physical equipment CNC system, sensors and virtual models, data-driven equipment status evaluation and equipment fault diagnosis are used. Through temperature monitoring, early warning is provided for equipment aging or power consumption problems.
[0014] In terms of real-time monitoring of the workshop, by establishing a three-dimensional model of the on-site manufacturing equipment and completing the virtual model mapping in the real space, the operating status monitoring and operation parameter monitoring of the manufacturing equipment are realized, and the current production details are displayed through a visual interface. The workshop staff uses the visual data to conduct production analysis.
[0015] Preferably, the equipment data module includes a data acquisition module, which monitors the operation status of the workshop in real time;
[0016] Among them, CNC equipment data includes basic information, alarm information and operation status information of CNC machine tools. Basic information of CNC machine tools includes the manufacturer name, model and equipment parameters of the CNC machine tools in the workshop. Alarm information includes the machine tool number of the fault, fault type, number of occurrences and occurrence rate information. Operation status information includes the spindle ratio and feed ratio, spindle speed, feed speed, machine operation mode, NC mode, CNC machine tool startup time, standby time, equipment and process requirements, and the number of processed parts during equipment operation.
[0017] Production material data includes product attributes, raw material types, and quantities. Warehouse inventory data includes the quantity of products entering and leaving the warehouse, and the quantity stored.
[0018] The sensor instrument data includes the data generated by the auxiliary equipment related to the robot arm, AGV car, wireless temperature collector, and three-axis vibration sensor, including the basic information and method parameters of the auxiliary equipment;
[0019] Other data include environmental data and location information. Environmental data includes the temperature, humidity and date of the workshop, and location information is the coordinates of the equipment in the workshop.
[0020] A method for constructing an intelligent workshop monitoring and decision-making support system based on augmented reality. In this method, a production workshop is used as a physical object. First, a virtual model of the workshop in a virtual scene is built, and the built model is optimized and the collision detection problem existing in the virtual model is processed. Secondly, the execution equipment, sensors and controllers of the workshop are connected through the MQTT protocol to collect real-time data of the workshop, and the data collected from the execution equipment, sensors and controllers of the workshop are transmitted to the system server. The model in the system is driven to move with the help of C# scripts, thereby completing the entire dynamic simulation work and realizing the function of real-time monitoring of the system.
[0021] Based on this development process, the first thing to be completed is the construction of the virtual scene, determining the modeling objects, modeling, optimization and collision detection methods; the second is to analyze the data source and type of the system, determine the data structure and the real-time data that needs to be collected, collect the real-time data generated in the workshop production process through the MQTT client / server mode, and then store it in the database and use C# scripts to write the real-time data into the virtual model through Socket communication. At the same time, by processing and processing the twin data stored in the database, provide data management and statistical related system service functions; finally, establish a dynamic simulation model of workshop production in Unity3D, so as to realize the workshop operation simulation and monitoring system based on digital twins;
[0022] (1) Construction of virtual scene:
[0023] (1.1) Analyze and determine the modeling solution, analyze the system design requirements, and decide to use SolidWorks and Unity3D to implement the modeling work;
[0024] (1.2) Establish a geometric model of the virtual workshop that is consistent with the real workshop:
[0025] First, determine the model structure and size according to the information in the equipment manual;
[0026] Secondly, in the modeling process, since the subsequent model needs to be connected with the data for movement, the overall model needs to be disassembled, and the individual equipment parts need to be modeled. After the sub-modules are modeled, they can be spliced. Before exporting the model, the model also needs to be optimized.
[0027] Import the built model into Unity for other subsequent processing of adding materials; create material balls by adding texture images and maps of real devices, and add corresponding materials to the read-in model to improve the virtual simulation effect of the model; then attach different parts of the map to the model by calling scripts in Unity3D to achieve the effect of virtual simulation; finally, make these models into presets to facilitate the subsequent scene construction work, thus completing the modeling operation of various geometric models in the virtual environment;
[0028] (1.3) Environmental elements for building a virtual workshop:
[0029] Use the preset body of the optimized geometric model of the workshop production equipment in the newly created scene to build the overall scene of the virtual workshop; in the overall scene construction, in order to achieve the authenticity of the model and the scene, the setting of the lights in the virtual scene is an indispensable step; add light to the virtual model by adding directional light sources, point light sources or spotlights to simulate the light source environment in the real physical environment;
[0030] (1.4) Model-driven:
[0031] The virtual model is driven by C# script based on the data of the real physical model;
[0032] (2) Data collection and transmission:
[0033] The data source in the system collects real data from the execution equipment, sensors and controllers in the workshop through the industrial Internet gateway, and publishes the data through the MQTT protocol. At the same time, on the system server side, an MQTT server is built to subscribe to the data published by the industrial Internet gateway, store and analyze the data. Finally, on the system application side, the data is finally displayed, analyzed and applied through C# scripts;
[0034] (3) System application service implementation:
[0035] The digital and virtual workshop models the three elements of the physical workshop: elements, behaviors, and rules, and obtains the mapped geometric model, behavior model, and rule model, thereby highly simulating and emulating the physical workshop. The simulation method is used to complete the full-dimensional simulation of the workshop driven by twin data.
[0036] The simulation of the production process is reflected in the motion simulation of the core assembly equipment in the workshop. The core assembly equipment in the workshop includes AGV carts, robots, and CNC equipment. For equipment with pre-set motion trajectories, the motion routes are set in advance, and for equipment with real-time motion trajectories, data is obtained through databases or Socket communications.
[0037] The simulation of production process is reflected in the support for process database resources. By connecting to the process database to obtain resource information, a workshop digital twin data model and a full-dimensional database of process information are formed; the process database information includes workshop equipment, workpieces, personnel, environment, workshop layout, equipment health model, workpiece quality inspection model, and production model; the simulation system and the process database are connected, and the equipment data and model information in the database are loaded into the simulation system; the resources in the database are divided into a tree structure according to the manufacturing method, and the production equipment resources belonging to them are selected from the sub-nodes at each level;
[0038] The simulation of equipment layout is reflected in the use of directed graph topology to describe the location of each equipment and facility in the workshop, and the use of dynamic loading methods of solid geometric models to achieve dynamic loading of standard three-dimensional model files, thereby realizing customized workshop layout and simulation optimization.
[0039] This application has achieved the following beneficial technical effects:
[0040] Compared with the existing technology, this application mainly realizes the full-view monitoring of the workshop, early warning of possible faults in the workshop, and analysis and management of production data; in terms of data transmission, the source of workshop data is analyzed and the data type is divided, and the real-time transmission of dynamic data of the workshop is realized by using the MQTT protocol; in terms of modeling, the model is lightweighted by reducing the number of facets and vertices of the model without affecting the system function;
[0041] This application mainly solves the remote online monitoring, analysis and management of workshop equipment status information, process parameter information, workshop logistics process, product quality information, energy consumption information, etc. On the other hand, combined with MR augmented reality technology, digital content that managers cannot directly experience in the real factory environment, such as big data prediction, workshop production line operation status, drawings, etc., is superimposed in front of them through smart devices to help workshop managers conduct real-time monitoring and intelligent assistance. Workshop managers can know the equipment operation status and workshop production situation in real time;
[0042] This system collects data on the manufacturing process of the smart workshop through the MR monitoring and decision-making support system of the smart workshop, establishes a data collection and monitoring system, and applies big data analysis technology and MR technology to realize the intelligence and transparency of the product manufacturing process, thereby improving product quality and the management capabilities of the enterprise. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 This is a functional application diagram of a workshop management system module in an augmented reality-based intelligent workshop monitoring and decision support system provided by the present application;
[0044] Figure 2 This is a data collection schematic diagram of an augmented reality-based intelligent workshop monitoring and decision-making support system provided by the present application;
[0045] Figure 3 This is a schematic diagram of the working principle of a mobile application terminal module in an augmented reality-based intelligent workshop monitoring and decision-making support system provided in this application. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.
[0047] This intelligent workshop monitoring and decision support system consists of four major modules, namely, equipment data module, MR system server module, mobile application module and real equipment scenario module;
[0048] The equipment data module collects and summarizes the status parameter data of mechanical equipment through the intelligent gateway, and transmits the data to the MR system server module through the MQTT Internet of Things protocol, in which it completes the model processing and data superposition work, builds the equipment MR virtual scene, and sends the corresponding scene content according to the needs of the mobile application module. The glasses of the mobile application module track the eyeballs, gestures and other information in real time, match the tracking identification, provide human-computer interaction, render the scene, and output the MR image.
[0049] The key points and protection points of the technical solution of this application are:
[0050] (1) The invention is based on the MR technology equipment detection, positioning and interactive system of the MR system server module, with the focus on detection and human-computer interaction, in which the equipment acquisition function module is used for the training of the detection network model on the one hand, and as the data input of the detection module before the interaction on the other hand;
[0051] The target detection module is used to detect and identify the images collected by the system camera;
[0052] The device positioning function analyzes and processes the detected images, combines the scene spatial position information, and calculates the physical spatial location of the target device within the detection threshold;
[0053] The main function of the model interaction function is to use some interactive method to provide MR experience for the rendered model;
[0054] The above adjacent functional modules are interconnected, and the lower modules use the data information of the upper modules as input;
[0055] (2) The invention is a mobile application module directly facing users. The mobile application module has strong interactivity and display, and can provide corresponding functions according to the goals that the system needs to achieve;
[0056] The workshop monitoring function can realize all-round and multi-angle monitoring of the workshop production process;
[0057] The production simulation function is the function of this system to simulate the production of real physical workshops, which needs to be completed through real-time data drive;
[0058] The data management and analysis function provides statistics and analysis functions for system data, including statistics on equipment operation status data, production status data, and production environment data.
[0059] The system early warning function is mainly derived from the data management and analysis functions. By analyzing the production environment data, when certain problems occur in the production environment data, it indicates that certain problems may occur in the system.
[0060] In order to realize the remote online monitoring, analysis and management of workshop equipment status information, process parameter information, workshop logistics process, product quality information, energy consumption information, etc., it is necessary to build a workshop remote monitoring system. On the other hand, combined with MR augmented reality technology, digital content that managers cannot directly experience in the real factory environment, such as big data prediction, workshop production line operation status, drawings, etc., are superimposed in front of them through smart devices to help workshop managers to conduct real-time monitoring and intelligent assistance. Workshop managers can know the equipment operation status and workshop production situation in real time, track every link, record every production step, query and back up historical data, and make statistics on the company's business resources, providing a basis for further data analysis, facilitating managers to make decisions, truly realizing the company's back-end data service platform, and realizing the informatization of enterprise resources. It mainly consists of the following modules:
[0061] 1). Workshop management system module
[0062] By analyzing the existing problems in the current workshop, the problems existing in the current workshop management system are proposed;
[0063] According to the current problem, determine the functional applications implemented by the following systems: Figure 1 As shown;
[0064] In terms of production planning in related workshops, the real-time data related to production equipment and production materials collected can be used to track the production status in real time and dynamically adjust the production plan, thereby enhancing the integration of the information and physical worlds and improving the anti-interference and intelligence of the production plan.
[0065] In terms of equipment fault detection in the production workshop, physical equipment such as CNC systems and sensors interact with virtual models in real time, and data is used to drive equipment status assessment and equipment fault diagnosis. Through temperature monitoring, early warning is provided for equipment aging or power consumption problems.
[0066] In terms of real-time workshop monitoring, by building a three-dimensional model of on-site manufacturing equipment and completing virtual model mapping in real space, the operating status monitoring and operation parameter monitoring of manufacturing equipment can be realized, the problem of blind spots in monitoring can be improved, different monitoring perspectives can be provided, and all-round monitoring of the operation status of the workshop can be achieved. The current production details can be displayed through a visual interface, such as current working hours, number of products produced, etc. Workshop staff can use the visualized data to perform production analysis and other tasks;
[0067] 2). Data acquisition module
[0068] The operation status of the workshop needs to be monitored in real time to ensure the realization of the system design purpose. For the simulation and monitoring system of workshop operations, the key to the system operation is to drive it through data, such as Figure 2 As shown;
[0069] CNC equipment data includes basic information, alarm information and operation status information of CNC machine tools. Basic information of CNC machine tools includes the manufacturer name, model, equipment parameters, etc. of the CNC machine tools in the workshop. Operation status information includes spindle ratio and feed ratio, spindle speed, feed speed, machine operation mode, NC mode, CNC machine startup time, standby time, equipment and process requirements, number of processed parts, etc. Alarm information includes the faulty machine tool number, fault type, number of occurrences, and occurrence rate information.
[0070] Production material data and warehouse inventory data. Production material data includes product attributes, raw material types, quantities, etc. Warehouse inventory mainly includes the quantity of products entering and leaving the warehouse, the quantity of storage, etc.
[0071] The sensor instrument data mainly includes the data generated by system-related auxiliary equipment such as robotic arms, AGV carts, wireless temperature collectors, and three-axis vibration sensors, including basic information and technical parameters of the auxiliary equipment;
[0072] Other data mainly include environmental data and location information. Environmental data includes the temperature, humidity and date of the workshop, and location information is the coordinates of the equipment in the workshop.
[0073] Analyze the sources and types of all data, and divide all data into two types according to the requirements for data real-time in system development: real-time data and non-real-time data. Non-real-time data is data that changes less during system operation, has a slower data update speed, and does not need to be acquired in real time. Non-real-time data includes some basic and general information related to workshop environment and equipment. Real-time data is data that is generated as workshop production proceeds and is constantly updated. It needs to be acquired in real time during system operation. Real-time data includes data generated by production equipment operation, data collected by sensors, etc.
[0074] 3).MR system server module
[0075] Through MR augmented reality technology, the virtual system can be realistically reproduced, providing users with a good visual experience and providing a good foundation for the implementation of the virtual model system at the rule model level;
[0076] 3.1) Hierarchy
[0077] The virtual model contains many three-dimensional models and there are certain dependencies between them. Due to the large number of models, the dependencies are more likely to be confused. Therefore, establishing a model with a good hierarchical relationship is helpful to enhance the logic and orderliness of the entire modeling process, and also helps to adjust the model during the development and maintenance process, which is convenient for secondary development of the model in the future.
[0078] 3.2) Interactivity
[0079] The virtual model is a module in the MR system server module that users can directly view and operate. It is also the module with the most frequent interactions. Users can understand the progress of workshop operations through the virtual model, and have a comprehensive understanding of workshop production conditions through multi-view monitoring. It also provides data analysis and mining functions during use. Therefore, the system is required to have a certain degree of interactivity, and provide higher convenience and flexibility on the basis of meeting the daily operations of system managers.
[0080] 3.3) Aesthetics
[0081] MR augmented reality technology can create holograms around the user, just like real objects. A hologram is a spatial image of a virtual object. Its visual effect is no different from that of a real object. Users can see holograms from various angles and distances. The difference between holograms and real objects is that holograms are made of light rather than real matter. When users touch holograms, they will not feel any physical resistance because they do not really exist. HoloLens renders holograms directly in the space in front of the user's eyes. Users can see both holograms and the surrounding environment, respond to gaze, gestures and voice commands, and interact with the real world around them.
[0082] This application builds and optimizes the virtual model of the workshop, performs holographic projection of the workshop operation, workshop process, equipment layout and other aspects, performs collision detection on objects that may collide in the production workshop in the model, and prevents the occurrence of model penetration problems in the model environment. Secondly, the real-time data generated by the workshop equipment and sensors is collected through the MQTT protocol, and the collected data is used to drive the movement of the virtual system through C# scripts. Finally, the collected data is analyzed and managed to complete the development of the mobile application module function;
[0083] 4). Mobile application module
[0084] The mobile application module is a user-oriented module provided by this system. This module has strong interactivity and display, and can provide corresponding functions according to the goals that the system needs to achieve, such as Figure 3 As shown, it mainly includes the following modules:
[0085] The data management and analysis function provides statistics and analysis functions for system data, including statistics on equipment operation status data, production status data, and production environment data. According to certain rules, it analyzes the optimal efficiency of the workshop, analyzes efficiency bottlenecks, and counts the operation efficiency and failures of CNC machine tools and other equipment in the workshop during operation time. Through these data, users can more easily understand the operation status of the entire workshop, adjust production plans or production progress, and improve the management efficiency of production managers.
[0086] The system early warning function is mainly derived from the data management and analysis functions. By analyzing the production environment data, when certain problems occur in the production environment data, it indicates that certain problems may occur in the system. For example, if the temperature data collected by the temperature sensor has been at a high level, then the equipment monitored by the temperature sensor may have problems such as excessive power or decreased mechanical strength of metal equipment. Through this monitoring, equipment problems can be discovered as early as possible.
[0087] The present application also provides a method for constructing an intelligent workshop monitoring and decision-making support system based on augmented reality. In this method, a production workshop is used as a physical object. First, a virtual model of the workshop in a virtual scene is built, and the built model is optimized and the collision detection problem existing in the virtual model is processed. Secondly, the execution equipment, sensors and controllers of the workshop are connected through the MQTT protocol to collect real-time data of the workshop, and the data collected from the execution equipment, sensors and controllers of the workshop are transmitted to the system server. The model in the system is driven to move with the help of C# scripts, thereby completing the entire dynamic simulation work and realizing the function of real-time monitoring of the system.
[0088] Based on this development process, the first thing to be completed is the construction of the virtual scene, determining the modeling objects, modeling, optimization and collision detection methods; the second is to analyze the data source and type of the system, determine the data structure and the real-time data that needs to be collected, collect the real-time data generated in the workshop production process through the MQTT client / server mode, and then store it in the database and use C# scripts to write the real-time data into the virtual model through Socket communication. At the same time, by processing and processing the twin data stored in the database, provide data management and statistical related system service functions; finally, establish a dynamic simulation model of workshop production in Unity3D, so as to realize the workshop operation simulation and monitoring system based on digital twins;
[0089] (1) Construction of virtual scene:
[0090] (1.1) Analyze and determine the modeling solution, analyze the system design requirements, and decide to use SolidWorks and Unity3D to implement the modeling work;
[0091] (1.2) Establish a geometric model of the virtual workshop that is consistent with the real workshop:
[0092] First, determine the model structure and size according to the information in the equipment manual;
[0093] Secondly, in the modeling process, since the subsequent model needs to be connected with the data for movement, the overall model needs to be disassembled, and the individual equipment parts need to be modeled. After the sub-modules are modeled, they can be spliced. Before exporting the model, the model also needs to be optimized.
[0094] Import the built model into Unity for other subsequent processing of adding materials; create material balls by adding texture images and maps of real devices, and add corresponding materials to the read-in model to improve the virtual simulation effect of the model; then attach different parts of the map to the model by calling scripts in Unity3D to achieve the effect of virtual simulation; finally, make these models into presets to facilitate the subsequent scene construction work, thus completing the modeling operation of various geometric models in the virtual environment;
[0095] (1.3) Environmental elements for building a virtual workshop:
[0096] Use the preset body of the optimized geometric model of the workshop production equipment in the newly created scene to build the overall scene of the virtual workshop; in the overall scene construction, in order to achieve the authenticity of the model and the scene, the setting of the lights in the virtual scene is an indispensable step; add light to the virtual model by adding directional light sources, point light sources or spotlights to simulate the light source environment in the real physical environment;
[0097] (1.4) Model-driven:
[0098] The virtual model is driven by C# script based on the data of the real physical model;
[0099] (2) Data collection and transmission:
[0100] The data source in the system collects real data from the execution equipment, sensors and controllers in the workshop through the industrial Internet gateway, and publishes the data through the MQTT protocol. At the same time, on the system server side, an MQTT server is built to subscribe to the data published by the industrial Internet gateway, store and analyze the data. Finally, on the system application side, the data is finally displayed, analyzed and applied through C# scripts;
[0101] (3) System application service implementation:
[0102] The digital and virtual workshop models the three elements of the physical workshop: elements, behaviors, and rules, and obtains the mapped geometric model, behavior model, and rule model, thereby highly simulating and emulating the physical workshop. The simulation method is used to complete the full-dimensional simulation of the workshop driven by twin data.
[0103] The simulation of the production process is reflected in the motion simulation of the core assembly equipment in the workshop. The core assembly equipment in the workshop includes AGV carts, robots, and CNC equipment. For equipment with pre-set motion trajectories, the motion routes are set in advance, and for equipment with real-time motion trajectories, data is obtained through databases or Socket communications.
[0104] The simulation of production process is reflected in the support for process database resources. By connecting to the process database to obtain resource information, a workshop digital twin data model and a full-dimensional database of process information are formed; the process database information includes workshop equipment, workpieces, personnel, environment, workshop layout, equipment health model, workpiece quality inspection model, and production model; the simulation system and the process database are connected, and the equipment data and model information in the database are loaded into the simulation system; the resources in the database are divided into a tree structure according to the manufacturing method, and the production equipment resources belonging to them are selected from the sub-nodes at each level;
[0105] The simulation of equipment layout is reflected in the use of directed graph topology to describe the location of each equipment and facility in the workshop, and the use of dynamic loading methods of solid geometric models to achieve dynamic loading of standard three-dimensional model files, thereby realizing customized workshop layout and simulation optimization.
[0106] The methods and devices not described in detail in this application are all prior art and will not be described in detail.
[0107] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An intelligent workshop monitoring and decision support system based on augmented reality, characterized in that: The system includes device data module, MR system server module, mobile application module and real device scenario module; The equipment data module collects and summarizes the status parameter data of mechanical equipment through the intelligent gateway, and then transmits the data to the MR system server module through the MQTT Internet of Things protocol. The model processing and data superposition work are completed in the MR system server module, and the equipment MR virtual scene is constructed. The MR virtual scene sends the corresponding scene content according to the needs of the mobile application module. The mobile application module tracks the information of eyeballs and gestures in real time through MR glasses, matches tracking identifiers, provides human-computer interaction, renders scenes, and outputs MR images.
2. The intelligent workshop monitoring and decision support system based on augmented reality according to claim 1, characterized in that: The equipment data module includes the workshop management system module, which includes: In terms of production planning in related workshops, the production status is tracked in real time and the production plan is adjusted dynamically through the real-time data related to the production equipment and the relevant data of the production materials collected; In terms of equipment fault detection in the production workshop, based on the real-time interaction between the physical equipment CNC system, sensors and virtual models, data-driven equipment status evaluation and equipment fault diagnosis are used. Through temperature monitoring, early warning is provided for equipment aging or power consumption problems. In terms of real-time monitoring of the workshop, by establishing a three-dimensional model of the on-site manufacturing equipment and completing the virtual model mapping in the real space, the operating status monitoring and operation parameter monitoring of the manufacturing equipment are realized, and the current production details are displayed through a visual interface. The workshop staff uses the visual data to conduct production analysis.
3. The intelligent workshop monitoring and decision support system based on augmented reality according to claim 1, characterized in that: The equipment data module includes a data acquisition module, which monitors the operation status of the workshop in real time; Among them, CNC equipment data includes basic information, alarm information and operation status information of CNC machine tools. Basic information of CNC machine tools includes the manufacturer name, model and equipment parameters of the CNC machine tools in the workshop. Alarm information includes the machine tool number of the fault, fault type, number of occurrences and occurrence rate information. Operation status information includes the spindle ratio and feed ratio, spindle speed, feed speed, machine operation mode, NC mode, CNC machine tool startup time, standby time, equipment and process requirements, and the number of processed parts during equipment operation. Production material data includes product attributes, raw material types, and quantities. Warehouse inventory data includes the quantity of products entering and leaving the warehouse, and the quantity stored. The sensor instrument data includes the data generated by the auxiliary equipment related to the robot arm, AGV car, wireless temperature collector, and three-axis vibration sensor, including the basic information and method parameters of the auxiliary equipment; Other data include environmental data and location information. Environmental data includes the temperature, humidity and date of the workshop, and location information is the coordinates of the equipment in the workshop.
4. A method for constructing an intelligent workshop monitoring and decision support system based on augmented reality, characterized in that: In this method, the production workshop is taken as the physical object. First, a virtual model of the workshop in the virtual scene is built, and the built model is optimized and the collision detection problem existing in the virtual model is processed. Secondly, the execution equipment, sensors and controllers of the workshop are connected through the MQTT protocol to collect the real-time data of the workshop. The data collected from the execution equipment, sensors and controllers of the workshop are transmitted to the system server. The model in the system is driven to move with the help of C# scripts, thereby completing the entire dynamic simulation work and realizing the function of real-time monitoring of the system. Based on this development process, the first thing to be completed is the construction of the virtual scene, determining the modeling objects, modeling, optimization and collision detection methods; Secondly, the data source and type of the system are analyzed to determine the data structure and real-time data that needs to be collected. The real-time data generated during the workshop production process is collected through the MQTT client / server mode, and then stored in the database. The real-time data is written into the virtual model through Socket communication using C# scripts. At the same time, the twin data stored in the database is processed and processed to provide data management and statistical related system service functions. Finally, a dynamic simulation model of workshop production is established in Unity3D, thereby realizing a workshop operation simulation and monitoring system based on digital twins. (1) Construction of virtual scene: (1.1) Analyze and determine the modeling solution, analyze the system design requirements, and decide to use SolidWorks and Unity3D to implement the modeling work; (1.2) Establish a geometric model of the virtual workshop that is consistent with the real workshop: First, determine the model structure and size according to the information in the equipment manual; Secondly, in the modeling process, since the subsequent model needs to be connected with the data for movement, the overall model needs to be disassembled, and the individual equipment parts need to be modeled. After the sub-modules are modeled, they can be spliced; before exporting the model, the model also needs to be optimized; Import the built model into Unity for other subsequent processing of adding materials; create material balls by adding texture images and maps of real devices, and add corresponding materials to the read-in model to improve the virtual simulation effect of the model; then attach different parts of the map to the model by calling scripts in Unity3D to achieve the effect of virtual simulation; finally, make these models into presets to facilitate the subsequent scene construction work, thus completing the modeling operation of various geometric models in the virtual environment; (1.3) Environmental elements for building a virtual workshop: Use the preset body of the optimized geometric model of the workshop production equipment in the newly created scene to build the overall scene of the virtual workshop; in the overall scene construction, in order to achieve the authenticity of the model and the scene, the setting of the lights in the virtual scene is an indispensable step; add light to the virtual model by adding directional light sources, point light sources or spotlights to simulate the light source environment in the real physical environment; (1.4) Model-driven: The virtual model is driven by C# script based on the data of the real physical model; (2) Data collection and transmission: The data source in the system collects real data from the execution equipment, sensors and controllers in the workshop through the industrial Internet gateway, and publishes the data through the MQTT protocol. At the same time, on the system server side, an MQTT server is built to subscribe to the data published by the industrial Internet gateway, store and analyze the data. Finally, on the system application side, the data is finally displayed, analyzed and applied through C# scripts; (3) System application service implementation: The digital and virtual workshop models the three elements of the physical workshop: elements, behaviors, and rules, and obtains the mapped geometric model, behavior model, and rule model, so as to simulate and simulate the physical workshop with high realism. The simulation method is used to complete the full-dimensional simulation of the workshop driven by twin data. The simulation of the production process is reflected in the motion simulation of the core assembly equipment in the workshop. The core assembly equipment in the workshop includes AGV carts, robots, and CNC equipment. For equipment with pre-set motion trajectories, the motion routes are set in advance, and for equipment with real-time motion trajectories, data is obtained through databases or Socket communications. The simulation of production process is reflected in the support for process database resources. By connecting to the process database to obtain resource information, a workshop digital twin data model and a full-dimensional database of process information are formed; the process database information includes workshop equipment, workpieces, personnel, environment, workshop layout, equipment health model, workpiece quality inspection model, and production model; the simulation system and the process database are connected, and the equipment data and model information in the database are loaded into the simulation system; the resources in the database are divided into a tree structure according to the manufacturing method, and the production equipment resources belonging to them are selected from the sub-nodes at each level; The simulation of equipment layout is reflected in the use of directed graph topology to describe the location of each equipment and facility in the workshop, and the use of dynamic loading methods of solid geometric models to achieve dynamic loading of standard three-dimensional model files, thereby realizing customized workshop layout and simulation optimization.
Citation Information
Cited By
Smart factory control method and device based on digital twinning, medium and product
CN119987303A
