Automatic sorting digital twin system and construction method

Abstract

The invention belongs to the technical field of automatic industrial sorting, and discloses an automatic sorting digital twin system and a construction method. The construction method comprises the following steps: constructing physical entities for executing different operation functions in automatic sorting based on a physical entity layer, and sending the physical entities to a digital twin platform layer; the digital twin platform layer constructs a three-dimensional virtual model synchronous with different operation functions executed by the physical entity through three-dimensional design software; the three-dimensional virtual model is dynamically updated in real time; and loading the three-dimensional virtual model to a three-dimensional engine of a visual application layer through a communication protocol or a transmission network, monitoring the real-time states of different operation functions of the physical entity in an industrial automatic sorting simulation scene, and visually displaying the real-time states through a user interface. The digital twinning platform of the automatic sorting system is constructed, and visual interaction, real-time data monitoring and virtual and real synchronous operation of the automatic sorting digital twinning system are achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of automated industrial sorting, and in particular relates to an automatic sorting digital twin system and a construction method thereof. Background Art

[0002] In the process of automated industrial sorting, the coordinated cooperation of different mechanical equipment can realize the automatic sorting of materials, which significantly improves the economic benefits. However, some enterprises have deficiencies in the monitoring and management of automated production, and lack good management methods and monitoring technologies. Current industrial monitoring mainly relies on traditional video monitoring, which cannot accurately and comprehensively monitor the operation of system equipment. In addition, since the management systems and control systems designed by different manufacturers and developers use different standards and protocols, and lack a unified integrated platform for information exchange, the industrial manufacturing information management system is heterogeneous and diversified, information exchange is difficult and untimely, and the information island phenomenon is serious. It is impossible to collect data, monitor the status and control industrial equipment in real time, which seriously affects the operation efficiency.

[0003] In view of the above problems, in order to promote the management of physical system operation and data and ensure the normal operation of the automated sorting system, the equipment operation status, operation parameters and surrounding environment of the automated sorting process can be monitored, so that the industrial sorting system can gradually develop from the current automation to the direction of system intelligence and information management. Since digital twin technology allows real-time data collection from actual equipment and systems and maps it to a virtual model, by real-time monitoring of the operating status of the sorting system, problems can be discovered and solved in a timely manner to avoid downtime and failures. Therefore, the present invention proposes to establish a digital twin monitoring system for automatic sorting scenarios.

[0004] At present, the difficulties in realizing the digital twin monitoring system of the automatic sorting process mainly include the following aspects: First, high-precision modeling and real-time data synchronization: It is necessary to carry out high-precision three-dimensional modeling of mechanical equipment and work scenes, and realize real-time data synchronization between physical equipment and digital twin models, integrate real-time monitoring and data-driven virtual modeling, and ensure low latency and high reliability of data transmission to keep the virtual model highly consistent with the actual equipment; second, data acquisition and transmission communication: There are a variety of communication interfaces and protocols in industrial manufacturing scenarios, which makes the data have multi-source heterogeneity. By using high-speed, low-latency data transmission protocols, the problem of multi-source heterogeneity of data can be solved, and real-time data processing of physical space and information space can be realized; third, monitoring of automated industrial manufacturing scenarios: It is necessary to develop modules that can monitor equipment status and system parameters in real time, integrate three-dimensional visualization monitoring, model motion simulation and real-time monitoring functions into one system, and realize collaborative work. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the disclosed embodiments of the present invention provide an automatic sorting digital twin system and a construction method. The purpose of the present invention is to promote the management of physical system operation and data and ensure the normal operation of the automated sorting system. The equipment operating status, operating parameters and surrounding environment of the automatic sorting process can be monitored, so that the industrial sorting system can gradually develop from the current automation to the direction of system intelligence and information management. Since digital twin technology allows real-time data collection from actual equipment and systems and maps it to a virtual model, by real-time monitoring of the operating status of the sorting system, problems can be discovered and solved in a timely manner to avoid downtime and failures. Therefore, the present invention proposes to establish a digital twin monitoring system for automatic sorting scenarios.

[0006] The technical solution is as follows: A method for constructing an automatic sorting digital twin system, comprising:

[0007] S1, build physical entities that perform different operating functions in automatic sorting based on the physical entity layer, and send the physical entities to the digital twin platform layer through a communication protocol or a transmission network; the different operating functions include automatic material transmission, robot handling, visual inspection, and human-computer interaction, and the physical entities include robots, conveyor belts, slide modules, and cameras;

[0008] S2, the digital twin platform layer receives the data information of the physical entity, and constructs a three-dimensional virtual model synchronized with the execution of different operating functions of the physical entity through three-dimensional design software; and dynamically updates the three-dimensional virtual model in real time to be consistent with the real-time status of different operating functions of the physical entity;

[0009] S3, loading the three-dimensional virtual model with consistent real-time status of different operating functions of the physical entity into the three-dimensional engine of the visualization application layer through the communication protocol or transmission network, monitoring the real-time status of different operating functions of the physical entity in the industrial automatic sorting simulation scenario, and visualizing it through the user interface.

[0010] In step S1, the automatic transmission, robot handling, visual inspection and human-computer interaction of the materials are respectively implemented through the material transmission module, robot handling module, visual inspection module and human-computer interaction module divided by the automatic sorting system, and the operation of the above modules is controlled by the PLC controller to complete the automatic sorting.

[0011] The material conveying module is located at the front end of the automatic sorting system and is used to move materials from the material placement area of ​​the conveyor belt to the material grabbing area of ​​the robot; the material conveying module includes a conveyor belt and a photoelectric detection sensor component. The conveyor belt continuously transports materials and is driven by an AC motor to adjust the speed to adapt to different material flows and processing requirements. The mode, forward rotation and reverse rotation of the conveyor belt are all controlled by a PLC controller;

[0012] The robot handling module is composed of a robot, a slide module and a pneumatic suction cup, and is used to complete the grabbing and placement of materials;

[0013] The visual inspection module is used for visual monitoring of materials;

[0014] The human-machine interaction module displays the human-machine interface through a touch screen, and realizes the switching between the automatic mode and the manual mode in the automatic sorting system through interface design;

[0015] The PLC controller also controls the robot, the slide module, the pneumatic suction cup and the camera to work together.

[0016] The visual inspection module uses template matching technology to compare the image to be inspected with a predefined standard template to find the area in the image to be inspected that matches the template; performs affine transformation on the image to ensure that the image is processed in a standard coordinate system to eliminate distortion or displacement that may occur during the shooting process; uses image binarization to convert the grayscale image into a binary image so that the target object and the background are obvious; counts the pixels of the binarized image to count the number of black pixels in a specific area, compares the statistical results with the preset standard to determine whether the image is qualified, and then sends the image inspection result to the PLC controller and saves the data.

[0017] In step S2, the 3D design software includes a Windows-based 3D CAD system SolidWorks, a 3D modeling software 3ds Max, and a 3D engine Unity;

[0018] Building a 3D virtual model includes:

[0019] (1) Check and sort the physical entity data, obtain the shape and size information of the physical entity, and measure and verify the physical data of each physical entity;

[0020] (2) Each physical entity is divided into multiple independent sub-modules, which are drawn using SolidWorks, and the three-dimensional virtual models drawn from the multiple independent sub-modules are assembled as a whole to be consistent with the physical entity in size, proportion, and shape;

[0021] (3) After saving the obtained 3D virtual model, import it into 3ds Max for lightweight processing to reduce the number of faces and vertices;

[0022] (4) Import the processed 3D virtual model into Unity in *.fbx format for loading and adjustment.

[0023] In step (3), lightweight processing includes:

[0024] 1) Export the geometric model file *.stl from SolidWorks and check whether the 3D virtual model information is complete;

[0025] 2) Use 3ds Max software to delete lines and faces that are not involved in actual operation control, streamline the model mesh, and reduce the number of hidden bodies;

[0026] In step (4), loading and adjustment are performed, including:

[0027] Loading is to import the robot's 3D virtual model into Unity. The robot base coordinate system is located at the origin of the world coordinate system. Through the setting of parent-child objects, the parts are connected to form the robot's sub-object hierarchy. After the setting is completed, all sub-object hierarchy models are dragged into the resource folder to convert each part into an object in Unity3D. The initial state of each joint is adjusted to be consistent with the initial state of the real robot. The joints of the robot's 3D virtual model are arranged in a parent-child structure.

[0028] Adjustment is to adjust the axis of the 3D virtual model and the conversion adjustment of the unit when importing the 3D virtual model from 3ds Max to Unity.

[0029] In step S3, it is loaded into Unity, a 3D engine of the visualization application layer, including: creating and exporting 3D virtual models of robots, conveyor belts, and slide modules in the industrial automatic sorting scene through SolidWorks, importing the 3D virtual models into 3ds Max, performing lightweight surface reduction processing, adjusting the center coordinates of the 3D virtual model, setting the hierarchy, and finally exporting it to a *.fbx format file, and loading the 3D virtual model into Unity.

[0030] During the lightweight and surface reduction process, the center coordinates of the 3D virtual model are adjusted and the hierarchical structure is set, including: exporting the geometric model file *.stl from SolidWorks to check whether the 3D virtual model information is complete; using 3ds Max software to delete lines and surfaces that are not involved in actual operation control, streamline the model mesh, and reduce the number of hidden bodies.

[0031] Another object of the present invention is to provide an automatic sorting digital twin system, the system comprising:

[0032] The physical entity layer is used to build the entity of the automatic sorting system and realize the control of the automatic sorting system to ensure the normal operation and sorting of the automatic sorting system;

[0033] The data transmission layer is located between the physical entity layer and the digital twin platform layer. It is used for data collection, transmission and synchronization. It uses communication protocols or data transmission networks to transmit data from the physical entity layer to the digital twin platform layer in real time to ensure the consistency and synchronization between the three-dimensional virtual model and the actual automatic sorting system.

[0034] The digital twin platform layer is used to establish a three-dimensional virtual model corresponding to the physical entity of the physical entity layer. By receiving data from the data transmission layer, the three-dimensional virtual model is dynamically updated to reflect the actual status and operation of the physical entity.

[0035] The visualization application layer (4) enables users to transparently and comprehensively monitor and analyze physical entities through the three-dimensional virtual model of the digital twin platform layer through the user interface, monitor the operating status of physical entities, view operating data, conduct environmental monitoring, optimize performance, and obtain decision support.

[0036] Furthermore, the automatic sorting system includes: a physical entity, a sensing part, a network communication, and a control and execution design;

[0037] The conveyor belt, robot, slide module and camera are physical entities used for actual sorting operations;

[0038] Material detection sensor, temperature and humidity sensor, air pump pressure sensor, and camera constitute the sensing part;

[0039] TCP / IP communication, RS485, and driver interface are defined as the network communication part, which is used to realize the communication interaction between automatic sorting equipment;

[0040] The control and execution design includes PLC controller, robot control system, human-machine interaction panel and visual inspection system to realize the control and execution of the corresponding automatic sorting physical entities.

[0041] Combining all the above technical solutions, the beneficial effects of the present invention are as follows: The present invention proposes the concept of using digital twin technology to establish a digital twin monitoring system for automatic sorting scenarios. Next, the architecture of the digital twin monitoring system was designed, which is mainly divided into four levels: physical entity layer, data transmission layer, digital twin platform layer and visualization application layer. According to this architectural design, the system solution of each layer is analyzed and designed in detail to ensure the accurate mapping between physical entities and virtual models in the digital twin system;

[0042] The present invention proposes a physical platform construction process for an automatic sorting system. In order to realize the automatic sorting of the system, the system is divided into four main working modules: material transmission module, robot handling module, visual inspection module and human-computer interaction module. First, a suitable PLC is selected according to the actual needs of the control system, and the input and output ports of the PLC are reasonably allocated to ensure the stability and efficient operation of the system. Next, according to the control process of the automatic sorting system, each module is program designed and tested to realize the functions of automatic material transmission, robot handling, visual inspection and human-computer interaction. The control mode of the system includes manual mode and automatic mode, so as to be flexibly adjusted under different operation requirements.

[0043] The present invention constructs a digital twin platform for an automatic sorting system. First, the requirements for establishing a virtual model are analyzed, and virtual models of each device are established according to these requirements. Three-dimensional models of robots, conveyor belts, and slide modules in industrial automatic sorting scenes are created and exported through SolidWorks, and these models are imported into 3ds Max, and lightweight surface reduction processing is performed, the center coordinates of the model are adjusted, and appropriate hierarchical structures are set. Optimization operations such as exporting to *.fbx format files and loading these models into Unity complete the construction of a digital twin platform for the automatic sorting system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure;

[0045] Figure 1 It is an architecture diagram of the automatic sorting digital twin system provided by an embodiment of the present invention;

[0046] Figure 2 It is a schematic diagram of the framework of the automatic sorting system (physical system) provided by an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of network communication in an automatic sorting system provided by an embodiment of the present invention;

[0048] Figure 4 It is a flow chart of a method for constructing an automatic sorting digital twin system provided by an embodiment of the present invention;

[0049] Figure 5 is a physical device modeling flow chart provided by an embodiment of the present invention;

[0050] Figure 6 It is a schematic diagram of the operation and control principle of the automatic sorting system provided by an embodiment of the present invention;

[0051] Figure 7It is a structural principle diagram of a PLC controller provided by an embodiment of the present invention;

[0052] Figure 8 It is a schematic diagram of a PLC controller control program of a material conveying module provided in an embodiment of the present invention;

[0053] Fig. 9 Adjust the interface diagram for the coordinate axis center of the three-dimensional virtual model;

[0054] Fig.10 Set up interface diagrams for system units;

[0055] Fig.11 The conveyor belt in the real-time linkage between the virtual space and the physical space of the automatic sorting system transports the materials to the designated location, and the robot carries the materials at this location;

[0056] Fig.12 This is a rendering of the robot moving materials from the conveyor belt to the visual inspection area in the real-time linkage between the virtual space and the physical space of the automatic sorting system;

[0057] Fig.13 The robot sorts the materials to the corresponding area according to the visual inspection results after the real-time linkage between the virtual space and the physical space of the automatic sorting system;

[0058] Fig.14 This is the flow chart of the automatic sorting system;

[0059] Fig.15 This is the interaction diagram between PLC and robot control;

[0060] Fig.16 Schematic diagram of the joint link;

[0061] Fig.17 This is the SCARA robot link coordinate system diagram;

[0062] Fig.18 Overall implementation framework diagram of automatic sorting digital twin;

[0063] Fig.19 This is the storage process diagram of twin data;

[0064] In the figure: 1. Physical entity layer; 2. Data transmission layer; 3. Digital twin platform layer; 4. Visualization application layer. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0066] The innovation of the present invention is that the present invention proposes and implements an automatic sorting monitoring system based on digital twin technology for the first time. The system ensures accurate mapping and real-time synchronization between physical devices and virtual models through a four-layer architecture: physical entity layer, data transmission layer, digital twin platform layer, and visualization application layer. In terms of the physical platform, an automated sorting system was constructed, covering multiple working modules such as material transfer, robot handling, visual inspection, and human-computer interaction, realizing the automated operation of the system. In addition, by using SolidWorks and 3ds Max software, key equipment models in the automatic sorting system, including robots, conveyor belts, and slide modules, were created and optimized, and these high-precision three-dimensional models were imported into the Unity engine, successfully building a digital twin platform, and providing a virtual, real-time interactive monitoring environment for the automatic sorting system. These innovations not only improve the monitoring efficiency and accuracy of the automatic sorting system, but also provide new technical solutions for the field of intelligent manufacturing.

[0067] Example 1: The automatic sorting digital twin system provided by the embodiment of the present invention uses digital twin technology to establish a three-dimensional visualization monitoring system. First, a digital twin monitoring system architecture is established. The automatic sorting digital twin system (digital twin monitoring system) constructed by the present invention is divided into four layers, namely, physical entity layer 1, data transmission layer 2, digital twin platform layer 3 and visualization application layer 4. These layers are responsible for different functions, and the hierarchical architecture is as follows: Figure 1 Specifically including:

[0068] Physical entity layer 1 is used to build the entity of the automatic sorting system and realize the control of the automatic sorting system to ensure the normal operation and sorting of the automatic sorting system; the physical entity layer 1 is the basic layer of the system, which is used to accurately build the entity of the automatic sorting system and realize the control of the system to ensure its normal operation and accurate sorting. This layer is the actual operation object of the digital twin system;

[0069] Data transmission layer 2, located between the physical entity layer 1 and the digital twin platform layer 3, is used for data collection, transmission and synchronization. It uses communication protocols or data transmission networks to transmit the data of the physical entity layer 1 to the digital twin platform layer 3 in real time to ensure the consistency and synchronization of the three-dimensional virtual model with the actual automatic sorting system;

[0070] Digital twin platform layer 3 is used to establish a three-dimensional virtual model corresponding to the physical entity of physical entity layer 1. By receiving data from data transmission layer 2, the three-dimensional virtual model is dynamically updated to reflect the actual status and operation of the physical entity.

[0071] The visualization application layer 4 enables users to transparently and comprehensively monitor and analyze physical entities through the three-dimensional virtual model of the digital twin platform layer 3 through the user interface, monitor the operating status of physical entities, view operating data, conduct environmental monitoring, optimize performance, and obtain decision support.

[0072] For example, in the automatic sorting process, combined with the needs of material handling and sorting, the automatic sorting system (physical system) is divided into four parts: physical equipment, perception part, network communication, and control and execution design. Figure 2 As shown;

[0073] Conveyor belts, robots, slide modules and cameras are physical equipment, i.e., automatic sorting equipment, used for actual sorting operations;

[0074] Material detection sensors, temperature and humidity sensors, air pump pressure sensors, cameras, etc. are the sensing parts;

[0075] TCP / IP communication, RS485, and driver interface are defined as the network communication part, which is used to realize the communication interaction between automatic sorting equipment;

[0076] The control and execution design includes a PLC controller, a robot control system, a human-computer interaction panel and a visual inspection system, which realizes the control and execution of the corresponding automatic sorting equipment; the physical entity layer 1 of the present invention is mainly reflected in the establishment, coordination and control of the automatic sorting system, and collects the real-time status data and environmental information of the sorting equipment through sensors and communication networks, and uploads the real-time data to the twin data service system.

[0077] The data transmission layer 2 is between the physical entity layer 1 and the digital twin platform layer 3 and is responsible for data collection, transmission and synchronization. It uses communication protocols and data transmission technology to transmit the data of the physical entity layer 1 to the digital twin platform layer 3 in real time to ensure the consistency and synchronization of the virtual model with the actual automatic sorting system; Figure 3 Network communication schematic diagram; including:

[0078] i) Unity's data collection of the robot and PLC's command transmission to the robot and camera are mainly based on the TCP / IP Socket interface;

[0079] ii) Unity collects the internal drive data of the PLC through the S7 protocol. The S7 protocol is designed specifically for Siemens PLCs and can provide efficient and low-latency communication. When transmitting large amounts of data, the protocol can reach up to 64K bytes, avoiding compatibility issues that may arise when using general protocols and providing efficient and low-latency communication performance;

[0080] iii) PLC uses Modbus RTU protocol to collect data from temperature and humidity sensors, single-phase electricity meters and noise sensors. By configuring parameters such as baud rate, start bit, parity bit, data bit and stop bit, high-speed data transmission is achieved, and Modbus TCP is used to enable Unity to periodically collect the perception data inside the PLC.

[0081] iv) PLC controls buttons, indicator lights, conveyor belts and slide modules through various methods such as switch quantity and PROFINET interface. This multi-interface control method enables PLC to accurately and flexibly manage and adjust various equipment to ensure efficient operation of the system.

[0082] The digital twin platform layer 3 establishes an accurate and hierarchical three-dimensional virtual model corresponding to the physical equipment of the physical entity layer 1, and dynamically updates the three-dimensional virtual model by receiving data from the data transmission layer 2 to reflect the actual state and operation of the physical entity;

[0083] Specifically, the digital twin platform layer 3 is an accurate mapping of the physical entity layer 1, and provides support for the operation of the physical entity layer by truly reflecting the operation of the physical entity layer in the digital twin platform layer. Among them, the virtual model in the digital twin platform layer includes the geometric shape, physical properties, behavior model and operating parameters of the physical entity. The present invention aims to simulate the operating state and behavior of the automatic sorting physical equipment, simulate it, and thus realize three-dimensional visual monitoring of the automatic sorting process.

[0084] Exemplarily, a 3D virtual model refers to an accurate digital representation of physical equipment in an automated sorting system built on digital twin technology. These 3D virtual models are created using 3D modeling software such as SolidWorks and 3ds Max, and are further processed and optimized in the Unity 3D development platform to achieve efficient and accurate operation in a virtual environment. Its functional principles are:

[0085] (1) Loading and displaying 3D virtual models: In Unity, the position, rotation, and scale of 3D virtual models are controlled through Transform components to achieve accurate placement and dynamic performance in virtual space.

[0086] (2) Data-driven dynamic update: Through real-time data collection (such as the robot's joint angles and speeds), the actions and states of the 3D virtual model can be updated in real time to reflect the current state of the physical device (physical entity).

[0087] (3) Interactivity: Users can interact with the 3D virtual model through the interface, such as simulating robot operations and adjusting equipment parameters.

[0088] (4) Visual monitoring: Combining real-time data and model animation, it provides an intuitive monitoring interface, allowing users to observe and analyze system status from different angles.

[0089] Exemplarily, the digital twin platform layer 3 first uses 3D modeling software to build a virtual model of each device. Considering that the complex model movement in the twin platform will affect the operating performance of the twin system, the virtual model is lightweighted to reduce the redundancy and complexity of the model. The lightweight model is exported and loaded into the digital twin development platform used. The model control script is edited and bound, and the logic processing rules are determined to realize the simulation function of the model;

[0090] Among them, the digital twin development platform used in the present invention is the Unity software platform, which has rich plug-ins, scripts and supporting document resources, which can accelerate the development process of the automatic sorting twin system. Its scalability allows seamless integration with IoT devices, databases, sensors, etc. in the sorting system, and realizes real-time synchronization and dynamic update of twin data. Among them, flexible programming interfaces and scripting languages ​​can be customized and developed according to the data transmission requirements in automatic sorting, enhancing the functions and performance of the twin system. The operation panel of the Unity software mainly includes 5 parts, among which the Scene panel is the main working area for editing and operating virtual models and scenes, the Game panel is used to display the real-time operation effect of the virtual model, the Hierarchy panel displays the hierarchical structure of all objects in the current scene, the Project panel displays all resources and files in the project, and the Inspector panel is used to view and edit the properties and components of the selected object. The integration of these panels in Unity makes the construction of the automatic sorting digital twin system more efficient and intuitive, and can accurately simulate and analyze the operation of the equipment.

[0091] In another exemplary embodiment, dynamically updating the virtual model includes:

[0092] First, data is collected in real time from physical devices (such as sensors, PLCs, etc.). These data include but are not limited to parameters such as position, speed, temperature, and pressure. Secondly, the collected raw data needs to be preprocessed to ensure the accuracy and availability of the data. Then, the processed data is transmitted to the digital twin platform via a wired network. Finally, on digital twin platforms such as Unity, the received data is mapped to the virtual model to achieve dynamic updating of the virtual model.

[0093] The visualization application layer 4 is the top user interface, which provides an integrated digital twin application system. Through this layer, users can realize transparent and comprehensive monitoring and analysis of physical equipment through three-dimensional simulation, monitor the operating status of physical equipment, view operating data, conduct environmental monitoring, optimize performance, and obtain decision support. In the digital twin system, the core of optimizing performance and obtaining decision support lies in the real-time dynamic mapping of physical equipment information to three-dimensional virtual space. The key to this process is to be able to monitor the operating status of physical equipment in real time, view operating data, conduct environmental monitoring, and optimize performance by analyzing these data. On the digital twin platform, the received data is mapped to the virtual model in real time. By updating the position, speed and direction of the virtual model in real time, the operator can clearly understand the dynamic behavior of the equipment and make adjustments accordingly to optimize the performance of the equipment. In addition, the system can also predict possible problems and maintenance needs of the equipment through data analysis and machine learning algorithms. This predictive maintenance reduces unexpected downtime and improves the reliability and life of the equipment. By simulating different operating scenarios, the system can also assist users in making decisions, such as providing scientific basis in resource allocation, production scheduling and energy optimization.

[0094] For example, the core of 3D visual monitoring and operation status is to dynamically map the physical equipment information to the 3D virtual space in real time, which is highly dependent on the precise support of twin data. The use of multi-source heterogeneous data transmission technology and efficient data storage solutions based on SQL Server database ensures seamless transmission and reliable storage of twin data. Subsequently, the client receives this data on the constructed digital twin platform and maps it to the 3D virtual model in real time, realizing the intuitive display of the operation status of actual equipment such as robots. 3D visual monitoring and operation status is the core of the digital twin system because it enables operators to comprehensively monitor the equipment from multiple angles and make data-based decisions quickly by dynamically mapping the status of physical equipment to the virtual space in real time and accurately. This real-time visualization capability not only enhances the interactivity of the system, allowing users to interact directly with the virtual model, but also provides strong support for decision-making by intuitively displaying the key operating parameters and environmental conditions of the equipment. In addition, it can also be used for fault diagnosis, performance prediction and maintenance optimization, and even as a training tool to improve employees' understanding of complex systems, thereby ensuring the efficient operation of the entire production process and the stability of the equipment.

[0095] Specifically, the visualization application layer 4 realizes all-round three-dimensional visualization monitoring, data monitoring, virtual and real synchronous operation, and historical status reproduction of physical entities by providing an integrated digital twin application system.

[0096] By using three-dimensional simulation to achieve visual monitoring of the system, the viewing angle can be switched according to specific needs, and the best viewing angle can be freely obtained in the virtual space. The interactivity of the monitoring system is enhanced, the status and behavior of the automatic sorting equipment can be tracked in real time, the sorting equipment can be operated synchronously in the virtual space and the physical space, and all-round dynamic monitoring of the sorting scene can be achieved, so that the operator can fully understand the equipment operation and the surrounding environment.

[0097] The operating status of the sorting equipment, system energy consumption and surrounding environment monitoring information are displayed in real time, and these monitoring data are integrated on one interface to intuitively monitor the overall operation of the automatic sorting system. This makes it convenient for engineering and technical personnel to grasp the overall information of the automatic sorting system, which not only simplifies information management, but also improves the accessibility and comprehensibility of data, and reduces the occurrence of information islands.

[0098] The system operation data is continuously stored in the database, and the historical data is used to drive the virtual model to reproduce the historical status to help users trace the root cause of the fault.

[0099] It can be seen from the above embodiments that in the process of automated industrial sorting, current industrial monitoring mainly relies on traditional video monitoring, which cannot accurately and comprehensively monitor the operation of system equipment. Enterprises have deficiencies in the monitoring and management of automated production, lack of good management methods and monitoring technologies, and because the management systems and control systems designed by different manufacturers and developers adopt different standards and protocols, there is a lack of a unified integrated platform for information exchange, resulting in heterogeneity and diversity in the industrial manufacturing information management system, difficult and untimely information exchange, serious information island phenomenon, and inability to accurately collect data, monitor status and control industrial equipment in real time, which seriously affects the efficiency of operational operations.

[0100] In contrast, the automatic sorting digital twin system of the present invention can use the digital twin three-dimensional visualization monitoring technology to more comprehensively monitor the operating status of the real-time automatic sorting system. According to specific needs, the viewing angle can be switched, and the best viewing angle can be freely obtained in the virtual space to monitor the operating status of the entire system equipment, thereby enhancing the interactivity of the monitoring system. In addition, by combining the three-dimensional visualization monitoring of the virtual model with the visual display of real-time operation data on the UI interface, the operating status of the current automatic sorting system can be more intuitively understood. This method provides a more intuitive monitoring method. The operator can intuitively grasp the system equipment energy consumption, the motion trajectory, speed, Cartesian coordinates, joint coordinates of the sorting robot, the operating mode of the conveyor belt, the real-time position and speed of the slide module and other parameters through a graphical interface, while monitoring the temperature, humidity, noise, etc. of the surrounding environment to reduce the impact of environmental factors on the operation of the system. By monitoring the operating status of the sorting system in real time, the work efficiency of the caregivers can be greatly improved, abnormal conditions in the system operation can be discovered in time, and the probability of errors can be reduced.

[0101] During the monitoring process, the present invention continuously stores the system operation data in the database, so that technicians can query and call the data at any time to analyze the system operation status. In particular, when the system fails, technicians can use historical data to drive the virtual model to reproduce the historical status, helping them to quickly locate the cause of the failure and perform troubleshooting. This continuous data storage method also provides more reliable data support for the long-term operation of the automatic sorting system, which helps to improve the reliability and stability of the system.

[0102] Embodiment 2, as Figure 4 As shown, the embodiment of the present invention provides a method for constructing an automatic sorting digital twin system, including:

[0103] S1, build physical entities that perform different operating functions in automatic sorting based on the physical entity layer, and send the physical entities to the digital twin platform layer through a communication protocol or a transmission network; the different operating functions include automatic material transmission, robot handling, visual inspection, and human-computer interaction, and the physical entities include robots, conveyor belts, slide modules, and cameras;

[0104] S2, the digital twin platform layer receives the data information of the physical entity, and constructs a three-dimensional virtual model synchronized with the execution of different operating functions of the physical entity through three-dimensional design software; and dynamically updates the three-dimensional virtual model in real time to be consistent with the real-time status of different operating functions of the physical entity;

[0105] S3, loading the three-dimensional virtual model with consistent real-time status of different operating functions of the physical entity into the three-dimensional engine of the visualization application layer through the communication protocol or transmission network, monitoring the real-time status of different operating functions of the physical entity in the industrial automatic sorting simulation scenario, and visualizing it through the user interface.

[0106] For example, in step S1, in order to realize the automatic sorting of the system, the automatic sorting system is divided into four main working modules: material transfer module, robot handling module, visual inspection module and human-computer interaction module. Figure 6 As shown;

[0107] First, select a suitable PLC according to the actual needs of the PLC controller, and reasonably allocate the input and output ports of the PLC to ensure the stability and efficient operation of the system. Next, according to the control process of the automatic sorting system, program and test each module to realize the functions of automatic material transmission, robot handling, visual inspection and human-computer interaction. The control mode of the system includes manual mode and automatic mode, so as to flexibly adjust under different operation requirements. The structure of the PLC controller is as follows: Figure 7 As shown;

[0108] For example, the material conveying module is located at the front end of the automatic sorting system and is responsible for moving the materials from the material placement area of ​​the conveyor belt to the material grabbing area of ​​the robot. The material conveying module mainly includes components such as a conveyor belt and a photoelectric detection sensor. The conveyor belt continuously transports materials and is driven by an AC motor to adjust its speed to adapt to different material flows and processing requirements. The mode, forward rotation and reverse rotation of the conveyor belt are all controlled by a PLC controller to ensure accurate material delivery. The PLC controller control program of the material conveying module is as follows: Figure 8 As shown. PLC controller adopts PLC controller;

[0109] The robot handling module is composed of a robot, a slide module and a pneumatic suction cup, etc., and requires PLC control of the robot, the slide module, the pneumatic suction cup and the camera to work together. The movement of the robot needs to be programmed in the ARM (Automation Resource Manager) software environment.

[0110] The visual inspection module is used for visual monitoring of materials. First, the image to be inspected is compared with a predefined standard template using template matching technology to find the area in the image to be inspected that matches the template. Then, an affine transformation is performed on the image to ensure that the image is processed in a standard coordinate system to eliminate distortion or displacement that may occur during the shooting process. The grayscale image is converted into a binary image using image binarization to make the target object and the background more obvious. Subsequently, the pixels of the binarized image are counted to count the number of black pixels in a specific area. The statistical results are compared with the preset standard to determine whether the image is qualified. Then, the image inspection results are sent to the PLC controller, and the data is saved for subsequent analysis and recording.

[0111] The human-machine interaction module of the present invention uses the Wellentong TK7080iP touch screen to display the human-machine interface, and completes the interface design through EasyBuilder Pro, which can realize the switching between the automatic mode and the manual mode in the automatic sorting system;

[0112] Manual mode allows the operator to directly control every part of the sorting system. It is usually used for system debugging, maintenance, parameter setting or manual operation when a failure occurs in the automatic mode. The operator can adjust the corresponding parameters through the human-computer interaction interface.

[0113] In automatic mode, the start and stop buttons on the human-computer interaction interface can be used to control the operation of automatic mode. After starting the automatic mode, the automatic mode can automatically complete the sorting task according to the preset procedures and rules without manual intervention, thus improving efficiency and accuracy. In addition, the interface has four main functions: equipment debugging, variable monitoring, alarm recording and user management. Equipment debugging provides manual control and settings of the equipment, supports debugging and troubleshooting; variable monitoring is used to display and monitor data in the PLC controller in real time, helping data analysis and status monitoring; alarm recording is used to record and manage alarm information, and provides the function of viewing and processing alarm history; user management is used to manage user accounts and permissions to ensure the security and operation specifications of the system.

[0114] As another example, in automatic mode, the operation of automatic mode can be controlled by the start and stop buttons on the human-computer interaction interface. After starting the automatic mode, the automatic mode can automatically complete the sorting task according to preset procedures and rules without human intervention, thereby improving efficiency and accuracy.

[0115] Exemplarily, in step S2, the digital twin platform layer 3 is an accurate mapping of the physical space of the physical entity layer 1, wherein the three-dimensional virtual model includes the geometry, physical properties, behavioral model, and operating parameters of the physical device (physical entity), aiming to simulate the operating state and behavior of the physical device of the physical entity layer 1, simulate and predict it, thereby helping to optimize and improve the production process.

[0116] The three-dimensional virtual model of the present invention is completed by SolidWorks, 3ds Max and Unity. The specific construction process is as follows:

[0117] For example, Figure 5 , the basic modeling process is as follows:

[0118] (1) Consult the manuals and related materials of each physical device for sorting, including robots, conveyor belts, slide modules, test benches and other physical devices in industrial manufacturing scenarios; understand their shapes, sizes and other information, and measure and verify the physical data of each device. For example, deeply understand the physical behavior and rules of sorting equipment (such as sorting robots), ensure the accuracy of the model, and perform kinematic analysis on the robots to prepare for the production of 3D virtual models.

[0119] (2) Each physical device model is split into multiple independent sub-modules, which are drawn using SolidWorks. The three-dimensional virtual models drawn from the multiple independent sub-modules are then assembled as a whole to ensure that they are consistent with the physical device entities in terms of size, proportion, and shape.

[0120] (3) After saving the three-dimensional virtual model obtained in step (2), import it into 3ds Max for lightweight processing to reduce the number of faces and vertices and reduce the redundancy of the model to improve loading speed and performance and enhance the response speed of the twin system.

[0121] (4) Import the processed 3D virtual model into Unity in *.fbx format for loading and adjustment.

[0122] Exemplarily, the lightweight processing includes:

[0123] 1) Export the geometry model file *.stl from SolidWorks and check whether the model information of each physical device is complete.

[0124] 2) Use 3ds Max software to delete lines and faces that are not involved in actual operation control, use professional optimization modules to streamline the model mesh, and reduce the number of hidden bodies. Taking the robot model as an example, the original model has 128,308 polygons and 79,635 vertices. After lightweight processing, the model has 52,610 polygons and 32,756 vertices.

[0125] For example, lightweight surface reduction processing, adjustment of model center coordinates, setting of appropriate hierarchical structures and other optimization operations can all be set in the software.

[0126] First, export the geometric model file *.stl from SolidWorks to check whether the 3D virtual model information is complete. Then, use 3ds Max software to delete the lines and faces that are not involved in the actual operation control, and use the professional optimization module to streamline the model mesh and reduce the number of hidden bodies. Taking the robot model as an example, the original model has 128308 polygons and 79635 vertices. After lightweight processing, the model has 52610 polygons and 32756 vertices.

[0127] When importing Unity model files, the coordinate axis center of some 3D virtual models is not at their geometric center, so appropriate adjustments are required. You can select the "Affect Axis Only" option on the hierarchy panel and adjust the position of the coordinate axis so that the coordinate system coincides with the axis center of the 3D virtual model. Fig. 9 As shown;

[0128] In addition, when importing 3D virtual models from 3ds Max to Unity, you need to pay attention to the unit conversion problem. By default, the unit ratio of Unity and 3ds Max is 100:1. The default unit of the model in 3ds Max is "meter", while the default unit of Unity is "centimeter". In order to ensure that the size of the model remains consistent during the import process, you need to set the system unit in 3dsMax to "meter" to match it with Unity's properties. The system unit setting is as follows: Fig.10 shown.

[0129] Exemplarily, in step S3, the requirements for establishing a 3D virtual model are first analyzed, and 3D virtual models of various physical entities (physical equipment) are established according to these requirements. 3D virtual models of robots, conveyor belts, and slide modules in industrial automatic sorting scenes are created and exported through SolidWorks, and these 3D virtual models are imported into 3ds Max for lightweight surface reduction, adjustment of the center coordinates of the 3D virtual model, setting of appropriate hierarchical structures, and other optimization operations. Finally, they are exported as *.fbx format files, and these 3D virtual models are loaded into Unity to complete the construction of the digital twin platform of the automatic sorting system.

[0130] Example 3, verification experiment.

[0131] When conducting a verification experiment, you first need to ensure that the automatic sorting equipment in the physical space is powered on, turn on the relevant servers, and start the Unity3D engine on the virtual side to establish a connection with the server. The automatic sorting physical platform has set the equipment operation path according to the automatic sorting requirements and the equipment's operating status and behavior rules. When the operating data of each device on the physical platform is received, it can be driven to run by mapping it to the virtual model through data analysis. In addition, on the right side of the Unity interface, you can configure the joint data of the sorting robot model to control the joint movement of the physical entity of the sorting robot. Through the virtual mapping and interactive fusion of these two aspects, an operating mechanism that combines virtuality and reality and interacts in real time is formed.

[0132] In order to clearly realize the comprehensive comparison and verification between the virtual model and the actual situation, the traditional video monitoring interface is embedded in the digital twin platform interface. Video monitoring can provide real-time pictures of the operation of the physical system, helping users to instantly understand the operating status of the system and compare it with the situation in the virtual space to ensure the consistency between the virtual model and the actual system. In addition, through the dual perspectives of video monitoring and 3D visualization, users can diagnose problems more accurately. Video monitoring can help identify abnormal situations in actual operations, while 3D visualization can analyze possible sources and solutions of problems. The combination of the two monitoring methods can provide more comprehensive data and information, help users conduct comprehensive analysis, and improve the monitoring efficiency and accuracy of the system.

[0133] When the automatic sorting physical platform and the twin system are started, the operation of the automatic sorting physical platform can be observed in the traditional video surveillance screen, and the operation of the virtual space can be observed in the automatic sorting virtual space operation interface. In order to clearly observe whether the equipment operation status of the physical space and the virtual space is consistent, during the operation of the automatic sorting system, three key actions in the automatic sorting process are intercepted to show its operation in the physical space and the virtual space, such as Fig.11 In the real-time linkage between the virtual space and the physical space of the automatic sorting system, the conveyor belt transports the materials to the designated location, where the robot moves the materials; Fig.12 In the real-time linkage between the virtual space and the physical space of the automatic sorting system, the robot moves the materials from the conveyor belt to the visual inspection area; Fig.13 In the real-time linkage between the virtual space and the physical space of the automatic sorting system, after visual inspection, the robot sorts the materials to the corresponding area according to the inspection results; these Figure 11-13 The image shows that the physical space and virtual space have achieved synchronous movement in the automatic sorting system, which greatly improves the work efficiency of caregivers, can detect abnormal situations in system operation more promptly, and reduce the probability of errors.

[0134] Example 4 is further described in detail to further describe the detailed process of building the physical platform of the automatic sorting system in step S1 and related technical features.

[0135] 1. Automatic sorting system process.

[0136] This automatic sorting system is built on the basis of programmable logic controller (PLC) technology. It uses advanced technology and algorithms to achieve automatic classification and accurate sorting of items. Through efficient and accurate operation processes, the system significantly improves the efficiency and processing speed of item classification and effectively reduces labor costs. Fig.14 This is a flow chart of the automatic sorting system. The following is a detailed description of the workflow of the automatic sorting system:

[0137] (1) Place the material in the material placement area of ​​the conveyor belt, click the human-machine interaction panel, control the conveyor belt to transport the material to the robot's material grabbing area, and use the photoelectric sensor to detect whether the material has been delivered to the right place.

[0138] (2) When the material is transported to the robot material grabbing area on the conveyor belt, the PLC controller sends a material grabbing instruction to the robot. The robot moves to the grabbing area, grabs the material, and then places it in the visual inspection area.

[0139] (3) The PLC controller sends a photo inspection command to the camera. The machine vision system takes a photo of the current material and performs inspection and analysis to detect whether it is qualified. After the inspection is completed, the machine vision system sends the inspection results to the PLC.

[0140] (4) After receiving the test results, the PLC will control the robot to perform sorting work according to the test results. If the visual inspection is qualified, the sorting robot will place the material in the defective area. If the inspection result is qualified, the sorting robot will place the material in the qualified area.

[0141] 2. PLC selection and I / O design. In the operation of the automatic sorting system, the PLC controller plays a core control role. Through careful programming, it can command multiple functional modules to work together. At the same time, it can also efficiently process the real-time collected data to ensure the stability and efficiency of the system operation. Therefore, choosing a suitable PLC model is crucial for the sorting system to successfully achieve its efficient and accurate goals.

[0142] 2.1PLC selection.

[0143] According to the control requirements of PLC on sorting equipment and communication between equipment in the automatic sorting system, the present invention selects SIMATIC S7-1200 CPU 1214C DC / DC / DC PLC as the control core. This series of PLCs is compact in design and powerful in function, and is suitable for scenarios with limited space. Its modular design allows users to add various expansion modules according to needs, such as digital and analog input / output modules and communication modules, to flexibly respond to different application scenarios; CPU 1214C is equipped with a high-performance processor that can quickly execute complex logic control and data processing tasks. It comes with 14 digital inputs and 10 digital outputs, which can directly connect sensors and actuators to meet the basic needs of the sorting system, and supports increasing the number of I / O points through expansion modules to meet more complex system requirements; it provides 50KB of working memory and 10KB of loading memory, which can store and run complex control programs, and has 2KB of retentive data storage to ensure the integrity and continuity of data in the event of a power failure. In addition, the S7-1200 series PLC has an integrated Ethernet interface and supports the PROFINET communication protocol, which facilitates data exchange with other devices and systems. It also supports multiple communication protocols, such as Modbus and S7 protocols, and can communicate with a variety of devices on site.

[0144] In summary, the SIMATIC S7-1200 CPU 1214C DC / DC / DC PLC was chosen as the controller for the automatic sorting system due to its compact design, high-performance processor, rich I / O options, flexible expansion capabilities, and sufficient storage and data processing capabilities. These features enable it to meet the needs of the automatic sorting system and ensure efficient, reliable and flexible operation of the system.

[0145] Since the system needs to collect data from multiple sensors, the CB 1241RS485 communication board was selected. The CB 1241 communication board is used in conjunction with the PLC and supports the RS485 communication protocol. The system connects multiple sensors to the communication board through the RS485 bus to achieve unified data collection and transmission. The RS485 bus has the characteristics of long transmission distance and strong anti-interference ability. It is suitable for multi-point communication and can effectively collect data from each sensor and transmit it to the central control unit for processing and analysis. This configuration ensures that the system can reliably monitor and manage the data of multiple sensors, providing a basis for further control and analysis. The PLC and communication board adopt the hardware structure of the S7-1200 series PLC, such as Figure 7 shown.

[0146] 2.2PLC input and output (I / 0) design.

[0147] The SIMATIC S7-1200 CPU 1214C PLC has 14 available input terminals and 10 available output terminals. The input and output terminals of the PLC are allocated according to the actual needs of the automatic sorting system. As shown in Table 1, the terminals used in the PLC design of this system include their specific models and address allocation. The table lists in detail the function and position of each terminal in the system.

[0148] Table 1 I / O allocation table

[0149]

[0150]

[0151] 3. System electrical design

[0152] 3.1 Power supply and control equipment.

[0153] The control cabinet of this system mainly contains Siemens SIMATIC S7-1200 control system, low voltage control and power distribution products and human-machine interaction panel. Table 2 shows the main control equipment and components of the control cabinet.

[0154] Table 2 Main control equipment of control cabinet

[0155]

[0156] The equipment in Table 2 realizes the control, monitoring and protection of the electrical equipment in the electric control box through the cooperation of various parts. Among them, the SIMATIC S7-1200 control system is the core control device, which communicates with other devices through multiple interfaces such as digital input and output modules, analog input and output modules and Ethernet communication modules. Low-voltage control and power distribution products provide necessary control and power distribution functions for electrical equipment. Circuit breakers are used for overload and short-circuit protection, contactors are used for motor start and stop, and control buttons and indicator lights are used for operation and monitoring. The human-machine interaction panel has a built-in MT8106iQ touch screen, buttons, and indicator lights, which can communicate with the PLC to realize real-time monitoring and operation of the PLC controller.

[0157] 3.2 Driving equipment.

[0158] The driving equipment includes a servo drive and an AC speed-regulating motor. The servo drive controls the operation of the servo motor to realize the control of the slide module, and determines the precise position and speed of the sorting robot through the slide module; the AC speed-regulating motor controls the speed of the motor by changing the input voltage and frequency of the motor, thereby realizing the control of the conveyor belt speed and operation mode.

[0159] 3.3 Execution equipment.

[0160] The executive equipment of the automatic sorting system mainly includes conveyor belts, slide modules, SCARA robots, pneumatic suction cups and cameras, etc. The movement of the slide module can accurately control the position and speed of the robot on the automatic sorting physical platform. The movement of the conveyor belt is used to transport materials. The pneumatic suction cup, cylinder and solenoid valve together constitute the material suction device. The movement of the robot and slide module is used to transport materials. The camera built at the end of the robot is used to identify and detect whether the material is qualified.

[0161] 4. System control program design.

[0162] 4.1 Material transfer design, PLC control program such as Figure 8 shown.

[0163] 4.2 Robot handling and sorting design.

[0164] (I) Robot motion control.

[0165] In ARM software, the robot's motion control is programmed using QRL (QKM Robot Language). QRL is a self-defined robot secondary development language based on Lua language, which supports single instruction execution, mathematical operations, logical operations, conditional control, loop control and thread control. It can automatically execute defined commands, perform complex operations, string processing and user-engineering interaction. By editing and sending command programs, users can develop robot integration projects in a simple and convenient manner.

[0166] To achieve PLC control of the robot, you can establish a connection between the PLC client and the robot server through the TCP / IP protocol. The following is the code for the robot server to wait for the client to connect:

[0167]

[0168] After the PLC and the robot establish a communication connection, the sending area and receiving area of ​​the PLC and the robot are initialized. The PLC writes the suction cup status, robot control instructions and slide position into the sending area, and the robot reads these instructions from the receiving area and performs the corresponding actions. After the execution is completed, the robot writes the suction cup control signal and the robot operation results into its sending area, and the PLC reads these results from the receiving area to update the status or send further instructions. Through this interactive process between the sending area and the receiving area, the PLC realizes precise control of the robot, slide module and suction cup, such as Fig.15 As shown in the diagram of the interaction between PLC and robot control.

[0169] After receiving data from the client, the robot parses and processes it, writes it into the global variable after processing, and performs corresponding tasks according to the value of the global variable. The code for data parsing is as follows:

[0170]

[0171]

[0172] According to the value of the global variable in the data exchange area, the robot performs different operations such as grabbing materials from the conveyor belt, placing materials in the visual inspection area and moving to the photo taking posture, grabbing materials from the visual inspection area, and sorting materials. This process uses GInt(18) as the transfer address of the robot control command and combines GInt(2) to confirm the result of the operation. According to different combinations of GInt(18) and GInt(2), the robot performs corresponding tasks at each workstation, thereby realizing automated material handling and visual inspection operations. Taking the control of the robot grabbing materials from the conveyor belt as an example, when GInt

[18] is 1 and GInt[2] is not 1, the robot moves to the top of the conveyor belt grabbing point. The following is the code for this part:

[0173]

[0174] (II) Control of the slide module.

[0175] The PLC control program for the servo motor of the slide module includes servo enable, servo return to zero, servo stop, jog control, absolute positioning, real-time position and speed monitoring and fault clearing functions. Servo enable is used to start the servo motor and prepare it to receive instructions; servo return to zero is used to reset the servo motor to the reference zero point; the servo stop function is used to immediately stop the operation of the servo motor; the jog control function is used to manually debug the position and speed of the servo motor; the absolute positioning control function is used to move the servo motor to a certain absolute position; real-time position and speed monitoring is used to monitor the position and speed of the servo motor in real time for easy operation and recording; the fault clearing function is used to clear the fault state of the servo motor and restore it to normal operation. These functions are realized through logical control to ensure the efficient operation and precise control of the servo motor of the slide module.

[0176] 4.3 Visual inspection design.

[0177] The visual inspection module is mainly responsible for identifying, detecting and classifying material trademarks according to preset standards. The module consists of A7500MG20 industrial camera, PLC controller, etc. The A7500MG20 camera is equipped with a high-performance image sensor, which can capture clear, high-resolution product images in real time on high-speed production lines to ensure that each product is thoroughly inspected.

[0178] Specifically, for visual monitoring of materials, first, the template matching technology is used to compare the image to be inspected with the predefined standard template to find the area in the image to be inspected that matches the template. Then, the image is affine transformed to ensure that the image is processed in a standard coordinate system to eliminate the distortion or displacement that may occur during the shooting process. The grayscale image is converted into a binary image using image binarization to make the target object and the background more obvious. The pixel count of the binarized image is then performed to count the number of black pixels in a specific area. The statistical results are compared with the preset standard to determine whether the image is qualified. The image detection results are then sent to the PLC, and the data is saved for subsequent analysis and recording.

[0179] Template matching finds the most similar area to the predefined template in the image to be detected by calculating the similarity measure. Let I(x,y) be the image to be detected, T(u,v) be the template image, and the template size be M×N. Move the template to a certain position (x0,y0) in the image. The similarity measure can be expressed as the normalized correlation coefficient between the template and the image at that position.

[0180]

[0181] In order to eliminate the influence of brightness changes, the normalized cross-correlation coefficient is used:

[0182]

[0183] In the formula, and are the average gray values ​​of the local area of ​​the image and the template respectively;

[0184] After template matching, the image is subjected to affine transformation, which maps a two-dimensional image from one coordinate system to another, including translation, rotation, scaling, and shearing. Through affine transformation, the target object in the image can be aligned to a standard reference coordinate system, eliminating geometric distortion caused by differences in shooting angle or position, and is represented by a 2*3 matrix:

[0185]

[0186] In the formula, (x, y) is the point in the original image, (x′, y′) is the transformed point, a, b, c, d are the rotation and scaling coefficients, and t x ,t y is the translation coefficient. By adjusting these parameters, image rotation, scaling, translation and other transformations can be achieved.

[0187] Subsequently, the grayscale image is converted to a black and white image using image binarization. By setting a threshold, the pixels in the image are divided into the target (black) and the background (white). Binarization can enhance the contrast of the image and make the target object more prominent. Let I(x,y) be the pixel value of the grayscale image and the binarization threshold be T. Then the binary image B(x,y) is defined as:

[0188]

[0189] The number of black pixels in the target area of ​​the binarized image is counted to evaluate the area or coverage of the target. By comparing this number with the preset standard value, it can be determined whether the target object is qualified. Let B(x,y) be the binarized image, the statistical area is S, and the total number of black pixels is:

[0190]

[0191] Count the number of all black pixels N in the binary image black Then compare this quantity with the preset standard value N threshold For comparison:

[0192]

[0193] In order to clearly display the test results, the API is used for interface development. An image display control is added to the interface to display the image taken by the camera and bind the image data source. Text boxes and chart controls are added to the interface, and the visual processing results are bound to the interface controls so that the results can be displayed in real time on the interface. The text box is used to display the current test results, the number of photos, the number of qualified times, the number of unqualified times and the qualified rate. The bar chart and pie chart on the right are used to display the data comparison between qualified and unqualified.

[0194] After the development is completed, a large number of trademark detection tests are carried out to ensure that the various functional modules of the interface can work properly. In the visual inspection interface, you can clearly see the camera image, current detection results, number of photos, number of qualified times (OK), number of unqualified times (NG), qualified rate, and qualified and unqualified data comparison bar charts and pie charts. For example: The current trademark Gilab has a qualified test result. 20 material trademarks have been photographed and tested, of which 15 material trademarks are qualified, 5 material trademarks are unqualified, and the qualified rate is 75%; another example: The current trademark is Gilab. The test result is unqualified. 22 material trademarks have been photographed and tested, of which 15 material trademarks are qualified, 7 material trademarks are unqualified, and the qualified rate is 68.18%.

[0195] 4.4 Human-computer interaction design. The system can switch between automatic mode and manual mode. The manual mode allows the operator to directly control every part of the sorting system. It is usually used for system debugging, maintenance, parameter setting or manual operation when the automatic mode fails. The operator can adjust the corresponding parameters through the human-computer interaction interface. In automatic mode, the operation of the automatic mode can be controlled by the start and stop buttons on the human-computer interaction interface. After starting the automatic mode, the automatic mode can automatically complete the sorting task according to the preset procedures and rules without manual intervention, improving efficiency and accuracy. In addition, the interface has four main functions: equipment debugging, variable monitoring, alarm recording and user management. Equipment debugging provides manual control and setting of the equipment, supports debugging and troubleshooting; variable monitoring is used to display and monitor the data in the PLC in real time, helping data analysis and status monitoring; alarm recording is used to record and manage alarm information, and provides the function of viewing and processing alarm history; user management is used to manage user accounts and permissions to ensure the security and operation specifications of the system.

[0196] In summary, this invention builds an automatic sorting physical platform based on the physical entity layer scheme of the automatic sorting twin system. First, the framework and process of the automatic sorting system are introduced, a suitable PLC is selected as the control core, and the input and output terminals are reasonably allocated according to the needs. Then, the electrical equipment and system control program in the automatic sorting system are described. Through these steps, the automatic sorting of materials is realized, laying a physical foundation for the development of the digital twin monitoring system.

[0197] Example 5, to further describe step S2 in detail, is described as follows.

[0198] The digital twin platform is an accurate mapping of physical entities in the real world. It provides support for the operation of the physical entity layer by truly simulating the behavior of the physical entity layer in the digital twin platform. Based on the theory of digital twin model construction, the present invention establishes a virtual model of the physical entity equipment and optimizes it, and builds a digital twin platform for the automatic sorting system based on the Unity engine.

[0199] 1. Requirements for establishing virtual models.

[0200] In order to ensure that the digital twin model can achieve synchronous mapping of the automatic sorting equipment, the following basic principles should be followed when constructing the virtual model of the digital twin platform;

[0201] (1) Accuracy. Accuracy is the core of the digital twin. It requires that the virtual models of physical equipment such as sorting robots, conveyor belts, and slide modules in the sorting system can be established strictly in accordance with the geometric and physical characteristics of the physical equipment to ensure the high fidelity and accuracy of the model.

[0202] (2) Lightweight. The automatic sorting digital twin includes not only the sorting robot and conveyor belt models, but also the supporting equipment models, workshop scene models, etc. If there are too many unnecessary details in the model and the redundancy is too high, the complex model movement will affect the computer's operating performance, reduce the clarity and fluidity of the system, and seriously affect the user experience. Therefore, on the basis of accurately depicting the physical entity equipment and restoring the actual scene with high fidelity, it is necessary to lightweight the model, optimize the model's vertices and faces, reduce the model's complexity and redundancy, reduce the computing load, and speed up the system's responsiveness. This will improve the operating speed and efficiency.

[0203] (3) The structural hierarchy remains consistent. The physical devices in the system have a certain hierarchical structure. The SCARA robot used in the present invention is mainly composed of a base, a first mechanical arm, a second mechanical arm, a spline screw shaft, an end flange and a bellows. The parent-child relationship between the front axis and the rear axis is that if the parent component moves, the child component will also rotate accordingly, ensuring the connectivity of the data between the parent-child relationship, and reflecting the structural hierarchy between the models in the construction of the digital twin model.

[0204] (4) Keep the motion method consistent. The motion method of the physical device must be consistent with the motion of the virtual model. Taking the sorting robot as an example, during the robot's motion, the rotation of each joint determines the position and posture of the robot's end, and the rotation direction and angle of each joint determine the movement of each axis. The motion method of the digital twin model and the relationship between the motion parameters of each joint and the end position and posture need to be consistent with the motion method of the robot's physical device.

[0205] 2. Basic process of virtual model building.

[0206] Designing and processing a virtual model of an automatic sorting system based on digital twins requires considering the authenticity of the model and its efficiency in operation. The key tasks of model building include modeling and assembly of equipment parts, lightweighting of the model, and loading of the virtual space model. Use SolidWorks, 3ds Max, and Unity to build the model and digital twin platform of the automatic sorting system to achieve accurate mapping of the physical entities of each device. The modeling process is as follows: Figure 5 .

[0207] 3. Kinematic analysis of sorting robot.

[0208] The key sorting equipment in the automatic sorting system of the present invention is a SCARA robot, and its main body components mainly include: a base, a first mechanical arm, a second mechanical arm, a spline screw shaft, an end flange and a bellows. The first mechanical arm and the second mechanical arm are two parallel link arms, the spline screw shaft is a vertical link arm, and the end flange is a tool end. The robot can move in a horizontal plane. The robot kinematic analysis is mainly divided into two aspects: on the basis of determining the parameters of each connecting rod of the robot, the forward kinematics is to know the variables of each motion joint and calculate the end effector posture; on the contrary, the inverse kinematics is to know the position of the mechanical arm and then inversely calculate the variables of each joint.

[0209] 3.1 Kinematic model of SCARA robot based on DH method.

[0210] The SCARA robot is composed of connecting rods connected by rotating and moving joints. A mathematical model is established for the robot, and the joints and connecting rods are simplified to facilitate kinematic analysis. Fig.16 The joint link diagram is shown.

[0211] The DH modeling method is a method for establishing the relationship between the joint model and the spatial posture. Each link i uses four characteristic parameters:

[0212] (a) Connecting rod length a i-1 : The vertical distance between the z-axis axes of adjacent joints;

[0213] (b) Joint offset d i : From x i-1 Along z i-1 Axis to x i distance;

[0214] (c) Connecting rod torsion angle a i-1 :From z i-1 to x i The corner along x i-1 The arrow direction of the axis is positive.

[0215] (d) Joint rotation angle θ i : From x i-1 to x i The corner along z i The direction of the arrow is positive.

[0216] The connecting rod coordinate system is Fig.17 The coordinate system of the SCARA robot link is shown in the figure. The z-axis direction of the rotating joints J1, J2, and J4 is determined by the right-hand rule, while the z-axis direction of the moving joint J3 is the direction of linear motion.

[0217] Table 3 shows the robot's link parameters. The variables of joints 1, 2, and 4 are θ i, the variable of joint 3 is d3.

[0218] Table 3 Robot connecting rod parameters

[0219]

[0220]

[0221] 3.2 Robot forward kinematics analysis

[0222] According to the robot's joint variables and link parameters, the transformation matrix can be obtained:

[0223]

[0224] In the formula, Rot(x,a i-1 ) represents a positive rotation of a:1 around the x-axis, Trans(x,a i-1 ) represents a-1 distance movement along the x-1 axis; Rot(z,θ i ) represents the positive rotation angle around the = axis, Trans(z,d i ) represents a movement of distance d along the z-axis.

[0225] According to Table 3 and Formula 7, the homogeneous transformation matrix between adjacent joints of the SCARA robot can be obtained as follows:

[0226]

[0227] According to formula (8), the transformation matrix of the terminal posture can be obtained as:

[0228]

[0229] Equation (9) is called the robot forward kinematics equation, where n, o, and m represent the direction cosines of the axes x4, y4, and z4 relative to the base coordinate axes x0, y0, and z0, respectively, and p represents the position vector of the axes x4, y4, and z4 relative to the base coordinate axes x0, y0, and z0. The end position of the robot can be obtained through the variables of the four joints and the link parameters.

[0230] 3.3 Robot inverse kinematics analysis.

[0231] When performing trajectory planning in Cartesian space, it is necessary to solve the inverse kinematics of the robot in real time. The inverse kinematics problem is exactly the opposite of the forward kinematics process, that is, given the end position of the SCARA robot, the coordinates of each joint in the joint space at this time are solved.

[0232] Now that the posture transformation matrix T is known, according to formula (4.3), the variables can be separated by left-multiplying the inverse matrix, and then the variables can be obtained by algebraic correspondence:

[0233] (1) Find the joint variable θ1

[0234] First, separate the variables. 0 Both sides of the T4 equation need to be multiplied on the left 0 T1 -1 , you can get:

[0235]

[0236] In the above equation, take the fourth element of the first row and the fourth element of the second row on both sides to be equal, and we get:

[0237]

[0238] Solving the above equation, we get

[0239]

[0240] Where A is:

[0241]

[0242] (2) Find the joint variable θ2

[0243]

[0244] (3) Find the joint variable d3

[0245] d3=-P z (15)

[0246] (4) Find the joint variable θ4

[0247] θ4=θ2-arcsin(-n x sinθ1+n y cosθ1) (16).

[0248] 4. Three-dimensional model building.

[0249] 4.1 Three-dimensional virtual model building.

[0250] Establishing a high-fidelity twin model is a key step in realizing the automatic sorting digital twin system. It is necessary to ensure that the geometric behavior and physical rules of each device are completely consistent with the actual device. By providing a dynamic simulation environment, the twin model realizes the integrated management of the automatic sorting operation cycle and model. The sorting equipment involved in the present invention mainly includes SCARA robots, slide modules and conveyor belts. Taking the establishment of the robot model as an example, the process is as follows:

[0251] (1) Use SolidWorks to draw sub-models such as the base, upper control cabinet, bellows, and each robotic arm, then determine the focus of each joint and ensure that the robot's mesh is aligned with the model to ensure the accuracy of the assembly and motion relationship definition. Select the surface, edge, or point of each sub-model and define the relationship between them, such as coaxial, parallel, distance, and angle, to ensure that each component is correctly connected and moves;

[0252] (2) Define the kinematic relationships of joints and links in the assembly to perform kinematic analysis and ensure that the motion and behavior of the robot model meet expectations;

[0253] (3) Define the parent-child relationship of each component to ensure the correct hierarchy of the assembly. Through these steps, the three-dimensional model of the robot is completed and the virtual model is accurately aligned at the joint position, laying the foundation for subsequent simulation and emulation work. After the robot model is established, other equipment models such as conveyor belts and slide modules are established in a similar way.

[0254] 4.2 Lightweight processing and format conversion.

[0255] The real-time response and execution efficiency of the digital twin system are closely related to the twin model. If the model is too complex, it may cause slow loading and slow response after being imported into the simulation software, and slow down some calculation modules (such as forward and inverse kinematics), which in turn affects the real-time synchronization of virtual entities and physical entities. By reducing the redundancy of the model, not only the time required for model loading and information transmission is reduced, but also the load pressure of the system can be reduced. Therefore, the operating speed of the system has been significantly improved, and the delay problem has been effectively controlled, which significantly improves the efficiency and real-time performance of online monitoring of the digital twin system.

[0256] (1) Lightweight processing.

[0257] (2) Adjust the model axis.

[0258] (3) System unit settings. Model files processed by 3ds Max need to be exported to *.fbx format files, otherwise they cannot be directly run in Unity. In addition, it is necessary to exclude unnecessary information in 3ds Max, such as lighting and animation, which is not conducive to the effective use of the model in Unity. At the same time, it will increase the amount of model data and affect the running efficiency of the model.

[0259] 4.3 Loading the virtual space model.

[0260] Import the virtual models of robots, conveyor belts, slide modules, etc. exported from 3ds Max into the Unity 3D development platform. By measuring the relative positions of each device in the physical platform, adjust the positions of each model in the virtual simulation scene to keep them consistent with the relative positions of each device in the actual physical platform, and integrate the virtual models of each device with the virtual environment scene. Take the imported robot model as an example, place the robot base coordinate system at the origin of the world coordinate system, and connect each part through the parent-child object setting to form the robot's child object hierarchy. After completing the settings, drag all models into the resource folder and convert each part into an object in Unity3D. Then, adjust the initial state of each joint to make it consistent with the initial state of the real robot. The joints of the sorting robot model are arranged in a "parent-child" structure. Load other device models in a similar way. When the model driver is running, the motion simulation in the virtual space can be consistent with the actual motion state.

[0261] The present invention describes in detail the construction method of the digital twin platform. First, the requirements and process for establishing the virtual model of the automatic sorting equipment are introduced, and the forward and inverse kinematics of the sorting robot are analyzed. Subsequently, SolidWorks is used to complete the part modeling and assembly of the automatic sorting equipment, and 3ds Max is used to perform lightweight processing and format conversion of the model. Finally, the model is imported into the Unity engine, and methods such as coordinate transformation, setting hierarchical relationships, and adding motion constraints are used to ensure the normal loading and operation of the model in the virtual environment, and an automatic sorting digital twin platform based on the Unity engine is successfully built.

[0262] Example 6, to further describe in detail the construction of the digital twin platform of the automatic sorting system in step S3, the specific description is as follows.

[0263] Through database comparison, SQL Server was selected as the database for system development to realize data storage and historical state reproduction functions. Finally, based on Unity and the previous twin data transmission and communication, the visual interaction, real-time monitoring data, and virtual-reality synchronous operation functions of the automatic sorting digital twin system were realized.

[0264] 1. The overall implementation framework of the automatic sorting digital twin, such as Fig.18 shown.

[0265] The goal of this invention is to design and implement a three-dimensional visual monitoring and real-time monitoring system for automatic sorting based on digital twins, and explore the combination of digital twin technology and manufacturing industry with automatic sorting scenarios as the entry point. The main design goals are as follows:

[0266] (1) Develop an automatic sorting system based on PLC control, which can automatically classify and sort items. Use visual inspection technology to efficiently and accurately complete item classification and sorting, thereby improving the efficiency and processing speed of cargo classification.

[0267] (2) Establish a virtual model corresponding to the physical entity equipment of the automatic sorting system, requiring the geometric dimensions and assembly relationships of the physical entity and the virtual system to be consistent, and build a digital twin platform based on Unity.

[0268] (3) A Socket interface based on TCP / IP protocol, S7 protocol and Modbus protocol were proposed to realize the real-time collection and transmission of robot operation data and PLC internal drive data and perception data.

[0269] (4) Based on the Unity software development platform, the digital twin system realizes real-time monitoring of the status and environment of the sorting equipment, synchronous movement of virtual and real, and data storage and historical status reproduction.

[0270] System status monitoring function: collects status data and environmental parameters of system equipment, combines the three-dimensional visual monitoring of the virtual model with the visual display of real-time operation data on the UI interface, and can more intuitively understand the current operation status of the automatic sorting system;

[0271] Virtual and real synchronous movement of twin system: the system equipment operation data is collected and transmitted to the digital twin platform, and the twin data is mapped to the virtual system in real time to achieve synchronization between the model and the physical entity;

[0272] Historical status reproduction: The data is collected and stored in the local database. The user enters the historical time node range that needs to be reproduced, and the stored twin data is used to trace back the historical operation status of the sorting robot in the system.

[0273] 2. Twin system development environment and architecture selection.

[0274] 2.1 Development environment.

[0275] Since the automated sorting scene digital twin monitoring system described in the present invention requires a large amount of data interaction and continuously updates the virtual model behavior through instructions, the hardware configuration requirements of the computer equipment are relatively high. Only in this way can the large amount of computing requirements generated during the operation of the system be met to avoid affecting the visualization effect. Table 4 shows the hardware configuration of the computer.

[0276] The digital twin monitoring system of the present invention is developed collaboratively using a variety of software. Therefore, when selecting development tools, its applicability and compatibility with the model must be considered. The software development environment configuration of the system is shown in Table 4.

[0277] Table 4 Computer hardware configuration

[0278]

[0279] 2.2 System architecture selection.

[0280] Currently, the commonly used system software architectures are mainly B / S architecture and C / S architecture. The B / S architecture (browser-server) divides the application into two parts: the browser and the server. The client accesses the server through the browser, and the server processes the client request and returns the result. It is characterized by simple deployment and maintenance, convenient upgrade, good compatibility, and is suitable for wide area network environments. It has high network dependence, but is limited by the performance and capabilities of the browser and has poor performance when processing complex calculations. The C / S architecture (client-server) divides the application into two parts: the client and the server. The client runs locally and requests data and services from the server. The server processes the request and returns the result. The advantages are that the client and the server can perform efficient data processing and transmission, and the client can perform complex calculations and processing, which reduces the burden on the server. It performs well in the local area network and has relatively low network dependence. As shown in Table 5.

[0281] Table 5 Software environment configuration

[0282]

[0283] Since the digital twin monitoring system for automated sorting scenarios developed in the present invention is mainly aimed at internal enterprises, almost all users are internal employees of the enterprise and can use the enterprise LAN. In addition, the system is required to have the characteristics of fast response speed and efficient operation speed. Therefore, the C / S architecture is adopted to build an automated sorting digital twin three-dimensional visualization monitoring platform.

[0284] The present invention is based on an automatic sorting digital twin platform built on Unity software. The platform has high compatibility and flexibility, and can adapt to a variety of client environments including PC and Android. The system can be packaged into a compressed file that is easy to transmit. Users only need to access it through the corresponding client to easily capture and monitor the working status of the automatic sorting system in real time, providing users with a real-time and comprehensive monitoring tool.

[0285] 3. Data storage and historical status reproduction.

[0286] 3.1 Database design.

[0287] Database management design is one of the core elements of the digital twin industrial sorting process monitoring system. In industrial systems, the classified storage of equipment operating parameters is essential for subsequent calls and viewing. Storing the historical operating parameters of industrial system equipment not only helps with equipment maintenance and performance optimization, but also improves equipment reliability and safety, and prepares for adding data online prediction modules. To this end, it is crucial to choose a suitable database to store and manage this data, taking into account multiple criteria such as performance, supported data types, persistence and stability, scalability, query capabilities, security, and ease of use. Table 6 lists the currently commonly used data types.

[0288] Table 6 Common data model categories

[0289]

[0290]

[0291] The twin data collected by the industrial automation sorting system is relational data, which is usually composed of multiple interrelated data, each of which contains a primary key and one or more foreign keys. The primary key is used to uniquely identify each piece of data, while the foreign key is used to associate other data. This data model can effectively simulate and express complex relationships in the real world, and provide efficient data query and flexible data management functions. The present invention chooses to use the SQL Server database to store data. The database is a powerful, reliable and easy-to-use relational database management system that is suitable for large-scale data storage and processing. In the system design, in order to ensure data persistence, a table named Robot is created in the database. The fields in the table are shown in Table 7, which shows only the fields of one axis.

[0292] Relational data usually consists of multiple interrelated data, each of which contains a primary key and one or more foreign keys. The primary key is used to uniquely identify each piece of data, while the foreign key is used to associate other data. This data model can effectively simulate and express complex relationships in the real world, providing efficient data query and flexible data management functions.

[0293] Table 7 Fields in the historical data table

[0294] Field Name Is it a primary key? Data Types meaning DataID yes int Data Identifiers RobotID no int Device Name CollectDate no datetime Data collection time RobotSpeed no float Robot speed JointSpeed no float Joint Speed JointCurrent no float Joint current JointAccleration no float Joint acceleration Pos_x no float End X coordinate Pos_y no float End Y coordinate Pos_z no float End Z coordinate

[0295] The storage process of twin data is as follows Fig.19As shown. First, instantiate the database operation class, then connect and open the database. According to the robot entity class data, first, instantiate the database operation class and connect to the database. Then, according to the robot entity class data, splice and execute SQL statements to insert the data into the data table. Execute this operation repeatedly until the twin data stops being received. Finally, close the database connection and release the database operation class instance to complete the storage of the twin data.

[0296] 3.2 Historical state reproduction.

[0297] The core of 3D visual monitoring and operation status is to dynamically map the physical world to the virtual space in real time, a process that is highly dependent on the precise support of twin data. The use of multi-source heterogeneous data transmission technology and an efficient data storage solution based on the SQL Server database ensures seamless transmission and reliable storage of twin data. Subsequently, the client receives this data on the constructed digital twin platform and maps it to the virtual model in real time, achieving an intuitive display of the operating status of actual equipment such as robots.

[0298] In terms of troubleshooting and status backtracking, the system provides a historical status playback function. Users can enter a specified time period through the user interface, and the system background automatically converts the input into a timestamp format, builds a corresponding database query statement, and stores all relevant data within this period. Subsequently, without continuously occupying the database connection, the system starts a new thread to process this data, drives the virtual model to reproduce the historical operation scene, and realizes high-fidelity visual playback of historical operation status. This function not only simplifies the troubleshooting process, but also allows users to intuitively understand the historical performance of the system, providing strong support for decision-making.

[0299] 4. Implementation and demonstration of twin system functions.

[0300] 4.1 Visual interaction.

[0301] The advantage of automatic sorting digital twin 3D visual monitoring is that it can avoid blind spots and improve the transparency of monitoring. Unlike real-time video monitoring, a notable feature of 3D visual monitoring is its strong interactivity, which can respond to human-computer interaction events. Scene roaming based on human-computer interaction mainly includes geometric transformation and response to external input events, which can be achieved through mouse clicks and keyboard operations. Geometric transformation is the key to realizing scene roaming. All model movements and scene changes are realized based on geometric transformation technology.

[0302] Geometric transformation is a technique for changing the shape, size, position and orientation of an image in two-dimensional or three-dimensional space. It achieves complex model transformation by combining geometric transformation operations such as translation, rotation and scaling into a matrix. This method can not only efficiently implement multiple transformations, but also reduce computer overhead during rendering. The principles of geometric transformation are as follows:

[0303] (1) Translation transformation

[0304] Translation transformation is used to change the position of the model in three-dimensional space. The displacement of the model is achieved by moving along the x, y, and z axes. In a three-dimensional environment, translation transformation requires matrix calculation. Since translation is not a linear transformation, homogeneous coordinates are used to represent three-dimensional points, and a 4×4 matrix is ​​used to represent the spatial transformation, as shown in formula (18):

[0305]

[0306] Where T is the translation matrix, T x 、T y 、T z are the translation distances of the virtual model along the x, y, and z axes respectively. If the coordinates of point a are (x, y, z), the coordinates after T translation are (x+T x ,y+T y ,z+T z ).

[0307] (2) Scaling transformation

[0308] Scaling transformation is used to change the size of the model by scaling it up or down in the x, y, and z directions. The scaling matrix is ​​shown in formula (19):

[0309]

[0310] Where S is the scaling matrix, α, β, and γ are the scaling ratios along the x, y, and z axes, respectively. If the coordinates of point a are (x, y, z), the coordinates after scaling by S are (xα, yβ, zγ).

[0311] (3) Rotation transformation

[0312] The rotation transformation is used to change the direction of the virtual model. The rotation can be performed around the x, y, and z axes respectively, where the rotation matrix is ​​shown in formula (20):

[0313] R'=R x *R y *R z (20)

[0314] Where R represents the rotation matrix, Rx, Ry, and Rz are the rotation matrices of the virtual model on the x, y, and z axes, respectively, and their expressions are shown in equations (21), (22), and (23), respectively.

[0315]

[0316] In formulas (4.4), (4.5), and (4.6), α, β, and γ are the angles of rotation of the model around the x, y, and z axes, respectively. If the coordinates of point a are (x, y, z), the new coordinates obtained after rotating around the x-axis by an angle of α are (x, ycosα-zsinα, ysinα+zcosα), the new coordinates obtained after rotating around the y-axis by an angle of β are (xcosβ+zsinβ, y, zcosβ-xsinβ), and the new coordinates obtained after rotating around the z-axis by an angle of γ are (xcosγ-ysinγ, ycosγ+xsinγ, z). By multiplying the translation, scaling, and rotation transformation matrices of the model, the total transformation matrix of the model can be obtained, and its matrix formula is shown in formula (24):

[0317] TRS=T×R×S (24)

[0318] In formula (24), TRS is the total transformation matrix of the virtual model, through which the movement of the digital twin model can be realized.

[0319] In the Unity software platform, the scene perspective and orientation display are controlled by the Camera component and the Transform component, and can be attached to any game object to move, rotate and scale at any time. The main parameters controlled include position and direction, which determine the position and orientation of the Camera in the scene. By adjusting the position and direction of the Camera, the Transform component can control the position and angle of the model, and achieve a multi-directional monitoring effect on the virtual model. The camera's field of view and viewing distance can be changed by clicking the mouse and scrolling the wheel; the camera's field of view can be adjusted by keyboard operation. The virtual space can be monitored from any angle and distance, achieving all-round three-dimensional visual monitoring of the twin system.

[0320] 4.2 Real-time monitoring status.

[0321] The digital twin real-time monitoring module realizes real-time monitoring of the robot speed, the speed and acceleration of each joint, the coordinate position of the end effector, the speed and position of the slide module and its surrounding environment. In the digital twin interface, the data of the model operation parameters can be updated in real time. The equipment status display interface supports real-time status monitoring and key parameter display of equipment such as robots, conveyor belts and slide modules; the system parameter display interface is used to display system voltage, current and power information; environmental parameter monitoring is used to monitor environmental changes around the system in real time.

[0322] (1) Display real-time monitoring data and develop a visual data monitoring integrated interface based on Unity's UGUI technology, such as equipment operation status, system energy consumption parameters, and environmental parameters. The digital twin monitoring interface contains multiple modules, namely: automatic sorting virtual space operation interface, traditional video monitoring interface, menu bar, sorting robot status monitoring, slide module and conveyor belt status monitoring, system equipment parameter monitoring, and environmental parameter monitoring. These interfaces are described in detail below. These interfaces are centrally managed in the menu bar. By centrally controlling the display and hiding of the interface, users can quickly switch between different monitoring views as needed, avoiding the tedious switching between multiple interfaces and improving the operating experience. When the system is abnormal or needs to focus on monitoring a certain part, the display and hiding of the interface can be quickly adjusted to focus on and deal with the problem in a timely manner. The robot operation status data monitoring interface, from top to bottom, is the connection control area, robot operation status area, robot operation parameter area, and robot speed control area. The connection control area can display the IP address of the data transmission server robot and control the opening and closing of its connection status; the robot operation status area displays the robot servo power-on, zero return operation, system mode and system status. There are two system modes, M represents macro mode and Q represents QRL mode. The system status includes start and stop; the robot operation parameter area displays the X / Y / Z coordinates of the robot in the Cartesian coordinate system and the four joint coordinates of the robot; the robot speed control area realizes the control of the physical robot speed in the virtual space. Through these functional modules, users can fully monitor and control the operation status of robots and other equipment. The PLC-controlled equipment status monitoring interface displays the operation status data monitoring of the conveyor belt and slide module controlled by the PLC. From top to bottom, they are the connection control area, the slide module status area, and the conveyor belt status area. The connection control area displays the IP address of the PLC controller on the data transmission server side, and can control the opening and closing of the PLC connection status; the slide module status area displays the real-time position, real-time speed, and high limit, low limit and origin status of the slide module; the conveyor belt status area displays the mode of the conveyor belt and the detection of materials on the conveyor belt. At this time, the conveyor belt is in a stopped state, and the photoelectric detection sensor on the conveyor belt detects the material.

[0323] (2) System equipment parameter monitoring.

[0324] By using a single-phase energy meter to collect system current, voltage and other data, and using a pressure sensor to collect and analyze the air pump pressure, data monitoring of the system energy consumption and air pump status is achieved. Users can clearly and intuitively view the cumulative energy consumption to understand the overall energy consumption; view power consumption to monitor instant power consumption; display the current voltage and current values ​​to ensure that the system voltage is within a safe range and prevent current overload; display the air pump pressure value in real time to monitor the working status and efficiency of the air pump. Through this interface, users can intuitively understand the system's energy consumption and the operating status of the air pump, and promptly discover and solve potential problems. This design not only improves the safety and efficiency of system operation, but also provides users with convenient management tools to help optimize energy use and maintain equipment.

[0325] (3) Environmental parameter monitoring.

[0326] Environmental parameter monitoring is used to obtain changes in parameters such as temperature, humidity, and noise in the actual environment during system operation, and to display the current parameters in real time on the UI. The environmental monitoring interface displays the current temperature, humidity, and noise data intuitively, helping users understand the changes in environmental parameters in real time. When the temperature, humidity, or noise exceeds the set threshold, the interface will sound an alarm to remind users to take action.

[0327] 4.3 Virtual and real run synchronously.

[0328] When conducting virtual-real synchronization verification, you first need to ensure that the automatic sorting equipment in the physical space is powered on, turn on the relevant servers, and start the Unity3D engine on the virtual side to establish a connection with the server. The automatic sorting physical platform has set the equipment operation path according to the automatic sorting requirements and the equipment's operating status and behavior rules. When the operating data of each device on the physical platform is received, it can be driven to run by mapping it to the virtual model through data analysis. In addition, on the right side of the Unity interface, you can configure the joint data of the sorting robot model to control the joint movement of the physical entity of the sorting robot. Through the virtual mapping and interactive fusion of these two aspects, an operating mechanism that combines virtuality and reality and interacts in real time is formed.

[0329] This paper mainly realizes the three-dimensional visualization application of the automatic sorting digital twin system. First, the overall implementation framework of the automatic sorting digital twin system is elaborated in detail, and the development environment and system architecture selection in the software development process are introduced. Then, various databases are compared, and SQL Server is selected as the database for system development, realizing data storage and historical state reproduction functions. Finally, based on Unity and the twin data transmission communication in Chapter 5, the visualization interaction, real-time monitoring data, and virtual-real synchronous operation functions of the automatic sorting digital twin system are realized, verifying the feasibility of the automatic sorting digital twin technology.

[0330] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0331] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for constructing an automatic sorting digital twin system, characterized in that: The method includes: S1, build physical entities that perform different operating functions in automatic sorting based on the physical entity layer, and send the physical entities to the digital twin platform layer through a communication protocol or a transmission network; the different operating functions include automatic material transmission, robot handling, visual inspection, and human-computer interaction, and the physical entities include robots, conveyor belts, slide modules, and cameras; S2, the digital twin platform layer receives the data information of the physical entity, and constructs a three-dimensional virtual model synchronized with the execution of different operating functions of the physical entity through three-dimensional design software; and dynamically updates the three-dimensional virtual model in real time to be consistent with the real-time status of different operating functions of the physical entity; S3, loading the three-dimensional virtual model with consistent real-time status of different operating functions of the physical entity into the three-dimensional engine of the visualization application layer through the communication protocol or transmission network, monitoring the real-time status of different operating functions of the physical entity in the industrial automatic sorting simulation scenario, and visualizing it through the user interface.

2. The method for constructing an automatic sorting digital twin system according to claim 1, characterized in that: In step S1, the automatic transmission, robot handling, visual inspection and human-computer interaction of the materials are respectively implemented through the material transmission module, robot handling module, visual inspection module and human-computer interaction module divided by the automatic sorting system, and the operation of the above modules is controlled by the PLC controller to complete the automatic sorting.

3. The method for constructing an automatic sorting digital twin system according to claim 2, characterized in that: The material conveying module is located at the front end of the automatic sorting system and is used to move materials from the material placement area of ​​the conveyor belt to the material grabbing area of ​​the robot; the material conveying module includes a conveyor belt and a photoelectric detection sensor component. The conveyor belt continuously transports materials and is driven by an AC motor to adjust the speed to adapt to different material flows and processing requirements. The mode, forward rotation and reverse rotation of the conveyor belt are all controlled by a PLC controller; The robot handling module is composed of a robot, a slide module and a pneumatic suction cup, and is used to complete the grabbing and placement of materials; The visual inspection module is used for visual monitoring of materials; The human-machine interaction module displays the human-machine interface through a touch screen, and realizes the switching between the automatic mode and the manual mode in the automatic sorting system through interface design; The PLC controller also controls the robot, the slide module, the pneumatic suction cup and the camera to work together.

4. The method for constructing an automatic sorting digital twin system according to claim 2, characterized in that: The visual inspection module uses template matching technology to compare the image to be inspected with a predefined standard template to find the area in the image to be inspected that matches the template; performs affine transformation on the image to ensure that the image is processed in a standard coordinate system to eliminate distortion or displacement that may occur during the shooting process; and uses image binarization to convert the grayscale image into a binary image so that the target object and the background are distinct; Count the pixels of the binarized image, count the number of black pixels in a specific area, compare the statistical results with the preset standard to determine whether the image is qualified, and then send the image detection results to the PLC controller and save the data.

5. The method for constructing an automatic sorting digital twin system according to claim 1, characterized in that: In step S2, the 3D design software includes a Windows-based 3D CAD system SolidWorks, a 3D modeling software 3ds Max, and a 3D engine Unity; Building a 3D virtual model includes: (1) Check and sort the physical entity data, obtain the shape and size information of the physical entity, and measure and verify the physical data of each physical entity; (2) Each physical entity is divided into multiple independent sub-modules, which are drawn using SolidWorks, and the three-dimensional virtual models drawn from the multiple independent sub-modules are assembled as a whole to be consistent with the physical entity in size, proportion, and shape; (3) After saving the obtained 3D virtual model, import it into 3ds Max for lightweight processing to reduce the number of faces and vertices; (4) Import the processed 3D virtual model into Unity in *.fbx format for loading and adjustment.

6. The method for constructing an automatic sorting digital twin system according to claim 5, characterized in that: In step (3), lightweight processing includes: 1) Export the geometric model file *.stl from SolidWorks and check whether the 3D virtual model information is complete; 2) Use 3ds Max software to delete lines and faces that are not involved in actual operation control, streamline the model mesh, and reduce the number of hidden bodies; In step (4), loading and adjustment are performed, including: Loading is to import the robot's 3D virtual model into Unity. The robot base coordinate system is located at the origin of the world coordinate system. Through the setting of parent-child objects, the parts are connected to form the robot's sub-object hierarchy. After the setting is completed, all sub-object hierarchy models are dragged into the resource folder to convert each part into an object in Unity3D. The initial state of each joint is adjusted to be consistent with the initial state of the real robot. The joints of the robot's 3D virtual model are arranged in a parent-child structure. Adjustment is to adjust the axis of the 3D virtual model and the conversion adjustment of the unit when importing the 3D virtual model from 3ds Max to Unity.

7. The method for constructing an automatic sorting digital twin system according to claim 1, characterized in that: In step S3, it is loaded into Unity, a 3D engine of the visualization application layer, including: creating and exporting 3D virtual models of robots, conveyor belts, and slide modules in the industrial automatic sorting scene through SolidWorks, importing the 3D virtual models into 3ds Max, performing lightweight surface reduction processing, adjusting the center coordinates of the 3D virtual model, setting the hierarchy, and finally exporting it to a *.fbx format file, and loading the 3D virtual model into Unity.

8. The method for constructing an automatic sorting digital twin system according to claim 7, characterized in that: During the lightweight and surface reduction process, the center coordinates of the 3D virtual model are adjusted and the hierarchical structure is set, including: exporting the geometric model file *.stl from SolidWorks to check whether the 3D virtual model information is complete; using 3ds Max software to delete lines and surfaces that are not involved in actual operation control, streamline the model mesh, and reduce the number of hidden bodies.

9. An automatic sorting digital twin system, characterized in that: The system is constructed using the construction method of the automatic sorting digital twin system according to any one of claims 1 to 8, and the system includes: The physical entity layer (1) is used to build the entity of the automatic sorting system and realize the control of the automatic sorting system to ensure the normal operation of the automatic sorting system and perform sorting; The data transmission layer (2) is located between the physical entity layer (1) and the digital twin platform layer (3) and is used for data collection, transmission and synchronization. The data of the physical entity layer (1) is transmitted to the digital twin platform layer (3) in real time by using a communication protocol or a data transmission network, so as to ensure the consistency and synchronization between the three-dimensional virtual model and the actual automatic sorting system. The digital twin platform layer (3) is used to establish a three-dimensional virtual model corresponding to the physical entity of the physical entity layer (1), and dynamically update the three-dimensional virtual model by receiving data from the data transmission layer (2) to reflect the actual state and operation of the physical entity; The visualization application layer (4) enables users to transparently and comprehensively monitor and analyze physical entities through the three-dimensional virtual model of the digital twin platform layer (3) through the user interface, monitor the operating status of the physical entities, view operating data, conduct environmental monitoring, optimize performance, and obtain decision support.

10. The automatic sorting digital twin system according to claim 9, characterized in that: The automatic sorting system includes: a physical entity, a sensing part, a network communication, and a control and execution design; The conveyor belt, robot, slide module and camera are physical entities used for actual sorting operations; Material detection sensor, temperature and humidity sensor, air pump pressure sensor, and camera constitute the sensing part; TCP / IP communication, RS485, and driver interface are defined as the network communication part, which is used to realize the communication interaction between automatic sorting equipment; The control and execution design includes PLC controller, robot control system, human-machine interaction panel and visual inspection system to realize the control and execution of the corresponding automatic sorting physical entities.

Images