A method for real-time monitoring of tool wear status in milling robots that integrates digital twins and deep learning

Abstract

This invention provides a real-time monitoring method for the wear state of milling robot tools that integrates digital twins and deep learning. Based on a digital twin system and a CNN-LSTM deep learning model, the method first uses a CNN network to extract features from the data. Then, the feature sequences extracted by the CNN are input into an LSTM network to model the sequence data and learn the temporal characteristics of tool wear state. The tool wear monitoring model is trained using a historical machining dataset. Through transfer learning technology, the model is retrained using actual machining data from the field to analyze and predict tool wear values ​​online.

Description

Technical Field

[0001] This invention belongs to the technical fields of industrial internet and machine learning, and specifically relates to a method for real-time monitoring of tool wear status of milling robots that integrates digital twins and deep learning. Background Technology

[0002] In the process of intelligent manufacturing, robots, as representative equipment of automation, intelligence, and digitalization, play a vital role in various manufacturing fields. In the field of material milling, robots have significant advantages in complex shapes, small batches, and less demanding precision applications due to their high flexibility, multiple degrees of freedom, and wide adaptability. However, in actual milling processes, robots often face various complex scenarios. Safety accidents could lead to equipment damage or personal injury risks, making simulation of the milling process essential. Digital twins, with their multi-dimensional, multi-physical, and multi-disciplinary attributes, as well as their high fidelity, virtual-real fusion, and real-time interaction characteristics, are gradually being integrated into intelligent manufacturing and are beginning to be applied to real-time monitoring and adjustment of robot operating status. Currently, the application of robot digital twin technology mainly focuses on robot dynamics modeling and simulation, machining process stiffness analysis and compensation, and posture planning, lacking research on overall simulation and making it difficult to fully understand the process and effects of robot milling. Summary of the Invention

[0003] To overcome the defects and shortcomings of existing technologies, this invention provides a real-time monitoring method for the wear state of milling robot tools that integrates digital twins and deep learning. Based on this method, the digital twin system is improved to achieve more comprehensive process support and effect simulation of robot milling.

[0004] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0005] A real-time monitoring method for tool wear status of milling robots integrating digital twins and deep learning is proposed. Based on a digital twin system and a CNN-LSTM deep learning model, the method first uses a CNN network to extract features from the data. Then, the feature sequences extracted by the CNN are input into an LSTM network to model the sequence data and learn the temporal characteristics of tool wear status. The tool wear monitoring model is trained using a historical machining dataset. Through transfer learning technology, the model is retrained using actual machining data to analyze and predict tool wear values ​​online.

[0006] Furthermore, the model transfer learning process includes: First, loading a pre-trained CNN-LSTM model in the source domain; second, retrieving and modifying the network structure of the original model, including adjusting the output layer (including fully connected layers) to meet the requirements of the target task; during this process, adjusting the parameters to ensure the model's performance on the new task; subsequently, merging the modified output layer with the first few layers of the original model to form a new network structure; then defining the training parameters for transfer learning and training the modified model using training data from the target domain to further fine-tune the model parameters to better adapt to the characteristics of the target task; finally, saving the model after transfer learning and applying it to the tool wear prediction task under actual working conditions.

[0007] Furthermore, the structure of the tool wear monitoring model includes a model data input layer, a convolutional layer, a pooling layer, an unfolded layer, an LSTM layer, a dropout layer, a fully connected layer, and an output layer. The input layer receives data, the convolutional and pooling layers are responsible for feature extraction and dimensionality reduction, the unfolded layer inputs the extracted feature sequence into the LSTM layer, the dropout layer is used to prevent overfitting, the fully connected layer is used for feature fusion and output, and finally the output layer gives the prediction result of the wear state.

[0008] Furthermore, a digital twin system for a milling robot employs the real-time tool wear state monitoring method described above; its application service layer includes a machining process simulation module, a tool wear state identification and prediction module, and a dynamic data information visualization and monitoring module; the machining process simulation module is used to display the effect of real-time milling and utilizes a material removal algorithm to ensure the accuracy and real-time performance of the machining process simulation; the dynamic data information visualization and monitoring module uses dynamically updated charts to statistically analyze and process various real-time data to achieve real-time monitoring of the robot's operating status and milling progress.

[0009] Furthermore, the machining process simulation module controls the virtual robot to move according to the path plan based on the twin data; calculates the distance between the tool cutting point and the workpiece mesh point to determine whether contact occurs; traverses the workpiece mesh and performs corresponding deformation processing on the contact part; the deformation processing is achieved by modifying the workpiece model mesh point; and refreshes the workpiece mesh to achieve the effect of simulating material removal.

[0010] Furthermore, the material removal algorithm defines a spherical bounding box to limit its spatial range. The spherical bounding box has its center point at the tool milling center, and its radius is adjusted based on the tool radius. The material removal algorithm based on the spherical bounding box calculates the three-dimensional spatial distance between the workpiece mesh points and the center point of the tool bounding box to obtain the mesh points located within the spherical bounding box, thereby limiting the space range of tool milling. The material removal algorithm is then executed based on the contact situation between the tool and the workpiece mesh points, performing material removal algorithm calculations on the mesh points within the bounding box to exclude mesh points that do not need to participate in the calculation and undergo deformation processing.

[0011] Furthermore, the material removal algorithm requires spatial coordinate transformation of the workpiece; spatial coordinate transformation transforms the workpiece mesh spatial coordinates into tool spatial coordinates, and milling judgment and mesh deformation are performed in the tool spatial coordinates. Finally, the workpiece mesh spatial coordinates are restored back to their own spatial coordinates for visualization.

[0012] The spatial coordinate transformation is accomplished based on a 4×4 homogeneous transformation matrix, which is used to represent the translation and rotation transformation of the spatial coordinates of the workpiece mesh in its own coordinate reference system and the tool coordinate reference system.

[0013] The coordinates of grid point P in the workpiece coordinate system are obtained by homogeneous transformation, enabling the tool to mill the workpiece at different angles.

[0014] Furthermore, the simulation model for the milling process is constructed by generating workpiece models and NC code files using software with CAD / CAM capabilities, building a workpiece mesh model, designing a dynamic milling algorithm to simulate the material removal process, converting the NC code into an end-effector trajectory, combining it with robot inverse kinematics, solving the motion parameters of each joint of the robot through the end-effector trajectory, performing simulation verification, and finally outputting the simulation results.

[0015] Furthermore, trajectory planning is achieved through the NC code parsing module. The NC code parsing module extracts information from the NC code text file, identifies and processes it, and converts it into motion parameters that the robot can recognize, so as to drive the simulated motion process of the robot milling.

[0016] Trajectory planning includes joint space trajectory planning and Cartesian space trajectory planning;

[0017] The joint space trajectory planning uses the S-shaped velocity interpolation function to interpolate the robot's joint space trajectory, so that the change curve of the joint variables transitions smoothly, thereby achieving the continuity of joint movement.

[0018] Cartesian space trajectory planning obtains the end-effector pose by solving the S-curve interpolation algorithm, and then inversely solves the joint angles by solving the end-effector pose, thereby controlling the robot to achieve the expected motion trajectory.

[0019] Furthermore, the NC code parsing module includes three steps: lexical analysis, syntax analysis, and data compilation, which are performed sequentially.

[0020] Lexical analysis uses Regex regular expression technology to match and analyze the instruction content in the code one by one; syntax analysis verifies the integrity and logical rationality of the NC code; data compilation parses the relevant data information, stores it in a data structure, and transmits it to an SQL Server database for management; the data structure contains information such as the spindle's interpolation motion mode, feed rate, spindle speed, and trajectory coordinate values.

[0021] The simulation model was constructed by drawing a 3D model of the workpiece blank using SolidWorks software and exporting it as an STL file. The STL file was then processed with 3ds Max software for mesh subdivision and texture mapping to enhance the realism and visual effect of the model, and the STL file was exported as an FBX file.

[0022] Import the FBX file into Unity3D software, run a material removal algorithm based on mesh deformation on the FBX file to achieve a milling simulation effect.

[0023] Compared to existing technologies, this invention and its preferred embodiment, the tool wear monitoring model, predicts tool wear status through a deep learning model; it uses transfer learning to adapt the tool wear monitoring model to tool wear prediction under the current working environment; and it achieves effective real-time monitoring of robot operation status and tool wear status. Furthermore, it establishes a complete digital twin system architecture for the milling robot through the high simulation capabilities of the Unity3D platform, achieving high-precision planning of the robot's machining trajectory and dynamic milling simulation, thus improving the visualization of milling processes. Through a dynamic milling simulation algorithm based on mesh deformation, it effectively simulates the material removal simulation effect during the milling process. The bounding box-based material removal algorithm limits the tool milling range to a certain area, effectively improving the efficiency of the material removal algorithm and simulation process; spatial coordinate transformation converts the mesh points in the workpiece coordinate system to the tool coordinate system, enabling the tool to mill the workpiece at different angles; and it provides strong support for the virtual-real mapping of the milling robot and the optimization of the machining process. Attached Figure Description

[0024] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0025] Figure 1This is a five-dimensional architecture diagram of a digital twin system for milling robots according to an embodiment of the present invention.

[0026] Figure 2 This is a flowchart illustrating the communication and data transmission management process between the virtual environment and physical robot devices in an embodiment of the present invention.

[0027] Figure 3 This is a flowchart illustrating the construction of a simulation model for the robot milling process in an embodiment of the present invention.

[0028] Figure 4 This is a flowchart of robot machining trajectory planning according to an embodiment of the present invention;

[0029] Figure 5 This is a flowchart of the data processing of the NC code parsing module in an embodiment of the present invention;

[0030] Figure 6 This is a flowchart of NC code lexical analysis according to an embodiment of the present invention;

[0031] Figure 7 This is a flowchart of NC code syntax analysis according to an embodiment of the present invention;

[0032] Figure 8 This is a flowchart of a milling simulation based on mesh deformation, according to an embodiment of the present invention.

[0033] Figure 9 This is a schematic diagram illustrating the determination of the distance between the cutting tool and the workpiece mesh in an embodiment of the present invention;

[0034] Figure 10 This is a structural diagram of the CNN-LSTM tool wear monitoring model according to an embodiment of the present invention;

[0035] Figure 11 This is a flowchart illustrating the model transfer learning process in an embodiment of the present invention.

[0036] Figure 12 This is a flowchart illustrating the real-time tool wear condition monitoring process according to an embodiment of the present invention.

[0037] Figure 13 This is a schematic diagram of the robot control interface of the digital twin system according to an embodiment of the present invention;

[0038] Figure 14 This is a simulation diagram of a robot milling operation using a digital twin system, as described in an embodiment of the present invention.

[0039] Figure 15 This is a schematic diagram of the three-dimensional visualization monitoring interface of the digital twin system according to an embodiment of the present invention;

[0040] Figure 16 This is a flowchart illustrating the virtual-real interaction process of a milling robot according to an embodiment of the present invention. Detailed Implementation

[0041] To make the features and advantages of this patent more apparent and understandable, specific embodiments are provided below for detailed explanation:

[0042] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0043] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0044] like Figures 1-12 As shown, this embodiment of the invention first provides a design scheme for a milling robot machining process simulation system based on digital twins. The system includes a physical layer, a virtual layer, a twin data layer, an application service layer, and the communication connections between them.

[0045] The physical layer includes physical entities, which include robots, electric spindles, cutting tools, workpieces, sensors, and data acquisition cards.

[0046] The virtual layer is a faithful mapping and simulation of physical entities in the virtual space, which can reflect their static and dynamic characteristics and keep updated synchronously with the physical entities;

[0047] The twin data layer includes static data and dynamic data. The twin data undergoes data processing and transmission management to drive the twin model's operation and to perform monitoring and feedback control.

[0048] The communication connection receives and monitors the operating status of physical devices through network communication interfaces and protocols, and then performs feedback control on the physical devices to achieve virtual-physical interaction;

[0049] The application service layer is used for simulation of robotic milling processes and identification and prediction of tool wear conditions, effectively realizing visualization of the milling process and optimizing robotic milling operations. For example... Figure 1 As shown.

[0050] The static data of the twin data layer mainly includes the robot's physical geometric attributes such as its geometry, size, and positional relationships. These attributes are determined during system initialization and do not change over time.

[0051] The dynamic data in the twin data layer mainly includes dynamic information such as the robot's six axes angle, speed, and acceleration. It is generated during system operation and reflects the robot's real-time status and behavior.

[0052] The transmission management of the digital twin data layer is achieved by accessing the physical robot through Socket communication to obtain robot running status data such as robot joint information, position coordinates, and spindle speed. By calling and executing specific Structured Query Language (SQL) statements, a connection is established with the SQL Server database, enabling the digital twin system to receive multi-source heterogeneous data from the physical space in real time.

[0053] The communication connection is based on a client / server architecture, consisting of a robot controller acting as the server and a digital twin system created by Unity3D as the client. TCP / IP protocol and Socket communication are used for interaction between the client and server. Utilizing the XCharts component, an external plugin in Unity3D, data is dynamically visualized through a UI interface in the form of line charts, text, etc. Figure 2 As shown.

[0054] The client uses multithreading technology. The main thread is responsible for establishing a Socket connection to the server and handling user interactions. Newly started threads are dedicated to receiving and parsing data from the server and performing subsequent processing. Once the data processing is complete, the new thread passes the result back to the main thread, which is responsible for updating the data and system response.

[0055] The processing simulation of the application service layer includes basic trajectory planning and visualization simulation; the processing simulation mainly displays the animation effect of real-time milling, and uses the corresponding material removal algorithm to ensure the accuracy and real-time performance of the processing simulation.

[0056] The simulation model construction process for robotic milling involves generating a workpiece model and NC code files using CAD / CAM software, constructing a workpiece mesh model, and designing a dynamic milling algorithm to simulate the material removal process. The NC code is then converted into an end-effector trajectory. Combined with robot inverse kinematics, the motion parameters of each robot joint are solved using the end-effector trajectory, and simulation verification is performed. Finally, the simulation results are output. Figure 3 As shown.

[0057] Trajectory planning includes joint space trajectory planning and Cartesian space trajectory planning;

[0058] The joint space trajectory planning uses the S-shaped velocity interpolation function to interpolate the robot's joint space trajectory, so that the change curve of the joint variables transitions smoothly, thereby achieving the continuity of joint movement.

[0059] Cartesian space trajectory planning obtains the end-effector pose by solving the S-curve interpolation algorithm, and then inversely solves the joint angles by solving the end-effector pose, thereby controlling the robot to achieve the expected motion trajectory.

[0060] The trajectory planning process is as follows: Import the NC code file, parse it, and read it as a string. After line-by-line lexical analysis, syntax analysis, and data compilation, the parsed data is uploaded to an SQL Server database for storage. Next, load the parsed code fields, read each field line by line, and perform S-curve interpolation on the interpolation instructions to complete the robot's end-effector trajectory planning and upload the end-effector pose data. Finally, perform inverse motion calculation on the robot's end-effector pose to obtain the angles of each joint, and store the data in the database for subsequent data-driven robot motion simulation. This completes the NC code compilation process. Figure 4 As shown.

[0061] The implementation of trajectory planning relies on the NC code parsing module. The NC code parsing module extracts valid information from the NC code text file, identifies and processes it, and converts it into motion parameters that the robot can recognize, thereby driving the simulated motion process of the robot's milling machining. NC code can be decomposed into specific character instructions, which mainly include program segment numbers and various function instruction words.

[0062] The NC code parsing module comprises three steps: lexical analysis, syntax analysis, and data compilation. The process proceeds sequentially: First, the NC code file is imported. After reading the text file, the system calls the lexical analysis module for processing. If the lexical analysis confirms no errors, the system further calls the syntax analysis module to perform syntax analysis on the code file. Code analysis is performed by iterating through all NC code fields until all NC code is compiled. If errors are detected in the NC code file during parsing, the system records the error information. After the NC code parsing module finishes compiling, the data is stored in an appropriate data structure and uploaded to a SQL Server database for management and retrieval. Figure 5 As shown.

[0063] Lexical analysis employs Regex regular expression technology to match and analyze the instructions in the code one by one. The system performs a normative check on the first letter of each instruction, verifying whether the numeric characters following the letter characters in the instruction are within the specified range. If the conditions are met, the lexical analysis of the instruction is determined to be compliant with relevant specifications, and the analyzed data is stored to prepare for the next step of syntax analysis. Figure 6 ;

[0064] Syntax analysis verifies the completeness and logical rationality of NC code. Taking the linear interpolation instruction G01 in NC code as an example, its standard form should be G01 X_Y_Z_F_. The interpolation instruction includes a feed rate F to control the tool feed rate. The G-series instructions need to match the corresponding coordinate instructions. Logical rationality verification includes checking the compliance between various instructions. For example, instructions G90 and G91 should be mutually exclusive and should not be used simultaneously in the same program segment. Also, if a program segment contains G53, it should not contain instructions such as G01 and G02. When NC code is found to be inconsistent with the plan, error messages and feedback are provided, such as... Figure 7 As shown.

[0065] Data compilation involves storing the relevant data information parsed from the syntax through an appropriate data structure and transmitting it to the SQL Server database for management. The data structure contains information such as the spindle's interpolation motion mode, feed rate, spindle speed, and trajectory coordinates.

[0066] The visualization simulation is based on a mesh deformation material removal algorithm, including a simulation model, simulation algorithm, and simulation verification;

[0067] The simulation model was constructed by drawing a 3D model of the workpiece blank using SolidWorks software and exporting it as an STL file. The STL file was then subjected to mesh subdivision and texture mapping using 3ds Max software to enhance the realism and visual effects of the model. The STL file was then exported as an FBX file. The FBX file was imported into Unity3D software, and a material removal algorithm based on mesh deformation was run on the FBX file to achieve the milling simulation effect.

[0068] The simulation algorithm uses a mesh deformation-based material removal algorithm to realize the milling simulation function. The essence of the mesh deformation-based material removal algorithm is to solve the positional changes of each point in the workpiece blank mesh under the action of the tool, and to reflect the changes in the workpiece shape by updating the point positions.

[0069] The simulation algorithm flow includes: controlling the virtual robot to move according to the planned path based on twin data; calculating the distance between the tool cutting point and the workpiece mesh point to determine whether contact has occurred; traversing the workpiece mesh and performing corresponding deformation processing on the contact parts; deformation processing is achieved by modifying the workpiece model mesh points; and refreshing the workpiece mesh to achieve the effect of simulating material removal. Figure 8 As shown.

[0070] Simulation verification involves driving a virtual robot to simulate the milling process of a physical robot, presenting the robot's milling trajectory to check for any abnormal trajectories.

[0071] As a preferred embodiment, a spherical bounding box is added at the milling center to determine the distance between the workpiece mesh points within the spherical bounding box and the milling center. The spherical bounding box has the milling center as its center, and its radius is adjusted based on the tool radius.

[0072] By defining a spherical bounding box, the three-dimensional spatial distance between the workpiece mesh points and the center point of the tool bounding box is calculated, thus obtaining the mesh points located within the spherical bounding box. This limits the space range of the tool milling operation, and the material removal algorithm is executed based on the contact between the tool and the workpiece mesh points. Figure 9 .

[0073] The material removal algorithm performs spatial coordinate transformation on the workpiece. This spatial coordinate transformation is based on a 4×4 homogeneous transformation matrix, which represents the translation and rotation transformation of the workpiece mesh's spatial coordinates within its own coordinate reference frame and the tool's coordinate reference frame.

[0074] Spatial coordinate transformation converts the workpiece mesh spatial coordinates into tool spatial coordinates. Milling decisions and mesh deformation are then performed in the tool spatial coordinates. Finally, the workpiece mesh spatial coordinates are restored back to their own spatial coordinates for visualization.

[0075] The tool wear condition identification and prediction module in the application service layer mainly predicts the tool wear condition through a deep learning model. It uses a digital twin system to adjust machining parameters such as spindle speed and feed rate and promptly reminds users to replace the tool to ensure the efficiency and quality of the machining process. A wear standard of 0.2mm is selected to define whether the tool has reached the dull stage.

[0076] As a preferred embodiment and a key design feature of this invention, a deep learning model is employed to predict tool wear based on milling force signals. Using a CNN-LSTM network structure, the CNN network is first used to extract features from the data. Then, the feature sequences extracted by the CNN are input into the LSTM network, which models the sequence data to learn the temporal characteristics of tool wear. The CNN-LSTM model combines convolutional and recurrent structures, enabling it to capture multi-scale features in the data at different levels.

[0077] The CNN-LSTM tool wear monitoring model consists of an input layer, convolutional layers, pooling layers, unfolded layers, LSTM layers, dropout layers, fully connected layers, and an output layer. The input layer receives the data; the convolutional and pooling layers are responsible for feature extraction and dimensionality reduction; the unfolded layer inputs the extracted feature sequences into the LSTM layer; the dropout layer prevents overfitting; the fully connected layer is used for feature fusion and output; and finally, the output layer provides a prediction of the wear state.

[0078] The CNN-LSTM-based deep learning model architecture consists of a three-layer convolutional neural network (CNN), a two-layer long short-term memory network (LSTM), and two fully connected layers. To enhance the network's local receptivity, a large kernel (25) is used in the first layer, with the kernel size gradually decreasing in the following three layers. To achieve feature dimensionality reduction, a max-pooling layer is added after each convolutional layer. Furthermore, to prevent overfitting, a Dropout layer with a retention probability of 0.25 is introduced between the LSTM layer and the fully connected layer. The structure is as follows: Figure 10 As shown.

[0079] The CNN-LSTM tool wear monitoring model uses transfer learning to adapt to tool wear prediction under the current working environment. The method first constructs a CNN-LSTM model, pre-trains the model using historical machining data, and then applies the model to the dataset under the current working conditions for training. The model's structural parameters are shared through different levels of similarity, thereby completing the model transfer process.

[0080] The model transfer learning process includes the following steps: First, loading a pre-trained CNN-LSTM model from the source domain. Second, retrieving and modifying the network structure of the original model, including adjusting the output layer, especially the fully connected layers, to meet the requirements of the target task. During this process, parameters need to be adjusted to ensure the model's performance on the new task. Then, merging the modified output layer with the first few layers of the original model to form a new network structure. Next, the training parameters for transfer learning need to be clearly defined, such as selecting an appropriate optimizer, setting the maximum number of iterations, and the learning rate. Then, training the modified model using training data from the target domain further fine-tunes the model parameters to better adapt it to the characteristics of the target task. Finally, saving the transfer-learned model and applying it to a tool wear prediction task under actual working conditions. The model transfer learning process is as follows: Figure 11 As shown.

[0081] The same dataset was used to compare the CNN model, the LSTM model, and the model combining CNN and LSTM (CNN-LSTM). Simultaneously, a dataset collected under actual working conditions was used to compare the CNN-LSTM model with the model that underwent transfer learning (CNN-LSTM-Transfer) to verify whether the transfer learning model can effectively predict tool wear under the current experimental environment. The evaluation metrics for each model are shown in Table 1.

[0082] Table 1 Comparison of Prediction Results of Tool Wear Monitoring Model

[0083]

[0084] The CNN-LSTM model outperformed standalone CNN or LSTM models in terms of evaluation metrics, demonstrating a more effective ability to predict tool wear. While the transfer-learning model showed a slight decrease in evaluation metrics compared to the original CNN-LSTM model, its prediction of the overall tool wear trend remained accurate and reliable. This indicates that the model can effectively predict tool wear conditions under various operating conditions, meeting the needs of tool wear monitoring during on-site machining.

[0085] In this embodiment, the dynamic data information visualization monitoring uses dynamically updated charts to statistically analyze and process various real-time data, thereby achieving effective real-time monitoring of the robot's operating status and tool wear status.

[0086] The dynamic data information visualization monitoring interface consists of text boxes, graphs, scenes, and milling robot models. It displays data such as robot joint motion parameters, equipment operating status, milling force signals, and tool wear prediction values ​​in real time through small windows.

[0087] The tool wear prediction value is generated by the digital twin system after real-time acquisition of sensor signals. The signal data is preprocessed and analyzed and then input into the tool wear prediction model to predict the tool wear state, which is then dynamically displayed in the Unity3D system interface.

[0088] The dynamic display of tool wear condition prediction is developed through a joint interactive approach using MATLAB and Unity3D software. The method employs a MATLAB-encapsulated DLL program to acquire signals in real time and process the data to predict the tool wear condition. The prediction results are then visualized in Unity3D. The real-time tool wear condition monitoring process is as follows: Figure 12 As shown.

[0089] The real-time tool wear condition monitoring process includes: configuring the NI DAQ driver in MATLAB software, and achieving real-time acquisition of data from the NI acquisition card by creating an NI session object and setting relevant NI acquisition parameters. During data acquisition, preprocessing operations such as downsampling and noise reduction filtering are performed on the data. Simultaneously, a communication connection is established between MATLAB and an SQL Server database for data transmission, management, and storage.

[0090] A tool wear monitoring model was built in MATLAB software. This model utilizes historical datasets, employs a CNN-LSTM network for initial training, and then undergoes transfer learning training to obtain a tool wear monitoring model more adapted to actual working conditions. Data acquisition and tool wear monitoring are integrated and encapsulated into MATLAB functions for subsequent use.

[0091] The tool wear monitoring function written in MATLAB is packaged into a DLL file, and this DLL file, along with related MATLAB DLL library files, is imported into Unity3D software. In Unity3D, C# code is used to call the DLL file, thereby driving the tool wear monitoring model. By acquiring data online and monitoring the tool status during machining, the tool wear curve is dynamically displayed for visualization analysis.

[0092] Based on the monitored tool wear status, the system adjusts machining parameters such as spindle speed and feed rate in real time. When the tool wear value exceeds the system's preset threshold, a warning window pops up to remind the operator to replace the tool in time and stop the machine to wait for the next operation. This effectively reduces accidents and losses caused by tool wear during machining, thereby improving machining efficiency.

[0093] The control interface of the digital twin system for milling robots is as follows: Figure 13 As shown, the robot control panel comprises three modules: Cartesian control, joint control, and NC control. In the Cartesian control module, the robot's initial and target end-effector poses and motion parameters are input. The system performs S-curve velocity interpolation planning on the end-effector pose and uses inverse kinematics to obtain the corresponding joint angles. Then, it controls the virtual robot's joints to simulate motion according to the given end-effector positions, and uses this joint angle data to drive the physical robot's motion. In the joint control module, the robot's initial and target joint angles are directly specified. The system directly controls the virtual robot's simulated motion using an interpolation algorithm and uses the joint angle data to drive the motion of the actual robot. In the NC control module, NC files are imported and compiled into robot motion instructions, thereby controlling the continuous motion simulation of the virtual robot and driving the physical robot to move continuously according to the NC code trajectory.

[0094] The milling simulation function of the digital twin system for milling robots imports NC code files into the digital twin system, compiles them into motion command code for the robot, and then performs milling simulation in a virtual environment. During the milling simulation, a dynamic milling simulation algorithm continuously updates the mesh point information of the workpiece model to simulate the removal of workpiece material and present the robot's milling trajectory to check for any abnormal trajectories. To verify the accuracy and practicality of this milling simulation function, pre-designed NC code was used in actual testing to simulate the virtual robot milling a workpiece blank according to a predetermined trajectory.

[0095] The obtained milling simulation effect is as follows Figure 14As shown, while maintaining a frame rate (FPS) of over 30, the visualization of workpiece material removal during milling simulation was achieved. This met the system's simulation accuracy and applicability requirements, and verified the reliability and accuracy of the milling simulation function.

[0096] The 3D visualization interface of the digital twin system for milling robots, such as Figure 15 The 3D visualization interface of the milling robot digital twin system consists of various text boxes, graphs, scenes, and a model of the milling robot. Within the data visualization interface, small windows display real-time data such as robot joint motion parameters, equipment operating status, and milling force signals. By dynamically updating charts to statistically analyze and process various real-time data, effective real-time monitoring of the robot's operating status and milling progress is achieved.

[0097] By using twin data-driven models, a virtual-real mapping is established between the virtual and physical entities of a robot, enabling virtual-real interaction. On one hand, the simulation-verified results of the virtual entity are mapped onto the physical robot, thereby controlling and driving the robot's milling motion. On the other hand, the twin data of the physical entity is used to drive the movement of the virtual entity, allowing it to reflect the behavior of the physical entity in real time, monitor its operating status, promptly detect anomalies, and provide feedback control. This improves the robustness and performance of the digital twin system, such as... Figure 16 As shown.

[0098] First, a workpiece model to be machined is built in SolidWorks software and imported into UG NX software for toolpath planning, generating a post-processing file of NC code for milling. Then, this NC code file is imported into a milling robot digital twin system built in Unity 3D software. In this system, the NC code parsing module parses it into robot end-effector motion commands, performs trajectory planning, and interpolates and inversely solves the results to obtain robot joint angles, driving the virtual robot's motion simulation and visualizing the toolpath. During the simulation, a dynamic milling simulation algorithm is used to remove material from the workpiece, obtaining the simulation results of the milling process. The correctness of the machining trajectory is verified during the simulation. Subsequently, the verified machining trajectory is exported for actual control of the physical robot to perform milling. During this actual machining process, the robot's operating status is continuously monitored, and the virtual robot is driven in real time to simulate the milling process of the physical robot. Through 3D visualization monitoring, collisions or other abnormal situations can be detected and warned in a timely manner, and necessary actions can be taken to handle them.

[0099] To achieve effective data transmission and management, and to facilitate the querying of historical data, relevant twin data is uploaded to a SQL Server database. This ensures data integrity and traceability, providing support for subsequent manufacturing optimization and decision-making.

[0100] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0105] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0106] This patent is not limited to the above-described preferred embodiment. Anyone can derive other forms of a method for real-time monitoring of tool wear status of milling robots that integrates digital twins and deep learning under the guidance of this patent. All equivalent changes and modifications made within the scope of this patent application shall be covered by this patent.

Claims

1. A digital twin system for a milling robot, characterized in that, The real-time tool wear monitoring method adopted includes: based on a digital twin system and a CNN-LSTM deep learning model, firstly, the CNN network is used to extract features from the data, then the feature sequences extracted by the CNN are input into the LSTM network, and the LSTM network is used to model the sequence data to learn the temporal characteristics of tool wear status; the tool wear monitoring model is trained using a machining history dataset; and the model is retrained using actual machining data from the field through transfer learning technology to analyze and predict tool wear values ​​online. The application service layer of the milling robot digital twin system includes a machining process simulation module, a tool wear state identification and prediction module, and a dynamic data information visualization and monitoring module. The machining process simulation module is used to display the effect of real-time milling and uses a material removal algorithm to ensure the accuracy and real-time performance of the machining process simulation. The dynamic data information visualization and monitoring module uses dynamically updated charts to perform statistical analysis on various real-time data to achieve real-time monitoring of the robot's operating status and milling progress. The machining process simulation module controls the virtual robot to move according to the path plan based on twin data; calculates the distance between the tool cutting point and the workpiece mesh point to determine whether contact occurs; traverses the workpiece mesh and performs corresponding deformation processing on the contact part; the deformation processing is achieved by modifying the workpiece model mesh point; refreshes the workpiece mesh to achieve the effect of material removal simulation; The material removal algorithm defines a spherical bounding box to limit its spatial range. The spherical bounding box is centered at the milling center point of the tool, and its radius is adjusted based on the tool radius. The material removal algorithm based on the spherical bounding box calculates the three-dimensional spatial distance between the workpiece mesh points and the center point of the tool bounding box to obtain the mesh points located within the spherical bounding box, thereby limiting the milling space of the tool. The material removal algorithm is then executed based on the contact between the tool and the workpiece mesh points, performing material removal algorithm calculations on the mesh points within the bounding box to exclude mesh points that do not need to participate in the calculation and undergo deformation processing.

2. The digital twin system for a milling robot according to claim 1, characterized in that: The material removal algorithm requires spatial coordinate transformation of the workpiece; the spatial coordinate transformation transforms the workpiece mesh spatial coordinates into tool spatial coordinates, and milling judgment and mesh deformation are performed in the tool spatial coordinates. Finally, the workpiece mesh spatial coordinates are restored back to their own spatial coordinates for visualization. The spatial coordinate transformation is accomplished based on a 4×4 homogeneous transformation matrix, which is used to represent the translation and rotation transformation of the spatial coordinates of the workpiece mesh in its own coordinate reference system and the tool coordinate reference system. The coordinates of grid point P in the workpiece coordinate system are obtained by homogeneous transformation, enabling the tool to mill the workpiece at different angles.

3. The digital twin system for a milling robot according to claim 1, characterized in that: The simulation model for the milling process is constructed by generating a workpiece model and NC code file using software with CAD / CAM capabilities, building a workpiece mesh model, designing a dynamic milling algorithm to simulate the material removal process, converting the NC code into an end-effector trajectory, combining it with robot inverse kinematics, solving the motion parameters of each joint of the robot through the end-effector trajectory, performing simulation verification, and finally outputting the simulation results.

4. The digital twin system for a milling robot according to claim 3, characterized in that: Trajectory planning is achieved through the NC code parsing module. The NC code parsing module extracts information from the NC code text file, identifies and processes it, and converts it into motion parameters that the robot can recognize, so as to drive the simulated motion process of the robot milling. Trajectory planning includes joint space trajectory planning and Cartesian space trajectory planning; The joint space trajectory planning uses the S-shaped velocity interpolation function to interpolate the robot's joint space trajectory, so that the change curve of the joint variables transitions smoothly, thereby achieving the continuity of joint movement. Cartesian space trajectory planning obtains the end-effector pose by solving the S-curve interpolation algorithm, and then inversely solves the joint angles by solving the end-effector pose, thereby controlling the robot to achieve the expected motion trajectory.

5. A digital twin system for a milling robot according to claim 4, characterized in that: The NC code parsing module consists of three steps: lexical analysis, syntax analysis, and data compilation, which are performed sequentially. Lexical analysis uses Regex regular expression technology to match and analyze the instructions in the code one by one; Syntax analysis verifies the completeness and logical rationality of NC code; data compilation parses the relevant data information, stores it in a data structure, and transmits it to an SQL Server database for management; the data structure contains information such as the spindle's interpolation motion mode, feed rate, spindle speed, and trajectory coordinate values. The simulation model was constructed by drawing a 3D model of the workpiece blank using SolidWorks software and exporting it as an STL format file. The STL file was meshed and textured using 3ds Max software to enhance the realism and visual effect of the model, and then the STL file was exported as an FBX file. Import the FBX file into Unity3D software, run a material removal algorithm based on mesh deformation on the FBX file to achieve a milling simulation effect.

6. A digital twin system for a milling robot according to claim 1, characterized in that: The model transfer learning process includes: First, loading a pre-trained CNN-LSTM model in the source domain; second, retrieving and modifying the network structure of the original model, including adjusting the output layer (including fully connected layers) to meet the requirements of the target task; during this process, adjusting the parameters to ensure the model's performance on the new task; subsequently, merging the modified output layer with the first few layers of the original model to form a new network structure; then defining the training parameters for transfer learning and training the modified model using training data from the target domain to further fine-tune the model parameters to better adapt to the characteristics of the target task; finally, saving the model after transfer learning and applying it to the tool wear prediction task under actual working conditions.

7. A digital twin system for a milling robot according to claim 1, characterized in that: The structure of the tool wear monitoring model includes a model data input layer, a convolutional layer, a pooling layer, an unfolded layer, an LSTM layer, a dropout layer, a fully connected layer, and an output layer. The input layer receives data, the convolutional and pooling layers are responsible for feature extraction and dimensionality reduction, the unfolded layer inputs the extracted feature sequence into the LSTM layer, the dropout layer is used to prevent overfitting, the fully connected layer is used for feature fusion and output, and finally the output layer gives the prediction result of the wear state.

Images