Robotic processing simulation system based on mixed reality
Through a hybrid reality-based robot processing simulation system, combined with a hybrid robot and camera, tool information is collected and processed in real time, the problem of unintuitive and poor accuracy of trajectory verification in milling processing of large-scale parts in the aerospace field is solved, and efficient and accurate are achieved verification of processing trajectory.
Patent Information
- Application Number
- CN202211258521.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-14
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-10-14
AI Technical Summary
In the milling of large-scale parts in the aerospace field, the traditional machining trajectory verification methods have problems such as unintuitive, inefficient and poor accuracy, and the existing mixed reality technology is difficult to effectively combine with robot processing, resulting in poor processing results.
A robot processing simulation system based on mixed reality is adopted, and through the combination of hybrid robots, cameras and computers, tool vibration information and end images are collected and processed in real time, combining virtual and real information to achieve accurate processing trajectory verification.
Real-time online detection of tool position and attitude is achieved, the milling accuracy of machining simulation is improved, verification time is reduced, the effect is intuitive, easy for operators to use, and the milling trajectory of aerospace components can be quickly verified.
Smart Images

Figure CN115609583B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an industrial hybrid robot processing trajectory verification system, and in particular to a robot processing simulation system based on mixed reality. Background Art
[0002] In order to reduce the weight of aircraft and improve its carrying capacity and maneuverability, more and more parts are used in the aerospace field, such as spacecraft cabins, aircraft fuselages, etc. The milling processing cost of large parts in aerospace is high and the processing difficulty is great. In the process of robot processing, wrong processing instructions will cause the tool to overcut, undercut, collide, etc. during the processing, thus scrapping the workpiece and tool, causing economic losses.
[0003] In order to solve the above problems, the processing trajectory needs to be verified after it is generated. The traditional methods for verifying the processing trajectory include the empty cutting method, the trial cutting method, and the processing path display method. The empty cutting method is based on human visual observation and can only ensure that the path is roughly correct. The trial cutting method will increase costs and is not suitable for the processing of high-value materials. The processing path display method can only guarantee the path and cannot show the actual effect after processing. The above methods are not intuitive and inefficient. The offline simulation method often cannot intuitively display some key information. In the actual production process, the stress changes, temperature fields, and vibrations during robot processing may cause the operator to ignore some key information, thus failing to achieve the desired processing effect.
[0004] Mixed reality technology is a new technology that integrates real-world information and virtual-world information into reality. It applies virtual information (visual information, sound, touch, etc.) that is difficult to experience in a certain time and space range in the real world to the real world through computer simulation. It is perceived by human senses, and the real environment and virtual objects are superimposed on the same screen or space in real time and exist simultaneously. The biggest advantage of mixed reality is that it is intuitive and convenient.
[0005] The existing mixed reality technology is: by modeling the machine tool and the workpiece, the operator can operate the virtual machine tool to process the virtual workpiece while wearing the mixed reality device, so as to achieve the effect of verifying the processing trajectory. The existing public technology focuses on the processing simulation of machine tools, and rarely combines with the emerging robot processing. In addition, the constructed virtual machine tool often cannot reproduce the real machine tool mechanism, which makes the virtual machine tool processing trajectory effect different from the real trajectory processing effect and poor accuracy. When the mixed reality device is running, it will freeze due to the large amount of calculation, affecting the real-time performance. Summary of the invention
[0006] The purpose of the present invention is to overcome the deficiencies in the prior art and to propose a robot machining simulation system based on mixed reality, which can achieve accurate and real-time machining simulation effects.
[0007] The objective of the present invention is achieved through the following technical solutions:
[0008] The present invention provides a robot processing simulation system based on mixed reality, comprising:
[0009] A hybrid robot, at the end of which a real tool is installed, the hybrid robot is used to execute the machining trajectory of the real tool to be verified input by the computer, and when the real tool performs cutting according to the machining trajectory to be verified, the sensor on the hybrid robot collects vibration information of the real tool and transmits it to the computer through the controller of the hybrid robot;
[0010] A camera, wherein the camera is used to collect in real time an image of a terminal of the hybrid robot during the hybrid robot cutting process according to the processing trajectory to be verified, and then output the image of the terminal of the hybrid robot to a computer;
[0011] A computer, the computer is used to receive the real tool vibration information input by the hybrid robot controller and output it to the UI panel display module in the information display module through the remote communication module, and receive the hybrid robot terminal image input by the camera in real time, and output the hybrid robot terminal image to the tool positioning module installed on the computer. The tool positioning module includes an object recognition module and a posture estimation module. The object recognition module is used to read the hybrid robot terminal image output by the camera and realize the detection of the real tool in the robot terminal image through a machine learning algorithm, and the real tool detected in the image is framed by a rectangular window, and the feature extraction algorithm is used to extract the key point features of the real tool from the framed area; the posture estimation module is used to input the extracted real tool key point features into the posture solver to solve the real tool posture, and then input the solved real tool posture into the filtering algorithm for stabilization processing, and the stabilized real tool posture is transmitted through the camera coordinate system {T b}Relative to the robot end coordinate system {T a The homogeneous transformation matrix of After transformation, the real tool is obtained in the camera coordinate system {T b}, the real tool is placed in the camera coordinate system {T b}The pose and homogeneous transformation matrix under Output to remote communication module;
[0012] A mixed reality device, in which a milling simulation module, an information display module and a command receiving and output module are installed; the milling simulation module includes a virtual-real coordination module and a processing milling module, in which a virtual tool and a virtual workpiece developed by Unity are arranged, and the virtual-real coordination module is used to read the posture and homogeneous transformation matrix of the real tool at the end of the robot transmitted by the posture estimation module through the remote communication module And read the camera coordinate system {T b}Relative to the mixed reality coordinate system{T c The homogeneous transformation matrix of According to the equation Solve the robot end coordinate system {T a}Transformation relative to the mixed reality coordinate system established at the mixed reality device location In the robot end coordinate system {T a The actual tool position under the coordinate system The real tool is transformed into the mixed reality coordinate system {T c}, the workpiece coordinate system {T d}Relative to the mixed reality coordinate system{T c Homogeneous transformation matrix Combining equations Solve the actual tool at the end of the robot in the workpiece coordinate system {T d}, the virtual workpiece is placed in the workpiece coordinate system {T d}, the position of the real tool and the virtual workpiece can be solved by the real tool position and the virtual workpiece position in the workpiece coordinate system and output to the processing and milling module;
[0013] The command receiving and outputting module is used to receive the start and stop commands input by the human-machine interaction module and output them to the controller of the hybrid robot. If the start and stop commands input are to start execution, the controller processes the machining trajectory of the real tool to be verified input by the computer into a control command recognizable by the hybrid robot, and drives the hybrid robot to execute the machining trajectory of the real tool to be verified;
[0014] The machining and milling module is used to read the coordinates of the real tool and the virtual workpiece in the workpiece coordinate system {T d}, and then the real tool is placed in the workpiece coordinate system {T d} is assigned to the virtual tool in real time, so that the real tool and the virtual tool in the mixed reality device overlap in position. After the hybrid robot receives the start signal output by the instruction receiving and output module, the real tool of the hybrid robot and the virtual tool in the mixed reality device move together according to the processing trajectory of the real tool to be verified, wherein the virtual tool and the virtual workpiece perform cutting collision intersection operation, and when a collision occurs, the part of the virtual workpiece that intersects with the virtual tool is removed, and the rendered object information of the virtual workpiece after cutting is output to the visual rendering module in the information display module in real time. During the cutting process, the sensor on the hybrid robot collects the vibration information of the real tool and transmits it to the computer through the controller of the hybrid robot; the virtual tool is equal in size and shape to the real tool at the end of the robot, and the virtual workpiece is equal in size and shape to the real workpiece to be processed;
[0015] The information display module includes a three-dimensional reconstruction module, a space anchoring module, a visual rendering module and a UI panel display module. The UI panel display module is provided with a virtual UI panel developed by Unity. The three-dimensional reconstruction module is used to read the scattered point cloud information of the surrounding environment of the mixed reality device, complete the three-dimensional reconstruction of the surrounding environment of the mixed reality device by performing point cloud fusion processing on the scattered point cloud information, obtain the overall environment point cloud information, and output the overall environment point cloud information to the space anchoring module;
[0016] The spatial anchoring module is used to read the overall environment point cloud information input by the 3D reconstruction module, and obtain the camera coordinate system {T b}Relative to the mixed reality coordinate system{T c The homogeneous transformation matrix of Will Output to the virtual-real coordination module in the milling simulation module;
[0017] The visual rendering module is used to read the rendering object information input by the processing and milling module, and project the virtual workpiece onto the lens of the mixed reality device through the visual algorithm rendering provided by the mixed reality device. The operator can observe the effect of the virtual workpiece during the cutting process through the mixed reality device to verify whether the real tool processing trajectory is correct:
[0018] The UI panel display module is used to receive the vibration information input by the computer through the remote communication module and the real tool posture information input by the tool positioning module, and display the real tool vibration information and the real tool posture information on the virtual UI panel. The operator can judge whether the processing trajectory needs to be optimized by observing the real tool vibration information on the virtual UI panel;
[0019] The human-computer interaction module is used to receive the gesture and voice data of a person collected by the sensor on the mixed reality device, perform semantic segmentation processing and conversion on the data, and then output the start and stop instructions to the instruction receiving and output module of the mixed reality device.
[0020] Compared with the prior art, the present invention has the following advantages:
[0021] 1. The present invention can detect the actual tool position and posture in real time online.
[0022] 2. The present invention can accurately place the virtual workpiece on the real fixture, cooperate with the camera to detect the position and posture of the real tool and assign it to the virtual tool, so that the virtual tool can replace the real tool to mill the virtual workpiece, and by observing the processing effect of the virtual workpiece, the problem of trajectory verification in the milling process of large parts can be solved.
[0023] 3. The present invention improves the milling accuracy of machining simulation, reduces the time for verifying machining trajectories, can quickly verify the milling trajectories of various aerospace parts, and has good economy.
[0024] 4. The invention is intuitive and easy for operators to use. It can collect and integrate various visible and invisible data during the milling process and display them to the operator in real time and accurately. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a schematic diagram of the composition of the robot processing simulation system based on mixed reality of the present invention;
[0026] Figure 2 It is a schematic diagram of the structure of a robot processing simulation system based on mixed reality of the present invention;
[0027] Figure 3 It is a schematic diagram of the process flow of the robot processing simulation system based on mixed reality of the present invention; DETAILED DESCRIPTION
[0028] The present invention is described in detail below with reference to the accompanying drawings and specific implementation processes.
[0029] like Figure 1 As shown, the robot processing simulation system based on mixed reality of the present invention includes:
[0030] A hybrid robot is provided with a real tool at the end thereof. The hybrid robot is used to execute the machining trajectory of the real tool to be verified input by the computer, and when the real tool performs cutting according to the machining trajectory to be verified, the sensor on the hybrid robot collects vibration information of the real tool and transmits it to the computer through the controller of the hybrid robot. The hybrid robot can select the existing TriMule-800 hybrid robot.
[0031] The camera is used to collect the image of the end of the hybrid robot in real time during the hybrid robot cutting process according to the processing trajectory to be verified, and then output the image of the end of the hybrid robot to the computer. The camera can be a Zed binocular camera.
[0032] A computer is used to receive the real tool vibration information input by the hybrid robot controller and output it to the UI panel display module in the information display module through the remote communication module, and receive the hybrid robot terminal image input by the camera in real time, and output the hybrid robot terminal image to the tool positioning module installed on the computer. The tool positioning module is used to identify the robot terminal tool in real time and solve the position and posture of the tool. The tool positioning module includes an object recognition module and a posture estimation module. The object recognition module is used to read the hybrid robot terminal image output by the camera and detect the real tool in the robot terminal image through a machine learning algorithm (such as an SVM algorithm, see the paper Vapnik, V. and Chervonenkis, A., 1964. A note on class of perceptron. Automation and Remote Control, 24. (Support vector machine)), select the real tool detected in the image using a rectangular window, and use a feature extraction algorithm (such as an ORB algorithm, see the paper Rublee E, Rabaud V, Konolige K, et al. ORB: An efficient alternative to SIFT or SURF [C]. International Conference on Computer Vision. IEEE Computer Society, 2011: 2564-2571 (An effective alternative to SIFT algorithm and SURF algorithm)) to extract key point features of the real tool from the selected area;
[0033] The pose estimation module is used to input the extracted key point features of the real tool into a pose solver (such as EPnP, see the paper Lepetit, V., Moreno-Noguer, F. & Fua, P. EPnP: An Accurate O(n) Solution to the PnPProblem. Int J ComputVis 81, 155–166 (2009) (An accurate solution to the PnP problem)) to solve the real tool pose, and then solve the real tool pose (the real tool pose at the end of the robot includes the tool in the robot end coordinate system {T aThe x, y, z axis positions and rotational postures around the x, y, z axis) are input into a filtering algorithm (such as Kalman filtering, see the paper Yoon Y, Kosaka A, Kak A C. A New Kalman-Filter-Based Framework for Fast and Accurate Visual Tracking of RigidObjects[J]. IEEE TRANSACTIONS ON ROBOTICS, 2008.) for stabilization processing, and the stabilized real tool posture is transmitted through the camera coordinate system {T b}Relative to the robot end coordinate system {T a The homogeneous transformation matrix of After transformation, the real tool is obtained in the camera coordinate system {T b}, the real tool is placed in the camera coordinate system {T b}The pose and homogeneous transformation matrix under Output to remote communication module.
[0034] Robot end coordinate system {T a} is: Take the center of mass of the tool at the end of the hybrid robot as the coordinate origin, and establish Z along the tool axis a Axis, establish X along the tool radial direction a axis, establish Y according to the right-hand theorem a axis.
[0035] Camera coordinate system {T b} is: Take the center of the main camera as the coordinate origin, and establish X along the horizontal center b Axis, establishes Y along the vertical direction b axis, establish Z according to the right-hand theorem b axis.
[0036] The tool positioning module has the functions of object recognition and posture estimation; wherein the object recognition function identifies the real tool at the end of the robot in the image input by the camera through a recognition algorithm, tracks the real tool at the end in real time and frames it with a rectangular frame; the posture estimation function uses the real tool image at the end of the robot selected in the previous step, and cooperates with the posture solver to obtain the real-time posture of the real tool at the end of the robot.
[0037] A mixed reality device, in which a milling simulation module, an information display module and a command receiving and output module are installed. The milling simulation module is used to realize the function of processing a virtual workpiece with a real tool. In order to realize this function, it is necessary to set a virtual tool so that the size, shape and position of the real tool and the virtual tool coincide, and let the virtual tool replace the real tool to perform milling on the virtual workpiece. At the same time, the virtual tool is set to be invisible so that the operator can see the effect of the real tool processing the virtual workpiece. The milling simulation module includes a virtual-reality coordination module and a processing milling module, in which a virtual tool and a virtual workpiece developed by Unity are provided. The virtual-reality coordination module is used to read the posture and homogeneous transformation matrix of the real tool at the end of the robot transmitted by the posture estimation module through the remote communication module (the existing TCP communication protocol can be used). And read the camera coordinate system {T b}Relative to the mixed reality coordinate system{T c The homogeneous transformation matrix of According to the equation Solve the transformation relationship between the robot end coordinate system and the mixed reality coordinate system In the robot end coordinate system {T a The actual tool position under the coordinate system The real tool is transformed into the mixed reality coordinate system {T c}. d}Relative to the mixed reality coordinate system{T c Homogeneous transformation matrix Combining equations Solve the actual tool at the end of the robot in the workpiece coordinate system {T d The virtual workpiece is placed in the workpiece coordinate system {T d The position is set under the control of the real tool position and the virtual workpiece position, and the posture relationship between the real tool and the virtual workpiece in the workpiece coordinate system can be solved and output to the machining and milling module.
[0038] Where: Mixed reality coordinate system {T c} is: Take the optical center of the camera on the mixed reality device as the center coordinate origin, and establish an X coordinate along the horizontal center of the camera plane. c Axis, establishes the Y direction along the vertical direction of the camera c axis, establish Z according to the right-hand theorem c axis.
[0039] Workpiece coordinate system {T d}: Take the center of the virtual workpiece as the coordinate origin and establish X d Axis, establish Z along the direction perpendicular to the bottom surface of the virtual workpieced Axis, according to the right-hand theorem to establish X d axis.
[0040] The instruction receiving and output module is used to receive the start and stop instructions (whether to start executing the processing trajectory) input by the human-computer interaction module and output it to the controller of the hybrid robot. If the input start and stop instructions are to start execution, the controller will process the processing trajectory of the real tool to be verified input by the computer into a control instruction recognizable by the hybrid robot, and drive the hybrid robot to execute the processing trajectory of the real tool to be verified.
[0041] The machining and milling module is used to read the coordinates of the real tool and the virtual workpiece in the workpiece coordinate system {T d}, and then the real tool is placed in the workpiece coordinate system {T d} is assigned to the virtual tool in real time, so that the real tool and the virtual tool in the mixed reality device overlap. After the hybrid robot receives the start signal output by the instruction receiving and output module, the real tool of the hybrid robot and the virtual tool in the mixed reality device move together according to the processing trajectory of the real tool to be verified. Among them, the virtual tool and the virtual workpiece perform cutting collision intersection calculation, and when a collision occurs, the part of the virtual workpiece that intersects with the virtual tool is removed. The rendered object information of the virtual workpiece after cutting is output to the visual rendering module in the information display module in real time. During the cutting process, the sensor on the hybrid robot collects the vibration information of the real tool and transmits it to the computer through the controller of the hybrid robot. The virtual tool is equal in size and shape to the real tool at the end of the robot; the virtual workpiece is equal in size and shape to the real workpiece to be processed.
[0042] The milling simulation module performs the following steps:
[0043] Step 101, the real tool pose of the robot end input by the mixed reality device is calculated according to the equation That is, the robot end coordinate system {T a}Relative to the mixed reality coordinate system{T c The homogeneous transformation matrix of Input from the tool pose estimation module, Input by the spatial anchoring module. The real tool pose at the end of the robot is obtained from the robot end coordinate system {T a} to the mixed reality coordinate system {T c} to align the coordinate system and set the coordinate system in the mixed reality coordinate system {T c The virtual tool under the robot and the real tool at the end of the robot are in the mixed reality coordinate system {T c}. By the workpiece coordinate system {T d}Relative to the mixed reality coordinate system{T cHomogeneous transformation matrix Combining equations The actual tool at the end of the robot can be solved in the workpiece coordinate system {T d}, so that the virtual tool and the virtual workpiece are unified in the same coordinate system {T d}, the position of the virtual tool relative to the virtual workpiece can be obtained from the real tool position and the virtual workpiece position, and then it can be determined whether the virtual tool collides with the virtual workpiece.
[0044] is the robot end coordinate system {T a}Relative to the mixed reality coordinate system{T c} homogeneous transformation matrix; is the camera coordinate system {T b}Relative to the mixed reality coordinate system{T c} homogeneous transformation matrix; is the workpiece coordinate system {T d}Relative to the mixed reality coordinate system{T c} Homogeneous transformation matrix; where {T a} represents the robot end coordinate system, {T b} represents the camera coordinate system, {T c} represents the mixed reality coordinate system, {T d} represents the workpiece coordinate system.
[0045] Step 102, by obtaining the position and posture of the virtual tool relative to the virtual workpiece, determine whether the virtual tool collides with the virtual workpiece; if a collision occurs, remove the part of the virtual workpiece that intersects with the virtual tool. Then output the rendering object information of the virtual workpiece (including the shape, size, and position and posture of the virtual workpiece) to the information display module in real time.
[0046] The information display module includes a three-dimensional reconstruction module, a spatial anchoring module, a visual rendering module and a UI panel display module. The UI panel display module is provided with a virtual UI panel developed by Unity. The three-dimensional reconstruction module is used to read the scattered point cloud information of the environment around the mixed reality device, complete the three-dimensional reconstruction of the environment around the mixed reality device by performing point cloud fusion processing on the scattered point cloud information, obtain the overall environment point cloud information, and output the overall environment point cloud information to the spatial anchoring module;
[0047] The spatial anchoring module is used to read the overall environment point cloud information input by the 3D reconstruction module, and obtain the camera coordinate system {T b}Relative to the mixed reality coordinate system{T c The homogeneous transformation matrix of Will Output to the virtual-real coordination module in the milling simulation module, Solve in the virtual-real coordination module The equation The required part of
[0048] The visual rendering module is used to read the rendering object information input by the processing and milling module, and project the virtual workpiece onto the lens of the mixed reality device through the visual algorithm rendering provided by the mixed reality device. The operator observes the effect of the virtual workpiece in the milling process through the mixed reality device to verify whether the real tool processing trajectory is correct;
[0049] The UI panel display module is used to receive the processing information (vibration information of the real tool) input by the computer through the remote communication module and the real tool posture information input by the tool positioning module, and display the real tool vibration information and the real tool posture information on the virtual UI panel. The reason for the vibration of the real tool is not the vibration generated when cutting the workpiece, but the vibration caused by the unreasonable design of the robot trajectory, which leads to the sudden change of speed and acceleration. Vibration is difficult to observe with the naked eye and requires sensor measurement. The operator can judge whether the processing trajectory needs to be optimized by observing the real tool vibration information on the virtual UI panel.
[0050] The human-computer interaction module can be the human-computer interaction module that comes with the mixed reality device, which is used to receive the gesture and voice data of the person collected by the sensor on the mixed reality device and perform semantic segmentation and conversion on the data (converting the gesture and voice into start and stop commands that can be executed by the device), and then output the start and stop commands to the command receiving and output module of the mixed reality device.
[0051] The human-computer interaction module is a module for people to interact with the virtual three-dimensional objects projected by the mixed reality device. By inputting people's gestures and voice data, the human-computer interaction module processes and converts the data, and then sends start and stop instructions to the mixed reality device. It has a gesture recognition module, a voice recognition module, and a key interaction module; wherein the gesture recognition module can recognize human hand gestures in real time, including the index finger clicking to represent a single click of a button, and a fist to represent the expansion of the main menu; the voice recognition module recognizes the voice instructions issued by people and performs corresponding operations; the key interaction module enables the operator to click the button on the virtual UI input panel on the human-computer interaction module to realize the corresponding function. The virtual UI input panel is used to input start and stop instructions to the mixed reality device through the human-computer interaction module, and the virtual UI input panel can be developed through Unity software.
[0052] like Figure 3 As shown, the working process of the mixed reality device is: receiving the real tool vibration information input by the computer through the remote communication module and outputting it to the UI panel display module in the information display module on the mixed reality device; receiving the real tool posture of the robot end transmitted by the tool positioning module through the remote communication module, and inputting the received real tool posture into the virtual-reality coordination module in the milling simulation module; the 3D reconstruction module collects the point cloud information of the surrounding environment through the camera on the mixed reality device to obtain the point cloud information of the overall environment and outputs it to the space anchor module; the space anchor module reads the overall environment point cloud information input by the 3D reconstruction module, and obtains it through the SLAM algorithm processing. And output to the virtual-real coordination module; the virtual-real coordination module reads the pose and homogeneous transformation matrix of the real tool at the end of the robot transmitted by the pose estimation module And read the camera coordinate system {T b}Relative to the mixed reality coordinate system{T c The homogeneous transformation matrix of The solved real tool and virtual workpiece are placed in the workpiece coordinate system {T d} is output to the machining and milling module; the machining and milling module is used to read the real tool and the virtual workpiece in the workpiece coordinate system {T d}, and output the rendered object information of the virtual workpiece after cutting to the visual rendering module in real time for rendering; the human-computer interaction module receives the gesture and voice data of the person collected by the sensor on the mixed reality device and performs semantic segmentation processing on the data, and after the processing is completed, outputs the start and stop instructions to the controller of the hybrid robot through the instruction receiving and output module on the mixed reality device. The mixed reality device can adopt the existing Microsoft HoloLens2 mixed reality glasses.
[0053] The following is an explanation of how the mixed reality-based robot machining simulation system works.
[0054] In the first step, the operator wears the mixed reality device and starts it. The system automatically starts the initialization operation, scans the surrounding environment through the 3D reconstruction module of the mixed reality device, and performs 3D reconstruction of the environment to determine the position of the mixed reality device in the real space, that is, the mixed reality coordinate system {T c}. At the same time, the position of the camera relative to the mixed reality device is obtained, that is, the camera coordinate system {T b}Relative to the mixed reality coordinate system{T c Homogeneous transformation matrix
[0055] In the second step, after the initialization is completed, the operator can see the virtual UI input panel of the human-computer interaction module through the lens on the mixed reality device. Click the connection button on the virtual UI input panel with your finger, and the mixed reality device can obtain the operator's connection command through the human-computer interaction module to start the remote communication module, remotely connect the computer to the mixed reality device, and then transmit data. The status bar on the virtual UI display panel can be used to observe whether the connection is successful.
[0056] In the third step, the operator starts the tool positioning module on the computer. The algorithm accurately calculates the position and posture of the real tool at the end of the robot through the camera input image, thereby obtaining the robot end coordinate system {T a}Relative to the camera coordinate system {T b Homogeneous transformation matrix The calculated The mixed reality device receives the data according to the equation Get the robot end coordinate system {T a}Relative to the mixed reality coordinate system{T c The homogeneous transformation matrix of
[0057] Step 4: Click the Generate button on the virtual UI panel to start the milling simulation module to generate virtual workpieces and virtual tools. After the generation is completed, the operator can see the virtual workpiece through the lens on the mixed reality device. Calculate the real tool in the mixed reality coordinate system {T c} is assigned to the virtual tool in real time, so that the real tool and the virtual tool coincide with each other in the mixed reality device. d}Relative to the mixed reality coordinate system{T c Homogeneous transformation matrix Combining equations The virtual tool can be solved in the workpiece coordinate system {T d}, so that the virtual tool and the virtual workpiece are unified in the workpiece coordinate system {T d}, you can get the position of the virtual tool relative to the virtual workpiece. The status bar on the virtual UI display panel can show the real tool position information and processing information.
[0058] In the fifth step, there are custom voice commands in the human-computer interaction module of the mixed reality device, and the operator can control the hybrid robot through voice to start executing the processing trajectory to be verified. After the hybrid robot receives the start signal output by the command receiving and output module, the real tool of the hybrid robot and the virtual tool in the mixed reality device move together according to the processing trajectory of the real tool to be verified. The virtual tool and the virtual workpiece perform cutting collision intersection operations, and the part of the virtual workpiece that intersects with the virtual tool is removed when a collision occurs. The operator can see the real tool milling the virtual workpiece through the lens of the mixed reality device (the virtual tool is invisible). The operator can verify whether the processing trajectory is correct by observing the effect of the virtual workpiece during the milling process through the mixed reality device.
[0059] Step 6: Through human-machine fist gesture recognition, the virtual UI display panel can follow the operator, so that the operator can walk around and observe the virtual workpiece while seeing the status bar on the virtual UI display panel.
Claims
1. The robot processing simulation system based on mixed reality is characterized by include: A hybrid robot, at the end of which a real tool is installed, the hybrid robot is used to execute the machining trajectory of the real tool to be verified input by the computer, and when the real tool performs cutting according to the machining trajectory to be verified, the sensor on the hybrid robot collects vibration information of the real tool and transmits it to the computer through the controller of the hybrid robot; A camera, wherein the camera is used to collect in real time an image of a terminal of the hybrid robot during the hybrid robot cutting process according to the processing trajectory to be verified, and then output the image of the terminal of the hybrid robot to a computer; A computer, the computer is used to receive the real tool vibration information input by the hybrid robot controller and output it to the UI panel display module in the information display module through the remote communication module, and receive the hybrid robot terminal image input by the camera in real time, and output the hybrid robot terminal image to the tool positioning module installed on the computer. The tool positioning module includes an object recognition module and a posture estimation module. The object recognition module is used to read the hybrid robot terminal image output by the camera and realize the detection of the real tool in the robot terminal image through a machine learning algorithm, and the real tool detected in the image is framed by a rectangular window, and the feature extraction algorithm is used to extract the key point features of the real tool from the framed area; the posture estimation module is used to input the extracted real tool key point features into the posture solver to solve the real tool posture, and then input the solved real tool posture into the filtering algorithm for stabilization processing, and the stabilized real tool posture is transmitted through the camera coordinate system {T b }Relative to the robot end coordinate system {T a The homogeneous transformation matrix of After transformation, the real tool is obtained in the camera coordinate system {T b }, the real tool is placed in the camera coordinate system {T b }The pose and homogeneous transformation matrix under Output to remote communication module; A mixed reality device, in which a milling simulation module, an information display module and a command receiving and output module are installed; the milling simulation module includes a virtual-real coordination module and a processing milling module, in which a virtual tool and a virtual workpiece developed by Unity are arranged, and the virtual-real coordination module is used to read the posture and homogeneous transformation matrix of the real tool at the end of the robot transmitted by the posture estimation module through the remote communication module And read the camera coordinate system {T b }Relative to the mixed reality coordinate system{T c The homogeneous transformation matrix of According to the equation Solve the robot end coordinate system {T a }Transformation relative to the mixed reality coordinate system established at the mixed reality device location In the robot end coordinate system {T a The actual tool position under the coordinate system The real tool is transformed into the mixed reality coordinate system {T c }, the workpiece coordinate system {T d }Relative to the mixed reality coordinate system{T c Homogeneous transformation matrix Combining equations Solve the actual tool at the end of the robot in the workpiece coordinate system {T d }, the virtual workpiece is placed in the workpiece coordinate system {T d }, the position of the real tool and the virtual workpiece can be solved by the real tool position and the virtual workpiece position in the workpiece coordinate system and output to the processing and milling module; The command receiving and outputting module is used to receive the start and stop commands input by the human-machine interaction module and output them to the controller of the hybrid robot. If the start and stop commands input are to start execution, the controller processes the machining trajectory of the real tool to be verified input by the computer into a control command recognizable by the hybrid robot, and drives the hybrid robot to execute the machining trajectory of the real tool to be verified; The machining and milling module is used to read the coordinates of the real tool and the virtual workpiece in the workpiece coordinate system {T d }, and then the real tool is placed in the workpiece coordinate system {T d } is assigned to the virtual tool in real time, so that the real tool and the virtual tool in the mixed reality device overlap in position. After the hybrid robot receives the start signal output by the instruction receiving and output module, the real tool of the hybrid robot and the virtual tool in the mixed reality device move together according to the processing trajectory of the real tool to be verified, wherein the virtual tool and the virtual workpiece perform cutting collision intersection operation, and when a collision occurs, the part of the virtual workpiece that intersects with the virtual tool is removed, and the rendered object information of the virtual workpiece after cutting is output to the visual rendering module in the information display module in real time. During the cutting process, the sensor on the hybrid robot collects the vibration information of the real tool and transmits it to the computer through the controller of the hybrid robot; the virtual tool is equal in size and shape to the real tool at the end of the robot, and the virtual workpiece is equal in size and shape to the real workpiece to be processed; The information display module includes a three-dimensional reconstruction module, a space anchoring module, a visual rendering module and a UI panel display module. The UI panel display module is provided with a virtual UI panel developed by Unity. The three-dimensional reconstruction module is used to read the scattered point cloud information of the surrounding environment of the mixed reality device, complete the three-dimensional reconstruction of the surrounding environment of the mixed reality device by performing point cloud fusion processing on the scattered point cloud information, obtain the overall environment point cloud information, and output the overall environment point cloud information to the space anchoring module; The spatial anchoring module is used to read the overall environment point cloud information input by the 3D reconstruction module, and obtain the camera coordinate system {T b }Relative to the mixed reality coordinate system{T c The homogeneous transformation matrix of Will Output to the virtual-real coordination module in the milling simulation module; The visual rendering module is used to read the rendering object information input by the processing and milling module, and project the virtual workpiece onto the lens of the mixed reality device through the visual algorithm rendering provided by the mixed reality device. The operator observes the effect of the virtual workpiece during the cutting process through the mixed reality device to verify whether the real tool processing trajectory is correct; The UI panel display module is used to receive the vibration information input by the computer through the remote communication module and the real tool posture information input by the tool positioning module, and display the real tool vibration information and the real tool posture information on the virtual UI panel. The operator can judge whether the processing trajectory needs to be optimized by observing the real tool vibration information on the virtual UI panel; The human-computer interaction module is used to receive the gesture and voice data of a person collected by the sensor on the mixed reality device, perform semantic segmentation processing and conversion on the data, and then output the start and stop instructions to the instruction receiving and output module of the mixed reality device.
