Mixed Reality based contact-free collaborative robot manipulation system

Abstract

The present invention relates to a mixed reality-based non-contact collaborative robot operation system. According to the present invention, a mixed reality-based non-contact collaborative robot operation system comprises: a position information acquisition unit that acquires initial coordinate values ​​for smart glasses, an image marker, an end tool of a robot arm, and a camera; a position matching unit that mutually aligns the acquired initial coordinate values ​​for the smart glasses, the image marker, the end tool of a robot arm, and the robot arm camera; an object creation unit that creates an object corresponding to the coordinate values ​​of the end tool of a robot arm in a three-dimensional space acquired through the position matching unit; a final position calculation unit that determines whether movement and rotation of the created object have occurred, and if movement and rotation have occurred, acquires position information for the object at the time when displacement occurred, and calculates final position information of the end tool of the robot arm using the acquired position information and robot arm joint information at the current time; and a control unit that generates a control signal according to the calculated final position information and transmits it to a device that controls the robot arm.

Description

Technology Field

[0001] The present invention relates to a mixed reality-based non-contact collaborative robot operation system, and more specifically, to a non-contact collaborative robot operation system that can operate the position of an end tool without physical contact with the robot by using smart glasses that support mixed reality. Background Technology

[0002] Extended Reality (XR) technology, which encompasses augmented reality, virtual reality, and mixed reality, is establishing its value in various fields such as industry, healthcare, education, and entertainment, alongside the advancement of hardware computing and graphics processing capabilities. In particular, Mixed Reality (MR) technology, which enables interaction by integrating the real world and the virtual world, is currently commercialized in the market based on smart glasses devices.

[0003] Along with the recent trend in the manufacturing sector shifting from mass production to multi-product, small-batch production, collaborative robots are being actively utilized in the workplace as robots that perform tasks collaboratively with humans in the same space. Because collaborative robots reduce the physical burden on workers and can accurately perform automated movements, they increase work efficiency in the logistics and manufacturing sectors.

[0004] The technology forming the background of the present invention is disclosed in Korean Published Patent No. 10-2021-0059980 (published May 26, 2021). The problem to be solved

[0005] As such, according to the present invention, the purpose is to provide a non-contact collaborative robot operation system that can manipulate the position of an end tool without physical contact with the robot by using smart glasses that support mixed reality. means of solving the problem

[0006] According to an embodiment of the present invention for achieving such technical challenges, a mixed reality-based non-contact collaborative robot operation system comprises: a position information acquisition unit that acquires initial coordinate values ​​for smart glasses, an image marker, an end tool of a robot arm, and a camera; a position matching unit that mutually aligns the acquired initial coordinate values ​​for the smart glasses, the image marker, the end tool of a robot arm, and the robot arm camera; an object creation unit that creates an object corresponding to the coordinate values ​​of the end tool of a robot arm in a three-dimensional space acquired through the position matching unit; a final position calculation unit that determines whether movement and rotation of the created object have occurred, and if movement and rotation have occurred, acquires position information for the object at the time when displacement occurred, and calculates final position information of the end tool of the robot arm using the acquired position information and robot arm joint information at the current time; and a control unit that generates a control signal according to the calculated final position information and transmits it to a device that controls the robot arm.

[0007] The above position matching unit can represent the relationship in three-dimensional space between the smart glasses and the end tool of the robot arm using the following mathematical formula.

[0008]

[0009] Here, H represents smart glasses, E represents the end tool of the robot arm, I represents an image marker, and C represents a camera.

[0010] The above final position calculation unit determines whether the object is moving using a signal obtained from the smart glasses, and can calculate the displacement of the end tool of the robot arm as a homogeneous matrix at each point in time when movement occurs.

[0011] The above object may be formed in a polyhedral shape excluding a sphere. Effects of the invention

[0012] As such, according to the present invention, virtual content is visualized by reflecting the user's perspective through a head-mounted display-type mixed reality-based smart glasses, and direct interaction is possible through hand and gesture recognition, thereby providing the user with excellent perspective of virtual objects and enabling intuitive interaction. Brief explanation of the drawing

[0013] FIG. 1 is a configuration diagram for explaining a non-contact collaborative robot operation system according to an embodiment of the present invention. FIG. 2 is an illustrative diagram for explaining a non-contact collaborative robot operation system according to an embodiment of the present invention. FIG. 3 is an example diagram illustrating an object created in a mixed reality space using a non-contact collaborative robot operation system according to an embodiment of the present invention. FIG. 4 is an illustrative diagram for explaining the displacement of an object in a non-contact collaborative robot operation system according to an embodiment of the present invention. Specific details for implementing the invention

[0014] Preferred embodiments according to the present invention will be described in detail below with reference to the attached drawings. In this process, the thickness of lines or the size of components shown in the drawings may be exaggerated for clarity and convenience of explanation.

[0015] Furthermore, the terms described below are defined in consideration of their functions within the present invention, and these may vary depending on the intent or practice of the user or operator. Therefore, the definitions of these terms should be based on the content throughout this specification.

[0016] Hereinafter, a non-contact collaborative robot operation system according to an embodiment of the present invention will be described in more detail using FIG. 1.

[0017] FIG. 1 is a configuration diagram for explaining a non-contact collaborative robot operation system according to an embodiment of the present invention.

[0018] As illustrated in FIG. 1, a non-contact collaborative robot operation system (100) according to an embodiment of the present invention includes a position information acquisition unit (110), a position matching unit (120), an object creation unit (130), a final position calculation unit (140), and a control unit (150).

[0019] The location information acquisition unit (110) acquires initial coordinate values ​​for the smart glasses, image marker, end tool of the robot arm, and camera.

[0020] FIG. 2 is an illustrative diagram for explaining a non-contact collaborative robot operation system according to an embodiment of the present invention.

[0021] As shown in FIG. 2, the present invention configures a 7-degree-of-freedom robot arm as a collaborative robot. Additionally, a camera is mounted on the end wrist joint of the robot arm.

[0022] The position matching unit (120) performs matching using the initial coordinate values ​​obtained to represent the initial coordinate system of the robot's end tool in the mixed reality space of a user wearing smart glasses.

[0023] To elaborate, the position matching part (120) is the relationship between the smart glasses and the robot's end tool through the following mathematical formula 1 ( It can represent ).

[0024]

[0025] Here, H represents smart glasses, E represents the end tool of the robot arm, I represents an image marker, and C represents a camera.

[0026] and, represents the relationship between smart glasses and image markers in three-dimensional space. The relationship between smart glasses and image markers in three-dimensional space ( ) can be acquired under the assumption that the camera of the robot arm and the smart glasses track the same image marker, and can be acquired in the form of a homogeneous matrix through the camera mounted on the smart glasses by applying Vuforia technology.

[0027] also, This represents the relationship between the camera and image marker of the botpal, and can be obtained in the form of a homogeneous matrix through camera pose estimation and camera calibration provided by the Open Computer Vision Library (OpenCV).

[0028] also, This represents the relationship between the robot arm's camera and the end tool.

[0029] The relationship between smart glasses and robot end tools ) can be calculated only once to initialize and represent the coordinate system of the end tool in the mixed reality space of a user wearing smart glasses.

[0030] The object generation unit (130) obtains the relationship between the smart glasses and the robot's end tool through the above mathematical formula 1 ( Creates an object in the mixed reality space using a homogeneous matrix based on ).

[0031] FIG. 3 is an example diagram illustrating an object created in a mixed reality space using a non-contact collaborative robot operation system according to an embodiment of the present invention.

[0032] As illustrated in FIG. 3, the object generation unit (130) aligns the coordinate system of the mixed reality space and the end tool of the robot arm to generate a cube-shaped object, and enables the generated object to be visualized through smart glasses. In the embodiment of the present invention, the object is described as a cube, but it is not limited thereto, and it may be formed in any shape among polyhedra excluding spherical shapes.

[0033] The user performs intuitive operations of moving and rotating by grabbing a visualized object while wearing smart glasses.

[0034] At this time, the smart glasses perform hand and gesture recognition functions and transmit the results to the non-contact collaborative robot operation system (100).

[0035] Then, the final position calculation unit (140) expresses the displacement of the object as a homogeneous matrix according to the hand and gesture recognition signals received from the smart glasses.

[0036] FIG. 4 is an illustrative diagram for explaining the displacement of an object in a non-contact collaborative robot operation system according to an embodiment of the present invention.

[0037] As shown in Fig. 4, when the user moves and rotates the cube object, a displacement occurs from the initial position.

[0038] At this time, the relationship between the position of the end tool at the time displacement occurred and the initial position of the end tool, calculated as a 4x4 homogeneous matrix It can be expressed as.

[0039] also, The final position calculation unit (140) contains quantitative information that the robot must move and rotate due to the occurrence of displacement. Using the information, the final position information of the end tool is calculated based on inverse kinematics through the coordinate values ​​at the time the displacement occurred and the current joint information of the robot arm.

[0040] Finally, the control unit (150) generates a control signal based on the calculated final position information and transmits it to a device that controls the robot arm.

[0041] As such, the non-contact collaborative robot operation system according to the present invention visualizes virtual content by reflecting the user's perspective through a mixed reality-based smart glass in the form of a head-mounted display and enables direct interaction through hand and gesture recognition, thereby providing the user with excellent perspective of virtual objects and enabling intuitive interaction.

[0042] The present invention has been described with reference to the embodiments illustrated in the drawings, but this is merely illustrative, and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the following claims. Explanation of the symbols

[0043] 100 : Non-contact collaborative robot operation system 110: Location information acquisition unit 120 : Position alignment part 130 : Object creation section 140 : Final position calculation unit 150 : Control unit

Claims

Claim 1 A non-contact collaborative robot operation system based on mixed reality, comprising: a position information acquisition unit for acquiring initial coordinate values ​​for smart glasses, an image marker, an end tool of a robot arm, and a camera; a position matching unit for mutually matching the acquired initial coordinate values ​​for the smart glasses, the image marker, the end tool of a robot arm, and the robot arm camera; an object creation unit for creating an object corresponding to the coordinate values ​​of the end tool of a robot arm in a three-dimensional space acquired through the position matching unit; a final position calculation unit for determining whether movement and rotation of the created object have occurred, and if movement and rotation have occurred, acquiring position information for the object at the time when displacement occurred, and calculating final position information of the end tool of the robot arm using the acquired position information and robot arm joint information at the current time; and a control unit for generating a control signal according to the calculated final position information and transmitting it to a device that controls the robot arm. Claim 2 In claim 1, the position matching unit represents the relationship in three-dimensional space between the smart glasses and the end tool of the robot arm using the following mathematical formula, a non-contact collaborative robot operating system: Here, H represents smart glasses, E represents the end tool of the robot arm, I represents an image marker, and C represents a camera. Claim 3 A non-contact collaborative robot operation system according to claim 1, wherein the final position calculation unit determines whether an object moves using a signal obtained from the smart glasses, and calculates the displacement of the end tool of the robot arm as a homogeneous matrix at each point in time when movement occurs. Claim 4 In claim 1, the object is a non-contact collaborative robot operating system formed in a polyhedral shape excluding a sphere.

Images