Mixed reality-based contactless collaborative robot manipulation system

Abstract

A mixed reality-based contactless collaborative robot manipulation system, according to the present invention, comprises: a position information acquisition unit which acquires initial coordinate values for smart glasses, an image marker, a robot arm end tool, and a camera; a position-coordinating unit which coordinates the acquired initial coordinate values for the smart glasses, the image marker, the robot arm end tool, and the robot arm camera; an object generation unit which generates an object in accordance with the coordinate values of the robot arm end tool in a three-dimensional space acquired through the position matching unit; a final position calculation unit which determines whether a movement and a rotation of the generated object has occurred, and if a movement and a rotation have occurred, acquires position information for the object at the time point when displacement occurred, and calculates final position information of the robot arm end tool by using the acquired position information and robot arm joint information at the current time point; and a control unit which generates a control signal based on the calculated final position information and transmits same to a robot arm control device. The present invention may also be utilized for mixed-reality technology-based contactless driving of a robot in a digital twin-based smart factory environment.

Description

Mixed reality-based non-contact collaborative robot manipulation system

[0001] The present invention relates to a non-contact collaborative robot manipulation system based on mixed reality, and more specifically, to a non-contact collaborative robot manipulation system that can manipulate the position of an end tool without physical contact with a robot by using smart glasses that support mixed reality.

[0002] Extended Reality (XR), encompassing augmented reality, virtual reality, and mixed reality, is gaining traction in a variety of fields, including industry, healthcare, education, and entertainment, thanks to the advancing computing and graphics processing capabilities of hardware. Mixed Reality (MR), in particular, enables seamless interaction between the real and virtual worlds, and is currently commercially available on the market using smart glasses.

[0003] With the recent shift in the manufacturing industry from mass production to low-volume, high-variety production, collaborative robots (CROs)—robots that work collaboratively with humans in the same space—are being actively utilized in workplaces. Collaborative robots reduce the physical burden on workers and, by enabling precise execution of automated movements, increase work efficiency in logistics and manufacturing.

[0004] Additionally, in a smart factory based on a digital twin environment, it provides technology that allows workers to directly operate robots while feeling a sense of presence based on a mixed reality environment even when they are not on site.

[0005] Thus, according to the present invention, a non-contact collaborative robot manipulation system capable of manipulating the position of an end tool without physical contact with the robot is provided using smart glasses that support mixed reality.

[0006] According to an embodiment of the present invention for achieving such a technical task, a mixed reality-based non-contact collaborative robot manipulation system includes a position information acquisition unit that acquires initial coordinate values ​​for smart glasses, an image marker, an end tool of a robotic arm, and a camera, a position alignment unit that mutually aligns the acquired initial coordinate values ​​for the smart glasses, the image marker, the end tool of the robotic arm, and the robotic arm camera, an object generation unit that generates an object corresponding to the coordinate values ​​of the end tool of the robotic arm in a three-dimensional space acquired through the position alignment unit, a final position calculation unit that determines whether movement and rotation of the generated object have occurred, and if movement and rotation have occurred, acquires position information on the object at the time when displacement occurred, and calculates final position information of the end tool of the robotic arm using the acquired position information and robotic arm joint information at the current time, and a control unit that generates a control signal according to the calculated final position information and transmits it to a device that controls the robotic arm.

[0007] The above position alignment part can represent the three-dimensional spatial relationship between the smart glass and the end tool of the robotic arm using the following mathematical formula.

[0008]

[0009] Here, H represents smart glasses, E represents the end tool of the robotic arm, I represents an image marker, and C represents a camera.

[0010] The above final position calculation unit determines whether an object has moved using a signal obtained from the smart glass, and can calculate the displacement of the end tool of the robot arm as a homogeneous matrix at each time movement occurs.

[0011] The above object can be formed in a polyhedral shape other than a sphere, to make it easier for the operator to recognize the rotation transformation of the object.

[0012] In this way, according to the present invention, virtual content is visualized by reflecting the user's perspective through mixed reality-based smart glasses in the form of a head-mounted display, and direct interaction is possible through hand and gesture recognition, thereby providing the user with an excellent sense of perspective of virtual objects and enabling intuitive interaction.

[0013] FIG. 1 is a schematic diagram illustrating a non-contact collaborative robot operation system according to an embodiment of the present invention.

[0014] FIG. 2 is an exemplary diagram for explaining a non-contact collaborative robot operation system according to an embodiment of the present invention.

[0015] FIG. 3 is an exemplary diagram illustrating an object created in a mixed reality space using a non-contact collaborative robot manipulation system according to an embodiment of the present invention.

[0016] FIG. 4 is an exemplary diagram for explaining displacement of an object in a non-contact collaborative robot manipulation system according to an embodiment of the present invention.

[0017] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. In this process, the thickness of lines and the sizes of components depicted in the drawings may be exaggerated for clarity and convenience of explanation.

[0018] Furthermore, the terms described below are defined based on their functions within the present invention, and may vary depending on the intent or custom of the user or operator. Therefore, the definitions of these terms should be based on the overall content of this specification.

[0019] Hereinafter, a non-contact collaborative robot operation system according to an embodiment of the present invention will be described in more detail using FIG. 1.

[0020] FIG. 1 is a schematic diagram illustrating a non-contact collaborative robot operation system according to an embodiment of the present invention.

[0021] As illustrated in FIG. 1, a non-contact collaborative robot operation system (100) according to an embodiment of the present invention includes a position information acquisition unit (110), a position matching unit (120), an object generation unit (130), a final position calculation unit (140), and a control unit (150).

[0022] The location information acquisition unit (110) acquires initial coordinate values ​​for the smart glasses, image marker, end tool of the robot arm, and camera.

[0023] FIG. 2 is an exemplary diagram for explaining a non-contact collaborative robot operation system according to an embodiment of the present invention.

[0024] As illustrated in Fig. 2, the present invention configures a 7-degree-of-freedom robotic arm as a collaborative robot. Additionally, a camera is mounted on the distal wrist joint of the robotic arm.

[0025] The position alignment unit (120) performs alignment using the acquired initial coordinate values ​​to express the initial coordinate system of the robot's terminal tool in the mixed reality space of the user wearing smart glasses.

[0026] To elaborate, the position alignment unit (120) is used to determine the relationship between the smart glass and the robot's end tool through the following mathematical expression 1. ) can be expressed.

[0027]

[0028] Here, H represents smart glasses, E represents the end tool of the robotic arm, I represents an image marker, and C represents a camera.

[0029] and, represents the three-dimensional spatial relationship between smart glasses and image markers. The three-dimensional spatial relationship between smart glasses and image markers ( ) can be obtained under the assumption that the camera of the robotic arm and the smart glasses track the same image marker, and can be obtained in the form of a homogeneous matrix through the camera mounted on the smart glasses by applying Vuforia technology.

[0030] also, It represents the relationship between the camera of the robotic arm and the image marker, and can be obtained in the form of a homogeneous matrix through camera pose estimation and camera calibration provided by the open source vision library (Open Computer Vision Library, OpenCV).

[0031] also, represents the relationship between the camera and the end tool of the robotic arm.

[0032] The relationship between smart glasses and robot end tools ( ) can be calculated only once to initialize and represent the coordinate system of the terminal tool in the mixed reality space of the user wearing the smart glasses.

[0033] The object generation unit (130) obtains the relationship between the smart glass and the robot's terminal tool through the above mathematical expression 1. ) to create an object in a mixed reality space.

[0034] FIG. 3 is an exemplary diagram illustrating an object created in a mixed reality space using a non-contact collaborative robot manipulation system according to an embodiment of the present invention.

[0035] As illustrated in FIG. 3, the object generation unit (130) aligns the coordinate system of the terminal tool of the robotic arm with the mixed reality space to generate a hexahedral object, and the generated object can be visualized through smart glasses. In the embodiment of the present invention, the object is described as a hexahedron, but it is not limited thereto, and any shape among polyhedrons other than a sphere may be formed.

[0036] Users wear smart glasses and perform intuitive manipulations such as grabbing visualized objects and moving and rotating them.

[0037] At this time, the smart glasses perform hand and gesture recognition functions and transmit the performed results to the non-contact collaborative robot operation system (100).

[0038] Then, the final position calculation unit (140) expresses the displacement of the object as a homogeneous matrix according to the hand and gesture body signals received from the smart glasses.

[0039] FIG. 4 is an exemplary diagram for explaining displacement of an object in a non-contact collaborative robot manipulation system according to an embodiment of the present invention.

[0040] As shown in Figure 4, when the user moves and rotates the hexahedral object, a displacement occurs from the initialized position.

[0041] At this time, the relationship between the position of the terminal tool at the time of displacement occurrence and the initial position of the terminal tool was calculated as a 4x4 homogeneous matrix. can be expressed as

[0042] also, It contains quantitative information on the displacement that occurs and the robot must move and rotate, and the final position calculation unit (140) Using the information, the final position information of the end tool is calculated according to inverse kinematics using the coordinate values ​​at the time when displacement occurred and the current joint information of the current robot arm.

[0043] Finally, the control unit (150) generates a control signal based on the calculated final position information and transmits it to a device that controls the robot arm.

[0044] In this way, the non-contact collaborative robot manipulation system according to the present invention visualizes virtual content by reflecting the user's perspective through mixed reality-based smart glasses in the form of a head-mounted display, and allows direct interaction through hand and gesture recognition, thereby providing the user with an excellent sense of perspective of virtual objects and enabling intuitive interaction.

[0045] While the present invention has been described with reference to the embodiments illustrated in the drawings, these are merely exemplary, and those skilled in the art will understand that various modifications and equivalent embodiments are possible. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the following claims.

Claims

1. In a non-contact collaborative robot manipulation system based on mixed reality, A position information acquisition unit that acquires initial coordinate values ​​for smart glasses, image markers, end tools of a robotic arm, and cameras; A position alignment unit that aligns the initial coordinate values ​​of the smart glasses, image marker, end tool of the robotic arm, and robotic arm camera obtained above, An object generation unit that generates an object corresponding to the coordinate values ​​of the end tool of the robot arm in the three-dimensional space obtained through the above position matching unit; A final position calculation unit that determines whether the above-mentioned object has moved or rotated, and if so, obtains position information about the object at the time of displacement, and calculates the final position information of the end tool of the robot arm using the obtained position information and the robot arm joint information at the current time. A non-contact collaborative robot manipulation system including a control unit that generates a control signal according to the final position information calculated above and transmits it to a device that controls a robot arm.

2. In paragraph 1, The above position matching part is, A non-contact collaborative robot manipulation system that represents the three-dimensional spatial relationship between smart glasses and the end tool of a robotic arm using the following mathematical formula: Here, H represents smart glasses, E represents the end tool of the robotic arm, I represents an image marker, and C represents a camera.

3. In paragraph 1, The final position calculation unit above is, Using the signal obtained from the above smart glasses, it is determined whether the object is moving, A non-contact collaborative robot manipulation system that calculates the displacement of the end tool of the robot arm as a homogeneous matrix at each point in time when movement occurs.

4. In paragraph 1, The above object is, A non-contact collaborative robot manipulation system formed in a polyhedral shape excluding a sphere.

Images