Remote manipulator control system through coordinate recognition using markers

Abstract

The present invention relates to a remote manipulator control device and system through coordinate recognition using a marker. More specifically, the invention relates to a remote manipulator control device and control system through coordinate recognition using a marker, wherein the movement of a marker pen is captured by an image capturing unit using a marker pen with a marker attached, the coordinates of the marker of the marker pen are calculated, and the movement of the manipulator is controlled through the coordinates of the marker.

Description

Technology Field

[0001] The present invention relates to a remote manipulator control device and system through coordinate recognition using a marker. More specifically, the invention relates to a remote manipulator control device and control system through coordinate recognition using a marker, wherein the movement of a marker pen is captured by an image capturing unit using a marker pen with a marker attached, the coordinates of the marker of the marker pen are calculated, and the movement of the manipulator is controlled through the coordinates of the marker. Background Technology

[0003] A manipulator is a programmable mechanical arm and a robot that performs the role of a human arm with a structure similar to that of a human arm. These manipulators are utilized in various fields, including manufacturing, healthcare, services, and agriculture. They are efficiently used for complex tasks that are difficult or dangerous for humans to perform, and are expected to expand into even more diverse fields in the future.

[0004] However, controlling the movements of such manipulators requires using physical devices like joysticks or writing complex code via programming languages. This approach creates a high barrier to entry as it demands a certain level of technical skill and learning from the user, and even if manipulators are controlled via image processing, expensive equipment is required.

[0005] Therefore, there is an urgent need for a system that allows anyone to easily operate a manipulator and control it at a low cost.

[0007] [Prior Art] KR Registered Patent Publication No. 10-2184935 (Published Dec. 01, 2020) The problem to be solved

[0009] The present invention has been devised to solve the above-mentioned problems, and aims to provide a remote manipulator control device and control system through coordinate recognition using markers, which can control the movement of the manipulator according to the movement of the marker pen by recognizing the marker pen and the marker on the grid board, calculating the coordinates, and transmitting them to the control unit, thereby enabling the manipulator to be operated simply and at a low cost. means of solving the problem

[0011] A remote manipulator control device and system using coordinate recognition with a marker according to the present invention, devised to achieve the above objective, comprises: a shooting device capable of capturing images and video; a marker pen capable of having a pen shape that can be wrapped around and grasped by hand and to which a marker can be attached; a grid board capable of having a marker attached and providing 2D coordinates; a manipulator capable of moving in the same manner as the movement of the marker pen; a manipulator control device capable of controlling the movement of the manipulator; and a user terminal capable of receiving an image labeled with an axis, coordinate value, and marker ID.

[0012] Additionally, the device includes: a shooting device capable of capturing video and images; a marker pen capable of having a pen shape that can be wrapped around and gripped by hand and to which a marker can be attached; a grid board capable of having a marker attached and providing 2D coordinates; a manipulator capable of moving in the same manner as the movement of the marker pen; a manipulator control device capable of controlling the movement of the manipulator; and a user terminal capable of receiving an image labeled with an axis, coordinate values, and a marker ID. The marker pen includes a first marker that can be positioned above the marker pen and can have a cuboid shape, a pen tip capable of contacting the grid board, a barrel capable of having a cylinder shape and being combined with the pen tip and the first marker, and a vibration generating part capable of generating vibration when the marker pen contacts the grid board.

[0013] Additionally, the device includes a shooting device capable of capturing video and images; a marker pen capable of having a pen shape that can be wrapped around and gripped by hand and to which a marker can be attached; a grid board capable of having a marker attached and providing 2D coordinates; a manipulator capable of moving in the same manner as the movement of the marker pen; a manipulator control device capable of controlling the movement of the manipulator; and a user terminal capable of receiving an image labeled with an axis, coordinate values, and a marker ID. The marker pen includes a first marker that can be positioned above the marker pen and can have a cuboid shape, a pen tip capable of contacting the grid board, a barrel capable of having a cylinder shape and being combined with the pen tip and the first marker, and a vibration generating part capable of generating vibration when the marker pen contacts the grid board. The grid board may be equipped with a second marker, and the second marker may be attached to four vertices of the grid board.

[0014] In addition, a shooting device capable of capturing video and images; a marker pen having a pen shape to which a marker can be attached and which can be wrapped and gripped by hand; a grid board to which a marker can be attached and which can provide 2D coordinates; a manipulator capable of moving in the same manner as the movement of the marker pen; and a manipulator control device capable of controlling the movement of the manipulator; The device includes a user terminal capable of receiving an image labeled with axes, coordinate values, and a marker ID; the marker pen comprises a first marker positioned above the marker pen and having a cuboidal shape, a pen tip capable of contacting a grid board, a barrel having a cylinderal shape and capable of combining with the pen tip and the first marker, and a vibration generating unit capable of generating vibration when the marker pen contacts the grid board; the grid board may be equipped with a second marker, and the second marker may be attached to four vertices of the grid board; and the manipulator control device comprises an image and video receiving unit capable of receiving images from a shooting device, a distortion correction unit capable of correcting distortion occurring in the image by calculating the focal length of the shooting device, a lens distortion correction value, and an image center point, a marker detection unit capable of detecting and extracting the first marker of the marker pen and the second marker of the grid board from the distortion-corrected image obtained through the distortion correction unit, and a moving unit capable of calculating the position and direction of the extracted first marker and second marker. It includes a position setting unit capable of setting the position of the pen tip of a marker pen, and a control unit capable of controlling the movement of a manipulator to be placed at the same position as the pen tip position obtained through the position setting unit.

[0015] In addition, a shooting device capable of capturing video and images; a marker pen having a pen shape to which a marker can be attached and which can be wrapped and gripped by hand; a grid board to which a marker can be attached and which can provide 2D coordinates; a manipulator capable of moving in the same manner as the movement of the marker pen; and a manipulator control device capable of controlling the movement of the manipulator; The device includes a user terminal capable of receiving an image labeled with axes, coordinate values, and a marker ID; the marker pen comprises a first marker positioned above the marker pen and having a cuboidal shape, a pen tip capable of contacting a grid board, a barrel having a cylinderal shape and capable of combining with the pen tip and the first marker, and a vibration generating unit capable of generating vibration when the marker pen contacts the grid board; the grid board may be equipped with a second marker, and the second marker may be attached to four vertices of the grid board; and the manipulator control device comprises an image and video receiving unit capable of receiving images from a shooting device, a distortion correction unit capable of correcting distortion occurring in the image by calculating the focal length of the shooting device, a lens distortion correction value, and an image center point, a marker detection unit capable of detecting and extracting the first marker of the marker pen and the second marker of the grid board from the distortion-corrected image obtained through the distortion correction unit, and a moving unit capable of calculating the position and direction of the extracted first marker and second marker. It includes a position setting unit capable of setting the position of the pen tip of a marker pen, a control unit capable of controlling the movement of a manipulator to be placed at the same position as the pen tip position obtained through the position setting unit, and a labeling unit capable of labeling coordinate values, axes, and marker IDs on an image and transmitting the labeled image to a user terminal. Effects of the invention

[0017] According to the present invention, the movement of a manipulator can be controlled according to the movement of the marker pen using a marker pen with a marker attached, allowing the user to intuitively operate the manipulator, thereby increasing productivity and efficiency.

[0019] In addition, the distortion correction unit can calculate the focal length of the imaging device, the lens distortion correction value, and the image center point, which has the effect of obtaining an accurate image with distortion corrected.

[0021] In addition, the marker pen includes a vibration generating part, so it can provide haptic feedback, which is an effect that conveys the feeling of actually touching something when the marker pen comes into contact with the grid board. Brief explanation of the drawing

[0023] FIG. 1 is a diagram illustrating the relationship between the components of a remote manipulator control device and a system through coordinate recognition using markers according to a preferred embodiment of the present invention. FIG. 2 is a drawing illustrating the components of a shooting device. FIG. 3 is a drawing showing a checkerboard. FIG. 4 is a drawing illustrating the components of a marker pen. FIG. 5 is a drawing illustrating the process of providing haptic feedback to a marker pen. Figure 6 is a drawing showing a grid board. FIG. 7 is a drawing illustrating the components of a manipulator. FIG. 8 is a drawing illustrating the components of a manipulator control device. FIG. 9 is a drawing illustrating the components of a positioning unit. FIG. 10 is a diagram illustrating the formula for calculating the rotation matrix. FIG. 11 is a drawing illustrating the components of a user terminal. FIG. 12 is a drawing illustrating a labeling image provided through a display unit. Specific details for implementing the invention

[0024] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, it should be noted that in assigning reference numerals to the components of each drawing, identical components are assigned the same reference numerals whenever possible, even if they are shown in different drawings. Furthermore, in describing the present invention, if it is determined that a detailed description of related known components or functions could obscure the essence of the invention, such detailed description is omitted. Additionally, while preferred embodiments of the present invention will be described below, it is understood that the technical concept of the present invention is not limited thereto and can be modified and implemented in various ways by those skilled in the art.

[0026] FIG. 1 is a diagram illustrating the relationship between the components of a remote manipulator control device (500) and a system through coordinate recognition using a marker according to a preferred embodiment of the present invention, FIG. 2 is a diagram illustrating the components of a shooting device, FIG. 3 is a diagram illustrating a checkerboard, FIG. 4 is a diagram illustrating the components of a marker pen, FIG. 5 is a diagram illustrating the process of providing haptic feedback to a marker pen, FIG. 6 is a diagram illustrating a grid board, FIG. 7 is a diagram illustrating the components of a manipulator, FIG. 8 is a diagram illustrating the components of a manipulator control device, FIG. 9 is a diagram illustrating the components of a position setting unit, FIG. 10 is a diagram illustrating an equation for calculating a rotation matrix, FIG. 11 is a diagram illustrating the components of a user terminal, and FIG. 12 is a diagram illustrating a labeling image provided through a display unit.

[0028] The present invention relates to a remote manipulator control device and system through coordinate recognition using a marker. More specifically, the invention relates to a remote manipulator control device (500) and control system through coordinate recognition using a marker, which can control the movement of a manipulator (400) by using a marker pen (200) with a marker attached to it, capturing the movement of the marker pen (200) in an image capturing unit (110), calculating the coordinates of the marker of the marker pen (200), and controlling the movement of the manipulator (400) through the coordinates of the marker.

[0030] The marker of the present invention may be a marker capable of detecting coordinates based on angle, direction, and distance. That is, it may be an ArUco marker that has a specific pattern to help a computer recognize and determine the position and direction. In this case, the marker may have unique x, y, and z coordinates and a marker ID representing the marker.

[0032] Referring to FIG. 1, a remote manipulator control device (500) and control system through coordinate recognition using a marker may include a shooting device (100), a marker pen (200), a grid board (300), a manipulator (400), a manipulator control device (500), and a user terminal (600).

[0034] The shooting device (100) can photograph a marker pen (200) equipped with a marker, a grid board (300), and a checkerboard, and can photograph the movement of a manipulator (400) according to the movement of the marker pen (200). The shooting device (100) may be provided in multiple units and may be fixed or moving to take pictures, and may include an image shooting unit (110), a checkerboard image shooting unit (120), a manipulator shooting unit (130), and a transmission unit (140) as shown in FIG. 2.

[0036] The video capturing unit (110) can capture the movement of the marker pen (200) in real time to acquire an image, and the image can be transmitted to the video and image receiving unit (510) of the manipulator control device (500) through the transmitting unit (140).

[0038] The checkerboard image capturing unit (120) can collect the entire image of the checkerboard attached to a flat surface.

[0039] As shown in Fig. 3, the checkerboard consists of grid stripes and may be a board with alternating black squares and marker squares or a black and white pattern, and can be attached to a flat surface such as a floor, wall, or desk.

[0040] At this time, it is desirable to collect 10 to 15 images of the checkerboard taken from different angles and distances.

[0042] The manipulator shooting unit (130) can capture the movement of the manipulator (400) corresponding to the movement of the marker pen (200). The manipulator (400) can be captured by fixing the shooting device (100) to obtain a video containing the entire movement of the manipulator (400), and the video captured through the manipulator shooting unit (130) can be transmitted to the display unit (610) of the user terminal (600) through the transmission unit (140).

[0044] The transmitting unit (140) can transmit video and images captured by the shooting device (100) to the manipulator control device (500) and the user terminal (600). Video captured through the video shooting unit (110) and multiple checkerboard images captured through the checkerboard image shooting unit (120) can be transmitted to the video and image receiving unit (510) of the manipulator control device (500), and video captured through the manipulator shooting unit (130) can be transmitted to the display unit (610) of the user terminal (600).

[0046] The marker pen (200) may have a pen shape to which a marker can be attached and which a user can wrap and grip with their hand, and may include a first marker (210), a barrel (220), a pen nib (230), and a vibration generating part (240) as shown in FIG. 4.

[0048] The first marker (210) can be located at the top part of the marker pen (200) and can collectively refer to five markers.

[0049] The five markers consist of a main marker, a first sub-marker, a second sub-marker, a third sub-marker, and a fourth sub-marker, and these can be positioned on the upper part of the marker pen (200) to form a cuboid shape.

[0050] The bottom of the cuboid can be combined with the top of the marker pen (200) barrel (220), and the main marker is located on the top of the cuboid, and the first sub-marker is located on the front of the cuboid, the second sub-marker on the left side, the third sub-marker on the back, and the fourth marker on the right side.

[0052] At this time, since the marker pen (200) is equipped with 5 markers, even if the pen is tilted and the recognition accuracy of the main marker located at the top decreases, it can be corrected through 4 sub-markers excluding the main marker. If the number of markers is reduced to 3 or fewer, the tilt recognition value may become unstable, and if there are 6 or more markers, the volume of the marker pen (200) increases, making it inconvenient to use.

[0053] In addition, if more than 6 markers are used by miniaturizing the markers, recognition of the markers may become unstable due to their small size.

[0055] The barrel (220) may be shaped like a cylinder and may be combined with the first marker (210) and the nib (230), and may include a first pressure part, a second pressure part, and a data transmission part.

[0056] The first and second pressure parts may be push buttons that can be pressed by applying force with a finger contact at the bottom of the barrel (220).

[0058] The first pressure part can reduce the distance between the first gripper and the second gripper of the gripper part (450) of the manipulator (400). In this case, when the first pressure part is pressed continuously, the distance can be reduced by a constant number of times. This can be achieved through the continuous pressing operation of the first pressure part until the first gripper and the second gripper of the gripper part (450) can come into contact with and grasp an object.

[0059] In addition, to prevent the first pressure unit from operating continuously even though the gripping is successfully achieved in the gripper part (450) of the manipulator (400), a warning notification may be transmitted to the notification receiving part (650) of the user terminal (600) using the pressure value measured by the pressure sensor of the gripper part (450) of the manipulator (400).

[0061] The second pressure part can increase the distance between the first gripper and the second gripper of the gripper part (450) of the manipulator (400). Depending on the number of times the second pressure part is pressed continuously, the distance between the first gripper and the second gripper can be increased by a constant amount, and once the first gripper and the second gripper of the gripper part (450) are extended to the maximum, the second pressure part will not be extended further even if the pressing action of the second pressure part continues.

[0062] After successfully gripping, in order to place the object where desired, the distance between the first clamp and the second clamp is increased through the pressing action of the second pressure part so that the object can be placed.

[0064] The data transmission unit can transmit whether the first pressurizing unit and the second pressurizing unit perform a pressing operation and the number of consecutive pressurizing operations to the control unit (550) of the manipulator control device (500), and if the pressing operation is performed continuously rather than once, it can immediately transmit to the manipulator control device (500) that the pressing operation has been performed so that the movement of the gripper unit (450) of the manipulator (400) can be controlled by the number of times it has been pressed.

[0066] The pen nib (230) can be shaped so that it narrows toward the bottom and has a pointed bottom end, and can be combined with the barrel (220). Additionally, the pen nib (230) can come into contact with the grid board (300), and the position of the manipulator (400) can be controlled by the position of the pen nib (230).

[0068] The vibration generating unit (240) can generate vibration when the pen tip (230) of the marker pen (200) comes into contact with the grid board (300). Specifically, vibration can be generated by a vibration motor, and the vibration generating unit (240) can be used interchangeably with the vibration motor. The vibration intensity and vibration period of the vibration motor can be controlled by the vibration control unit (620) of the user terminal (600).

[0070] In addition, as shown in FIG. 5, the vibration generating unit (240) can generate vibrations in the marker pen (200) to provide haptic feedback.

[0071] Haptic feedback is a technology that stimulates a person's sense of touch with vibrations to convey the feeling of actually touching something.

[0073] If the marker pen (200) is equipped with a vibration motor to provide haptic feedback, it can provide the feeling of actually performing the work of the manipulator (400) rather than remotely.

[0075] A grid board (300) may have markers attached and may provide 2D coordinates. As shown in FIG. 6, the grid board (300) is a board in which horizontal and vertical lines intersect at right angles at equal intervals. Markers may be attached to the four vertices of the grid board (300), and it may include a second marker (310) and a range of motion control unit. At this time, the grid board (300) may be attached to a flat surface such as a ceiling or floor for use.

[0077] The second marker (310) can be collectively referred to as the markers attached to the four vertices of the grid board (300), and may include the first corner marker (311), the second corner marker (312), the third corner marker (313), and the fourth corner marker (314) in a clockwise direction starting from the top left marker.

[0079] The range of motion control unit can control the range of motion of the marker pen (200) moving on the grid board (300).

[0080] A column can be installed at the center of each of the four ends of the grid board (300), and the center of the marker pen (200) barrel (220) can be connected from the top part of the column with a string of elastic material.

[0081] The four lines connected to the marker pen (200) can control the movement of the marker pen (200) so that it does not go beyond the range where coordinate detection is possible according to the size of the grid board (300) and the operating range of the manipulator. That is, when the marker pen (200) attempts to go beyond the range where coordinate detection is possible or the operating range of the manipulator (400), the marker pen (200) can be pulled by the four lines to prevent it from going beyond.

[0082] At this time, the height of the column, the elasticity of the string, and the length of the string may vary depending on the range in which coordinate detection is possible and the operating range of the manipulator (400).

[0084] The manipulator (400) is a mechanical arm with a structure similar to a human arm and can be operated by a command from a manipulator control device. The manipulator (400) can move in the same way as the movement of the marker pen (200).

[0085] Referring to FIG. 7, the manipulator (400) may include a fixed part (410), a link arm (420), a first joint part (430), a second joint part (440), and a gripper part (450).

[0087] The fixed part (410) can fix the manipulator (400) to a flat floor or a desk and can provide stability to the movement of the manipulator.

[0088] The parts of the manipulator (400), excluding the fixed part (410), can be rotated.

[0089] The link arm (420) can be rotated and may be provided in multiple numbers and may be connected to other link arms (420) through joints.

[0090] The first joint part (430) can connect the link arm (420) and the link arm (420), and the second joint part (440) can connect the link arm (420) and the gripper part (450).

[0091] Movement can be provided to the manipulator (400) by the first joint part (430) and the second joint part (440), and the movement can be controlled according to the manipulator control device (500).

[0093] The gripper part (450) can grasp an object at the outermost part of the manipulator (400), may be equipped with a pressure sensor, and may include a first gripper, a second gripper, and a pressure data transmitter.

[0095] The gripper part (450) can have a clamp shape and can have a clamp shape by the first clamp located on the left and the second clamp located on the right.

[0097] The object can be grasped as the distance between the first and second grippers decreases and they converge toward the object, and the object can be released as the distance between the first and second grippers increases and they move to both sides.

[0098] At this time, the movement of the first clamp and the second clamp can be controlled by the number of times the first pressure part and the second pressure part of the marker pen (200) are pressed.

[0100] The pressure data transmitter can transmit the pressure value applied to the object, measured by the pressure sensor when the gripper part (450) grasps the object, to the warning notification part (560) of the manipulator control device (500).

[0102] The manipulator control device (500) can control the movement of the manipulator (400) by recognizing a marker through an image and providing the coordinates of the marker, and may include an image and image receiving unit (510), a distortion correction unit (520), a marker detection unit (530), a position setting unit (540), a control unit (550), a warning notification unit (560), a labeling unit (570), and a data collection unit (580) as shown in FIG. 8.

[0104] The video and image receiving unit (510) can receive a video of a marker captured according to the movement of the marker pen (200) from the transmitting unit (140) of the shooting device (100) and a plurality of checkerboard images captured through the checkerboard image shooting unit (120), and can provide the received video and images to the distortion correction unit (520).

[0106] The distortion correction unit (520) can correct distortion occurring in the image by calculating the focal length of the shooting device (100), the lens distortion correction value, and the image center point. That is, the distortion correction unit (520) can correct distortion occurring during image shooting by mathematically modeling the relationship between the 3D coordinates of the real world and the 2D coordinates on the image captured by the shooting device (100).

[0108] Image coordinates of each corner can be extracted from a plurality of checkerboard images obtained through the video and image receiving unit (510). The image coordinates are 2D coordinates indicating where the corner captured by the shooting device (100) is located in the image, and the corner of the checkerboard may refer to an intersection point formed by the meeting of squares with a black and white pattern.

[0110] After extracting image coordinates, they can be matched with real-world 3D coordinates to determine the position of each corner of the checkerboard in the real world, and the camera's intrinsic parameters, such as focal length, optical center, and distortion coefficient, can be obtained as output values ​​based on the image coordinates and real 3D coordinates using the calibrateCamera function.

[0112] The distortion coefficient is a value that explains lens distortion, representing the degree of distortion of optically curved straight lines numerically; by using the distortion coefficient to correct distorted images, a corrected image can be obtained at the same ratio as the actual image.

[0114] That is, the distorted coordinates of the image obtained through the image and video receiving unit (510) can be converted into corrected coordinates with the distortion removed using a distortion coefficient, and the image can be reconstructed using the converted coordinates to obtain an image with the distortion corrected.

[0115] The distorted image can be transmitted to the marker detection unit (530).

[0117] The marker detection unit (530) can detect and extract the first marker (210) of the marker pen (200) and the second marker (310) of the grid board (300) from the distortion-corrected image provided by the distortion correction unit (520).

[0118] At this time, five markers constituting the first marker (210) of the marker pen (200) can be detected, and the position coordinates can be determined based on one marker that is best recognized among these five markers.

[0119] The remaining four coordinates extracted from the first marker (210) can be used to remove outliers occurring in the movement path.

[0121] The position setting unit (540) can set the position of the moving marker pen (200) by calculating the position and direction of the extracted first marker (210) and second marker (310), and may include a pose estimation unit (541), an outlier removal unit (542), and an axis estimation unit (543) as shown in FIG. 9.

[0123] The pose estimation unit (541) can obtain a movement vector and a rotation vector and can estimate the position and direction of the marker.

[0124] The movement vector can indicate where the marker is relative to the camera. That is, it can indicate how far to the left / right (X-axis), up / down (Y-axis), and front / back (Z-axis) the marker is from the shooting device (100).

[0126] The rotation vector can represent the angle at which the marker is rotated relative to the imaging device (100). That is, it can indicate which direction the marker is tilted.

[0128] Translation vectors and rotation vectors can be obtained using the solbePnP function. The solbePnP function is a function that can obtain rotation vectors and translation vectors using 2D image coordinates and 3D world coordinates.

[0130] In this case, the rotation vector includes the rotation axis and the rotation angle, so to use it intuitively, it can be converted into a rotation matrix (R) through the Rodrigues transformation. Through the transformed rotation matrix, the direction in which the marker is moving in the camera coordinate system can be determined.

[0132] The rotation matrix can be obtained using the equation shown in Fig. 10.

[0134] The coordinates can be obtained through a rotation matrix and a translation vector that transforms the rotation vector. The direction of the marker pen (200) can be obtained through the rotation matrix, and the marker position of the marker pen (200) can be obtained through the translation vector. By moving the marker pen (200) in the obtained direction by the length of the marker pen (200), the coordinates of the pen tip (230) of the marker pen (200) can be obtained.

[0135] The position of the manipulator (400) can be controlled through the coordinates of the pen tip (230) of the marker pen (200).

[0137] The outlier removal unit (542) can remove outliers that occur in the calculation of the marker's coordinates.

[0138] If an outlier value occurs during the calculation of the coordinates of the first marker (210), the outlier value can be removed by selecting the median value among the five coordinates using a median value filter.

[0139] In other words, median filtering can be applied using a median filter used to remove noise from images or signals based on coordinates that are converted in real time; this removes outliers and prevents the path from becoming unstable.

[0141] The axis estimation unit (543) can define points to draw the axis and connect the defined points to represent the axis.

[0142] After defining points at regular intervals to represent the axis in the unique x, y, and z coordinates of the first marker (210) of the marker pen (200), the 3D axis can be converted into a 2D image of the camera view through the cv2.projectPoints code and the axis can be represented on the image by connecting the points.

[0144] The control unit (550) can control the movement of the manipulator (400) so that it is placed at the same position as the marker pen (200) obtained through the position setting unit (540). Additionally, the manipulator (400) can be controlled to move along the same path as the data input through the input unit (640) of the user terminal (600).

[0145] In addition, data regarding whether the first pressure part and the second pressure part perform a pressing action and the number of times the pressing action is performed consecutively is received from the data transmission part of the marker pen (200), and the distance between the first clamp and the second clamp of the gripper part (450) can be adjusted and controlled accordingly.

[0147] The warning notification unit (560) can transmit a warning notification including a request to stop pressurization to the user terminal (600) when the pressure value received from the pressure data transmission unit of the gripper unit (450) is greater than or equal to the pressure reference value, and can be equipped with pressure reference data.

[0148] Pressure reference data may be data regarding the maximum pressure that each object can withstand without deformation when pressure is applied.

[0150] The labeling unit (570) can label the coordinate values ​​and axes obtained through the pose estimation unit (541) and the axis estimation unit (543) onto the image, and can also label the ID of the marker.

[0151] The coordinate values ​​and axes obtained through the position setting unit (540) are labeled in real time, and the labeled image can be transmitted to the display unit (610) of the user terminal (600).

[0153] The data storage unit (580) can store images corrected through the distortion correction unit (520) and labeled images, and can store coordinate values, axes, and marker IDs obtained from the position setting unit (540). In the case of coordinate values, they are stored in chronological order so that the path of movement of the manipulator (400) can be known. The data stored by the data storage unit (580) can be transmitted to the data receiving unit (630) of the user terminal (600).

[0155] The user terminal (600) may include a general PC such as a general desktop or laptop, and may include mobile terminals such as a smartphone, tablet PC, PDA (Personal Digital Assistant), and mobile communication terminal.

[0157] Additionally, the user terminal (600) can receive an image labeled with an axis, coordinate value, and marker ID, and may include a display unit (610), a vibration control unit (620), a data receiving unit (630), and an input unit (640) as shown in FIG. 11.

[0159] The display unit (610) can receive a labeled image through the labeling unit (570) of the manipulator control device (500) and provide it visually. As shown in FIG. 12, the image may include the movement of the marker pen (200), the coordinate values ​​of the first marker (210), and the axes, and the coordinate values ​​may be provided by inputting the coordinate values ​​detected in real time to match the values ​​detected in real time.

[0160] Through the display unit (610), the path of movement of the manipulator (400) can be checked and coordinates can be received in real time.

[0162] Additionally, the display unit (610) can receive and visually provide a video of the manipulator (400) moving according to the movement of the marker pen (200). The video of the manipulator (400) movement obtained through the manipulator shooting unit (130) of the shooting device (100) can be viewed through the display unit (610), and through this, it is possible to check in real time whether the manipulator (400) is moving in the same path as the marker pen (200), and even if the manipulator (400) is in a dangerous area that the user cannot enter, the movement of the manipulator (400) can be checked through the display unit (610).

[0164] In addition, the labeled image obtained through the labeling unit (570) of the manipulator control device (500) and the image obtained through the manipulator shooting unit (130) of the shooting device (100) can be provided to the display unit (610) separately or simultaneously.

[0165] That is, only the labeled image obtained through the labeling unit (570) of the manipulator control device (500) can be provided, or only the image obtained through the manipulator shooting unit (130) of the shooting device (100) can be provided, and two images can be provided as images of a predetermined size each, thereby providing two images simultaneously.

[0166] When two videos are provided simultaneously, the movement of the marker pen (200) and the movement of the manipulator (400) can be checked together, and there is an effect of being able to accurately check whether the movement of the manipulator (400) according to the marker pen (200) is correct.

[0168] The vibration control unit (620) can control the vibration of the marker pen (200). It can control the vibration intensity of the vibration motor for haptic feedback and select the vibration cycle. The vibration cycle can be short or long, and can be set so that no vibration occurs when haptic feedback is not needed.

[0170] The data receiving unit (630) can receive data stored from the data storage unit (580) of the manipulator control device (500). It can receive data stored in chronological order of the coordinates of the first marker (210) of the marker pen (200) indicating the path, and through this data, it can verify whether the intended path has moved accurately, and if the same path is required, it can receive data for the required path from the data storage unit (580) of the manipulator control device (500).

[0172] The input unit (640) can control the movement of the manipulator (400) by inputting the same path as the data obtained through the data receiving unit (630) and transmitting it to the control unit (550) of the manipulator control device (500).

[0173] If the user terminal (600) is a smartphone, the input unit (640) can be input through contact with a finger touching the display unit (610), and if the user terminal (600) is a desktop, input can be input through a keyboard or mouse.

[0175] The notification receiving unit (650) can receive a warning notification from the warning notification unit (560) of the manipulator control device (500).

[0176] The warning notification may include the pressure value applied to the object and a request to stop the first pressurizing unit.

[0178] Hereinafter, the operation method of a remote manipulator control device (500) and a system using coordinate recognition with a marker according to a preferred embodiment of the present invention will be described in detail.

[0180] A checkerboard is attached to a flat surface, and 10 to 15 images of the checkerboard are obtained by taking pictures from multiple angles and distances through the checkerboard image collection unit (120) of the shooting device (100).

[0182] The checkerboard image obtained through the checkerboard image collection unit (120) is transmitted to the image and image receiving unit (510) through the transmission unit (140), and the image and image receiving unit (510) provides a plurality of checkerboard images to the distortion correction unit (520), and the distortion correction unit (520) extracts the image coordinates of the corners, which are 2D coordinates, from the checkerboard images.

[0184] After extracting image coordinates, the image coordinates are matched with actual 3D coordinates, and the camera's intrinsic parameters, such as focal length, optical center, and distortion coefficient, are obtained based on the image coordinates and actual 3D coordinates through the calibrateCamera function.

[0186] Afterwards, the movement of the marker pen (200) moving on the grid board (300) is captured in real time through the image capturing unit (110) of the shooting device (100) and transmitted to the image and image receiving unit (510) through the transmitting unit (140), and the image and image receiving unit (510) provides the image to the distortion correction unit (520).

[0188] The distortion correction unit (520) uses a distortion coefficient to convert 2D coordinates into corrected coordinates from which distortion has been removed, and reconstructs the image through the converted coordinates to obtain a distortion-corrected image.

[0190] A distorted image is transmitted to a marker detection unit (530) of a manipulator control device (500), and the marker detection unit (530) detects and extracts the first marker (210) of the marker pen (200) and the second marker (310) of the grid board (300) from the image, and the position setting unit (540) can calculate the position and direction of the marker extracted by the marker detection unit (530).

[0192] That is, in the pose estimation unit (541) of the position setting unit (540), the translation vector and rotation vector are obtained using the solbePnP function, and the coordinates are obtained through the rotation matrix and translation vector that transform the rotation vector.

[0193] The direction of the marker pen (200) can be obtained through a rotation matrix, and the marker position of the marker pen (200) can be obtained through a movement vector, and the coordinates of the pen tip (230) can be obtained by moving the marker pen (200) in the obtained direction by the length of the marker pen (200).

[0195] After obtaining the coordinates of the pen tip (230), the control unit (550) controls the movement of the manipulator (400) so that it is positioned at the same coordinates as the pen tip (230).

[0197] In addition, after obtaining the coordinates, points for drawing the axis are defined from the unique x, y, and z coordinates of the marker through the axis estimation unit (543) of the position setting unit (540), and the axis is represented by connecting these points, and then the marker ID, coordinate values, and axis are labeled on the image through the labeling unit (570), and the labeled image is transmitted to the user terminal (600).

[0199] The display unit (610) of the user terminal (600) receives an image labeled with a marker ID, coordinate values, and axes, and provides the labeled image so that the moving path of the marker pen (200) and the marker ID, coordinate values, and axes can be checked. In addition, the display unit (610) receives an image containing the movement of the manipulator (400) from the manipulator shooting unit (130) of the shooting device (100) so that the labeled image and the manipulator (400) image can be checked simultaneously.

[0201] The foregoing description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications, changes, and substitutions within the scope of the essential characteristics of the present invention. Accordingly, the embodiments disclosed in the present invention and the accompanying drawings are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by such embodiments and accompanying drawings. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols

[0203] 100 - Camera 200 - Marker pen 300 - Grid board 400 - Manipulator 500 - Manipulator controller 600 - User terminal 110 - Video Recording Unit 120 - Checkerboard Image Recording Unit 130 - Manipulator imaging unit 140 - Transmitter unit 210 - 1st Marker 220 - Barrel 230 - Pen nib 240 - Vibration generating part 310 - 2nd Marker 410 - Fixed part 420 - Link arm 430 - 1st joint 440 - 2nd joint 450 - Gripper 510 - Video and image receiver 520 - Distortion correction unit 530 - Marker detection unit 540 - Position setting unit 550 - Control unit 560 - Warning notification unit 570 - Labeling Unit 580 - Data Storage Unit 610 - Display unit 620 - Vibration control unit 630 - Data receiving unit 640 - Input unit 650 - Notification receiver

Claims

Claim 1 delete Claim 2 delete Claim 3 delete Claim 4 A shooting device capable of capturing video and images; a marker pen having a pen shape to which a marker can be attached and which can be wrapped and grasped by hand; a grid board to which a marker can be attached and which can provide 2D coordinates; a manipulator capable of moving in the same manner as the movement of the marker pen; and a manipulator control device capable of controlling the movement of the manipulator. The device includes a user terminal capable of receiving an image labeled with an axis, coordinate values, and a marker ID; the marker pen comprises a first marker positioned above the marker pen and having a cuboidal shape, a pen tip capable of contacting a grid board, a barrel having a cylinderal shape and capable of combining with the pen tip and the first marker, and a vibration generating unit capable of generating vibration when the marker pen contacts the grid board; the grid board may be equipped with a second marker, and the second marker may be attached to four vertices of the grid board; and the manipulator control device comprises an image and video receiving unit capable of receiving images from a shooting device, a distortion correction unit capable of correcting distortion occurring in the image by calculating the focal length of the shooting device, a lens distortion correction value, and an image center point, a marker detection unit capable of detecting and extracting the first marker of the marker pen and the second marker of the grid board from the distortion-corrected image obtained through the distortion correction unit, and the extracted first marker and second marker A remote manipulator control system using coordinate recognition with a marker, comprising: a position setting unit capable of setting the position of a pen tip of a moving marker pen by calculating position and direction; and a control unit capable of controlling the movement of a manipulator so that it is placed at the same position as the position of the pen tip obtained through the position setting unit. Claim 5 A shooting device capable of capturing video and images; a marker pen having a pen shape to which a marker can be attached and which can be wrapped and grasped by hand; a grid board to which a marker can be attached and which can provide 2D coordinates; a manipulator capable of moving in the same manner as the movement of the marker pen; and a manipulator control device capable of controlling the movement of the manipulator. The device includes a user terminal capable of receiving an image labeled with an axis, coordinate values, and a marker ID; the marker pen comprises a first marker positioned above the marker pen and having a cuboidal shape, a pen tip capable of contacting a grid board, a barrel having a cylinderal shape and capable of combining with the pen tip and the first marker, and a vibration generating unit capable of generating vibration when the marker pen contacts the grid board; the grid board may be equipped with a second marker, and the second marker may be attached to four vertices of the grid board; and the manipulator control device comprises an image and video receiving unit capable of receiving images from a shooting device, a distortion correction unit capable of correcting distortion occurring in the image by calculating the focal length of the shooting device, a lens distortion correction value, and an image center point, a marker detection unit capable of detecting and extracting the first marker of the marker pen and the second marker of the grid board from the distortion-corrected image obtained through the distortion correction unit, and the extracted first marker and second marker A remote manipulator control system using coordinate recognition with a marker, comprising: a position setting unit capable of setting the position of a pen tip of a moving marker pen by calculating position and direction; a control unit capable of controlling the movement of a manipulator so that it is placed at the same position as the pen tip position obtained through the position setting unit; and a labeling unit capable of labeling coordinate values, axes, and marker IDs on an image and transmitting the labeled image to a user terminal.

Images