Robot posture estimation method and system

Abstract

To disclose a robot posture estimation method.SOLUTION: A robot posture estimation method of the present invention includes the steps of: generating a first marker and at least one or more second markers and attaching the markers to a robot; collecting a video including at least one of the first marker and the plurality of second markers; estimating position and rotation information of the first marker and the second marker; deriving the position and rotation information of the first marker from a relative positional relation between the first marker and the second marker when the first marker is not estimated; and estimating a posture of the robot on the basis of the position and rotation information of the first marker.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a method and system for estimating the posture of a robot.

Background Art

[0002] A collaborative robot can improve the productivity of a factory by collaborating with other devices and workers. However, since it works with people, there is a need for a technology that can grasp such defects in advance and maintain and repair them. If defects are diagnosed using internal data produced for the drive and control of a collaborative robot, it can proceed without installing additional hardware. However, the data that can be collected varies depending on the type of robot, and since the movement is corrected internally for accurate driving, there is a possibility that the characteristics of the defects cannot be correctly reflected in the internal data. Also, even for robots of the same type, the defect detection criteria change depending on the program and environment to be executed.

[0003] As a method for solving such problems, an external vision sensor is also used to detect abnormalities in the robot. When using an external vision sensor of the type that attaches a marker to the robot, an independent defect detection system can be constructed for the operation of the robot.

[0004] By utilizing this, the movement path of the robot can be estimated through continuous shooting of the marker, and abnormalities and defects can be detected by comparing with normal driving. However, when the marker is blocked, the position of the robot cannot be estimated, and in the conventional method, only the position of the marker is estimated, and the accurate posture of the joints and links of the robot cannot be estimated.

[0005] Also, when using multiple cameras to recognize the position of the marker, the installation of the cameras may be restricted depending on the working environment.

Summary of the Invention

Problems to be Solved by the Invention

[0006] The technical problem that the present invention aims to solve is to provide a robot posture estimation method and system that applies at least one marker to a robot, accurately estimates the position of a marker using other markers even when one marker is obstructed, and estimates the robot's posture based on this result. [Means for solving the problem]

[0007] To solve the aforementioned technical problems, a robot posture estimation method according to one embodiment of the present invention can be performed by one or more processors of a computer device and may include the steps of: generating a first marker and at least one second marker and attaching them to the robot; collecting an image including the first marker and at least one of the plurality of second markers; estimating the position and rotation information of the first marker and the second marker; if the first marker is not estimated, deriving the position and rotation information of the first marker from the relative positional relationship between the first marker and the second marker; and estimating the posture of the robot based on the position and rotation information of the first marker.

[0008] In one embodiment of the present invention, the first marker may be attached to the end effector of a robot.

[0009] One embodiment of the present invention further includes the step of setting the positional relationship between the first marker and the second marker attached to the robot, wherein the positional relationship can be derived by analyzing an image generated by simultaneously photographing the first marker and the second marker.

[0010] In one embodiment of the present invention, the state information of the first marker is derived in multiple units based on its positional relationship with each of the multiple second markers, and the further step may include selecting one of the multiple first marker state information units derived.

[0011] In one embodiment of the present invention, the posture of the robot can be estimated by a machine learning module that has been pre-trained on the relationship between the state information of the first marker and the posture of the robot.

[0012] To solve the aforementioned technical problems, a robot posture estimation system according to one embodiment of the present invention may include: a first marker and at least one second marker attached to a robot; a sensing unit that collects images including the first marker and at least one of the plurality of second markers; and a main control unit that estimates the position and rotation information of the first marker and the second marker, and if the first marker is not estimated, derives the position and rotation information of the first marker from the relative positional relationship between the first marker and the second marker, and estimates the posture of the robot based on the position and rotation information of the first marker. [Effects of the Invention]

[0013] This invention applies at least one marker to a robot, allowing for accurate estimation of the marker positions even when one marker is obstructed, and enabling the estimation of the robot's posture based on these results. [Brief explanation of the drawing]

[0014] [Figure 1] This shows a robot posture estimation system according to one embodiment of the present invention. [Figure 2] This document illustrates a method for estimating the posture of a robot according to one embodiment of the present invention. [Figure 3] This diagram shows in detail some of the configurations of a robot posture estimation method according to one embodiment of the present invention. [Figure 4] This diagram shows in detail some of the configurations of a robot posture estimation method according to one embodiment of the present invention. [Figure 5] This diagram shows in detail some of the configurations of a robot posture estimation method according to one embodiment of the present invention. [Figure 6]This shows in detail a part of the configuration of a method for estimating the posture of a robot according to an embodiment of the present invention. [Figure 7] This shows the state of attachment of markers to the robot. [Figure 8] This shows the positional relationship among the first marker, the second marker, and the sensing unit. [Figure 9] This is a graph comparing before and after data preprocessing. [Figure 10] These are graphs and tables showing the positional relationship between markers. [Figure 11] This shows the state information of markers in a graph.

Embodiments for Carrying Out the Invention

[0015] The present invention can be subjected to various transformations and can have various embodiments. Here, specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all transformations, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0016] In describing the present invention, if it is determined that a specific description of related known technologies may obscure the gist of the present invention, the detailed description thereof will be omitted.

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0018] FIG. 1 shows a posture estimation system of a robot according to an embodiment of the present invention.

[0019] Referring to FIG. 1, a posture estimation system of a robot according to an embodiment of the present invention can include a main control unit 110, a sensor control unit 120, a sensing unit 130, a robot control unit 140, a first marker 210, and a second marker 220.

[0020] The main control unit 110 is connected to and controls the sensor control unit 120 and the robot control unit 140.

[0021] The main control unit 110 can connect to the administrator terminal 300 via a wired / wireless network to send and receive information.

[0022] The main control unit 110 may be a control device including a processor, memory, input interface, etc. For example, it may consist of a desktop computer, a laptop computer, or a similar device that performs the same functions.

[0023] The sensor control unit 120 can control the sensing unit 130 under the control of the main control unit 110.

[0024] Although the sensor control unit 120 is shown as being physically separated from the main control unit 110, it may also be composed of a program embedded in the main control unit 110. In this case, the main control unit 110 can directly control the sensing unit 130.

[0025] The sensing unit 130 is connected to the sensor control unit 120 and operates under the control of the sensor control unit 120.

[0026] The sensing unit 130 may include an image sensor. For example, the sensing unit 130 may include a camera.

[0027] In one embodiment of the present invention, the sensing unit 130 may include only one camera. However, multiple cameras may be incorporated to improve image quality such as resolution, but their positions may be uniquely determined. Here, a unique position may mean a position where a single physical object is located, even if it is not exactly the same location. In other words, image sensing using multiple cameras that are spatially very far apart may not be the main objective of the present invention. However, this is not a limitation.

[0028] The robot control unit 140 is connected to the main control unit 110 and the robot, and can control the robot through the control of the main control unit 110. However, the embodiment is not necessarily limited to this one, and the robot control unit 140 may be arranged, connected, and configured to control the robot independently of the main control unit 110.

[0029] In this embodiment, the robot control unit 140 may also be composed of a program embedded in the main control unit 110.

[0030] The first marker 210 is attached to the robot, and its image can be acquired by the sensing unit 130.

[0031] Specifically, the first marker 210 can be attached to the robot's end-effector.

[0032] An end effector refers to a part of a robot that has a function that directly interacts with the workpiece during its operation. For example, grippers, welding torches, spray guns, and nut runners are all end effectors.

[0033] The first marker 210 is attached to the end effector in order to estimate the TCP (Tool Center Point) position, and the first marker 210 is M TCP We will also display it as follows:

[0034] The second marker 220 is attached to the robot, and its image can be acquired by the sensing unit 130.

[0035] The second marker 220 may be attached so as to have a relative positional relationship with the first marker 210.

[0036] For example, even if the second marker 220 is attached to the end effector, it may be attached to the four sides of the end effector rather than to the TCP.

[0037] The second marker 220 is an auxiliary means for estimating the position of the first marker 210 even when the first marker 210 is not sensed, M SUB We will also display it as follows:

[0038] Figure 7 shows the state of marker attachment to the robot. Referring to Figure 7, in one embodiment of the present invention, the first marker 210 is attached to the TCP position, which is the upper center of the end effector, and four second markers 220 are attached around the side of the end effector at 90-degree intervals.

[0039] Although not shown in the drawings, the administrator terminal 300 may include a control unit, a communication unit, a storage unit, and an input unit.

[0040] The control unit is connected to and can control the communication unit, storage unit, and input unit.

[0041] The communications unit can send and receive information with the main control unit 110.

[0042] The storage unit can store necessary information to facilitate information processing. The storage unit can store applications that may be installed on the administrator terminal 300.

[0043] The input section is an input interface for controlling the administrator terminal 300, and may consist of a keyboard or a touchscreen.

[0044] The administrator terminal configured in this way could be, for example, a smartphone, a tablet PC, or a notebook computer, and is not limited to the above embodiment as long as it can perform the functions described above.

[0045] The person who owns and uses the administrator terminal 300 may be the administrator operating the robot posture estimation system of the present invention. The administrator terminal 300 may be used to receive relevant information from the system and to set and transmit variables necessary for the system.

[0046] In the following, a robot posture estimation method will be described, primarily focusing on a robot posture estimation system according to one embodiment of the present invention. Unless otherwise specified, the robot posture estimation method according to one embodiment of the present invention can be understood as being performed by the collaborative work of the robot posture estimation system or its subordinate components.

[0047] Figure 2 shows a robot posture estimation method according to one embodiment of the present invention.

[0048] Referring to Figure 2, a marker is set on the robot at step S100.

[0049] Figure 3 shows the S100 stage in detail.

[0050] Referring to Figure 3, the first marker 210 and the second marker 220 are generated at step S110.

[0051] Each marker can be generated using the same method. For example, each marker can be generated according to the ArUco method. An ArUco marker can consist of a n*n size 2D bit pattern and a black border region surrounding it.

[0052] In step S120, the generated marks are attached to the robot. Mark 1 may be attached to the TCP, which is the center of the robot's end effector, and marks 2 may be attached along the sides of the robot's end effector at 90-degree intervals.

[0053] At step S130, the relative positions of the first marker 210 and the second marker 220 are set.

[0054] The relative positions of the first marker 210 and the second marker 220 can be set by directly determining a state conversion value. The state includes position and rotation information. The state conversion value can be input to the main control unit 110 directly or through the administrator terminal 300.

[0055] The relative positions of the first marker 210 and the second marker 220 may be set by simultaneously capturing images of them by the sensing unit 130 and then deriving state conversion values ​​through image processing.

[0056] Let the transformation vector of each marker be denoted as t=[x, y, z], and the rotation vector as r=[a, b, c].

[0057] The variables required for the transformation can be expressed in the respective matrices as shown in [Equation 1], [Equation 2], and [Equation 3].

[0058]

number

[0059]

number

[0060]

number

[0061] Rodrigues Rotation allows three-dimensional coordinate transformations to be represented by a combination of a rotation matrix and a transformation vector.

[0062] Figure 8 shows the positional relationship between the first marker, the second marker, and the sensing unit 130.

[0063] JPEG0007854732000004.jpg29161

[0064]

number

[0065]

number

[0066] Referring again to Figure 3, the positional relationship between the sensing unit 130 and each marker is set in step S140.

[0067] The positional relationship between the sensing unit 130 and each marker can be obtained through image processing even if it is not directly set. Here, "obtained" can mean that not only the positional relationship but also the state information of each marker is output based on that reference.

[0068] In this embodiment, the positional relationship between the sensing unit 130 and each marker can be set by directly determining the state conversion value. The state conversion value can be input directly to the main control unit 110 or through the administrator terminal 300.

[0069] Referring again to Figure 2, in step S200, the sensing unit 130 collects images of the robot including the marker.

[0070] The video acquired by the sensing unit 130 can be collected on a frame-by-frame basis.

[0071] The image acquired by the sensing unit 130 may include an image of the robot, and at least one of the first marker 210 and the second marker 220.

[0072] At stage S300, the main control unit 110 derives the status information for each marker.

[0073] Figure 4 shows the S300 stage in detail.

[0074] Referring to Figure 4, at stage S310, the main control unit 110 estimates the state information of each marker. The estimation of the state information of each marker will be described later with reference to Figure 5.

[0075] In step S310, the position and rotation information of the first marker 210 and the second marker 220 can be estimated.

[0076] At step S320, the main control unit 110 determines whether the first marker 210 can be identified. If it can be identified, the process proceeds to step S340; otherwise, the process proceeds to step S330.

[0077] Here, the statement that the first marker 210 is not identified may mean that the state information of the first marker 210 cannot be estimated using the method in step S310, due to circumstances such as the first marker 210 not being captured by the camera because of changes in the robot's movement.

[0078] At step S330, the main control unit 110 applies the positional relationship between the first marker 210 and the second marker 220 to derive the state information of the first marker 210.

[0079] JPEG0007854732000007.jpg44161

[0080]

number

[0081]

number

[0082] [Mathematical Equation 6] and [Mathematical Equation 7] allow us to determine the rotation matrix and transformation vector of the first marker 210 in the coordinate system of the sensing unit 130, thereby enabling us to identify the relative state information of the first marker 210 with respect to the sensing unit 130.

[0083] In this embodiment of the present invention, the second marker 220 consists of four elements. Therefore, the rotation matrix and transformation vector (or the resulting state information) of the first marker 210 obtained by the above process can be stored in four sets. In this case, the optimal value can be selected for the state information of the first marker 210 according to the set criteria.

[0084] In step S340, the control unit determines the state information for each marker. Determining the state information may involve simply determining the estimated state information for each marker, or it may involve selecting one of the multiple candidate state information options.

[0085] For example, if the first marker 210 is identified at step S320, the process can proceed to step S330 to generate state information for the first marker 210 using the second marker 220. In this case, the state information for the first marker 210 can be stored in sets of up to five. The main control unit 110 can select the optimal state information from the five sets according to the set criteria.

[0086] Optimal state information, for example, when multiple frames exist, may be the value closest to the correct position between the position at a previous time and the position at a later time. The correct position can be set in various ways, such as the average value or median value. However, although the correct position takes into account the robot's natural movement, unnatural movement information such as shaking can also occur. Therefore, even if the movement is not natural, if a large number of sets of state information show similar information, that information may be selected as the optimal state information.

[0087] For example, if the first marker 210 is not identified, one of the state information of the first marker 210 derived from up to four second markers 220 generated in step S330 may be selected and saved.

[0088] Figure 5 shows the S310 stage in detail.

[0089] In step S311, the main control unit 110 sets an adaptive critical value for the sensed image and performs binary conversion. The adaptive critical value means that the optimal critical value is dynamically adjusted according to the brightness of the image and the lighting conditions. The marker region is converted into a black and white binary image based on the critical value through binary conversion.

[0090] Dynamically adjusting the critical value of an image can be done, for example, by following Otsu's algorithm.

[0091] In step S312, the main control unit 110 detects the contour.

[0092] Contour detection is the process of searching for the contour boundaries of an object in a binary-evolved image, specifically the outer outline composed of the object's pixels. Contour detection can be performed using contour detection and approximation algorithms. Through this process, the contour lines of markers can be extracted in an image to identify individual markers.

[0093] In step S313, the main control unit 110 verifies the effectiveness of the marker. If the effectiveness is verified, the process proceeds to step S314; otherwise, it returns to step S311.

[0094] Validity verification is the process of selecting reliable markers by checking their pattern, orientation, size, etc. In the embodiment of the present invention, it is confirmed whether the contour line of the marker detected in the contour detection stage is a valid ArUco marker. Since ArUco markers are made in a certain format, they have a specific pattern and rules, and these are used to verify the validity of the detected markers.

[0095] If the validity of the marker cannot be verified, steps S311 to S313 can be repeated. However, repeating the same operation on the same frame may result in repeated errors, and in the embodiments of the present invention, the possibility of error in the first marker 210 is assumed, so the number of repetitions can be limited to a set number.

[0096] In step S314, the main control unit 110 estimates the state information of the markers based on the markers verified by the contour lines.

[0097] State information estimation is the process of estimating the object's 3D pose (position and orientation) by calculating the position of markers relative to the camera.

[0098] Through this process, the 3D state information of the marker is estimated from the 2D image obtained from the sensing unit 130. At this time, the extracted 3D state information of the marker is its position relative to the sensing unit 130.

[0099] At step S500, the main control unit 110 estimates the robot's posture based on at least one of the determined state information of the first marker 210 and the second marker 220. Estimating the robot's posture may involve estimating the movement of multiple joints and links included in the robot.

[0100] Figure 6 shows in detail a part of the configuration of a robot posture estimation method according to one embodiment of the present invention.

[0101] Referring to Figure 6, at step S510, the main control unit 110 preprocesses data relating to the state information of the first marker 210 and / or the second marker 220.

[0102] Data preprocessing may include, for example, correction or filtering of data that is above a critical value (i.e., contains noise).

[0103] The SWAI (Sliding Window Average Interpolation) technique can be applied to process the noise.

[0104] Specifically, when using an external vision sensor to photograph markers attached to a robot, factors such as the light intensity at the robot's location, partial obstruction of the marker due to singularities, and recognition errors due to the shooting distance can occur. These environmental factors are reflected in the image, resulting in positional data containing noise such as outliers or missing values ​​in the positional data at the time of marker detection. The left side of Figure 9 shows an example of such noisy data. Sliding window average interpolation and min-maximum normalization techniques can be applied to process the noise contained in the time-series data.

[0105] The left side of Figure 9 shows the data before preprocessing of the coordinate values ​​of the first marker 210, while the right side shows the data after preprocessing.

[0106] Referring to Figure 9, it can be seen that the data's information values ​​were smoothly concatenated through preprocessing.

[0107] Figure 10 shows the positional relationship between markers applied to an actual robot, and Figure 11 shows the state information of those markers.

[0108] In each graph, indices M6, M8, M10, and M11 each represent the second marker 220. Index M7 represents the first marker 220.

[0109] Figure 10 shows the relationship between the second marker 220 and the first marker 210, verified using actual robot data.

[0110] On the other hand, referring to the top of Figure 11, it can be seen that some markers were obscured (not sensed) by the robot's movement, and this data is shown parallel to the horizontal axis. By deriving and organizing the position of the first marker 210 from the data of the observable second marker 220, a continuous and smooth graph like the one at the bottom of Figure 11 can be derived.

[0111] In the S510 stage, the main control unit 110 inputs the pre-processed data into the machine learning module to estimate the robot's posture.

[0112] Robot posture estimation can be performed by first generating a first marker 210 and training data related to posture, training a machine learning module with this data, and then applying the pre-processed input data.

[0113] The robot's pose estimation may be performed by first determining the training data using state information and pose datasets for the first marker 210 and the second marker 220, and then training a machine learning module with this data.

[0114] The terms used in this application are used solely to describe specific embodiments and are not intended to limit the invention. In this application, terms such as “includes” or “having” are intended to specify the presence of features, figures, stages, operations, components, parts, or combinations thereof as described in the specification, and should be understood not to preemptively exclude the possibility of the presence or addition of one or more other features, figures, stages, operations, components, parts, or combinations thereof. [Explanation of symbols]

[0115] 110: Main Control Unit 120: Sensor Control Unit 130: Sensing Department 140: Robot Control Unit 200: Collaborative Robots 210: First marker 220: Second marker 300: Administrator terminal

Claims

1. In a robot pose estimation method performed by one or more processors of a computer device, A step of generating a first marker and at least one second marker and attaching them to the robot; A step of collecting video including the first marker and at least one of the plurality of second markers; A step of estimating the position and rotation information of the first marker and the second marker, respectively; If the first marker is not estimated, the step of deriving position and rotation information of the first marker based on the relative positional relationship between the first marker and the second marker, including a rotation matrix and a transformation vector; and A method for estimating the posture of a robot, comprising the step of estimating the posture of the robot based on the position and rotation information of the first marker.

2. The method for estimating the posture of a robot according to claim 1, characterized in that the first marker is attached to the end effector of the robot.

3. The step of setting the positional relationship between the first marker and the second marker attached to the robot; further includes, The method for estimating the posture of a robot according to claim 1, characterized in that the positional relationship is derived by analyzing an image generated by simultaneously photographing the first marker and the second marker.

4. The state information of the first marker is derived in multiple units based on its positional relationship with each of the multiple second markers. The step of selecting one of the first marker state information derived from the aforementioned multiple values; The robot posture estimation method according to claim 1, further comprising the following:

5. The robot posture estimation method according to claim 1, characterized in that the posture of the robot is estimated by a machine learning module that has been pre-trained on the relationship between the state information of the first marker and the posture of the robot.

6. A first marker and at least one second marker attached to the robot; A sensing unit that collects video including the first marker and at least one of the plurality of second markers; and A robot posture estimation system comprising: a main control unit that estimates the position and rotation information of the first marker and the second marker, and, if the first marker is not estimated, derives the position and rotation information of the first marker based on a pre-set relative positional relationship between the first marker and the second marker, including a rotation matrix and a transformation vector; and estimates the posture of the robot based on the position and rotation information of the first marker.

Images