Robot posture estimation method and system

Abstract

To disclose a robot posture estimation method.SOLUTION: A robot posture estimation method of the present invention includes the steps of: generating a first marker and at least one or more second markers and attaching the markers to a robot; collecting a video including at least one of the first marker and the plurality of second markers; estimating position and rotation information of the first marker and the second marker; deriving the position and rotation information of the first marker from a relative positional relation between the first marker and the second marker when the first marker is not estimated; and estimating a posture of the robot on the basis of the position and rotation information of the first marker.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a method and a system for estimating the posture of a robot.

Background Art

[0002] A collaborative robot can improve the productivity of a factory by collaborating with other devices and workers. However, since it works with people, it is necessary to have a technology that can grasp the defects in advance and maintain and repair them. If defects are diagnosed using internal data generated for driving and controlling the collaborative robot, it can proceed without installing additional hardware. However, the data that can be collected varies depending on the type of robot, and since the movement is corrected internally for accurate driving, there is a possibility that the characteristics of the defects cannot be correctly reflected in the internal data. Also, even for the same type of robot, the defect detection criteria change depending on the program and environment being executed.

[0003] As a method for solving such problems, an external vision sensor is also used to detect abnormalities in the robot. When using an external vision sensor of the type that attaches markers to the robot, an independent defect detection system can be constructed for the operation of the robot.

[0004] By utilizing this, the movement path of the robot can be estimated through continuous photographing of the markers, and abnormalities and defects can be detected by comparing with normal driving. However, when the markers are blocked, the position of the robot cannot be estimated, and in the conventional method, only the position of the markers is estimated, and the exact postures of the joints and links of the robot cannot be estimated.

[0005] Also, when using multiple cameras to recognize the positions of the markers, the installation of the cameras may be restricted depending on the working environment.

Summary of the Invention

Problems to be Solved by the Invention

[0006] The technical problem to be solved by the present invention is to apply at least one or more markers to a robot, accurately estimate the position of a marker by other markers even when one marker is blocked, and provide a robot pose estimation method and system capable of estimating the pose of the robot based on this result.

Means for Solving the Problem

[0007] To solve the above technical problem, a robot pose estimation method according to an embodiment of the present invention can be executed by one or more processors of a computer device, and includes the steps of generating a first marker and at least one or more second markers and attaching them to the robot; collecting an image including at least one of the first marker and the plurality of second markers; estimating the position and rotation information of each of the first marker and the second marker; when the first marker is not estimated, deriving the position and rotation information of the first marker based on the relative positional relationship between the first marker and the second marker; and estimating the pose of the robot based on the position and rotation information of the first marker.

[0008] In one embodiment of the present invention, the first marker can be attached to an end effector of the robot.

[0009] In one embodiment of the present invention, it further includes the step of setting the positional relationship between the first marker and the second marker attached to the robot; and the positional relationship can be derived by analyzing an image generated by simultaneously photographing the first marker and the second marker.

[0010] In one embodiment of the present invention, it can further include the step of deriving a plurality of pieces of state information of the first marker based on the positional relationship between each of the plurality of second markers, and selecting any one of the plurality of pieces of derived state information of the first marker.

[0011] In one embodiment of the present invention, the posture of the robot can be estimated by a machine learning module in which the relationship between the state information of the first marker and the posture of the robot has been learned in advance.

[0012] To solve the above technical problem, a robot posture estimation system according to an embodiment of the present invention may include: a first marker and at least one second marker attached to the robot; a sensing unit configured to collect an image including at least one of the first marker and the plurality of second markers; and a main control unit configured to estimate the position and rotation information of each of the first marker and the second marker, derive the position and rotation information of the first marker based on the relative positional relationship between the first marker and the second marker when the first marker is not estimated, and estimate the posture of the robot based on the position and rotation information of the first marker.

Advantages of the Invention

[0013] The present invention applies at least one marker to the robot, and can accurately estimate the position of the marker even when one marker is blocked, and estimate the posture of the robot based on this result.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Mode for Carrying Out the Invention

[0015] The present invention can be subjected to various conversions and can have various embodiments. Specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all conversions, equivalents, and alternatives included in the spirit and technical scope of the present invention.

[0016] In describing the present invention, if it is determined that a specific description of related known technologies may obscure the gist of the present invention, the detailed description thereof will be omitted.

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0018] FIG. 1 shows a posture estimation system for a robot according to an embodiment of the present invention.

[0019] Referring to FIG. 1, a posture estimation system for a robot according to an embodiment of the present invention may include a main control unit 110, a sensor control unit 120, a sensing unit 130, a robot control unit 140, a first marker 210, and a second marker 220.

[0020] The main control unit 110 is connected to the sensor control unit 120 and the robot control unit 140 to control them.

[0021] The main control unit 110 can be connected to the administrator terminal 300 via a wired / wireless network to send and receive information.

[0022] The main control unit 110 can be a control device including a processor, a memory, an input interface, etc. For example, it can be composed of a desktop computer, a laptop computer, or a device performing similar functions.

[0023] The sensor control unit 120 can control the sensing unit 130 under the control of the main control unit 110.

[0024] Although the sensor control unit 120 is illustrated as being physically separated from the main control unit 110, it may be composed of a program embedded in the main control unit 110. In this case, the main control unit 110 can directly control the sensing unit 130.

[0025] The sensing unit 130 is connected to the sensor control unit 120 and operates under the control of the sensor control unit 120.

[0026] The sensing unit 130 can include an image sensor. For example, the sensing unit 130 can include a camera.

[0027] In one embodiment of the present invention, the sensing unit 130 can include only one camera. However, in order to improve image quality such as resolution, a plurality of cameras can also be built in, and their positions can be determined to be one. Here, one position can mean a position where one physical entity is arranged, not necessarily exactly the same position. That is, image sensing by a plurality of cameras that are spatially very far apart may be different from the main purpose of the present invention. However, it is not limited.

[0028] The robot control unit 140 is connected to the main control unit 110 and the robot, and can control the robot under the control of the main control unit 110. However, it is not necessarily limited to this embodiment, and the robot control unit 140 can be arranged, connected, and set to control the robot independently of the main control unit 110.

[0029] According to an embodiment, the robot control unit 140 may also be composed of a program embedded in the main control unit 110.

[0030] The first marker 210 is attached to the robot, and its image can be acquired by the sensing unit 130.

[0031] Specifically, the first marker 210 can be attached to the end - effector of the robot.

[0032] The end - effector means a part that has the function of directly acting on the work object when the robot performs work. For example, a gripper, a welding torch, a spray gun, a nut runner, etc. correspond to the end - effector.

[0033] Attaching the first marker 210 to the end - effector is for estimating the TCP (Tool Center Point) position, and the first marker 210 is also denoted as M TCP and will also be denoted as such.

[0034] The second marker 220 is attached to the robot, and its image can be acquired by the sensing unit 130.

[0035] The second marker 220 can be attached so as to have a relative positional relationship with respect to the first marker 210.

[0036] For example, the second marker 220 can be attached to the end - effector, or can be attached to the four directions of the lateral surface of the end - effector instead of the TCP.

[0037] The second marker 220 is an auxiliary means for estimating the position of the first marker 210 and the like even when the first marker 210 is not sensed, and M SUB will also be displayed.

[0038] FIG. 7 shows the state of attachment of the markers to the robot. Referring to FIG. 7, in one embodiment of the present invention, the first marker 210 is at the TCP position at the upper center of the end effector, and four second markers 220 are attached at 90-degree intervals along the periphery of the side surface of the end effector.

[0039] Although not shown in the drawings, the administrator terminal 300 can include a control unit, a communication unit, a storage unit, and an input unit.

[0040] The control unit is connected to the communication unit, the storage unit, and the input unit and can control these.

[0041] The communication unit can transmit and receive information to and from the main control unit 110.

[0042] The storage unit can store necessary information so as to provide convenience for information processing. The storage unit can store an application installed in the administrator terminal 300.

[0043] The input unit is an input interface for controlling the administrator terminal 300 and can be composed of a keyboard or a touch screen.

[0044] The administrator terminal configured in this way can be, for example, any one of a smartphone, a tablet PC, and a notebook computer, and is not limited to the above embodiments as long as it can perform the above functions.

[0045] Those who own and use the administrator terminal 300 can be administrators who operate the robot posture estimation system of the present invention. The administrator terminal 300 can be used to receive relevant information from the system and set and transmit the necessary variables to the system.

[0046] Hereinafter, a robot posture estimation method will be described mainly based on the robot posture estimation system according to an embodiment of the present invention. Unless otherwise specified, the robot posture estimation method according to an embodiment of the present invention can be understood to be performed by the collaborative work of the robot posture estimation system or its subordinate components.

[0047] FIG. 2 shows a robot posture estimation method according to an embodiment of the present invention.

[0048] Referring to FIG. 2, in step S100, a marker is set on the robot.

[0049] FIG. 3 shows step S100 in detail.

[0050] Referring to FIG. 3, in step S110, a first marker 210 and a second marker 220 are respectively generated.

[0051] Each marker can be generated in the same manner. For example, each marker can be generated according to the ArUco method. The ArUco method marker can be composed of a two-dimensional bit pattern of size n*n and a black edge region surrounding it.

[0052] In step S120, the generated marks are attached to the robot. The first mark can be attached to the TCP which is the center of the robot's end effector, and the second mark can be attached at 90-degree intervals along the side surface of the robot's end effector.

[0053] In step S130, the positional relationship between the first marker 210 and the second marker 220 is set.

[0054] The positional relationship between the first marker 210 and the second marker 220 can be set by directly determining the state conversion value. The state includes position and rotation information. The state conversion value can be input directly to the main control unit 110 or through the administrator terminal 300.

[0055] The positional relationship between the first marker 210 and the second marker 220 may be set by deriving the state conversion value through image processing after simultaneously photographing them by the sensing unit 130.

[0056] Let the conversion vector of each marker be t = [x, y, z] and the rotation vector be r = [a, b, c].

[0057] If the variables required for the conversion are expressed by their respective matrices, they are as shown in [Equation 1], [Equation 2], and [Equation 3] respectively.

[0058]

Number

[0059]

Number

[0060]

Number

[0061] The three-dimensional coordinate transformation can be expressed by the combination of the rotation matrix and the conversion vector by Rodrigues Rotation.

[0062] Figure 8 shows the positional relationship among the first marker, the second marker, and the sensing unit 130.

[0063] JPEG2025105426000005.jpg29161

[0064]

Number

[0065]

Number

[0066] Referring to FIG. 3 again, at step S140, the positional relationship between the sensing unit 130 and each marker is set.

[0067] Even if the positional relationship between the sensing unit 130 and each marker is not set directly, it can be obtained by image processing. Here, being obtained can mean that not only the positional relationship but also the status information of each marker is output according to that standard.

[0068] According to the embodiment, the positional relationship between the sensing unit 130 and each marker can be set by directly determining the state conversion value. The state conversion value can be input directly to the main control unit 110 or through the administrator terminal 300.

[0069] Referring to FIG. 2 again, at step S200, the sensing unit 130 collects the video of the robot including the markers.

[0070] The video obtained by the sensing unit 130 can be collected in frame units.

[0071] The video obtained by the sensing unit 130 may include at least one of the image of the robot, the first marker 210, and the second marker 220.

[0072] At step S300, the main control unit 110 derives the status information of each marker.

[0073] FIG. 4 shows step S300 in detail.

[0074] Referring to FIG. 4, at step S310, the main control unit 110 estimates the state information of each marker. The estimation of the state information of each marker will be described later with reference to FIG. 5.

[0075] At step S310, the positions and rotation information of the first marker 210 and the second marker 220 can be estimated.

[0076] At step S320, the main control unit 110 determines whether the first marker 210 is discriminated. If it is discriminated, the process proceeds to step S340; if not, the process proceeds to step S330.

[0077] Here, the fact that the first marker 210 is not discriminated may mean that the state information of the first marker 210 cannot be estimated by the method of step S310 due to a situation such as the first marker 210 not being captured by the camera due to a change in the operation of the robot.

[0078] At step S330, the main control unit 110 applies the positional relationship between the first marker 210 and the second marker 220 to derive the state information of the first marker 210.

[0079] JPEG2025105426000008.jpg44161

[0080]

Number

[0081]

Number

[0082] According to [Equation 6] and [Equation 7], the rotation matrix and transformation vector in the coordinate system of the sensing unit 130 of the first marker 210 can be obtained, so that the relative state information with respect to the sensing unit 130 of the first marker 210 can be specified.

[0083] In an embodiment of the present invention, the second marker 220 is composed of four elements. Therefore, the rotation matrix and transformation vector (or the state information thereby obtained) of the first marker 210 obtained by the above process can be stored in four sets. In this case, an optimal value can be selected from the state information of the first marker 210 according to a set criterion.

[0084] In step S340, the control unit determines the state information of each marker. Determining the state information may directly determine the estimated state information of the marker, or may select one from a plurality of candidates for the state information.

[0085] For example, even when the first marker 210 is discriminated in step S320, the process can proceed to step S330 to generate the state information of the first marker 210 by the second marker 220. In this case, the state information of the first marker 210 can be stored in up to five sets. The main control unit 110 can select the optimal state information from the five sets according to a set criterion.

[0086] The optimal state information can be, for example, the value closest to an appropriate position between the position at a previous time and the position at a subsequent time when there are a plurality of frames. The appropriate position can be set in various ways such as an average value, a median value, etc. However, although the natural movement of the robot is considered for the appropriate position, since unnatural movement information such as shaking can actually occur, if a majority of sets of the state information indicate similar information even if it is not natural movement, that information may be selected as the optimal state information.

[0087] For example, when the first marker 210 is not discriminated, any one of the state information of the first marker 210 derived from the maximum four second markers 220 generated in step S330 can be selected and stored.

[0088] FIG. 5 shows step S310 in detail.

[0089] In step S311, the main control unit 110 sets an adaptive threshold for the sensed image and evolves it. The adaptive threshold means dynamically adjusting the optimal threshold according to the brightness of the image and the lighting conditions. The marker area is evolved based on the threshold and converted into a binary image of black and white.

[0090] Dynamically adjusting the threshold of the image can be achieved, for example, according to the Otsu Algorithm.

[0091] In step S312, the main control unit 110 senses the contour.

[0092] Sensing the contour is an operation of searching for the contour boundary of an object in the evolved image, and is a process of finding the outer contour line composed of the pixels of the object. Contour line detection can be performed using a contour detection and approximation algorithm. Through this, the contour line of the marker for identifying individual markers in the image can be extracted.

[0093] In step S313, the main control unit 110 verifies the validity of the marker. If the validity is verified, it proceeds to step S314, and if not, it returns to step S311.

[0094] Validity verification is an operation of checking the pattern, direction, size, etc. of the marker to select a reliable marker. In an embodiment of the present invention, it is confirmed whether the contour line of the marker detected in the contour sensing stage is a valid ArUco marker. Since the ArUco marker is made in a certain format and has a specific pattern and rule, the validity of the detected marker is verified using this.

[0095] If the validity of the marker is not verified, steps S311 to S313 can be repeated. However, if the same operation is repeated for the same frame, errors can be repeated. In an embodiment of the present invention, assuming the error possibility of the first marker 210, the number of repetitions can be limited to the set number of times.

[0096] In step S314, the main control unit 110 estimates the state information of the marker based on the marker verified by the contour line.

[0097] The estimation of the state information is a process of calculating the position of the marker with respect to the camera and estimating the 3D pose (position and orientation) of the object.

[0098] Through this, the three-dimensional state information of the marker is estimated from the two-dimensional video obtained from the sensing unit 130. At this time, the extracted three-dimensional state information of the marker is the position based on the sensing unit 130.

[0099] In step S500, the main control unit 110 estimates the posture of the robot based on at least one of the state information of the determined first marker 210 and the second marker 220. Estimating the posture of the robot may mean estimating the movements of a plurality of joints and links included in the robot.

[0100] FIG. 6 shows in detail a part of the configuration of a method for estimating the posture of a robot according to an embodiment of the present invention.

[0101] Referring to FIG. 6, in step S510, the main control unit 110 preprocesses the data regarding the state information of the first marker 210 and / or the second marker 220.

[0102] The preprocessing of the data can include, for example, correction or filtering for data above a threshold value (including noise).

[0103] The processing of noise can apply the SWAI (Sliding Window Average Interpolation) technique.

[0104] Specifically, when photographing a marker attached to a robot using an external vision sensor, factors such as the intensity of light at the location where the robot is installed, partial occlusion of the marker due to singularities, and recognition errors due to the photographing distance may occur. Such environmental factors are reflected in the video, and position data containing noise such as outliers or missing values with respect to the position data at the time of marker detection can be obtained. The left side of FIG. 9 shows an example of such data containing noise. Sliding Window Average Interpolation and Min-Max Normalization techniques can be applied to process the noise contained in the time-series data.

[0105] The left side of FIG. 9 shows the data before preprocessing the coordinate values of the first marker 210, and the right side shows the data after preprocessing.

[0106] Referring to FIG. 9, it can be seen that the information values of the data are smoothly connected by preprocessing.

[0107] FIG. 10 shows the positional relationship between the markers applied to the actual robot, and FIG. 11 shows the state information of the markers.

[0108] In each graph, the indices M6, M8, M10, and M11 respectively indicate the second marker 220. The index M7 indicates the first marker 220.

[0109] FIG. 10 verifies the relationship between the second marker 220 and the first marker 210 using actual robot data.

[0110] On the other hand, referring to the upper end of FIG. 11, it can be seen that data in which some markers were occluded (not sensed) due to the operation of the robot is also shown (the corresponding data is shown parallel to the horizontal axis). When the position of the first marker 210 is derived and organized based on the observable data of the second marker 220, a continuous and smooth graph as shown in the graph at the lower end of FIG. 11 can be derived.

[0111] In the S510 stage, the main control unit 110 inputs the preprocessed data into the machine learning module to estimate the posture of the robot.

[0112] For the posture estimation of the robot, after generating the learning data related to the first marker 210 and the posture in advance and having the machine learning module learn it, the preprocessed input data can be applied.

[0113] For the posture estimation of the robot, after determining the learning data based on the state information of each of the first marker 210 and the second marker 220 and the data set related to the posture, the machine learning module may be made to learn.

[0114] The terms used in this application are merely used to explain specific embodiments and are not intended to limit the present invention. In this application, terms such as "including" or "having" are intended to specify the existence of the features, numbers, steps, operations, components, parts described in the specification or combinations thereof, and should be understood not to preclude in advance the possibility of the existence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

Explanation of Reference Numerals

[0115] 110: Main control unit 120: Sensor control unit 130: Sensing unit 140: Robot control unit 200: Collaborative robot 210: First marker 220: Second marker 300: Administrator terminal

Claims

1. In a method for estimating the posture of a robot executed by one or more processors of a computer device, generating a first marker and at least one or more second markers and attaching them to the robot; collecting an image including at least one of the first marker and the plurality of second markers; estimating the position and rotation information of each of the first marker and the second marker; when the first marker is not estimated, deriving the position and rotation information of the first marker based on the relative positional relationship between the first marker and the second marker; and estimating the posture of the robot based on the position and rotation information of the first marker; A method for estimating the posture of a robot, characterized by comprising:

2. The method for estimating the posture of a robot according to claim 1, wherein the first marker is attached to an end effector of the robot.

3. further comprising setting a positional relationship between the first marker and the second marker attached to the robot; The method for estimating the posture of a robot according to claim 1, wherein the positional relationship is derived by analyzing an image generated by simultaneously photographing the first marker and the second marker.

4. The state information of the first marker is derived in plural by the positional relationship with each of the plurality of second markers, The method for estimating the posture of a robot according to claim 1, further comprising selecting any one of the plurality of pieces of state information of the first marker derived.

5. The method for estimating the posture of a robot according to claim 1, wherein the posture of the robot is estimated by a machine learning module in which the relationship between the state information of the first marker and the posture of the robot has been learned in advance.

6. A first marker and at least one or more second markers attached to a robot; a sensing unit that collects an image including at least one of the first marker and the plurality of second markers; and A main control unit that estimates the position and rotation information of each of the first marker and the second marker, and when the first marker is not estimated, derives the position and rotation information of the first marker based on the relative positional relationship between the first marker and the second marker, and estimates the posture of the robot based on the position and rotation information of the first marker; A robot posture estimation system, characterized by including.

Images