Method and system for estimating position and posture of object

Abstract

To provide a method and a system for estimating the position and posture of an object in real time with a small operation amount by using a camera provided in a robot.SOLUTION: The method includes capturing an image of an object by a camera provided in a robot, recognizing a landmark indicating at least one surface included in the object from the image of the object by a controller, determining a main surface by using the recognized landmark by the controller, determining a position of the landmark located on the main surface by the controller, calculating a final center position and a final normal vector of the main surface by using the determined position of the landmark and a specification of the object by the controller, and estimating a position and a posture of the object by the controller. The position of the center and the posture of the object are calculated by using the position of the final center of the main surface, the final normal vector, and the specification of the object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a method and system for estimating the position and orientation of an object, and more particularly to a method and system for estimating the position and orientation of an object in real time using a camera provided on a robot. [Background technology]

[0002] Recently, the use of autonomous robots has been increasing. These robots can perform a variety of tasks (such as charging vehicles parked in parking lots) in addition to transporting people and cargo. Tasks performed by robots can be divided into individual tasks performed by a single robot and integrated tasks performed by multiple robots working together. In order for multiple robots to perform integrated tasks, one robot can be connected to or follow another robot. In order for one robot to be connected to or follow another robot, the robot must know the position and posture of the other robot.

[0003] There are various methods for determining the position and orientation of an object. In one example, a controller (processor) installed in a robot calculates the position and orientation of an object in real time through deep learning using values ​​detected by a camera or LIDAR. However, calculating the position and orientation of an object in real time requires a large amount of calculation, making it difficult to apply to embedded systems with low calculation performance. In another example, an object transmits its position and orientation to a robot in real time. However, if communication between the object and robot is interrupted, the robot is unable to determine the position and orientation of the object.

[0004] The matters described in this background art section are prepared to enhance understanding of the background of the invention, and may include matters that are not prior art already known to those having ordinary skill in the art to which this technology pertains. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2024-150627 Summary of the Invention [Problem to be solved by the invention]

[0006] SUMMARY OF THE INVENTION An embodiment of the present invention provides a method and system for estimating the position and orientation of an object in real time with a small amount of calculation using a camera provided on a robot. [Means for solving the problem]

[0007] According to an embodiment of the present invention, there is provided a method for estimating the position and orientation of an object, wherein the object includes at least one surface, and at least one marker indicating the surface is provided near a vertex of the surface, and a width and height between the at least one marker and a center of the surface, and a width, height, and depth between the center of the surface and a center of the object are stored in advance as object specifications.

[0008] The posture estimation method may include the steps of: capturing an image of the object using a camera provided to the robot; recognizing the markers from the image of the object by a controller; determining a main surface using the recognized markers by the controller; determining positions of the markers located on the main surface by the controller; calculating a final center position and a final normal vector of the main surface by the determined marker positions and specifications of the object by the controller; and calculating a center position and posture of the object by the controller using the final center position and the final normal vector of the main surface and specifications of the object.

[0009] The indicia may be light emitting diodes that emit light of one color set by the surface on which the indicia is provided.

[0010] The image of the object captured by the camera may include information about the color of the LED in the image, information about pixel data of the LED in the image, and / or information about a transformation relationship between the pixel data of the LED and three-dimensional data, and the three-dimensional data may include depth data.

[0011] In the step of recognizing the sign from the image of the object, at least the color of light emitted by the light emitting diode and pixel data of the vertices of a bounding box of N*N size corresponding to the light emitting diode may be detected.

[0012] The step of determining the main surface using the recognized mark may include the step of determining, as the main surface, a surface on which the largest number of vertices out of all detected vertices are located.

[0013] The step of determining the position of the marker located on the major surface may include the steps of: generating 3D data of vertices of a bounding box corresponding to the LED located on the major surface; calculating 2D data of the center of at least the i-th LED and a normal vector based on the 3D data of the vertices of the bounding box corresponding to the i-th LED and the size of the bounding box; determining whether the i-th LED is located left or right in the width direction while moving the i-th LED left or right in the width direction; and determining whether the i-th LED is located up or down in the height direction by determining whether the height of the center of the i-th LED relative to the camera is above or below the height of the center of the object.

[0014] The step of determining whether the i-th LED is located left or right in the width direction while moving the i-th LED left or right in the width direction may include the steps of: calculating a difference between a pre-movement normal vector and a post-movement normal vector while moving the i-th LED left or right by a set distance in the width direction; determining that the i-th LED is located on the side of the movement direction if the difference between the pre-movement normal vector and the post-movement normal vector is greater than a set value; and determining that the i-th LED is located on the opposite side of the movement direction if the difference between the pre-movement normal vector and the post-movement normal vector is equal to or less than a set value.

[0015] The step of calculating the final center position and the final normal vector of the main surface using the determined position of the marker and the specifications of the object may include the steps of calculating the center position and the normal vector of the main surface for all light emitting diodes located on the main surface based on data of the centers of the light emitting diodes and the specifications of the object, and calculating the center position and the normal vector of the main surface using the center positions and normal vectors of the main surface calculated for all light emitting diodes located on the main surface.

[0016] The step of calculating the center position and normal vector of the main surface using the center position and the normal vector of the main surface calculated for all light-emitting diodes located on the main surface may include the step of filtering the center position and normal vector of the main surface calculated for all light-emitting diodes located on the main surface.

[0017] The method may further include controlling the robot based on the position and orientation of the center of the object by the controller.

[0018] According to another embodiment of the present invention, there is provided a system for estimating the position and orientation of an object, wherein the object includes at least one surface, and at least one marker indicating the surface is provided near a vertex of each surface, and a width and height between the at least one marker and a center of the surface, and a width, height, and depth between the center of the surface and a center of the object are stored in advance as object specifications.

[0019] The posture estimation system may include a robot including a driving unit that generates a driving force and configured to follow or combine with the object, a camera attached to the robot and configured to capture an image of the object within a field of view, and a controller connected to the camera to receive the image of the object from the camera and configured to control the robot.

[0020] The controller may be configured to recognize the markers from an image of the object, determine a principal plane using the recognized markers, determine the positions of the markers located on the principal plane, calculate a final center position and a final normal vector of the principal plane using the determined marker positions and specifications of the object, and calculate a center position and orientation of the object using the final center position and final normal vector of the principal plane and specifications of the object.

[0021] The indicia may be light emitting diodes that emit light of one color set by the surface on which the indicia is provided.

[0022] The image of the object captured by the camera may include information about the color of the LED in the image, information about pixel data of the LED in the image, and / or information about a transformation relationship between the pixel data of the LED and three-dimensional data, and the three-dimensional data may include depth data.

[0023] When recognizing the sign from the image of the object, the controller can detect at least the color of light emitted by the light emitting diode and pixel data of the vertices of a bounding box of N*N size corresponding to the light emitting diode.

[0024] When determining the main surface using the recognized markers, the controller may determine the surface on which the most vertices of all detected vertices are located as the main surface.

[0025] When determining the position of the marker located on the major surface, the controller may be configured to generate the 3D data of the vertices of the bounding box corresponding to the LEDs located on the major surface, calculate at least 2D data of the center of the i-th LED and a normal vector based on the 3D data of the vertices of the bounding box corresponding to the i-th LED and the size of the bounding box, determine whether the i-th LED is located left or right in the width direction while moving the i-th LED left or right, and determine whether the i-th LED is located up or down in the height direction by determining whether the height of the center of the i-th LED relative to the camera is above or below the height of the center of the object.

[0026] When determining whether the i-th LED is located to the left or right in the width direction while moving the i-th LED to the left or right in the width direction, the controller may be configured to calculate a difference between a pre-movement normal vector and a post-movement normal vector while moving the i-th LED to the left or right by a set distance in the width direction, and determine that the i-th LED is located on the side of the movement direction if the difference between the pre-movement normal vector and the post-movement normal vector is greater than a set value, or determine that the i-th LED is located on the opposite side of the movement direction if the difference between the pre-movement normal vector and the post-movement normal vector is equal to or less than a set value.

[0027] When calculating the final center position and final normal vector of the main surface using the determined marker positions and object specifications, the controller may be configured to calculate the center position and normal vector of the main surface for all light emitting diodes located on the main surface based on data of the centers of the light emitting diodes and the object specifications, and to calculate the center position and normal vector of the main surface using the calculated center positions and normal vectors of the main surface for all light emitting diodes located on the main surface.

[0028] When calculating the center position and normal vector of the main surface using the center position and normal vector of the main surface calculated for all light-emitting diodes located on the main surface, the controller may be configured to filter the center position and normal vector of the main surface calculated for all light-emitting diodes located on the main surface.

[0029] The controller may be further configured to control the robot based on a position and a posture of a center of an object. [Effects of the Invention]

[0030] According to the present invention, an LED indicating the corresponding surface is attached to each surface of an object, and the LED is recognized by a camera mounted on a robot. The recognized LED is used to determine the main surface, and the position and orientation of the object can be calculated based on the 3D data of the LEDs positioned on the main surface and the specifications of the object. Therefore, the position and orientation of the object can be estimated in real time with a small amount of calculation.

[0031] In addition, the estimated position and orientation of the object can be used to control connection with the object or tracking of the object. Other effects that can be obtained or are expected to be obtained by the embodiments of the present invention will be directly or implicitly disclosed in the detailed description of the embodiments of the present invention, i.e., various effects expected by the embodiments of the present invention will be disclosed in the detailed description below. [Brief explanation of the drawings]

[0032] The embodiments herein may be better understood with reference to the following description taken in conjunction with the accompanying drawings, where like reference numbers indicate identical or functionally similar elements and wherein: [Figure 1] 1 is a block diagram of an object position and pose estimation system according to an embodiment of the present invention; [Figure 2] 1 is a schematic diagram illustrating one surface (e.g., a side surface) of an object according to one example. [Figure 3] 1 is a schematic diagram illustrating another surface (e.g., the front or back surface) of an object according to one example. [Figure 4] 1 is a flowchart of a method for estimating the position and orientation of an object according to an embodiment of the present invention. [Figure 5] 5 is a specific flowchart of step S150 in FIG. 4. [Figure 6] 10 is a schematic diagram illustrating the specification of one aspect of an object according to one example. [Figure 7] 10 is a diagram illustrating a schematic view of a center position of a light emitting diode disposed on one surface of an object according to an example. [Figure 8] 10A and 10B schematically illustrate the position and normal vector of the center of one surface of an object, and the position and orientation of the center of the object, according to one example.

[0033] It should be understood that the above-referenced drawings are not necessarily drawn to scale, but rather present somewhat simplified representations of various preferred features illustrating the underlying principles of the present disclosure. For example, specific design features of the present disclosure, including specific dimensions, orientations, locations, and shapes, are determined in part by the particular intended application and environment of use. DETAILED DESCRIPTION OF THE INVENTION

[0034] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the present invention. As used herein, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. The terms "comprises" and / or "comprises," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but should also be understood to not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of the associated listed items.

[0035] As used herein, the term "robot" or other similar terms includes general land-based robots, including passenger cars (including sport utility vehicles) (SUVs), buses, trucks, and various commercial vehicles; marine-based robots, including various boats and ships; and airborne robots, including aircraft, drones, and the like, and includes any object that can move under power from a power source. Additionally, the term "robot" or other similar terms used herein is understood to include hybrid-powered robots, electric-powered robots, plug-in hybrid-powered robots, hydrogen-powered robots, and other alternative-fuel (e.g., fuels derived from sources other than petroleum) robots. As referred to herein, hybrid-powered robots include robots with two or more power sources, such as gasoline-powered and electric-powered robots. Robots according to embodiments of the present invention include robots that are driven somewhat autonomously and / or automatically, as well as manually driven robots.

[0036] Additionally, it is understood that one or more of the methods or aspects thereof described below can be performed by at least one or more controllers. The term "controller" can refer to a hardware device including a memory and a processor. The memory is configured to store program instructions, and the processor is specifically programmed to execute the program instructions to perform one or more processes described in more detail below. The controller can control the operation of a unit, module, component, device, or the like, as described herein. It is also understood that the methods described below can be performed by a device that includes a controller along with one or more other components, as will be appreciated by those skilled in the art.

[0037] The controller of the present disclosure may also be implemented as a non-transitory computer-readable storage medium containing executable program instructions for execution by a processor. Examples of computer-readable storage media include, but are not limited to, ROM, RAM, compact disc (CD) ROM, magnetic tape, floppy disk, flash drive, smart card, and optical data storage device. The computer-readable storage medium may also be distributed across a computer network so that the program instructions are stored and executed in a distributed manner, such as in a telematics server or controller area network (CAN).

[0038] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a block diagram of an object position and orientation estimation system according to an embodiment of the present invention. As shown in FIG. 1, the object position and orientation estimation system according to the embodiment of the present invention includes a camera 10, a controller 20, and a robot 30.

[0039] The camera 10 is attached to the robot 30 and can capture an image of an object 40 within the field of view of the camera 10. The image of the object 40 can include information about the object 40, such as information about the color of a feature point (e.g., an LED) within the object 40, information about pixel data of the feature point within the object 40 (pixel data of the vertices of a bounding box corresponding to the LED), and / or information about the transformation relationship between the pixel data of the feature point within the object 40 and the 3D data of the feature point. The 3D data includes depth data. Here, the bounding box may be an N*N regular rectangle corresponding to the LED, but other appropriate bounding box shapes can also be used. For ease of explanation, a bounding box with an N*N regular rectangle shape will be exemplified herein. In one example, the pixel data of the feature point represents the coordinates (Px, Py) of the vertices of the bounding box on the 2D image. The three-dimensional data may include an x-coordinate in a first direction (corresponding to the width direction), a y-coordinate in a second direction (corresponding to the height direction) perpendicular to the first direction, and a z-coordinate in a third direction (corresponding to the depth direction) perpendicular to the first and second directions, but is not limited thereto and may be any three-dimensional data including depth data known to those skilled in the art.

[0040] The controller 20 is connected to the camera 10 and can receive an image of the object 40 from the camera 10. The controller 20 can recognize feature points, etc. from the image of the object 40. More specifically, the controller 20 can recognize, from the image of the object 40, the color of the feature point (e.g., an LED) in the object 40, pixel data of the feature point in the object 40 (pixel data of the vertices of a bounding box corresponding to the LED), and / or 3D data of the feature point in the object 40 (3D data of the vertices of a bounding box corresponding to the LED).

[0041] The controller 20 can determine the main surface of the object 40 based on information about the colors of feature points in the object 40, and can determine the posture and position of the object 40 based on three-dimensional data of the feature points on the main surface. The controller 20 can also control the operation of the robot 30 based on the posture and position of the object 40. The controller 20 is configured to perform each step of the method for estimating the position and posture of the object 40 according to an embodiment of the present invention. The controller 20 can be attached to the robot 30 or provided remotely from the robot 30.

[0042] The robot 30 is connected to the controller 20, and its operation can be controlled by the controller 20. The robot 30 includes at least the controller 20, an energy storage system, a sensor system, and a drive unit 32. The sensor system also includes at least a camera 10.

[0043] The energy storage system is mounted in the robot 30 and can receive and store electrical energy from a charging station or discharge electrical energy to drive the drive unit 32 under the control of the controller 20 .

[0044] The driving unit 32 is attached to the robot 30 and receives power from the energy storage system to move the robot 30. The driving unit 32 may include, but is not limited to, at least one wheel and at least one driving motor connected to the at least one wheel to rotate the at least one wheel. The driving unit 32 may further include a steering device to steer the robot 30.

[0045] Meanwhile, in order to implement a method for estimating the position and orientation of an object 40 according to another embodiment of the present invention using the system for estimating the position and orientation of an object 40 according to an embodiment of the present invention, the object 40 includes at least one surface recognizable by the camera 10. FIG. 2 is a schematic diagram illustrating one surface (e.g., a side surface) of the object according to one example, and FIG. 3 is a schematic diagram illustrating another surface (e.g., a front or rear surface) of the object according to one example. As shown in FIGS. 2 and 3, the object 40 includes multiple surfaces recognizable by the camera 10, and each surface is provided with a mark indicating the surface. In one example, the mark may be a light-emitting diode (LED) 44a, 44b that emits light of one color. For example, a red-emitting LED 44a may be provided on one surface (e.g., a side surface) of the object 40, and a green-emitting LED 44b may be provided on the other surface (e.g., the front or rear surface) of the object 40. The camera 10 can identify the surface of the object 40 that is being photographed by the camera 10 by identifying the color emitted by the light emitting diodes 44a and 44b. For example, if the camera 10 identifies red, it can identify that it is photographing the side of the object 40. If the camera 10 identifies green, it can identify that it is photographing the front or rear of the object 40. Furthermore, if the camera 10 identifies both red and green, it can identify that it is simultaneously photographing the side and front or the side and rear of the object 40. In this case, the controller 20 can determine one of the side, front, or rear of the object 40 as the main surface of the object 40. Here, the main surface refers to the surface with the largest number of light emitting diodes 44a and 44b, which indicates one of the light emitting diodes 44a and 44b identified by the camera 10. If the number of LEDs 44a indicating one surface is equal to the number of LEDs 44b indicating the other surface, any one of the two surfaces can be set as the main surface. The indicators are not limited to LEDs, but may be any indicators that can indicate each surface of the object 40.

[0046] Furthermore, the object 40 may be movable. Accordingly, the object 40 may have a driving unit 42 similar to the driving unit 32 of the robot 30. In one example, the object 40 may be one type of robot 30. However, the object 40 is not limited to a movable object 40, and may be a fixed object fixed to the ground, such as a charging station.

[0047] Hereinafter, a method for estimating the position and posture of an object according to another embodiment of the present invention will be described in detail with reference to FIGS.

[0048] FIG. 4 is a flowchart of an object position and orientation estimation method according to an embodiment of the present invention, FIG. 5 is a specific flowchart of step S150 of FIG. 4, FIG. 6 schematically shows the specifications of one surface of an object according to one example, FIG. 7 schematically shows the center position of an LED located on one surface of an object according to one example, and FIG. 8 schematically shows the center position and normal vector of one surface of an object, and the center position and orientation of the object according to one example.

[0049] 4, the object position and orientation estimation method according to an embodiment of the present invention begins with step S100. In step S100, the camera 10 captures an image of the object 40 within its field of view and transmits the image of the object 40 to the controller 20. Here, the object 40 includes a plurality of surfaces that can be recognized by the camera 10, and each surface is provided with at least one marker indicating the surface, and the feature points of the object 40 may include the at least one marker. In one example, the markers may be light-emitting diodes 44a and 44b provided near the vertices of the surfaces (e.g., the upper left vertex, the lower left vertex, the upper right vertex, and the lower right vertex). The light-emitting diodes 44a and 44b may emit light of a specific color depending on the surface to which they are attached, and the image of the object 40 includes information about the color of the light emitted by the light-emitting diodes 44a and 44b.

[0050] The image of the object 40 also includes information about pixel data of the vertices of the bounding box corresponding to the LEDs 44a and 44b, and / or information about the transformation relationship between the pixel data of the vertices of the bounding box corresponding to the LEDs 44a and 44b and the 3D data of the vertices of the bounding box of the LEDs 44a and 44b. Here, the bounding box may be an N*N regular rectangle corresponding to the LEDs, and therefore one bounding box may have four vertices.

[0051] The controller 20 recognizes the LEDs 44a and 44b from the image of the object 40 (S110) and stores data on the vertices of the bounding boxes corresponding to the recognized LEDs 44a and 44b (S120). Here, "recognizing the LEDs 44a and 44b" means detecting at least the color of light emitted by the LEDs 44a and 44b and pixel data on the vertices of the bounding boxes corresponding to the LEDs 44a and 44b. In addition, the data on the vertices of the bounding boxes corresponding to the LEDs 44a and 44b may be stored in the form of (Px, Py, wi, hi, m). Here, Px and Py indicate pixel data of the vertices of the bounding box, m is a number indicating the face of the object 40 (e.g., m=1 for the front face, m=2 for the left face, m=3 for the rear face, and m=4 for the right face), and wi and hi respectively indicate the width and height between the center of the bounding box corresponding to the i-th LED 44a and the center 46 of the face on which the LED 44a is located, as shown in Fig. 6. wi and hi may be stored in advance in the memory of the controller 20 as specifications of the object 40. Once the data of the vertices of the bounding boxes corresponding to the LEDs 44a and 44b has been saved, the controller 20 determines the principal face using the data of all the recognized vertices (S130). More specifically, the controller 20 determines the m with the largest number from the data of all the recognized vertices, and determines the face indicated by m as the principal face. Also, if the number of one m and the number of another m are the same and largest, the controller 20 can set any one of the faces indicated by one m and the other m as the principal face.

[0052] Once the main surface is determined, the controller 20 generates 3D data of the vertices of a bounding box corresponding to the LEDs 44a located on the main surface (S140). The controller 20 generates 3D data of the vertices of the bounding box corresponding to the LEDs 44a located on the main surface based on the data of the vertices of the bounding box stored in step S120 and the transformation relationship between the pixel data of the vertices of the bounding box included in the image of the object 40 and the 3D data of the vertices.

[0053] Thereafter, the controller 20 calculates the final center position and the final normal vector of the main surface using the 3D data of the vertices of the bounding box corresponding to the LED 44a located on the main surface (S150). Step S150 will be described in more detail with reference to FIG. 5.

[0054] As shown in FIG. 5, step S150 begins with step S200. In step S200, controller 20 determines whether the number (n) of LEDs 44a located on the main surface is greater than 1. If the number (n) of LEDs 44a located on the main surface is greater than 1 in step S200, controller 20 assigns 1 to i (S210) and calculates at least two-dimensional data (xic, yic) and a normal vector (xin, yin, zin) of the center of the i-th LED 44a (S220), as shown in FIG. 7. The two-dimensional data (xic, yic) and normal vector (xin, yin, zin) of the center of the i-th LED 44a can be calculated using a pre-stored algorithm based on the three-dimensional data (xij, yij, zij) of the vertices of a bounding box corresponding to the i-th LED 44a and the size of the bounding box. Here, the subscript ij denotes the jth vertex of the i-th LED 44a. The algorithm for calculating the 3D data (xic, yic, zic) of the center of a bounding box of known size (N*N) and the normal vector (xin, yin, zin) from the 3D data (xij, yij, zij) of the vertices of the bounding box is well known to those skilled in the art, and therefore a detailed description of the algorithm will be omitted.

[0055] Once at least two-dimensional data (xic, yic) and normal vector (xin, yin, zin) of the center of the i-th LED 44a have been calculated, the controller 20 moves the bounding box corresponding to the i-th LED 44a to the left by a set distance (S230) and determines whether the difference between the normal vector before and after the movement exceeds a set value (S240). If the center of the i-th LED 44a is located to the left of the main surface, moving the LED 44a to the left by the set distance will position the LED 44a on a surface other than the main surface. As a result, the difference between the normal vector before and after the movement will be greater than the set value, and the i-th LED 44a is determined to be located on the left side (S245). On the other hand, if the center of the i-th LED 44a is located to the right of the main surface, even if the LED 44a is moved to the left by the set distance, the LED 44a will still be located on the main surface. As a result, the difference between the pre-movement normal vector and the post-movement normal vector is smaller than a set value, and it is determined that the i-th LED 44a is positioned on the right side (S250). The set distance can be set according to the distance between the LED 44a attached to a given surface and the left edge of that surface. There are various methods for calculating the difference between the pre-movement normal vector and the post-movement normal vector, and the set value can be set according to the selected method.

[0056] In this specification, whether the i-th LED 44a is on the left or right side is determined by moving the bounding box corresponding to the i-th LED 44a a set distance to the left. However, whether the i-th LED 44a is on the left or right side can also be determined by moving the bounding box corresponding to the i-th LED 44a a set distance to the right. In this case, if the difference between the pre-movement normal vector and the post-movement normal vector exceeds a set value, the i-th LED 44a is determined to be on the right side. If the difference between the pre-movement normal vector and the post-movement normal vector is equal to or less than the set value, the i-th LED 44a is determined to be on the left side.

[0057] The controller 20 also determines whether the y-value of the center of the i-th LED 44a is greater than half the height of the object 40 minus the mounting height of the camera 10 from the ground (S260). That is, it determines whether the height of the center of the i-th LED 44a relative to the camera 10 is higher than the height of the center of the object 40. If the height of the center of the i-th LED 44a relative to the camera 10 is higher than the height of the center of the object 40 ("Yes" in step S260), the controller 20 determines that the i-th LED 44a is located on the upper side (S265). If the height of the center of the i-th LED 44a relative to the camera 10 is lower than the height of the center of the object 40 ("No" in step S260), the controller 20 determines that the i-th LED 44a is located on the lower side (S270).

[0058] Once the position of the i-th LED 44a has been determined, the controller 20 calculates the position (xec, yec, zec) and normal vector (xnc, ync, znc) of the center 46 of the major surface based on the center data (xic, yic, zic) of the i-th LED 44a and the object specifications (e.g., at least wi, hi) (S280) (see FIG. 8). For example, the two-dimensional position of the center 46 of the major surface is calculated by adding the two-dimensional data (xic, yic) of the center of the i-th LED 44a and the object specifications (wi, hi), and the depth data (zec) of the center 46 of the major surface can be calculated from the depth data at the two-dimensional position (xec, yec) of the center 46 of the major surface. The normal vector (xnc, ync, znc) of the center 46 of the major surface can be calculated from the normal vector at the position (xec, yec, zec) of the center 46 of the major surface.

[0059] Thereafter, the controller 20 determines whether i is equal to the number (n) of LEDs 44a located on the major surface (S290). If i is not equal to the number (n) of LEDs 44a located on the major surface at step S290, the controller 20 increments i by 1 (S300) and returns to step S220, repeating steps S220 through S280. If i is equal to the number (n) of LEDs 44a located on the major surface at step S290, the controller 20 calculates the position (xec, yec, zec) and final normal vectors (xnc, ync, znc) of the main surface center 46 using the position (xec, yec, zec) and normal vectors (xnc, ync, znc) of the main surface center 46 calculated based on all of the LEDs 44a located on the major surface (S310). In step S310, the position (xec, yec, zec) and normal vector (xnc, ync, znc) of the center 46 of the main surface calculated based on all the LEDs 44a located on the main surface are corrected through at least one filter (e.g., Kalman filter, extended Kalman filter, etc.) to calculate the final position (xec, yec, zec) and final normal vector (xnc, ync, znc) of the center 46 of the main surface.

[0060] On the other hand, if the number (n) of LEDs 44a located on the main surface is 1 in step S200, the controller 20 proceeds to step S310 to calculate the center position and normal vector of the LED 44a using the position (xec, yec, zec) and final normal vector (xnc, ync, znc) of the final center 46 of the main surface. Thereafter, the controller 20 proceeds to step S160.

[0061] 4 again, in step S160, the controller 20 calculates the orientation (xmc, ymc, zmc) of the object 40 and the position (xec, yec, zec) of the center 48 of the object 40 using the position (xec, yec, zec) and final normal vector (xnc, ync, znc) of the final center 46 of the main surface, and the specifications of the object 40 (e.g., the width, height, and dimensions between the center of the main surface and the center of the object 40). The algorithm for calculating the orientation (xmc, ymc, zmc) of the object 40 and the position (xec, yec, zec) of the center 48 of the object 40 using the position (xec, yec, zec) and final normal vector (xnc, ync, znc) of the final center 46 of the main surface, and the specifications of the object 40 is well known to those skilled in the art, and therefore, a description thereof will be omitted.

[0062] Thereafter, the controller 20 controls the posture or movement of the robot 30 based on the posture (xmc, ymc, zmc) of the object 40 and the position (xec, yec, zec) of the center 48 of the object 40 (S170). In one example, the controller 20 can control the driving unit 32 of the robot 30 so that the robot 30 follows the position (xec, yec, zec) of the center 48 of the object 40. In another example, the controller 20 can control the posture and movement of the robot 30 according to the posture (xmc, ymc, zmc) of the object 40 and the position (xec, yec, zec) of the center 48 of the object 40, thereby binding the robot 30 to the object 40.

[0063] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above embodiments and includes all modifications that can be easily made by a person having ordinary skill in the art to which the invention pertains and that are deemed equivalent to the embodiments of the present invention. [Explanation of symbols]

[0064] 10 Camera 20 Controller 30 Robot 32 Drive unit 40 Object 42 Drive unit 44a, 44b Light-emitting diode

Claims

1. In the method for estimating the position and orientation of an object, The object includes at least one surface, and at least one mark indicating the surface is provided on each surface near a vertex of the surface, and a width and height between the at least one mark and a center of the surface, and a width, height, and depth between the center of the surface and a center of the object are stored in advance as object specifications; The posture estimation method includes: capturing an image of the object using a camera provided in the robot; recognizing the mark from the image of the object by a controller; determining, by the controller, a major surface using the recognized landmarks; determining, by the controller, the location of the indicia located on the major surface; calculating, by the controller, a final center position and a final normal vector of the principal surface using the determined marker position and the object specification; and calculating, by the controller, the position and orientation of the center of the object using the final center position and the final normal vector of the principal surface and specifications of the object.

2. The method of claim 1 , wherein the marker is a light-emitting diode that emits light of a color determined by a surface on which the marker is provided.

3. 3. The object position and orientation estimation method of claim 2, wherein the image of the object captured by the camera includes information about the color of the light emitting diode in the image, information about pixel data of the light emitting diode in the image, and / or information about a transformation relationship between the pixel data of the light emitting diode and three-dimensional data, and the three-dimensional data includes depth data.

4. 4. The method of claim 3, wherein, in the step of recognizing the sign from the image of the object, at least the color of light emitted by the light emitting diode and pixel data of vertices of a bounding box of N*N size corresponding to the light emitting diode are detected.

5. 5. The method of claim 4, wherein determining the main surface using the recognized landmarks comprises determining a surface on which the most vertices of all detected vertices are located as the main surface.

6. The step of determining the location of the markings located on the major surface comprises: generating three-dimensional data of vertices of a bounding box corresponding to the light emitting diode located on the main surface; calculating three-dimensional data of vertices of the bounding box corresponding to the i-th LED and two-dimensional data of the center and normal vector of at least the i-th LED based on the size of the bounding box; determining whether the i-th LED is located left or right in the width direction while moving the i-th LED left or right in the width direction; and 5. The method of claim 4, further comprising determining whether the i-th LED is located above or below the center of the object in a height direction by determining whether the height of the center of the i-th LED is above or below the center of the object with respect to the camera.

7. The step of determining whether the i-th LED is positioned left or right in the width direction while moving the i-th LED left or right in the width direction includes: Calculating a difference between a normal vector before and a normal vector after the movement while moving the i-th LED to the left or right by a set distance in a width direction; determining that the i-th LED is located in the moving direction if a difference between the pre-movement normal vector and the post-movement normal vector is greater than a set value; and 7. The method of claim 6, further comprising determining that the i-th LED is located on the opposite side of the movement direction if a difference between the pre-movement normal vector and the post-movement normal vector is equal to or less than a set value.

8. calculating the final center position and the final normal vector of the principal surface using the determined marker position and the object specification, calculating the center position and the normal vector of the main surface for all light emitting diodes located on the main surface based on the center data of the light emitting diodes and the specifications of the object; and 7. The method of claim 6, further comprising: calculating the center position and the normal vector of the main surface using the center positions and the normal vectors of the main surface calculated for all light emitting diodes located on the main surface.

9. calculating the center position and normal vector of the main surface using the center positions and normal vectors of the main surface calculated for all light emitting diodes located on the main surface, 9. The method of claim 8, further comprising filtering the center positions and normal vectors of the main surface calculated for all light emitting diodes positioned on the main surface.

10. The method of claim 1 , further comprising controlling the robot based on the position and orientation of the center of the object by the controller.

11. In an object position and orientation estimation system, The object includes at least one surface, and at least one mark indicating the surface is provided on each surface near a vertex of the surface, and a width and height between the at least one mark and a center of the surface, and a width, height, and depth between the center of the surface and a center of the object are stored in advance as object specifications; The pose estimation system includes: a robot including a driving unit that generates a driving force and configured to follow or combine with the object; a camera mounted on the robot and configured to capture images of the object within its field of view; and a controller connected to the camera to receive an image of the object from the camera and configured to control the robot; The controller is Recognizing the sign from the image of the object; determining a major surface using the recognized landmarks; determining the location of the indicia on the major surface; Calculating the final center position and the final normal vector of the main surface using the determined marker position and the object specification; The object position and orientation estimation system is configured to calculate the position and orientation of the center of the object using the final center position and final normal vector of the main surface and specifications of the object.

12. The object position and orientation estimation system of claim 11, wherein the marker is a light-emitting diode that emits light of a color determined by a surface on which the marker is provided.

13. 13. The object position and orientation estimation system of claim 12, wherein the image of the object captured by the camera includes information about the color of the light-emitting diode in the image, information about pixel data of the light-emitting diode in the image, and / or information about a transformation relationship between the pixel data of the light-emitting diode and three-dimensional data, and the three-dimensional data includes depth data.

14. 14. The object position and orientation estimation system of claim 13, wherein when recognizing the marker from the image of the object, the controller detects at least the color of light emitted by the light emitting diode and pixel data of vertices of a bounding box of N*N size corresponding to the light emitting diode.

15. 15. The object position and orientation estimation system of claim 14, wherein when determining the main surface using the recognized landmarks, the controller determines the surface on which the most vertices of all detected vertices are located as the main surface.

16. When determining the position of a mark located on the major surface, the controller: generating the three-dimensional data of the vertices of the bounding box corresponding to the light emitting diodes located on the main surface; calculating two-dimensional data and a normal vector of at least the center of the i-th LED based on three-dimensional data of vertices of the bounding box corresponding to the i-th LED and the size of the bounding box; determining whether the i-th LED is located left or right in the width direction while moving the i-th LED left or right in the width direction; 15. The object position and orientation estimation system of claim 14, wherein the system is configured to determine whether the i-th LED is located above or below the center of the object in a height direction by determining whether the height of the center of the i-th LED is above or below the center of the object with respect to the camera.

17. When determining whether the i-th LED is located left or right in the width direction while moving the i-th LED left or right in the width direction, the controller The i-th LED is moved left or right in the width direction by a set distance, and the difference between the normal vector before the movement and the normal vector after the movement is calculated; If the difference between the pre-movement normal vector and the post-movement normal vector is greater than a set value, it is determined that the i-th LED is located in the moving direction; 17. The object position and orientation estimation system of claim 16, further comprising: determining that the i-th LED is located on the opposite side of the movement direction if a difference between the pre-movement normal vector and the post-movement normal vector is equal to or less than a set value.

18. When calculating the final center position and the final normal vector of the main surface using the determined marker position and object specifications, the controller For all light emitting diodes located on the main surface, calculate the center position and normal vector of the main surface based on the center data of the light emitting diode and the specifications of the object; 17. The object position and orientation estimation system of claim 16, configured to calculate the center position and normal vector of the main surface using the center positions and normal vectors of the main surface calculated for all light emitting diodes located on the main surface.

19. When calculating the center position and the normal vector of the main surface using the center positions and the normal vectors of the main surface calculated for all light emitting diodes located on the main surface, The controller 20. The object position and orientation estimation system of claim 18, further comprising: filtering the positions of the centers of the main surfaces and the normal vectors calculated for all light-emitting diodes located on the main surfaces.

20. The system of claim 11 , wherein the controller is further configured to control the robot based on the position and orientation of the center of the object.

Images