Robot posture measurement method and robot system in which multiple robots interact

Abstract

The present invention relates to a robot posture measuring method and a robot system using the robot posture measuring method. The robot posture measuring method according to the present invention includes the steps of: moving at least one of the first robot and the second robot so that the first robot and the second robot are adjacent to each other; outputting a visual marker on a display of the second robot; photographing the second robot by the first robot; detecting the visual marker included in the image photographed by the first robot and calculating the posture of the visual marker included in the image; and calculating the posture of the second robot using coordinate data of the second robot and the posture of the visual marker included in the image.

Description

Technical Field

[0001] The present invention relates to a method for measuring the posture of a robot and a robot system using the method for measuring the posture of a robot, and more particularly to a method for measuring the posture of a partner robot for interaction between robots and a robot system for performing interaction.

Background Art

[0002] With the development of technology, various service devices have emerged. In particular, recently, technological development for robots that perform various tasks or services has been actively carried out.

[0003] Furthermore, recently, with the development of artificial intelligence technology, cloud technology, etc., it has become possible to control robots more precisely and safely, and thereby the utilization rate of robots has been gradually increasing. In particular, due to technological development, robots have reached a level where they can coexist safely with humans in indoor spaces.

[0004] Therefore, recently, robots have been replacing human tasks or operations, and in particular, in indoor spaces, various methods in which robots directly provide services to humans have been actively studied.

[0005] For example, in public places such as airports, train stations, and department stores, robots provide navigation services, and in restaurants, robots provide food delivery services. Furthermore, in office spaces, shared living spaces, etc., robots provide delivery services such as delivering mail and home deliveries. In addition to this, robots also provide various services such as cleaning services, security services, and logistics processing services. It is expected that the types and scope of services provided by robots will increase dramatically in the future, and the service provision level is also expected to continue to develop.

[0006] Such robots provide a variety of services not only in outdoor spaces but also in indoor spaces of buildings such as offices, apartments, department stores, schools, hospitals, and amusement facilities. In this case, the robots are controlled to move around the indoor spaces of the building and provide a variety of services.

[0007] On the other hand, for robots to provide diverse services or live in indoor spaces, they need to cooperate with each other to perform tasks. For example, Korean Published Patent No. 10-2010-0086093 (Autonomous Driving Cluster Robot Position Control System) discloses a system in which a master robot forms a cluster with multiple subordinate robots and drives autonomously. Thus, gradually developing from driving technology, there has been a surge in efforts to realize diverse services by controlling the interaction between robots.

[0008] Therefore, essential research is needed to enable robots to accurately and quickly measure their position and orientation with each other, in order to provide higher-level services using robots through robot interaction. [Overview of the Initiative] [Problems that the invention aims to solve]

[0009] This invention provides a method and system for rapidly and accurately measuring the posture of a robot.

[0010] More specifically, the present invention provides a method for measuring the posture of a robot using a display and visual markers attached to the robot. [Means for solving the problem]

[0011] To solve the above problems, the robot posture measurement method and robot system using the present invention employ a process in which a visual marker is output to the display of the target robot, and the robot photographs it to recognize the posture of the target robot.

[0012] Specifically, the robot posture measurement method of the present invention includes the steps of: moving at least one of the first robot and the second robot so that they are adjacent to each other; outputting a visual marker to the display of the second robot; the first robot photographing the second robot; detecting a visual marker included in the image taken by the first robot and calculating the pose of the visual marker included in the image; and calculating the posture of the second robot using the coordinate data of the second robot and the pose of the visual marker included in the image.

[0013] In one embodiment of the present invention, the robot posture measurement method may further include the step of monitoring a request signal in which a second robot requests the output of a visual marker.

[0014] The request signal may be transmitted from a server communicating with the first robot, or from the first robot to the second robot.

[0015] The step of outputting a visual marker may involve, upon receiving a request signal while image information is being output to the display, either turning off the output of the image information and outputting the visual marker, or outputting the visual marker together with the image information.

[0016] The step of outputting a visual marker may be modified if a request signal is received while the display is not activated, by activating the display and outputting the visual marker.

[0017] On the other hand, the present invention discloses a robot position measurement method that includes the steps of: the robot moving in a specific space where a display is arranged; image information being displayed on the display; a server communicating with the robot or the robot transmitting a request signal to a control system that controls the display requesting the output of a visual marker; controlling the display in response to the request signal so that the control system outputs a visual marker; and the robot estimating its current position using the visual marker output on the display.

[0018] Furthermore, the present invention discloses a robot system in which multiple robots interact, comprising: a first robot equipped with a camera; a second robot that interacts with the first robot and is equipped with a display; the system outputs a visual marker to the display of the second robot; the first robot photographs the second robot; the first robot detects the visual marker contained in the image it has taken; the second robot calculates the pose of the visual marker contained in the image; the second robot estimates its pose using its coordinate data and the pose of the visual marker contained in the image; and the first robot controls its movement using the estimated pose of the second robot. [Effects of the Invention]

[0019] The robot posture measurement method and robot system using the robot posture measurement method according to the present invention detect a marker output to a display, measure the marker posture, compare it with the robot's shape data, convert it into the robot's posture, and recognize the posture of the target robot, thereby enabling quick and accurate measurement of the target robot's posture in a simple manner.

[0020] In this way, by rapidly measuring the precise robot posture through a new process, it becomes possible to provide human-like service in a variety of robotic services, such as delivery, serving food, and customer service.

[0021] In addition, in order to accurately measure the posture of a robot using a visual marker, the present invention enables the measurement of the posture of a partner robot by utilizing a display for interaction with humans. In this case, a visual marker is output on a display fixed in a specific space such as signage, and the position of the robot that has photographed the visual marker can be accurately and quickly measured.

Brief Description of the Drawings

[0022] [Figure 1] It is a conceptual diagram for explaining the interaction of the robot according to the present invention. [Figure 2a] It is a diagram showing an example of the robot presented in the concept of FIG. 1. [Figure 2b] It is a diagram showing an example of the robot presented in the concept of FIG. 1. [Figure 3] It is a flowchart for explaining the method of measuring the posture of the robot of the present invention. [Figure 4] It is a conceptual diagram showing the concept of the robot in FIG. 2a photographing a visual marker to measure the posture of the robot in FIG. 2b. [Figure 5] It is a conceptual diagram for detecting the pose of a visual marker. [Figure 6] It is a conceptual diagram showing an example of conversion to the posture of a robot. [Figure 7a] It is a conceptual diagram showing another embodiment of the present invention. [Figure 7b] It is a conceptual diagram showing another embodiment of the present invention. [Figure 8] It is image data showing actual data realized using the process of the present invention. [Figure 9] It is a conceptual diagram for measuring the position of a robot in another embodiment of the present invention.

Modes for Carrying Out the Invention

[0023] The embodiments disclosed herein will be described in detail below with reference to the accompanying drawings, but identical or similar components will be given the same reference numeral regardless of the reference numerals in the drawings, and redundant descriptions thereof will be omitted. The suffixes “module” and “part” used for components in the following description are added or mixed together solely for the purpose of facilitating the preparation of the specification and do not have any distinguishing meaning or role in themselves. Furthermore, in describing the embodiments disclosed herein, if it is determined that a specific description of the relevant prior art may obscure the gist of the embodiments disclosed herein, such detailed description will be omitted. In addition, the accompanying drawings are provided to facilitate the understanding of the embodiments disclosed herein and should be understood not to limit the technical ideas disclosed herein, but to include all modifications, equivalents and substitutions that fall within the concept and technical scope of the present invention.

[0024] Terms including ordinal numbers, such as "first," "second," etc., can be used to describe a variety of components, but the components are not limited by these terms. These terms are used solely for the purpose of distinguishing one component from another.

[0025] When one component is said to be “connected” or “linked” to another component, it should be understood that it may be directly connected or linked to the other component, but there may also be other components between them. On the other hand, when one component is said to be “directly connected” or “directly linked” to another component, it should be understood that there are no other components between them.

[0026] Unless otherwise indicated by the context, singular expressions include plural forms.

[0027] In this application, terms such as “includes” or “having” should be understood to indicate the presence of features, figures, steps, actions, components, parts, or combinations thereof as described in the specification, and not to preemptively exclude the possibility of the presence or addition of one or more other features, figures, steps, actions, components, parts, or combinations thereof.

[0028] The present invention relates to a method for measuring the posture of a robot and a robot system using the same, and more specifically, to a method for measuring the posture of robots for interaction between robots and a robot system for performing interaction.

[0029] Here, interaction can be defined as a type of communication that can occur between humans, between humans and matter, between humans and systems, and between systems. Specifically, it is a compound word of "inter" (mutual) and "action" (action, operation), meaning an action in which two or more objects influence each other, and can refer to all actions characterized primarily by the bidirectional nature and real-time nature of the influence.

[0030] Furthermore, in this specification, a robot means i) a machine having functions similar to those of a human, or ii) a mechanical device that can be operated by one or more computer programs and automatically performs a complex series of tasks. As another example, a robot may mean a machine that has form and the ability to think for itself. Moreover, the robots described herein can be realized in forms such as android robots that resemble humans in appearance and behavior, humanoid robots that have a body structure similar to that of a human, and cyborgs whose bodies have been modified with machinery.

[0031] As technology advances, the use of robots is gradually increasing. Traditionally, robots have been used in specialized industrial fields (for example, in the field of industrial automation), but they are gradually transforming into service robots that can perform useful tasks for humans and equipment.

[0032] In this case, the services provided by the robot can be implemented within the building. As an example of this, the robot according to the present invention may be controlled based on at least one of a cloud server and a control unit provided in the robot itself, and may be configured to travel within the building or to provide services corresponding to assigned tasks.

[0033] The building in this invention is a building where robots and humans coexist, and the robot is configured to navigate while avoiding obstacles such as people, items used by people (e.g., strollers and carts), and animals, and may optionally be configured to output notification information related to the robot's movement. The movement of such a robot may be performed to avoid obstacles based on at least one of a cloud server and a control unit provided in the robot. The cloud server can control the robot to move around the building while avoiding obstacles based on information received through various sensors provided in the robot (e.g., cameras (image sensors), proximity sensors, infrared sensors, etc.). However, this invention is not necessarily limited thereto, and the services and processes described in this invention can also be applied outdoors.

[0034] On the other hand, the robot of the present invention may be configured to provide services to people or target objects present within a building.

[0035] The types of services a robot provides can vary from robot to robot. In other words, there are many different types of robots depending on their intended use, each with a different structure, and each robot is equipped with a program suited to that use.

[0036] For example, a robot may be deployed to provide at least one of the following services: delivery to buildings, logistics operations, guidance, interpretation, parking assistance, security, crime prevention, guarding, public safety, cleaning, disease prevention, disinfection, washing, beverage production, food production, serving, fire suppression, medical assistance, and entertainment services. The services provided by the robot may be diverse beyond the examples given above.

[0037] On the other hand, the cloud server can take into account the intended use of each robot, assign appropriate tasks to the robots, and control the robots to ensure that the assigned tasks are performed.

[0038] At least some of the robots described in this invention are cloud servers Ba The robot can move or perform tasks under control, minimizing the amount of data processed by the robot itself for movement or task performance. In this invention, such a robot may be referred to as a brainless robot. Such a brainless robot may rely on a cloud server for control of at least some of its activities, such as moving around a building, performing tasks, charging, waiting, and cleaning.

[0039] However, in this specification, we will not distinguish and name unintelligible robots separately, but will refer to them all collectively as "robots."

[0040] Furthermore, in the present invention, a building may be provided with a variety of infrastructure (or equipment infrastructure) to perform the services provided by the robot.

[0041] Infrastructure or facility infrastructure refers to facilities provided in a building for purposes such as providing services, moving robots, maintaining functionality, and maintaining cleanliness, and its types and forms are extremely diverse. For example, infrastructure provided in a building can include a variety of things such as mobility equipment (e.g., robot pathways, elevators, escalators, etc.), charging equipment, communication equipment, cleaning equipment, and structures (e.g., stairs, etc.). Thus, the aforementioned infrastructure or facility infrastructure is defined as the infrastructure used by robots to perform their assigned tasks.

[0042] In this invention, robots are configured to cooperate with each other to perform assigned tasks. This cooperation is achieved through interaction between robots, and the invention provides a method for measuring the posture of the robots for this purpose.

[0043] As an example of this, as shown in Figure 1, robots can cooperate with each other to deliver goods (e.g., mail, packages, etc.) to users or to perform services such as serving meals.

[0044] A robot providing such a service may be configured such that the first robot 100 transmits items to the second robot 200 in order to perform the service. For example, this could involve transmitting items to another robot for delivery, picking them up from another robot, or, as another example, transmitting a tray to another robot after serving food.

[0045] More specifically, the first robot 100 is configured to pick up items 10, such as delivery goods or trays, at a fixed location, while the second robot 200 is configured to primarily move around to transport the items 10. Thus, the first robot 100 may be assigned a first task, and the second robot 200 may be assigned a second task. For example, the task assigned to the second robot 200 may be to receive goods from a user and transport them to the first robot 100 in order to provide a corresponding service (goods delivery). The task assigned to the first robot 100 may be to pick up the items 10 transported by the first robot 100 and load them into a loading box.

[0046] In this case, the second robot 200 travels through the building and moves to a specific location 11 adjacent to the first robot 100, while still holding the item 10 transmitted by the user. The first robot 100 recognizes when the second robot 200 approaches the specific location 11, picks up the item 10 transported by the first robot 100, and loads it into a loading box (not shown). In this way, the first robot 100 and the second robot 200 cooperate with each other to complete a task in a specific section for the delivery of the item 10, and can be called a single robotic system.

[0047] On the other hand, in order for the first robot 100 to accurately pick up the item 10 transferred by the second robot 200, the first robot 100 must accurately detect the orientation of the second robot 200. In particular, referring to Figures 1, 2a, and 2b together, the first robot 100 and the second robot 200 may have different shapes from each other. That is, the shapes of the first robot 100 and the second robot 200 are different from each other to suit their respective tasks, and in such cases, it is necessary to measure the orientation of the other robot more easily and accurately in order to perform the task.

[0048] The robot presented in the concept of Figure 1 will be explained in more detail below with reference to Figures 2a and 2b.

[0049] Figures 2a and 2b show examples of robots presented in the concept of Figure 1, respectively.

[0050] Referring to Figure 2a, for example, the first robot 100 may include at least one of the body section 110, the sensing section, and the communication section.

[0051] The body portion may be equipped with an end effector 111, a manipulator 112, an actuator (not shown), etc., for picking up the item 10.

[0052] The end effector 111 is an end device of a robot arm designed to interact with the environment, and may be a gripper, pincer, scalpel, etc. Such an end effector 111 may be provided separately from the robot and used as a peripheral device or accessory for the robot.

[0053] The manipulator 112 may be a device that integrates links, joints, and other structural elements and mechanisms with the main body and arms of the robotic device. In the case of a multi-joint robot, the manipulator 112 may have multiple joints that perform the rotational motion of motors.

[0054] In this case, the manipulator 112 and the end effector 111 can form an arm with 6 or more degrees of freedom and a hand with 1 or more degrees of freedom. Furthermore, in order to realize the robot of the present invention as a two-armed robot or a robot with multiple arms, the arms and hands may exist in multiple pairs.

[0055] Furthermore, the actuator is a device that converts electrical, chemical, or thermal energy into rotational or linear motion, and may be a motor, pneumatic cylinder, artificial muscle, or the like.

[0056] In this manner, the first robot 100 controls the end effector 111, the manipulator 112, and the actuators to pick up and move the item 10.

[0057] The sensing unit may include one or more sensors for sensing at least one of the following: information within the robot (particularly the robot's operating state), information about the surrounding environment of the robot, the robot's position information, and user information.

[0058] For example, the sensing unit may include a camera (image sensor) 121, a proximity sensor, a biosensor, an infrared sensor, a laser scanner (LiDAR sensor), an RGBD sensor, a geomagnetic sensor, an ultrasonic sensor, an inertial sensor, a UWB sensor, a microphone, and the like.

[0059] The camera 121 may be positioned at the upper end of the robot's body and configured to primarily capture images of the surrounding environment facing downwards. In this case, the camera 121 may be configured to be tiltable, allowing the shooting angle to be adjusted.

[0060] The robot's communication unit is configured to send and receive wireless signals for wireless communication between the robot and the building's communication equipment, between the robot and other robots, or between the robot and a cloud server. For example, the communication unit may include a wireless internet module, a short-range communication module, a location information module, and the like.

[0061] Referring to Figure 2b, for example, the second robot 200 may include at least one of the following: a body 210, a drive unit 220, a sensing unit 230, a communication unit, and a display 240.

[0062] The body portion 210 includes an external case (casing, housing, cover, etc.). In this embodiment, the case can be divided into multiple parts, and various electronic components are incorporated into the space formed by the case. In this case, the body portion 210 may be configured in different forms depending on the various services exemplified in the present invention. For example, in the case of a robot that provides delivery services, a storage box for storing goods 10 may be provided on the upper part of the body portion 210.

[0063] The drive unit 220 is configured to perform specific operations based on control commands transmitted from the cloud server or the control unit of the second robot 200.

[0064] The drive unit 220 provides a means for the robot's body to move within a specific space in connection with its movement. More specifically, the drive unit 220 includes a motor and a plurality of wheels, and by combining these, the robot To It performs the functions of driving, changing direction, and rotating. However, the present invention is not necessarily limited thereto, and the drive unit 220 may include at least one of an end effector, a manipulator, and an actuator to perform other operations other than driving, such as picking up.

[0065] The sensing unit 230, like the first robot 100, may include one or more sensors for sensing at least one of the following: information within the robot (in particular, the robot's operating state), information about the surrounding environment of the robot, the robot's position information, and user information.

[0066] The second robot 200 can perform visual localization by moving through a specific space, acquiring images with the camera, and then comparing them with three-dimensional map data.

[0067] Furthermore, the communication unit, like the first robot 100, is configured to allow the robot to send and receive wireless signals in order to communicate wirelessly between the robot and the building's communication equipment, between the robot and other robots, or between the robot and a cloud server. As an example, the communication unit may include a wireless internet module, a short-range communication module, a location information module, and the like.

[0068] The display 240 is positioned facing upward on the front or top surface of the second robot 200, and may output images and graphic objects. In this invention, there are no restrictions on the type of display 240. The display 240 may operate as a monitor and be configured as a touchscreen. In this case, the display 240 can perform both the role of outputting information and the role of receiving information.

[0069] On the other hand, as explained with reference to Figure 1, the first robot 100 and the second robot 200 each have a control unit or can be controlled by a cloud server. For example, the cloud server or the control unit of the second robot 200 can control the display 240 to output image information. The image information may include not only information provided to the user but also information provided to the other robot, for example, the first robot 100.

[0070] In this invention, interaction between the first robot 100 and the second robot 200 is realized by outputting a visual marker to the display 240 of the second robot 200, and the first robot 100 photographing it to measure the posture of the other robot. (See Figures 3 to 3 below.) 8 Refer to the documentation for a more detailed explanation of the process of measuring the robot's posture in order to achieve such interactions.

[0071] Figure 3 is a flowchart illustrating the robot posture measurement method of the present invention.

[0072] In the present invention, a new method for measuring the posture of a robot is presented in which the first robot 100 photographs a visual marker output from the second robot 200, and the first robot 100 recognizes the posture of the second robot 200. In this case, the posture of the robot may include coordinates indicating position and directions indicating orientation.

[0073] In this invention, a method for measuring the posture of an opposing robot is described using a pair of robots (first robot 100 and second robot 200) as an example, but the invention is not limited thereto. For example, one robot may photograph the visual markers of multiple robots, or multiple robots may photograph the visual markers of one robot.

[0074] Referring to Figure 3, in the robot posture measurement method of the present invention, a step (S111) may be performed first in which at least one of the first robot 100 and the second robot 200 moves so that the first robot 100 and the second robot 200 are adjacent to each other.

[0075] For example, the first robot 100 may be fixed in a specific position, and the second robot 200 may move to the vicinity of the first robot 100 by transporting an object or the like, or the first robot 100 and the second robot 200 may move to a specific location so that they are adjacent to each other at that specific location.

[0076] In this case, the first robot 100 may be equipped with a camera 121, and the second robot 200 may be equipped with a display 240. However, the first robot 100 and the second robot 200 exemplified in this invention are named for convenience of explanation, and the first robot 100 may be equipped with a display, and the second robot 200 may be equipped with a camera. Furthermore, it is also possible for both the first robot 100 and the second robot 200 to be equipped with a camera and a display.

[0077] For example, the second robot 200 can output various image information to people in its vicinity using the display 240. This image information may include graphic objects for interaction with people or video information for advertisements, etc.

[0078] Next, a step (S112) may be performed in which the second robot 200 is requested to output a visual marker.

[0079] The first robot 100, or a server communicating with the first robot 100, such as a cloud server, can request the second robot 200 to output a visual marker. Specifically, the first robot 100 may transmit a visual marker request signal to the second robot 200 in order to perform a task, or the server may transmit the request signal to the second robot 200.

[0080] As another example, for cooperation between the first robot 100 and the second robot 200, if the cloud server determines that the posture data of the second robot 200 is necessary, it can transmit a request signal to the second robot 200 requesting the output of a visual marker. For example, the cloud server may sense that the first robot 100 is approaching the second robot 200 and transmit a request signal to the second robot 200 requesting the output of a visual marker, and the second robot 200 may receive the request signal and output the visual marker.

[0081] In this case, the robot posture measurement method of the present invention may further include the step of monitoring a request signal that the second robot 200 requests the output of the visual marker.

[0082] When the request signal is transmitted in the request step (S112), the second robot 200 can periodically monitor the request signal in order to receive the transmitted request signal and recognize the request to output the visual marker.

[0083] The monitoring of the request signal may be performed when the second robot 200 moves to a specific location. For example, when the second robot 200 moves to the specific location to deliver an item to the first robot 100, the control unit of the second robot 200 can determine this and start monitoring the request signal.

[0084] Next, the step of outputting a visual marker to the display 240 of the second robot 200 (S113) may be performed.

[0085] When the second robot 200 receives the request signal, it displays the visual marker using the display 240.

[0086] The aforementioned visual marker may be, for example, a Visual Fiducial Marker. The Visual Fiducial Marker is a mark generated in an image for use as a reference or measurement point, and may be a publicly available marker that anyone can freely use, such as Apriltag or Aruco. As another example, the visual marker may be an Artificial Marker, which may be a variety of markers designed directly by the user.

[0087] In this case, the step of outputting the visual marker (S113) can be performed by activating the display 240 and outputting the visual marker if the request signal is received while the display 240 is not activated. As a specific example, the display 240 may be activated in response to the request signal while not activated, and the visual marker may be output.

[0088] As another example, in the step of outputting the visual marker (S113), if the request signal is received while image information is being output to the display 240, the output of the image information can be turned off and the visual marker can be output, or the visual marker can be output together with the image information. That is, it is also possible to output an image or the like while the display 240 is activated, and then stop the output of the image in response to the request signal and output the visual marker.

[0089] On the other hand, it is also possible to terminate the output of the visual marker to the display 240 of the second robot 200 after a certain amount of time has elapsed since the visual marker was output. In this case, the certain amount of time may be set to be longer than the time it takes for the first robot 100 to perform control to photograph the second robot 200.

[0090] Next, the first robot 100 takes a photograph of the second robot 200 (S114).

[0091] The first robot 100 uses the camera 121 to photograph the second robot 200.

[0092] For example, the second robot 200 may transmit a feedback signal to the first robot 100 indicating that it has outputted the visual marker, and the first robot 100 may photograph the second robot 200 in response to the feedback signal. Alternatively, the feedback signal may be transmitted from the second robot 200 to a cloud server, and the cloud server may generate a control command to photograph the second robot 200 and transmit it to the first robot 100.

[0093] In this case, the camera 121 may be a sensor attached to the first robot 100, but the present invention is not necessarily limited thereto. As an example, cameras may be placed in the space where the robot is located. There is no limit to the number of cameras placed in the space, and multiple cameras may be placed in the space. There are various types of cameras placed in the space, for example, the cameras placed in the space may be CCTV (closed circuit television).

[0094] Furthermore, the building's communication equipment can communicate directly with cameras placed in the space. The building's communication equipment may also be configured to communicate with a video control system that controls the cameras. In this case, the camera can be controlled by the video control system to detect when the second robot 200 arrives at the specific location and to photograph the first robot 100. In this case, the video control system may be provided as a separate control system or may be configured as part of the cloud server described above.

[0095] Referring to Figure 3, the first robot 100 detects the visual markers contained in the image and calculates the pose of the visual markers contained in the image (S115).

[0096] Here, the visual marker may include at least one Apriltag, as shown in the figure. The Apriltag may consist of two-dimensional information as a visual reference marker. Multiple Apriltags may be combined to constitute the visual marker.

[0097] Such markers are detected from images captured by the first robot 100, and the pose of the visual markers is calculated using the detected visual markers. The detection of the pose of the visual markers may be performed using the PnP (Perspective-n-Point) method.

[0098] The aforementioned PnP method is a technique that estimates the camera pose (e.g., camera position, angle, and direction) using the perspective-n-point algorithm. Using this, the orientation of the visual marker, which is the input image, is calculated relative to the camera coordinate system.

[0099] Next, the pose of the second robot 200 is calculated using the coordinate data of the second robot 200 and the pose of the visual marker included in the image (S 116 ) may be performed.

[0100] For example, the orientation of the visual marker relative to the camera coordinate system can be converted to the orientation of the second robot 200 using the shape data of the second robot 200, such as CAD model data.

[0101] As another example, it is also possible to convert the marker orientation relative to the camera coordinate system to the robot orientation using a calibration method. In this case, the calibration method may be a calibration method between the camera and wheel odometry.

[0102] As described above, in the present invention, when the opposing robot outputs a visual marker, it is possible to photograph it to measure the opposing robot's posture and use that posture to interact with the opposing robot. For example, the first robot 100 and the second robot 200 can either transmit an item to the other, or both robots can interact with each other, such as shaking hands.

[0103] The following describes in more detail each step of the robot posture measurement method of the present invention, with examples.

[0104] Figure 4 is a conceptual diagram showing the concept of the robot in Figure 2a taking a visual marker to measure the posture of the robot in Figure 2b; Figure 5 is a conceptual diagram showing the detection of the pose of the visual marker; Figure 6 is a conceptual diagram showing an example of conversion to the robot's posture; Figures 7a and 7b are conceptual diagrams showing other embodiments of the present invention; and Figure 8 is image data showing actual data realized using the process of the present invention.

[0105] First, referring to Figure 4, the second robot 200 outputs a visual marker 250, and the first robot 100 photographs the outputted visual marker 250.

[0106] To facilitate and accurately capture the visual marker 250, the display 240 of the second robot 200 that outputs the visual marker 250 may be positioned facing upward on the front or top surface of the second robot 200, and the first robot 100 may be equipped with a camera 121 that photographs the second robot 200 facing downward.

[0107] To achieve such relative positions, the display 240 of the second robot 200 may have a view area 241 in the form of a surface inclined with respect to the vertical. On the other hand, the camera 121 of the first robot 100 may be positioned at the upper end of the body of the first robot 100 and positioned downward so as to have a field of view toward the view area 241. In this case, the field of view may be the angle of view (FoV) of the camera 121.

[0108] Furthermore, in order for the first robot 100 to photograph the second robot 200 more accurately, the positions and orientations of the first robot 100 and the second robot 200 can be controlled so that they face each other.

[0109] At this time, the visual marker 250 output to the display 240 is in the center of the display 240. prescribed It can be output in the following size. In this case, the pixel pitch (pixel pitch, mm) of the display 240 and the size of the marker image (pixels) are used to create the visual marker 250. prescribed It can be output in the following size; for example, the width w of the visual marker 250 may be 175 mm and the height h may be 91 mm.

[0110] First, the first robot 100 can detect the visual markers 250 included in the captured image. In this case, the control unit of the first robot 100 can identify at least one marker reference point included in the visual markers 250 based on a pre-configured marker detection algorithm (or program). The marker reference point may mean at least one corner point corresponding to each corner included in the visual markers 250.

[0111] Specifically, the control unit of the first robot 100 can recognize the visual marker 250 from the captured image and identify (extract) at least one marker reference point (P, or corner point, a, b, c, d, e, f, g, h) corresponding to each corner of the recognized marker.

[0112] Furthermore, based on the identified marker reference point P, the control unit of the first robot 100 can determine the marker coordinate system {W} of the visual marker 250. That is, it can measure the degree of freedom information (or 6-degree-of-freedom posture) of the marker coordinate system {W} relative to the camera coordinate system {C}.

[0113] The marker coordinate system {W} may include three-dimensional axes with one of the specified marker reference points (e.g., a, b, c, d, e, f, g, h) as the origin. Specific location of the camera The corresponding marker reference point may be used as the origin.

[0114] Next, the control unit of the first robot 100 can estimate the orientation of the visual marker 250. Therefore, in the step of calculating the orientation of the visual marker included in the image (S115), the detected visual marker 250 can be used to estimate the 6-degree-of-freedom orientation of the visual marker 250 with respect to the coordinate system of the camera 121 that photographed the second robot 200.

[0115] Such a camera 121 may be configured to have a camera coordinate system {C}, as shown in the figure. The camera coordinate system {C} may be a three-dimensional coordinate system with a specific point on the camera 121 as its origin. Alternatively, as shown in the figure, a marker coordinate system {W} may be set as another reference coordinate system, based on markers included in the image captured by the camera 121. In this case, the marker coordinate system {W} may be a three-dimensional coordinate system with a specific point on the markers included in the image as its origin.

[0116] In this case, the visual marker 250 may have a reference coordinate system at the point of the marker, and the degree of freedom information of the marker can be obtained by reflecting the relative positional relationship between the reference coordinate system of the camera 121 and the reference coordinate system of the marker.

[0117] In this way, the degree of freedom information (6 degrees of freedom orientation) of the marker coordinate system {W} can be extracted with reference to the camera coordinate system {C} of the camera 121. As a result, the control unit can extract the degree to which the visual marker 250 or the marker coordinate system {W} contained in the captured image has been rotated and translated with respect to the camera coordinate system {C}.

[0118] For example, by identifying the pixel coordinates corresponding to the marker reference point, and using the pixel coordinates corresponding to the marker reference point and the camera coordinate system {C} of the camera 121, the degree of freedom orientation of the marker coordinate system {W} relative to the camera coordinate system {C} can be determined.

[0119] More specifically, control The department , camera coordinate system { C To extract the degrees of freedom information (or relative positional relationship) of the marker coordinate system {W} for}, PnP may be calculated.

[0120] On the other hand, the camera coordinate system { C The relative position of the marker coordinate system {W} to} is such that the marker coordinate system {W} is in the camera coordinate system { CThis can represent the degree of rotation and transformation applied to {}. Such information regarding the degree of rotation and transformation can correspond to the degrees of freedom information (degrees of freedom orientation) of the marker coordinate system {W} relative to the camera coordinate system {C}.

[0121] In order to extract the degrees of freedom information (degrees of freedom orientation) of the marker coordinate system {W} relative to the camera coordinate system {C}, the pixel coordinates (u, v) corresponding to at least one marker reference point of the visual marker 250 are obtained. 、 By substituting the camera's three-dimensional coordinates (x, y, z) into the PnP equation, we can extract the degrees of freedom information (degrees of freedom orientation) of the marker coordinate system {W} relative to the camera coordinate system {C}.

[0122] Specifically, the 6-degree-of-freedom orientation of the marker coordinate system relative to the camera coordinate system is calculated. This can be calculated by defining the pixel coordinates of the corner points of the April tag using the PnP equation in equation (1) below.

[0123]

number

[0124] In this case, the left-hand side is the pixel coordinates of the corner points of the April tag, the first matrix on the right-hand side is for camera parameters, the second matrix is ​​the 6-degree-of-freedom orientation of the marker coordinate system relative to the camera coordinate system, and the third matrix is ​​for real-world coordinate values. Furthermore, since the April tag provides corner points in the video using open source, the pixel coordinates of the aforementioned corner points can be calculated.

[0125] When the posture of the visual marker 250 is measured, it is converted into the posture of the second robot 200.

[0126] First, as shown in Figure 6, the pose of the visual marker 250 can be converted to the robot's pose using a CAD model, which is the shape data of the second robot 200. For example, the three-dimensional coordinates (x, y, z) of the corner points of the visual marker 250 relative to the reference coordinate system of the CAD model can be extracted. For such coordinate transformations, the shape data of the second robot 200 may be stored in the first robot 100 or the cloud server described above.

[0127] Furthermore, as shown in the figure, a robot coordinate system {O}, which is set with the robot as the reference, may be set as one of the reference coordinate systems. The robot coordinate system {O} may be a three-dimensional coordinate system with a specific point on the second robot 200 as its origin.

[0128] Furthermore, it goes without saying that the degrees of freedom information of the second robot 200 is obtained by reflecting the relative positional relationship between the marker's reference coordinate system and the second robot 200's reference coordinate system. In other words, in the present invention, it is possible to collect degrees of freedom information for at least one of the camera 121, the marker, and the robot based on the mutually relative positional relationship between the camera 121's reference coordinate system, the marker's reference coordinate system, and the robot's reference coordinate system.

[0129] The aforementioned coordinate transformation can be calculated by defining the pixel coordinates of the corner points of the April tag using the PnP equation and the following equation (2).

[0130]

number

[0131] In this case, the left side is the pixel coordinates of the corner points of the April tag, the first matrix on the right side is for the camera parameters, the second matrix is ​​the 6-degree-of-freedom orientation of the robot coordinate system (i.e., the coordinate system of the second robot 200) relative to the camera coordinate system, and the third matrix is ​​for the real-world coordinate values.

[0132] As shown in Figure 6, the posture of the second robot 200 can be defined by calculating the x, y, and z coordinate values ​​for the corner point Pf at the lower right end of the visual marker 250, based on the robot's reference coordinate system. In this way, the 3D coordinates of the points of the visual marker 250 included in the image are extracted relative to the reference coordinate system of the CAD model of the second robot 200.

[0133] Converting the orientation of the visual marker 250 to the robot's orientation can also be done by other methods. (Figure 7) a ,figure 7b and figure 8 By referring to this, it is possible to convert the marker orientation relative to the camera coordinate system into the robot orientation using a calibration method.

[0134] As an example of this, the pose of the visual marker 250 included in the image can be converted to the pose of the second robot 200 using odometry information from the camera 121 of the first robot 100 and the wheel 220 of the second robot 200.

[0135] In this case, to ensure proper operation in a system that uses a camera attached to a mobile robot to explore its surroundings, corrections are made to estimate the internal and external parameters of the camera, the relative pose between the camera and the mobile robot's frame, and the mobile robot's movement parameters. At this time, an odometry method is applied, which involves reconstructing the configuration (i.e., position and orientation) of the mobile robot based on measurements from an encoder on the wheel 220. Starting from a known configuration, the robot's current position and orientation are calculated by the time integral of the vehicle speed corresponding to the measured speed of the wheel 220. In this invention, the robot's posture can be calculated by applying such a method to the relationship between the reference coordinate system of the second robot 200, the corner points of the visual marker 250, and the wheel odometry.

[0136] For example, wheel odometry Oi T Oj With this definition, the relationship in equation (3) below holds, and the orientation of the visual marker 250 can be defined by equation (4) below.

[0137]

number

[0138]

number

[0139] Using equations (3) and (4), the orientation of the visual marker 250 can be converted to the robot's orientation.

[0140] According to the method described above, the first robot 100 can measure the posture of the second robot 200 using the visual marker 250. As an example of this, Figures 8(a), (b), and (c) show examples in which the CAD model (Mesh) of the robot is projected onto the image based on the measured posture.

[0141] In this way, after measuring the posture of the second robot 200, the first robot 100 calculates the position of the item transported by the second robot 200, picks up the item, and then performs the assigned task.

[0142] Through the process described above, the robot posture measurement method and robot system using the present invention can recognize the posture of an opposing robot, thereby enabling rapid and accurate measurement of the opposing robot's posture in a simple manner.

[0143] On the other hand, the process described above may be modified to measure the position of oneself rather than the position of the target robot. Such modifications will be explained below with reference to Figure 9.

[0144] Figure 9 is a conceptual diagram illustrating how to measure the position of a robot in another embodiment of the present invention.

[0145] In real indoor environments, significant changes occur over time, making it extremely difficult to accurately estimate the pose information of a robot's camera. For example, areas without texture, changes in appearance due to content changes on signage, changes in crowding and occlusion over time, and changes in the dynamic environment all make it difficult to estimate the camera pose for an image.

[0146] Referring to Figures 1 to 8, the posture measurement using the visual markers described above can also be applied to estimating the camera pose of a robot moving in an indoor space.

[0147] Referring to Figure 9(a), first, the robot 300 can move within the indoor space 20 where the display 410 is located. In this case, the robot 300 may be the second robot 200 illustrated above.

[0148] As the aforementioned display 410 Digital signage While this is given as an example, other types of displays are also possible. In this case, image information may be displayed on the display 410, and the image information may be advertising information, guidance information, video content, etc.

[0149] When the robot 300 is moving through the indoor space 20, the display 410 can be detected using the camera 321 attached to the robot 300. When the display 410 is detected, the robot 300 transmits a request signal to the control system that controls the display 410, requesting the output of the visual marker 420. At this time, the control system can monitor the reception of the request signal.

[0150] The control system may be a control system for a building having an indoor space, or it may be an external server such as a cloud server.

[0151] As another example, the system may be equipped with a separate server that operates the robot, which can sense when the robot is approaching the display and transmit a request signal to the control system requesting the output of the visual marker 420.

[0152] Referring to Figure 9(b), in response to the request signal, the control system controls the display 410 to output the visual marker 420.

[0153] The visual marker 420 may be an exemplary visual marker as described with reference to Figures 1 to 8, and may be output to the display 410 together with the image information output from the display 410. As an example, the visual marker 420 may be output overlapping the image information at the lower end of one side of the display 410.

[0154] Next, the robot 300 estimates its current position using the visual marker 420 output to the display 410. In this case, the robot 300's pose can also be estimated. The estimation of the robot 300's current position and pose may be performed by visual localization using the indoor space feature map.

[0155] Using this position and estimated pose, the precise position of the robot 300 can be estimated. In this case, the robot 300 can perform various services based on the position.

[0156] As illustrated in this example, the present invention can output a visual marker on a display fixed in a specific space, such as a signage screen, and accurately and quickly measure the position of a robot that has photographed the visual marker.

[0157] The robot posture measurement method and robot system using the same described above are not limited to the configuration and methods of the above embodiments, and the above embodiments can be configured by selectively combining all or part of each embodiment so that various modifications can be made.

Claims

1. A step in which at least one of the first robot and the second robot moves so that the first robot and the second robot are adjacent to each other, The steps include outputting a visual marker to the display of the second robot, The first robot photographs the second robot, The steps include detecting visual markers included in an image captured by the first robot and calculating the pose of the visual markers included in the image, The step of calculating the pose of the second robot using the pose of the visual marker included in the image relative to the camera coordinate system of the first robot, A method for measuring the posture of a robot.

2. The robot posture measurement method according to claim 1, further comprising the step of monitoring a request signal in which the second robot requests the output of the visual marker.

3. The robot posture measurement method according to claim 2, characterized in that the request signal is transmitted from a server communicating with the first robot or from the first robot to the second robot.

4. The step of outputting the aforementioned visual marker is: When the request signal is received while image information is being output to the display, the output of the image information is turned off and the visual marker is output, or the visual marker is output together with the image information. The method for measuring the posture of a robot according to claim 2.

5. The step of outputting the aforementioned visual marker is: The display is characterized by activating the display and outputting the visual marker when it receives the request signal while the display is not activated. The method for measuring the posture of a robot according to claim 2.

6. The robot posture measurement method according to claim 1, characterized in that the output of the visual marker is terminated after a certain amount of time has elapsed since the output of the visual marker to the display of the second robot.

7. The robot posture measurement method according to claim 1, characterized in that the display is positioned facing upward on the front or top surface of the second robot, and the first robot is equipped with a camera that photographs the second robot facing downward.

8. The robot posture measurement method according to claim 1, characterized in that, in the step of calculating the posture of the visual marker included in the aforementioned image, the six-degree-of-freedom posture of the visual marker with respect to the coordinate system of the camera that photographed the second robot is estimated using the detected visual marker.

9. The robot posture measurement method according to claim 1, characterized in that, in the step of calculating the posture of the second robot, the 3D coordinates of the points of the visual markers included in the image are extracted with respect to the reference coordinate system of the CAD model of the second robot.

10. The robot posture measurement method according to claim 1, characterized in that, in the step of calculating the posture of the second robot, the posture of a visual marker included in the image is converted to the posture of the second robot using the camera of the first robot and odometry information for the wheels of the second robot.

11. The first robot is equipped with a camera, The system includes a second robot that interacts with the first robot and is equipped with a display, The second robot outputs a visual marker to its display, and the first robot photographs the second robot. The first robot detects the visual markers contained in the image it captures, calculates the pose of the visual markers contained in the image, and estimates the pose of the second robot using the pose of the visual markers contained in the image relative to the camera coordinate system of the first robot. The operation of the first robot is controlled using the estimated posture of the second robot. A robotic system in which multiple robots interact with each other.

12. The first robot and the second robot are controlled to interact with each other. The first robot is characterized by moving the end effector to a set position of the second robot using the estimated posture of the second robot. A robot system in which a plurality of robots interact as described in claim 11.

13. A robot system in which a plurality of robots interact, as described in claim 11, characterized in that the second robot receives a request signal from the first robot or from a server that communicates with the first robot, requesting the output of the visual marker.

Images